On Sat, Jul 2, 2011 at 2:12 AM, Kevin Constantine <kevin.constant...@disneyanimation.com> wrote: > On 06/30/2011 12:11 AM, Michal Hocko wrote: >> >> On Wed 29-06-11 12:32:13, Kevin Constantine wrote: >>> >>> On 06/29/2011 07:20 AM, Michal Hocko wrote: >>>> >>>> Hi, >>>> >>>> On Thu 23-06-11 19:10:39, Kevin Constantine wrote: >>>>> >>>>> On 06/23/2011 05:51 PM, Balbir Singh wrote: >>>>>> >>>>>> If you don't mind, can I ask how you are using libcgroup, what is the >>>>>> use case? >>>>>> >>>>> >>>>> We're using cgroups in our render environment to pin batch jobs and >>>>> their child processes to individual cpu's to avoid processes hopping >>>>> around to different cores and flushing their caches. Basically each >>>>> batch job gets its own cgroup with the cpuset controller and we >>>>> allocate >>>>> a single cpu to that cgroup. >>>>> >>>>> We're also using them to run batch jobs on our interactive workstations >>>>> without impacting the end user. We're carving off a few cores and a >>>>> few >>>>> GB's of RAM from each workstation to run batch jobs on throughout the >>>>> day. >>>>> >>>>> Basically the batch scheduler is dynamically creating and destroying >>>>> cgroups pretty regularly. I wrote code to do all of that and then >>>>> realized that libcgroup exists, so now I'm going back through and >>>>> refactoring to use the api as much as possible. >>>> >>>> I am wondering why you need to write any code for that. Isn't cgred >>>> (man cgrules.conf) what you need? >>>> >>> >>> I think cgred makes sense for managing machines where I know what >>> kind of workload to expect. My understanding of it is that cgroups >>> are created ahead of time with cgconfig.conf and then the rules map >>> processes into those cgroups. >> >> I may be wrong because I have never played with %p or %u for group name >> specification but they sound dynamic enough to require on demand cgroups >> creation. >> > > What's the %p %u you're talking about? I haven't seen that syntax in what > I've read. > >>> The problem is that I have no idea what kind of workload to expect >>> on most of these machines. The processes could be named just about >>> anything, and they could come from just about any user. >> >> How do you do that in your code then? Do you have any active monitoring >> running which shuffles jobs around? >> > > Each client machine has a daemon that communicates with a central scheduler. > host configuration is also stored centrally. I can say something like all > HP DL380's will present 16GB and CPU's 0-3 to the queue. A cgroup will get > created with the memory subsystem and the cpuset subsystem. We'll set > memory.limit_in_bytes to "16G", and cpuset.cpus to 0-3 (we'll also set > cpuset.mems to whatever the parent cgroup has, but that's another issue to > discuss).
You mean inheriting cpuset values from parent? So, you use the full machine and the limit is set to the size of the machine? The client daemon is then responsible for keeping track of what > cpu's are currently pinned and which ones are free. New batch jobs will get > a new cgroup created in /cgroup/cpuset/batchJobs/123456 and have cpuset.cpus > set to the appropriate pinned cpu (or cpu's for a multithreaded job). The > process will be started up in cpuset:/batchJobs/12345, and memory:/batchJobs > cgroups. When a job completes, cpuset:/batchJobs/12345 gets destroyed. > Are you using notifications to free up the cgroup? >>> >>> I could set up a config so that anything that gets run from the >>> scheduler's client daemon process lands in a cgroup. But then we >>> run into the situation where I want each batch process to live in >>> its own cgroup and have one or more cpu's pinned to that cgroup >>> (depending on how many cpu's the job asks for). I'm not sure how to >>> configure something dynamic like that with these rules. >> >> So if I understand that correctly, you are missing some way how to setup >> dynamically created groups, right? This sounds like an interesting idea >> because if you create a group without initialization then you get only >> inherited values from the parent group which is very controller >> specific. >> I am not sure whether there is a way to communicate an application's >> requirements to the daemon, though. >> Thanks for sharing your experience, we'll use this to write better APIs Balbir ------------------------------------------------------------------------------ All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2 _______________________________________________ Libcg-devel mailing list Libcg-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/libcg-devel
