On 06/30/2011 12:11 AM, Michal Hocko wrote: > On Wed 29-06-11 12:32:13, Kevin Constantine wrote: >> On 06/29/2011 07:20 AM, Michal Hocko wrote: >>> Hi, >>> >>> On Thu 23-06-11 19:10:39, Kevin Constantine wrote: >>>> On 06/23/2011 05:51 PM, Balbir Singh wrote: >>>>> If you don't mind, can I ask how you are using libcgroup, what is the use >>>>> case? >>>>> >>>> >>>> We're using cgroups in our render environment to pin batch jobs and >>>> their child processes to individual cpu's to avoid processes hopping >>>> around to different cores and flushing their caches. Basically each >>>> batch job gets its own cgroup with the cpuset controller and we allocate >>>> a single cpu to that cgroup. >>>> >>>> We're also using them to run batch jobs on our interactive workstations >>>> without impacting the end user. We're carving off a few cores and a few >>>> GB's of RAM from each workstation to run batch jobs on throughout the day. >>>> >>>> Basically the batch scheduler is dynamically creating and destroying >>>> cgroups pretty regularly. I wrote code to do all of that and then >>>> realized that libcgroup exists, so now I'm going back through and >>>> refactoring to use the api as much as possible. >>> >>> I am wondering why you need to write any code for that. Isn't cgred >>> (man cgrules.conf) what you need? >>> >> >> I think cgred makes sense for managing machines where I know what >> kind of workload to expect. My understanding of it is that cgroups >> are created ahead of time with cgconfig.conf and then the rules map >> processes into those cgroups. > > I may be wrong because I have never played with %p or %u for group name > specification but they sound dynamic enough to require on demand cgroups > creation. >
What's the %p %u you're talking about? I haven't seen that syntax in what I've read. >> The problem is that I have no idea what kind of workload to expect >> on most of these machines. The processes could be named just about >> anything, and they could come from just about any user. > > How do you do that in your code then? Do you have any active monitoring > running which shuffles jobs around? > Each client machine has a daemon that communicates with a central scheduler. host configuration is also stored centrally. I can say something like all HP DL380's will present 16GB and CPU's 0-3 to the queue. A cgroup will get created with the memory subsystem and the cpuset subsystem. We'll set memory.limit_in_bytes to "16G", and cpuset.cpus to 0-3 (we'll also set cpuset.mems to whatever the parent cgroup has, but that's another issue to discuss). The client daemon is then responsible for keeping track of what cpu's are currently pinned and which ones are free. New batch jobs will get a new cgroup created in /cgroup/cpuset/batchJobs/123456 and have cpuset.cpus set to the appropriate pinned cpu (or cpu's for a multithreaded job). The process will be started up in cpuset:/batchJobs/12345, and memory:/batchJobs cgroups. When a job completes, cpuset:/batchJobs/12345 gets destroyed. >> >> I could set up a config so that anything that gets run from the >> scheduler's client daemon process lands in a cgroup. But then we >> run into the situation where I want each batch process to live in >> its own cgroup and have one or more cpu's pinned to that cgroup >> (depending on how many cpu's the job asks for). I'm not sure how to >> configure something dynamic like that with these rules. > > So if I understand that correctly, you are missing some way how to setup > dynamically created groups, right? This sounds like an interesting idea > because if you create a group without initialization then you get only > inherited values from the parent group which is very controller > specific. > I am not sure whether there is a way to communicate an application's > requirements to the daemon, though. > Yeah, communication between the application and the daemon would be a challenge for sure. -kevin >> It seems easier to add cgroup functionality to the scheduler than it >> does to come up with some configuration management scheme for >> creating and maintaining cgrules.conf files all over the place. > > Probably yes. > >> cgconfig/cgred feels like something that you'd set up and kind of >> forget about on a server where I've got an ftp service and an email >> service and want to keep them in resource boxes. In our >> environment, rules might change depending on what time of day it is. >> For example, at night the memory and number of cpus allocated to >> batch jobs on a machine might get increased since there's very >> little interactive work happening. > > > Yes, that is the reason I was asking for your workload. Maybe some of > the ideas you are already using can be pushed to the daemon as well. > For example I would really love if I could be able to specify groups > parameters for those rules that use %[uUgGpP] because they can hardly be > created in advance. > >> >> -kevin > > Thanks ------------------------------------------------------------------------------ All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2 _______________________________________________ Libcg-devel mailing list Libcg-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/libcg-devel
