Michel, Child processes are automatically added to the cgroup of their parent, so when proctrack/cgroup is used forked processes are included in the proctrack container. The cgroup for each job step contains the pids of all tasks in the step and all offspring of those tasks.
The jobacct_gather plugin accounts for child processes by scanning the /proc table. See _get_offspring_data(). Martin [slurm-dev] Re: Monitoring pids dynamically Michel Bourget to: slurm-dev 02/24/2012 02:47 PM Cc: Danny Auble From: Michel Bourget <[email protected]> To: "slurm-dev" <[email protected]> Cc: Danny Auble <[email protected]> Please respond to "slurm-dev" <[email protected]> On 02/24/2012 01:24 PM, Danny Auble wrote: > Michel are you using JobAcctGatherFrequency? Most people set it to 30, > but we have tested it down to 5 without noticing any real noise, > obviously it is application dependent though. Nothing should be set in > stone. Yes, I use a non-zero JobAcctGatherFrequency. The problem is not there. > Everything should be dynamic. If you aren't polling with > JobAcctGatherFrequency then the job_acct_gather plugin will only look at > the beginning and at the end of the step (which is probably what you are > seeing). > Of course, it polls but it polls a set-in-stone accounting tasks established ( well, at least in 2.3.3 iirc ) by jobacct_gather_g_add_task ( from _fork_all_tasks ). In the case of jobacct_common_add_task() ( gather_linux ), this is set once. The jobacct_gather_linux task_list will never *intersect* with new children pids proctrack could possibly discover on its side. ( Just saw Moe's reply ... Lemme continue there ) PS: I should have mentioned about the specific jobacct_gather/linux oriented question. > Danny > > On 02/24/12 09:08, Michel Bourget wrote: >> Hi all, >> >> was the issue of monitoring pids coming-and-going-away addressed ( or >> debated ) in >> the past ( or the future tbd) in regards to proctrack and job_acct_gather ? >> >> I mean, since pids can fork() children and go away later, proctrack seems >> not to able to dynamically track this since it's "on-demand". Same for >> jobacct_gather since it's set "in stone" when a step is launched. >> And, because proctrack is on-demand and jobacct_gather pids are set in stone >> at the beginning, on-demand newly discovered pids never intersect >> with those jobacct pids. >> >> Maybe an approach like using the kernel process socket connector, >> based on an initial set of pids ( monitor fork() and exit() ), and then >> proctrack/job_act_gather using that list instead, would be useful >> and feasible ? In that case, I would think additional information >> relative to the obtained pid list would be something in the lines of: >> >> pid_list_t { >> a_lock; // Global list lock >> int n; // # of records >> pid_info_t *info; // Obvious >> more ? >> } >> >> pid_info_t { >> a_lock; // Record lock >> int is_active; // 0 means pids once live but now gone >> struct jobacctinfo; // acct for that pid so far. >> more ? >> } >> >> Given the above, proctrack services would key on pid where active=1. >> And jobacct_gather services would key on jobacctinfo gathered so far, >> regardless of is_active.And I would risk to state proctrack and >> jobacct_gather could be independent of each other, which is not the case >> today, I believe. >> >> I have to admit the above would allow a lot more easily to inject >> out-of-band pids to slurm. I can think of those using mpirun >> in an salloc, or similar. "Similar" is about the sgimpi >> implementation I maintain here at SGI. I understand it >> sounds SGI-specific but I believe there is a generic value >> in the above-mentioned approach that would benefit to SLURM in >> general. >> >> Hopefully, I hope I am not off track ;-) >> >> Too evil ? Not worth ? Comments ? >> -- ----------------------------------------------------------- Michel Bourget - SGI - Linux Software Engineering "Past BIOS POST, everything else is extra" (travis) -----------------------------------------------------------
