This may not be exactly what you're looking for but it could be a start. We're looking at adding modifying ssh_config and sshd_config to propagate SLURM_JOB_ID for jobs that use ssh to spawn processes (credit to our sysadmin Lloyd Brown for that one). Then we will use something like a script in /etc/profile.d to add the process to the correct cgroup if it's launched via ssh and has $SLURM_JOB_ID set. We're not using cgroups yet (still have some CentOS 5) so I don't have exact implementation details at this point. Then the cgroups should work for resource control and, I assume, accounting if using the correct plugin.
This may not catch 100% of everything, but we would probably have something look for all user processes that are not part of a cgroup and add them to the user cgroup. I don't think accounting could work in that case, but that would help catch and control "rogue processes" that aren't accounted for under SLURM. Epilog or a cron could clean up all of a user's processes after they don't have jobs on the node anymore. I don't know if SLURM has something like Torque's tm_adopt, but that could work in lieu of cgroups for accounting if you don't happen to use cgroups. tm_adopt allowed you to add a random process to be accounted for under Torque, even if it wasn't launched under Torque. We used to have a wrapper script for ssh that did just that when we used Torque and Moab. Ryan P.S. We've only been using SLURM for a few weeks so you might want to double-check the accuracy and viability of my statements :) On 02/21/2013 12:57 PM, Moe Jette wrote: > Slurm only tracks the processes that it's daemons launch (most MPI > implementations can launch their tasks using slurm). Anything launched > outside of Slurm can be killed as part of a job prolog, but accounting > and job step management are outside of Slurm's control. > > Quoting Michael Colonno <[email protected]>: > >> SLURM gurus ~ >> >> I'm trying to configure a commercial MPI code to run through SLURM. >> I can launch this code through either srun or sbatch without any >> issues (the good) but the processes manage to run completely >> disconnected from SLURM's notice (the bad). i.e. the job is running >> just fine but SLURM thinks it's completed and hence does not report >> anything running. I'm guessing this is due to the fact that this >> tool runs a pre-processing-type executable and then launches >> sub-processes to solve (MPI on a local system) without connecting >> the process IDs(?) In any event, I'm guessing I'm not the first >> person to run into this. Is there a recommended solution to >> configure SLURM to track codes like this? >> >> Thanks, >> ~Mike C. >> >> -- Ryan Cox Operations Director Fulton Supercomputing Lab Brigham Young University
