I believe you're understanding the problem correctly. Especially with MPI, you need some mechanism to launch processes on remote hosts. With scheduler integration of some kind (eg. srun with SLURM, TM-API with Torque, etc.), then the MPI implementation can work with the scheduling tool to do this, and everyone is happy. But lacking that, the MPI implementation still needs to have a fallback, which is usually SSH.
Having said that, in my experience most commercial codes that use MPI, just package/utilize some MPI implementation that's out there on the market. Many of those MPI implementations do have SLURM integration. It would be helpful to know what the commercial software is, and perhaps more importantly, what MPI implementation it is using. Also, some commercial codes also have a way to specify (via a cli parameter) a drop-in replacement for ssh. In theory, you could probably create a simple wrapper around srun to make the syntax more like ssh, and use that. There are probably other solutions out there too. Lloyd Brown Systems Administrator Fulton Supercomputing Lab Brigham Young University http://marylou.byu.edu On 02/21/2013 02:06 PM, Michael Colonno wrote: > > Thanks for the reply. I'm not 100% clear on the below so let me be more > specific. I'm launching the code via srun (for example). The code launches, > runs a few different executables in order, and eventually launches a few MPI > processes though its own MPI implementation. I have no control over the > source code nor what syntax is used to launch the sub-processes. srun > launches these processes and then reports the job completed; this is the only > tool that behaves this way (others seem to track processes even if not > launched through SLURM). Is the conclusion that if the sub-processes are not > launched explicitly via SLURM (but are child processes of a SLURM-launched > process) there is nothing that can be done at the SLURM level to prevent > SLURM from relinquishing the resources before the job is completed? > > Thanks, > ~Mike C. > > -----Original Message----- > From: Moe Jette [mailto:[email protected]] > Sent: Thursday, February 21, 2013 11:00 AM > To: slurm-dev; Michael Colonno > Subject: Re: [slurm-dev] untracked processes > > Slurm only tracks the processes that it's daemons launch (most MPI > implementations can launch their tasks using slurm). Anything launched > outside of Slurm can be killed as part of a job prolog, but accounting and > job step management are outside of Slurm's control. > > Quoting Michael Colonno <[email protected]>: > >> >> SLURM gurus ~ >> >> I'm trying to configure a commercial MPI code to run through SLURM. >> I can launch this code through either srun or sbatch without any >> issues (the good) but the processes manage to run completely >> disconnected from SLURM's notice (the bad). i.e. the job is running >> just fine but SLURM thinks it's completed and hence does not report >> anything running. I'm guessing this is due to the fact that this tool >> runs a pre-processing-type executable and then launches sub-processes >> to solve (MPI on a local system) without connecting the process IDs(?) >> In any event, I'm guessing I'm not the first person to run into this. >> Is there a recommended solution to configure SLURM to track codes like >> this? >> >> Thanks, >> ~Mike C. >> >> > >
