Moe,
Yes, that change does allow bash to run under salloc when salloc is
within a script. Other shells ("tcsh", "ksh", and "zsh") on my system
would also fail to run, and now are ok.
I sympathize with Gerrit's problem of hung jobs, and I understand his
motivation for trying to manage the potentially orphaned processes. But I
have to contend with our users as well, many of whom have been happily
(until now) using salloc in scripts, or running it non-interactively as
background jobs, without encountering any such dire consequences.
As you mentioned, perhaps the job control changes should be in conditional
code instead of general code, since the problem seems to be specific to
Cray? What about other users of SLURM with Cray systems? Doesn't LLNL
itself run SLURM on Cray? Have you all encountered the same blockages of
resources that motivated Gerrit to add these changes?
-Don Albert-
[email protected]
06/02/2011 06:47 AM
To
[email protected]
cc
[email protected]
Subject
Re: [slurm-dev] Running Salloc From Bash Fails Under 2.2.3?
Don,
I believe that the patch below will fix the problem that you have
reported and return salloc to something very similar to its previous
behavior. It would cause problems on Cray systems, so this logic
might be something that is conditionally compiled. Perhaps you could
try this change and see how it works for you.
diff --git a/src/salloc/salloc.c b/src/salloc/salloc.c
index 4cbc9c5..092668f 100644
--- a/src/salloc/salloc.c
+++ b/src/salloc/salloc.c
@@ -219,10 +219,6 @@ int main(int argc, char *argv[])
* a) input is from a terminal (stdin has valid termios
attributes),
* b) controlling terminal exists (non-negative tpgid),
* c) salloc is not run in allocation-only (--no-shell) mode,
- * d) salloc runs in its own process group (true in interactive
- * shells that support job control),
- * e) salloc has been configured at compile-time to support
background
- * execution and is not currently in the background process
group.
*/
if (tcgetattr(STDIN_FILENO, &saved_tty_attributes) < 0) {
/*
@@ -234,9 +230,8 @@ int main(int argc, char *argv[])
error("no controlling terminal: please set
--no-shell");
exit(error_exit);
}
- } else if ((!opt.no_shell) && (pid == getpgrp())) {
- if (tpgid == pid)
- is_interactive = true;
+ } else if (!opt.no_shell) {
+ is_interactive = true;
#ifdef SALLOC_RUN_FOREGROUND
while (tcgetpgrp(STDIN_FILENO) != pid) {
if (!is_interactive) {
Quoting Gerrit Renker <[email protected]>:
> Don,
>
> as I stated earlier, I am sorry that this broke things for you, but ...
>
> There was on major and one minor reason for submitting the set of job
> control patches. The minor one is such that I would have no problems if
> the SLURM developers would withdraw job control, and is: without job
> control, there will always be situations where users are forced to use
> kill -9 or similar to get rid of salloc. Since salloc is then not in
> control of the sub-processes it runs.
>
> The major reason for keeping this is the proprietary system we use, the
> technical details are below. I am not writing these to enter into a
> discussion of Cray features, but hope that the work at ORNL proceeds to
> replace ALPS with SLURM. Then the problem described below would not
> exist. That problem basically forced us to either
> * disable interactive sessions on our systems (unacceptable since both
> researchers and newcomers rely on interactive sessions to test out
> how a particular combination of parameters work) or
> * use job control as a trade-off.
>
> The situation in December was such that it took only a single mistake of
one
> user to make an entire multi-cabinet system unusable, i.e. if SLURM were
a
> firewall or malware protection program, we would not even have this
> discussion.
>
> In December, when job control had not been added, we twice had stretches
in
> the order of 10-12 hours where the machine became blocked and no new
jobs
> would run, the messed up salloc sessions both happened in the evening.
>
>> Perhaps I was not clear in my description of the problem, but the
patch
>> you supplied most emphatically does *not* solve the problem that the
bash
>> shell is crashing and getting into the SIGTTIN loop before ever issuing
>> its own prompt to the user! The "/bin/bash" command should have
>> executed the shell and allowed the user to enter commands, and not
>> immediately terminated.
>>
> Have you tried this with other shells? This is coming from bash itself,
which
> tries to become the foreground process in order to perform its job
control.
> When run within a job script, salloc does not allow another process to
come
> into the foreground, only if run in interactive mode. The behaviour
> is expected,
> it may be that other shells less aggressively try to move into the
> foreground.
>
>> You seem to imply that it is somehow illegitimate to execute the
"salloc"
>> within a script. I submit that it is almost second nature for
>> Linux/Unix programmers to create various "wrapper" scripts to invoke
>> commands (including "salloc") with certain fixed parameters, while
>> allowing easy substitution of other parameters.
> If I understand you correctly, you mean a shell invoking a shell such as
> % cat a
> #!/bin/bash
> /bin/sh -c /bin/bash
> % ./a
> % ps f -o pid,ppid,sid,pgid,tpgid,stat,wchan,cmd
> PID PPID SID PGID TPGID STAT WCHAN CMD
> 25480 25477 25480 25480 26024 Ss wait -bash
> 25739 25480 25480 25739 26024 S wait \_ sh a
> 25740 25739 25480 25740 26024 S wait \_ /bin/bash
> 26024 25740 25480 26024 26024 R+ - \_ ps f -o
> pid,ppid,sid,pgid,tpgid,stat,wchan,cmd
>
> The three nested shells all have the same session ID, the login
> shell 25480 remains session leader.
> The bottom-level shell 25740 has backgrounded itself so that the ps
> command it forked can be the
> foreground process (tpgid = 26024).
>
>> The "salloc" command itself is essentially a wrapper which allows a
>> user to invoke a specific
>> command or shell after calling SLURM to reserve some resources. I
>> see no reason that salloc
>> should not be able to be executed within a script.
> The above use case uses shell scripts to construct interactive
> sessions. I have found no non-ugly
> way of allowing this, and believe also that sh or bash resort to
> some tricks and heuristics that
> make this possible (i.e. not sure that sh/bash will always get this
right).
>
> The reason is the "some resources". When a user kills the salloc
> process, the remote processes
> running via aprun on compute nodes are not at the same time
> terminated. Once the salloc session
> is in addition either terminated via scancel or has naturally timed
> out, it would be time to free
> up the resources reserved for this session. Very likely SLURM itself
> could handle this situation.
>
> However, on Cray the compute nodes are not under control of SLURM,
> but the Basil batch system layer.
> At the end of the salloc session, this layer receives a notification
> that the job is done. However,
> it will refuse to launch any new reservations until it has not
> cleaned up the existing ones. But it
> can not clean up the existing ones since the remote orphaned
> processes are still executing. Until
> an operator comes in (e.g. during the middle of the night) and
> cleans up the orphaned processes, no
> new jobs will run since the old reservation sits in "pending cancel"
state.
>
> Hence we do not allow other processes to obtain control over salloc
> unless salloc is running in an
> interactive mode.
>
> We run SLURM on 3 XT systems, including our main production system,
> and one XE system. When we
> migrated we had many novice users. Since introduction of the job
> control into salloc, we have had
> not a single case of machine-unusable-time caused by orphaned child
> processes of salloc.
>
> Hence if you have your way, we would need to fork in order to keep
> things running.
>
