I think that Hydra should kept 0,1,2 open and dup them to /dev/null so that any children’s file descriptor will be greater than 2. This is standard Unix way.
Thanks /David/Bigagli [email protected] =============================================================== Slurm User Group Meeting, 15-16 September 2015, Washington D.C. http://slurm.schedmd.com/slurm_ug_agenda.html > On 25 Jul 2015, at 05:05, Aaron Knister <[email protected]> wrote: > > tl;dr - if srun is exec'd by a process that has fd 0 close()'d an external > library (in my case nss_sssd) can open a file/socket that ends up being given > fd 0. srun then thinks this is supposed to be stdin for the job and will read > it and pass the data on to other tasks which can cause a whole host of > problems especially if the remote ranks aren't expecting data on stdin. > > Is it reasonable for srun to check and see if fd 0 is open pretty early in > the game before it calls functions from external nss libraries or spank > plugins and use that to determine if stdin should be poll()'d or read()? That > makes me wonder if I couldn't just implement functionality like this in a > spank plugin-- if fd 0 is closed, then open /dev/null and dup it on fd 0. > > > Here's the long version: > > I've been working with the devs offline on the stdin shutdown issue (I highly > recommend a SchedMD support contract-- they're great!), but I did discover a > higher level issue that was triggering this. I mentioned that mvapich's hydra > launcher was being very chatty and spamming srun with \377 characters. I did > some investigation into what was going on here and am posting this back to > the list in case others find it helpful. I'll be opening a ticket with the > mvapich folks at some point soon to hopefully get this change integrated into > mvapich. > > What I discovered was that the hydra launcher close()'s stdin before it > exec's srun. That's all fine and dandy but then srun calls some function that > ropes in the nss libraries (and for us the libnss_sssd since we use sssd for > authorization, which is pretty great I might add). What happens here is that > sssd opens it's cache file and when open() is called it gets the lowest file > descriptor which is 0. srun then I guess looks to see if stdin is attached > and as far as it's concerned it is. srun is then reading > /var/lib/sss/mc/passwd and sending that data to every SLURM task. > > stdin is never read by the spawned tasks (hydra_pmi_proxy) so there's data > outstanding on the socket when shutdown is called by stepd and then when > stepd attempts to send the EOF for stderr we hit the issue the reproducer > below highlights. > > If I set the HYDRA_LAUNCHER_EXTRA_ARGS environment variable to "--input none" > in the batch script the problem doesn't happen. I think the "--input none" > argument is something hydra should append by default since it's just going to > close stdin before calling srun. > > This affects the latest mvapich (2.1) as well as older versions. > > -Aaron > > On Mon, Jul 20, 2015 at 6:25 PM, Aaron Knister <[email protected] > <mailto:[email protected]>> wrote: > Dear Slurm Devs, > > I've got a good one for ya that Lyn Gerner discovered. It seems as though > when a large amount of data is piped into srun, somewhat of a race condition > occurs that causes spurious error messages (and seemingly job aborts) like > these: > > srun: error: _server_read: fd 13 got error or unexpected eof reading header > srun: error: step_launch_notify_io_failure: aborting, io error with > slurmstepd on node 0 > > We discovered this when using mvapich 1.8.1 which seems to forward disturbing > amounts of I/O (largely huge chunks of \377 chars) to srun. The sequence of > events seems to be: > > - task exits, _task_write returns an error and the fd gets closed > - slurmstepd notices stdout to the task has disappeared and sends eof message > to srun > - srun reads eof message > - srun forwards i/o from whatever's feeding it, to slurmstepd > - slurmstepd notices stderr has gone away and queues up eof to send to srun > - slurmstepd calls shutdown() on the receive side of the socket > - slurmstepd attempts to send EOF to srun, however if a socket descriptor is > shutdown with data pending further writes to the socket will trigger a TCP RST > - srun sees -ECONNRESET on its end of the socket > > > A reproducer is as simple as this: > > dd if=/dev/urandom bs=1K | srun -n1 sleep 5 > > Interesting to note is this doesn't seem to happen on 2.6 kernels. I'm not > quite sure what to make of that. This article was *really* helpful in > tracking this down: > http://blog.netherlabs.nl/articles/2009/01/18/the-ultimate-so_linger-page-or-why-is-my-tcp-not-reliable > > <http://blog.netherlabs.nl/articles/2009/01/18/the-ultimate-so_linger-page-or-why-is-my-tcp-not-reliable>. > > I've attached 2 rtf files of debug output from srun and slurmstepd with > spacing and highlighting to illustrate what I found. > > I'm not really sure how to fix this but I figure you guys are much more well > versed when it comes to socket communications. I'll try and give it some > thought if I get some time later tonight. > > Best, > Aaron >
