tl;dr - if srun is exec'd by a process that has fd 0 close()'d an external library (in my case nss_sssd) can open a file/socket that ends up being given fd 0. srun then thinks this is supposed to be stdin for the job and will read it and pass the data on to other tasks which can cause a whole host of problems especially if the remote ranks aren't expecting data on stdin.
Is it reasonable for srun to check and see if fd 0 is open pretty early in the game before it calls functions from external nss libraries or spank plugins and use that to determine if stdin should be poll()'d or read()? That makes me wonder if I couldn't just implement functionality like this in a spank plugin-- if fd 0 is closed, then open /dev/null and dup it on fd 0. Here's the long version: I've been working with the devs offline on the stdin shutdown issue (I highly recommend a SchedMD support contract-- they're great!), but I did discover a higher level issue that was triggering this. I mentioned that mvapich's hydra launcher was being very chatty and spamming srun with \377 characters. I did some investigation into what was going on here and am posting this back to the list in case others find it helpful. I'll be opening a ticket with the mvapich folks at some point soon to hopefully get this change integrated into mvapich. What I discovered was that the hydra launcher close()'s stdin before it exec's srun. That's all fine and dandy but then srun calls some function that ropes in the nss libraries (and for us the libnss_sssd since we use sssd for authorization, which is pretty great I might add). What happens here is that sssd opens it's cache file and when open() is called it gets the lowest file descriptor which is 0. srun then I guess looks to see if stdin is attached and as far as it's concerned it is. srun is then reading /var/lib/sss/mc/passwd and sending that data to every SLURM task. stdin is never read by the spawned tasks (hydra_pmi_proxy) so there's data outstanding on the socket when shutdown is called by stepd and then when stepd attempts to send the EOF for stderr we hit the issue the reproducer below highlights. If I set the HYDRA_LAUNCHER_EXTRA_ARGS environment variable to "--input none" in the batch script the problem doesn't happen. I think the "--input none" argument is something hydra should append by default since it's just going to close stdin before calling srun. This affects the latest mvapich (2.1) as well as older versions. -Aaron On Mon, Jul 20, 2015 at 6:25 PM, Aaron Knister <[email protected]> wrote: > Dear Slurm Devs, > > I've got a good one for ya that Lyn Gerner discovered. It seems as though > when a large amount of data is piped into srun, somewhat of a race > condition occurs that causes spurious error messages (and seemingly job > aborts) like these: > > srun: error: _server_read: fd 13 got error or unexpected eof reading header > srun: error: step_launch_notify_io_failure: aborting, io error with > slurmstepd on node 0 > > We discovered this when using mvapich 1.8.1 which seems to forward > disturbing amounts of I/O (largely huge chunks of \377 chars) to srun. The > sequence of events seems to be: > > - task exits, _task_write returns an error and the fd gets closed > - slurmstepd notices stdout to the task has disappeared and sends eof > message to srun > - srun reads eof message > - srun forwards i/o from whatever's feeding it, to slurmstepd > - slurmstepd notices stderr has gone away and queues up eof to send to srun > - slurmstepd calls shutdown() on the receive side of the socket > - slurmstepd attempts to send EOF to srun, however if a socket descriptor > is shutdown with data pending further writes to the socket will trigger a > TCP RST > - srun sees -ECONNRESET on its end of the socket > > > A reproducer is as simple as this: > > dd if=/dev/urandom bs=1K | srun -n1 sleep 5 > > Interesting to note is this doesn't seem to happen on 2.6 kernels. I'm not > quite sure what to make of that. This article was *really* helpful in > tracking this down: > http://blog.netherlabs.nl/articles/2009/01/18/the-ultimate-so_linger-page-or-why-is-my-tcp-not-reliable > . > > I've attached 2 rtf files of debug output from srun and slurmstepd with > spacing and highlighting to illustrate what I found. > > I'm not really sure how to fix this but I figure you guys are much more > well versed when it comes to socket communications. I'll try and give it > some thought if I get some time later tonight. > > Best, > Aaron >
