Thanks David. That was my gut feeling and what I think I've seen before but I wasn't sure.
On Mon, Jul 27, 2015 at 3:19 PM, David Bigagli <[email protected]> wrote: > I think that Hydra should kept 0,1,2 open and dup them to /dev/null so > that any children’s > file descriptor will be greater than 2. This is standard Unix way. > > Thanks > > /David/Bigagli > [email protected] > =============================================================== > Slurm User Group Meeting, 15-16 September 2015, Washington D.C. > http://slurm.schedmd.com/slurm_ug_agenda.html > > On 25 Jul 2015, at 05:05, Aaron Knister <[email protected]> wrote: > > tl;dr - if srun is exec'd by a process that has fd 0 close()'d an > external library (in my case nss_sssd) can open a file/socket that ends up > being given fd 0. srun then thinks this is supposed to be stdin for the job > and will read it and pass the data on to other tasks which can cause a > whole host of problems especially if the remote ranks aren't expecting data > on stdin. > > Is it reasonable for srun to check and see if fd 0 is open pretty early in > the game before it calls functions from external nss libraries or spank > plugins and use that to determine if stdin should be poll()'d or read()? > That makes me wonder if I couldn't just implement functionality like this > in a spank plugin-- if fd 0 is closed, then open /dev/null and dup it on fd > 0. > > > Here's the long version: > > I've been working with the devs offline on the stdin shutdown issue (I > highly recommend a SchedMD support contract-- they're great!), but I did > discover a higher level issue that was triggering this. I mentioned that > mvapich's hydra launcher was being very chatty and spamming srun with \377 > characters. I did some investigation into what was going on here and am > posting this back to the list in case others find it helpful. I'll be > opening a ticket with the mvapich folks at some point soon to hopefully get > this change integrated into mvapich. > > What I discovered was that the hydra launcher close()'s stdin before it > exec's srun. That's all fine and dandy but then srun calls some function > that ropes in the nss libraries (and for us the libnss_sssd since we use > sssd for authorization, which is pretty great I might add). What happens > here is that sssd opens it's cache file and when open() is called it gets > the lowest file descriptor which is 0. srun then I guess looks to see if > stdin is attached and as far as it's concerned it is. srun is then reading > /var/lib/sss/mc/passwd and sending that data to every SLURM task. > > stdin is never read by the spawned tasks (hydra_pmi_proxy) so there's data > outstanding on the socket when shutdown is called by stepd and then when > stepd attempts to send the EOF for stderr we hit the issue the reproducer > below highlights. > > If I set the HYDRA_LAUNCHER_EXTRA_ARGS environment variable to "--input > none" in the batch script the problem doesn't happen. I think the "--input > none" argument is something hydra should append by default since it's just > going to close stdin before calling srun. > > This affects the latest mvapich (2.1) as well as older versions. > > -Aaron > > On Mon, Jul 20, 2015 at 6:25 PM, Aaron Knister <[email protected]> > wrote: > >> Dear Slurm Devs, >> >> I've got a good one for ya that Lyn Gerner discovered. It seems as though >> when a large amount of data is piped into srun, somewhat of a race >> condition occurs that causes spurious error messages (and seemingly job >> aborts) like these: >> >> srun: error: _server_read: fd 13 got error or unexpected eof reading >> header >> srun: error: step_launch_notify_io_failure: aborting, io error with >> slurmstepd on node 0 >> >> We discovered this when using mvapich 1.8.1 which seems to forward >> disturbing amounts of I/O (largely huge chunks of \377 chars) to srun. The >> sequence of events seems to be: >> >> - task exits, _task_write returns an error and the fd gets closed >> - slurmstepd notices stdout to the task has disappeared and sends eof >> message to srun >> - srun reads eof message >> - srun forwards i/o from whatever's feeding it, to slurmstepd >> - slurmstepd notices stderr has gone away and queues up eof to send to >> srun >> - slurmstepd calls shutdown() on the receive side of the socket >> - slurmstepd attempts to send EOF to srun, however if a socket descriptor >> is shutdown with data pending further writes to the socket will trigger a >> TCP RST >> - srun sees -ECONNRESET on its end of the socket >> >> >> A reproducer is as simple as this: >> >> dd if=/dev/urandom bs=1K | srun -n1 sleep 5 >> >> Interesting to note is this doesn't seem to happen on 2.6 kernels. I'm >> not quite sure what to make of that. This article was *really* helpful in >> tracking this down: >> http://blog.netherlabs.nl/articles/2009/01/18/the-ultimate-so_linger-page-or-why-is-my-tcp-not-reliable >> . >> >> I've attached 2 rtf files of debug output from srun and slurmstepd with >> spacing and highlighting to illustrate what I found. >> >> I'm not really sure how to fix this but I figure you guys are much more >> well versed when it comes to socket communications. I'll try and give it >> some thought if I get some time later tonight. >> >> Best, >> Aaron >> > > >
