Dear Slurm Devs, I've got a good one for ya that Lyn Gerner discovered. It seems as though when a large amount of data is piped into srun, somewhat of a race condition occurs that causes spurious error messages (and seemingly job aborts) like these:
srun: error: _server_read: fd 13 got error or unexpected eof reading header srun: error: step_launch_notify_io_failure: aborting, io error with slurmstepd on node 0 We discovered this when using mvapich 1.8.1 which seems to forward disturbing amounts of I/O (largely huge chunks of \377 chars) to srun. The sequence of events seems to be: - task exits, _task_write returns an error and the fd gets closed - slurmstepd notices stdout to the task has disappeared and sends eof message to srun - srun reads eof message - srun forwards i/o from whatever's feeding it, to slurmstepd - slurmstepd notices stderr has gone away and queues up eof to send to srun - slurmstepd calls shutdown() on the receive side of the socket - slurmstepd attempts to send EOF to srun, however if a socket descriptor is shutdown with data pending further writes to the socket will trigger a TCP RST - srun sees -ECONNRESET on its end of the socket A reproducer is as simple as this: dd if=/dev/urandom bs=1K | srun -n1 sleep 5 Interesting to note is this doesn't seem to happen on 2.6 kernels. I'm not quite sure what to make of that. This article was *really* helpful in tracking this down: http://blog.netherlabs.nl/articles/2009/01/18/the-ultimate-so_linger-page-or-why-is-my-tcp-not-reliable
