I don't believe any recent slurm changes will change this behavior. I
suspend the key message to investigate here is: "stdin openpty:
Permission denied"
If you want to check the code, the openpty call is in
src/slurmd/slurmstepd/io.c
Moe Jette
Quoting Michael Gutteridge <michael.gutteri...@gmail.com>:
We have a problem submitting interactive jobs to a new environment
we're building out. We are upgrading our nodes from opensuse 11.3 to
12.1, and something about that upgrade is breaking pty allocation.
All nodes have slurm 2.3.2-1.2chaos- submitting "srun --pty" to an
11.3 node works fine, but if I submit to a 12.1 host using "srun
--pty" I get:
slapshot: srun --pty -N1 -n1 -w puck1 /bin/bash -i
srun: Job is in held state, pending scheduler release
srun: job 286 queued and waiting for resources
srun: job 286 has been allocated resources
... about ten minutes passes here...
srun: error: timeout waiting for task launch
srun: Job step 286.0 aborted before step completely launched.
srun: Job step aborted: Waiting up to 2 seconds for job step to finish.
srun: error: Timed out waiting for job step to complete
slapshot:
The host "puck1" is a 12.1 host. In it's log I see:
[2012-01-24T10:30:36] debug: task_slurmd_launch_request: 286.0 0
[2012-01-24T10:30:36] launch task 286.0 request from
34152.34152@140.107.43.101 (port 51922)
[2012-01-24T10:30:36] debug: Checking credential with 256 bytes of sig data
[2012-01-24T10:30:36] switch NONE plugin loaded
[2012-01-24T10:30:36] [286.0] Message thread started pid = 22345
[2012-01-24T10:30:36] [286.0] task NONE plugin loaded
[2012-01-24T10:30:36] debug: task_slurmd_reserve_resources: 286 0
[2012-01-24T10:30:36] [286.0] Checkpoint plugin loaded: checkpoint/none
[2012-01-24T10:30:36] [286.0] mpi type = openmpi
[2012-01-24T10:30:36] [286.0] stdin uses a pty object
[2012-01-24T10:30:36] [286.0] stdin openpty: Permission denied
[2012-01-24T10:30:36] [286.0] IO setup failed: Permission denied
[2012-01-24T10:30:36] [286.0] auth plugin for Munge
(http://home.gna.org/munge/) loaded
[2012-01-24T10:30:36] [286.0] Message thread exited
[2012-01-24T10:30:36] [286.0] done with job
[2012-01-24T10:40:38] debug: _rpc_terminate_job, uid = 6281
[2012-01-24T10:40:38] debug: task_slurmd_release_resources: 286
[2012-01-24T10:40:38] debug: credential for job 286 revoked
If I submit to an opensuse 11.3 node this works just fine. It also
works fine if I submit the job as root:
slapshot: sudo srun --pty -N1 -n1 -w puck1 /bin/bash -i
srun: Job is in held state, pending scheduler release
srun: job 288 queued and waiting for resources
srun: job 288 has been allocated resources
puck1 # exit
slapshot:
[2012-01-24T10:45:35] debug: task_slurmd_launch_request: 288.0 0
[2012-01-24T10:45:35] launch task 288.0 request from
0.0@140.107.43.101 (port 14291)
[2012-01-24T10:45:35] debug: Checking credential with 256 bytes of sig data
[2012-01-24T10:45:35] switch NONE plugin loaded
[2012-01-24T10:45:35] [288.0] Message thread started pid = 22676
[2012-01-24T10:45:35] debug: task_slurmd_reserve_resources: 288 0
[2012-01-24T10:45:35] [288.0] task NONE plugin loaded
[2012-01-24T10:45:35] [288.0] Checkpoint plugin loaded: checkpoint/none
[2012-01-24T10:45:35] [288.0] mpi type = openmpi
[2012-01-24T10:45:35] [288.0] stdin uses a pty object
[2012-01-24T10:45:35] [288.0] init pty size 42:80
[2012-01-24T10:45:35] [288.0] in _window_manager
[2012-01-24T10:45:35] [288.0] debug level = 2
[2012-01-24T10:45:35] [288.0] IO handler started pid=22676
[2012-01-24T10:45:35] [288.0] task 0 (22680) started 2012-01-24T10:45:35
[2012-01-24T10:45:35] [288.0] Sending launch resp rc=0
[2012-01-24T10:45:35] [288.0] auth plugin for Munge
(http://home.gna.org/munge/) loaded
[2012-01-24T10:45:35] [288.0] mpi type = (null)
[2012-01-24T10:45:35] [288.0] Using mpi/openmpi
[2012-01-24T10:45:35] [288.0] task_pre_launch: 288.0, task 0
[2012-01-24T10:45:35] [288.0] Couldn't find SLURM_RLIMIT_CPU in environment
[2012-01-24T10:45:35] [288.0] Couldn't find SLURM_RLIMIT_FSIZE in environment
[2012-01-24T10:45:35] [288.0] Couldn't find SLURM_RLIMIT_DATA in environment
[2012-01-24T10:45:35] [288.0] Couldn't find SLURM_RLIMIT_STACK in environment
[2012-01-24T10:45:35] [288.0] Couldn't find SLURM_RLIMIT_CORE in environment
[2012-01-24T10:45:35] [288.0] Couldn't find SLURM_RLIMIT_RSS in environment
[2012-01-24T10:45:35] [288.0] Couldn't find SLURM_RLIMIT_NPROC in environment
[2012-01-24T10:45:35] [288.0] Couldn't find SLURM_RLIMIT_NOFILE in
environment
[2012-01-24T10:45:35] [288.0] Couldn't find SLURM_RLIMIT_MEMLOCK in
environment
[2012-01-24T10:45:35] [288.0] Couldn't find SLURM_RLIMIT_AS in environment
[2012-01-24T10:45:35] [288.0] Handling REQUEST_INFO
[2012-01-24T10:45:35] [288.0] Handling REQUEST_SIGNAL_CONTAINER
[2012-01-24T10:45:35] [288.0] _handle_signal_container for job 288.0
[2012-01-24T10:46:00] [288.0] task 0 (22680) exited with exit code 0.
[2012-01-24T10:46:00] [288.0] task_post_term: 288.0, task 0
[2012-01-24T10:46:00] [288.0] Waiting for IO
[2012-01-24T10:46:00] [288.0] Closing debug channel
[2012-01-24T10:46:00] [288.0] IO handler exited, rc=0
[2012-01-24T10:46:00] [288.0] _slurm_recv_timeout at 0 of 4, recv zero bytes
[2012-01-24T10:46:00] [288.0] read window size error: Zero Bytes were
transmitted or received
[2012-01-24T10:46:00] debug: _rpc_terminate_job, uid = 6281
[2012-01-24T10:46:00] debug: task_slurmd_release_resources: 288
[2012-01-24T10:46:00] debug: credential for job 288 revoked
[2012-01-24T10:46:00] [288.0] Handling REQUEST_SIGNAL_CONTAINER
[2012-01-24T10:46:00] [288.0] _handle_signal_container for job 288.0
[2012-01-24T10:46:00] [288.0] Sent signal 18 to 288.0
[2012-01-24T10:46:00] [288.0] Handling REQUEST_SIGNAL_CONTAINER
[2012-01-24T10:46:00] [288.0] _handle_signal_container for job 288.0
[2012-01-24T10:46:00] [288.0] Sent signal 15 to 288.0
[2012-01-24T10:46:00] [288.0] Handling REQUEST_STATE
[2012-01-24T10:46:00] [288.0] Message thread exited
[2012-01-24T10:46:00] [288.0] done with job
[2012-01-24T10:46:01] debug: Waiting for job 288's prolog to complete
[2012-01-24T10:46:01] debug: Finished wait for job 288's prolog to complete
[2012-01-24T10:46:01] debug: completed epilog for jobid 288
[2012-01-24T10:46:01] debug: Job 288: sent epilog complete msg: rc = 0
I am using the Moab scheduler, but I get the same behaviour if I
switch to the built-in. I see that 2.3.3 came out this morning- I
didn't immediately see anything that I think would impact this
problem, so I'll hold off on upgrading right now.
Can anyone provide some insight into how to further diagnose this
problem? It's a permissions issue, but I can't fathom why 12.1 and
11.3 would be so different.
Thanks
Michael
