Trey,
that was the right hint. As our cluster is pretty much closed down from
the outside I don't have a problem opening the firewall for requests
coming from the nodes.
Now that I allowed that it works like a charm.
Thanks a lot,
Uwe
Am 10.07.2014 19:55, schrieb Trey Dockendorf:
Uwe,
I ran into something similar. Our submit host (login node) could not execute srun
commands, they would simply hang. I found that it was due to my iptables rules being
"too strict". I had to allow all incoming traffic from our private network to
the login node. Once that was done all srun commands began working. My controller host
is also on this private network, so I'm not sure if it was the compute node or the slurm
controller that need the access through the firewall. If your attempting to setup
iptables as restrictive as possible, start by allowing incoming from the controller and
see if that fixes the issue.
You can use something like the following to log all iptables DROPs which can be
viewed in dmesg:
<ACCEPT rules>
-A INPUT -m limit --limit 2/min -j LOG --log-prefix "IPTABLES: INPUT DROPPED "
--log-level 7
-A INPUT -j DROP
- Trey
=============================
Trey Dockendorf
Systems Analyst I
Texas A&M University
Academy for Advanced Telecommunications and Learning Technologies
Phone: (979)458-2396
Email: [email protected]
Jabber: [email protected]
----- Original Message -----
From: "Uwe Sauter" <[email protected]>
To: "slurm-dev" <[email protected]>
Sent: Thursday, July 10, 2014 9:41:15 AM
Subject: [slurm-dev] srun not executing command on node
Hi all,
I have a cluster which is set up as following
hostname description
cl4intern admin server, running slurmctld and slurmdbd
cl4fr1 frontend, not running any slurm service but has slurm
installed
n01 -- n54 compute nodes
slurm.conf is shared on all hosts via NFS. cl4fr1 does not appear in
slurm.conf. It should only act as submit node.
Queuing job from cl4fr1 using sbatch is working as expected. But if I
run "srun -N1 -n1 -p mypartition hostname" I do not get the output of
"hostname" but just a terminal window that is waiting for something
to
happen.
> srun -N1 -n1 -p mypartition hostname
With "scontrol show jobs" I can see that a node was allocated but
this
is just sitting around, doing nothing.
JobId=2681 Name=df
UserId=myuser(15001) GroupId=mygroup(15000)
Priority=4294901684 Nice=0 Account=(null) QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=0 ExitCode=0:0
RunTime=00:00:04 TimeLimit=UNLIMITED TimeMin=N/A
SubmitTime=2014-07-10T16:30:26 EligibleTime=2014-07-10T16:30:26
StartTime=2014-07-10T16:30:26 EndTime=Unknown
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=mypartition AllocNode:Sid=cl4fr1:18795
ReqNodeList=(null) ExcNodeList=(null)
NodeList=n52
BatchHost=n52
NumNodes=1 NumCPUs=40 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=0
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
Features=(null) Gres=(null) Reservation=(null)
Shared=0 Contiguous=0 Licenses=(null) Network=(null)
Command=/bin/hostname
WorkDir=/home/myuser
I can cancel this job with CRTL+C without problem:
^Csrun: interrupt (one more within 1 sec to abort)
srun: task 0: unknown
^Csrun: sending Ctrl-C to job 2681.0
srun: Job step 2681.0 aborted before step completely launched.
srun: Job step aborted: Waiting up to 2 seconds for job step to
finish.
srun: error: Timed out waiting for job step to complete
A combination of salloc + env | grep SLURM_JOB_NODELIST + ssh to node
+
srun -p mypartition -N1 -n1 hostname does execute, but on a different
node.
Can someone point me into a direction where to investigate further?
Did
you already have this issue? Is it even possible to have a submit
node
that does not apper in the SLURM configuration??
Thanks in advance,
Uwe Sauter
