[slurm-users] Requested partition configuration not available now

2018-05-16 Thread Mahmood Naderan
Hi, After creating an account and a partition, I get an error that requested partition configuration not available now. Although I restarted the services on all nodes, I wonder why that happen? [root@rocks7 ~]# rocks run host compute-0-0 "systemctl restart slurmd" [root@rocks7 ~]# rocks run host c

Re: [slurm-users] Requested partition configuration not available now

2018-05-16 Thread Werner Saar
Hi Mahmood, this question is related to the slurm-roll. The command rocks sync slurm has more tasks: 1. Rebuild of 411 is forced 2. on compute nodes, the command /etc/slurm/slurm-prep.sh start is executed 3. on compute nodes, slurmd is restarted 4. slurmctld is restarted. Step 1 and 2 are requi

Re: [slurm-users] Question about sacct

2018-05-16 Thread Chris Bridson (NBI)
Is accounting setup to use a slurmdbd/database backend or file (AccountingStorageType)? 3 minutes could make sense if data are being stored in a (large) flat file. From: slurm-users [mailto:slurm-users-boun...@lists.schedmd.com] On Behalf Of Zohar Roe MLM Sent: 16 May 2018 07:52 To: 'slurm-user

Re: [slurm-users] Requested partition configuration not available now

2018-05-16 Thread Mahmood Naderan
Yes I did that prior to my first email. However, I thought that is similar to the service restart bug in the roll. As you can see below, still the configuration is said to be not available [mahmood@rocks7 ~]$ su Password: [root@rocks7 mahmood]# rocks sync slurm [root@rocks7 mahmood]# exit exit [

Re: [slurm-users] Requested partition configuration not available now

2018-05-16 Thread John Hearns
Mahmood, you should check that the slurm.conf files are identical on the head node and the compute nodes after you run the rocks sync. On 16 May 2018 at 11:07, Mahmood Naderan wrote: > Yes I did that prior to my first email. However, I thought that is > similar to the service restart bug in

[slurm-users] Unable to access job_desc.environment (NULL) from Lua job submission script

2018-05-16 Thread Pablo Llopis
Dear Slurm users, I am trying to write a lua job submission script that sets certain environment variables depending on the partition where the job is being submitted to. When I try to set the environment, I get the following error in the slurmctld log: error: _set_job_env_field: job_desc->envi

Re: [slurm-users] Question about sacct

2018-05-16 Thread Zohar Roe MLM
Hi Chris, The AccountingStorageType is to a file. You probably are right about the long time because of the size (A few giga in size). Maybe this also affect my first question about the wrong status (Since the reading time takes too long). Is there any problem cleaning this file every month (e

Re: [slurm-users] Requested partition configuration not available now

2018-05-16 Thread Mahmood Naderan
Yes they are the same. [root@rocks7 ~]# cp /etc/slurm/slurm.conf rocks7 [root@rocks7 ~]# scp compute-0-0:/etc/slurm/slurm.conf compute-0-0 slurm.conf 100% 2465 3.6MB/s 00:00 [root@rocks7 ~]# scp compute-0-1:/etc/slurm/slurm.conf compute-0-1 slurm.conf 100% 2465 4.7MB/s 00:00 [root@

Re: [slurm-users] Requested partition configuration not available now

2018-05-16 Thread Mahmood Naderan
Interesting thing I found! As I checked the log, I saw part_policy_valid_acct: job's account not permitted to use this partition (RUBY allows Y8 not y8) However, in the command I use "-A Y8" and I am sure about that. The parts file contains PartitionName=RUBY AllowAccounts=Y8 Nodes=compute-

[slurm-users] SLURM nodes flap in "Not responding" status when iptables firewall enabled

2018-05-16 Thread Sean Caron
Hi all, Does anyone use SLURM in a scenario where there is an iptables firewall on the compute nodes on the same network it uses to communicate with the SLURM controller and DBD machine? I have the very basic situation where ... 1. There is no iptables firewall enabled at all on the SLURM contro

Re: [slurm-users] SLURM nodes flap in "Not responding" status when iptables firewall enabled

2018-05-16 Thread Alex Chekholko
Add a logging rule to your iptables and look at what traffic is actually being blocked? On Wed, May 16, 2018 at 11:11 AM Sean Caron wrote: > Hi all, > > Does anyone use SLURM in a scenario where there is an iptables firewall on > the compute nodes on the same network it uses to communicate with

Re: [slurm-users] SLURM nodes flap in "Not responding" status when iptables firewall enabled

2018-05-16 Thread Sean Caron
I see some chatter on 6818/TCP from the compute node to the SLURM controller, and from the SLURM controller to the compute node. The policy is to permit all packets inbound from SLURM controller regardless of port and protocol, and perform no filtering whatsoever on any output packets to anywhere.

Re: [slurm-users] x11 for interactive jobs

2018-05-16 Thread Nadav Toledo
Maybe you've got a mistake? replace: echo -e "optional\tx11.so" >> ./plugstack.conf with echo -e "optional\x11.so" >> ./plugstack.conf On 15/05/2018 21:35, Mahmood Naderan wrote: Hi, I followed the steps described in [1]. However, srun

[slurm-users] X11 debug

2018-05-16 Thread Nadav Toledo
Hello everyone, After fighting with x11 forwarding couple of weeks, I think i've got a few tips that can help others. I am using slurm 17.11.6 with builtin x11 forwarding with ubuntu server distro, all servers in cluster share /home via beegfs. slurm was compile