Re: [slurm-users] Major newbie - Slurm/jupyterhub
Assuming you need a scheduler for whatever size your user population is, so they need literal JupyterHub, or would they all be satisfied running regular Jupyter notebooks? On May 4, 2020, at 7:25 PM, Lisa Kay Weihl wrote: External Email Warning This email originated from outside the university. Please use caution when opening attachments, clicking links, or responding to requests. I have a single server with 2 cpu, 384gb memory and 4 gpu (GeForce RTX 2080 Ti). Use is to be for GPU ML computing and python based data science. One faculty wants jupyter notebooks, other faculty member is used to using CUDA for GPU but has only done it on a workstation in his lab with a GUI. New faculty member coming in has used nvidia-docker container for GPU (I think on a large cluster, we are just getting started) I'm charged with making all this work and hopefully all at once. Right now I'll take one thing working. So I managed to get Slurm-20.02.1 installed with CUDA-10.2 on CentOS 7 (SE Linux enabled). I posted once before about having trouble getting that combination correct and I finally worked that out. Most of the tests in the test suite seem to run okay. I'm trying to start with very basic Slurm configuration so I haven't enabled accounting. For reference here is my slurm.conf # slurm.conf file generated by configurator easy.html. # Put this file on all nodes of your cluster. # See the slurm.conf man page for more information. # SlurmctldHost=cs-host #authentication AuthType=auth/munge CacheGroups = 0 CryptoType=crypto/munge #Add GPU support GresTypes=gpu # #MailProg=/bin/mail MpiDefault=none #MpiParams=ports=#-# #service ProctrackType=proctrack/cgroup ReturnToService=1 SlurmctldPidFile=/var/run/slurmctld.pid #SlurmctldPort=6817 SlurmdPidFile=/var/run/slurmd.pid #SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmUser=slurm #SlurmdUser=root StateSaveLocation=/var/spool/slurmctld SwitchType=switch/none TaskPlugin=task/affinity # # # TIMERS #KillWait=30 #MinJobAge=300 #SlurmctldTimeout=120 SlurmdTimeout=1800 # # # SCHEDULING SchedulerType=sched/backfill SelectType=select/cons_tres SelectTypeParameters=CR_Core_Memory PriorityType=priority/multifactor PriorityDecayHalfLife=3-0 PriorityMaxAge=7-0 PriorityFavorSmall=YES PriorityWeightAge=1000 PriorityWeightFairshare=0 PriorityWeightJobSize=125 PriorityWeightPartition=1000 PriorityWeightQOS=0 # # # LOGGING AND ACCOUNTING AccountingStorageType=accounting_storage/none ClusterName=cs-host #JobAcctGatherFrequency=30 JobAcctGatherType=jobacct_gather/none SlurmctldDebug=info SlurmctldLogFile=/var/log/slurmctld.log #SlurmdDebug=info SlurmdLogFile=/var/log/slurmd.log # # # COMPUTE NODES NodeName=cs-host CPUs=24 RealMemory=385405 Sockets=2 CoresPerSocket=6 ThreadsPerCore=2 State=UNKNOWN Gres=gpu:4 #PARTITIONS PartitionName=DEFAULT Nodes=cs-host Shared=FORCE:1 Default=YES MaxTime=INFINITE State=UP PartitionName=faculty Priority=10 Default=YES I have jupyterhub running as part of RedHat SCL. It works fine with no integration with Slurm. Now I'm trying to use batchspawner to start a server for the user. Right now I'm just trying one configuration from within the jupyterhub_config.py and trying to keep it simple (see below). When I connect I get this error: 500: Internal Server Error Error in Authenticator.pre_spawn_start: RuntimeError The Jupyter batch job has disappeared while pending in the queue or died immediately after starting. In the jupyterhub.log: [I 2020-05-04 19:47:58.604 JupyterHub base:707] User logged in: csadmin [I 2020-05-04 19:47:58.606 JupyterHub log:174] 302 POST /hub/login?next= -> /hub/spawn (csadmin@127.0.0.1) 227.13ms [I 2020-05-04 19:47:58.748 JupyterHub batchspawner:248] Spawner submitting job using sudo -E -u csadmin sbatch --parsable [I 2020-05-04 19:47:58.749 JupyterHub batchspawner:249] Spawner submitted script: #!/bin/bash #SBATCH --partition=faculty #SBATCH --time=8:00:00 #SBATCH --output=/home/csadmin/jupyterhub_slurmspawner_%j.log #SBATCH --job-name=jupyterhub-spawner #SBATCH --cpus-per-task=1 #SBATCH --chdir=/home/csadmin #SBATCH --uid=csadmin env which jupyterhub-singleuser batchspawner-singleuser jupyterhub-singleuser --ip=0.0.0.0 [I 2020-05-04 19:47:58.831 JupyterHub batchspawner:252] Job submitted. cmd: sudo -E -u csadmin sbatch --parsable output: 7117 [W 2020-05-04 19:47:59.481 JupyterHub batchspawner:377] Job neither pending nor running. [E 2020-05-04 19:47:59.482 JupyterHub user:640] Unhandled error starting csadmin's server: The Jupyter batch job has disappeared while pending in the queue or died immediately after starting. [W 2020-05-04 19:47:59.518 JupyterHub web:1782] 500 GET /hub/spawn (127.0.0.1): Error in Authenticator.pre_spawn_start: RuntimeError The Jupyter batch job has disappeared while pending in the queue or
[slurm-users] Major newbie - Slurm/jupyterhub
I have a single server with 2 cpu, 384gb memory and 4 gpu (GeForce RTX 2080 Ti).
Use is to be for GPU ML computing and python based data science.
One faculty wants jupyter notebooks, other faculty member is used to using CUDA
for GPU but has only done it on a workstation in his lab with a GUI. New
faculty member coming in has used nvidia-docker container for GPU (I think on a
large cluster, we are just getting started)
I'm charged with making all this work and hopefully all at once. Right now I'll
take one thing working.
So I managed to get Slurm-20.02.1 installed with CUDA-10.2 on CentOS 7 (SE
Linux enabled). I posted once before about having trouble getting that
combination correct and I finally worked that out. Most of the tests in the
test suite seem to run okay. I'm trying to start with very basic Slurm
configuration so I haven't enabled accounting.
For reference here is my slurm.conf
# slurm.conf file generated by configurator easy.html.
# Put this file on all nodes of your cluster.
# See the slurm.conf man page for more information.
#
SlurmctldHost=cs-host
#authentication
AuthType=auth/munge
CacheGroups = 0
CryptoType=crypto/munge
#Add GPU support
GresTypes=gpu
#
#MailProg=/bin/mail
MpiDefault=none
#MpiParams=ports=#-#
#service
ProctrackType=proctrack/cgroup
ReturnToService=1
SlurmctldPidFile=/var/run/slurmctld.pid
#SlurmctldPort=6817
SlurmdPidFile=/var/run/slurmd.pid
#SlurmdPort=6818
SlurmdSpoolDir=/var/spool/slurmd
SlurmUser=slurm
#SlurmdUser=root
StateSaveLocation=/var/spool/slurmctld
SwitchType=switch/none
TaskPlugin=task/affinity
#
#
# TIMERS
#KillWait=30
#MinJobAge=300
#SlurmctldTimeout=120
SlurmdTimeout=1800
#
#
# SCHEDULING
SchedulerType=sched/backfill
SelectType=select/cons_tres
SelectTypeParameters=CR_Core_Memory
PriorityType=priority/multifactor
PriorityDecayHalfLife=3-0
PriorityMaxAge=7-0
PriorityFavorSmall=YES
PriorityWeightAge=1000
PriorityWeightFairshare=0
PriorityWeightJobSize=125
PriorityWeightPartition=1000
PriorityWeightQOS=0
#
#
# LOGGING AND ACCOUNTING
AccountingStorageType=accounting_storage/none
ClusterName=cs-host
#JobAcctGatherFrequency=30
JobAcctGatherType=jobacct_gather/none
SlurmctldDebug=info
SlurmctldLogFile=/var/log/slurmctld.log
#SlurmdDebug=info
SlurmdLogFile=/var/log/slurmd.log
#
#
# COMPUTE NODES
NodeName=cs-host CPUs=24 RealMemory=385405 Sockets=2 CoresPerSocket=6
ThreadsPerCore=2 State=UNKNOWN Gres=gpu:4
#PARTITIONS
PartitionName=DEFAULT Nodes=cs-host Shared=FORCE:1 Default=YES MaxTime=INFINITE
State=UP
PartitionName=faculty Priority=10 Default=YES
I have jupyterhub running as part of RedHat SCL. It works fine with no
integration with Slurm. Now I'm trying to use batchspawner to start a server
for the user. Right now I'm just trying one configuration from within the
jupyterhub_config.py and trying to keep it simple (see below).
When I connect I get this error:
500: Internal Server Error
Error in Authenticator.pre_spawn_start: RuntimeError The Jupyter batch job has
disappeared while pending in the queue or died immediately after starting.
In the jupyterhub.log:
[I 2020-05-04 19:47:58.604 JupyterHub base:707] User logged in: csadmin
[I 2020-05-04 19:47:58.606 JupyterHub log:174] 302 POST /hub/login?next= ->
/hub/spawn (csadmin@127.0.0.1) 227.13ms
[I 2020-05-04 19:47:58.748 JupyterHub batchspawner:248] Spawner submitting job
using sudo -E -u csadmin sbatch --parsable
[I 2020-05-04 19:47:58.749 JupyterHub batchspawner:249] Spawner submitted
script:
#!/bin/bash
#SBATCH --partition=faculty
#SBATCH --time=8:00:00
#SBATCH --output=/home/csadmin/jupyterhub_slurmspawner_%j.log
#SBATCH --job-name=jupyterhub-spawner
#SBATCH --cpus-per-task=1
#SBATCH --chdir=/home/csadmin
#SBATCH --uid=csadmin
env
which jupyterhub-singleuser
batchspawner-singleuser jupyterhub-singleuser --ip=0.0.0.0
[I 2020-05-04 19:47:58.831 JupyterHub batchspawner:252] Job submitted. cmd:
sudo -E -u csadmin sbatch --parsable output: 7117
[W 2020-05-04 19:47:59.481 JupyterHub batchspawner:377] Job neither pending
nor running.
[E 2020-05-04 19:47:59.482 JupyterHub user:640] Unhandled error starting
csadmin's server: The Jupyter batch job has disappeared while pending in the
queue or died immediately after starting.
[W 2020-05-04 19:47:59.518 JupyterHub web:1782] 500 GET /hub/spawn (127.0.0.1):
Error in Authenticator.pre_spawn_start: RuntimeError The Jupyter batch job has
disappeared while pending in the queue or died immediately after starting.
[E 2020-05-04 19:47:59.521 JupyterHub log:166] {
"X-Forwarded-Host": "localhost:8000",
"X-Forwarded-Proto": "http",
"X-Forwarded-Port": "8000",
"X-Forwarded-For": "127.0.0.1",
"Cookie": "jupyterhub-hub-login=[secret]; _xsrf=[secret];
jupyterhub-session-id=[secret]",
"Accept-Language": "en-US,en;q=0.9",
"Accept-Encoding": "gzip, deflate, br",
Re: [slurm-users] not allocating jobs even resources are free
Thanks Denial for detailed Description Regards Navin On Sun, May 3, 2020, 13:35 Daniel Letai wrote: > > On 29/04/2020 12:00:13, navin srivastava wrote: > > Thanks Daniel. > > All jobs went into run state so unable to provide the details but > definitely will reach out later if we see similar issue. > > i am more interested to understand the FIFO with Fair Tree.it will be good > if anybody provide some insight on this combination and also if we will > enable the backfilling here how the behaviour will change. > > what is the role of the Fair tree here? > > Fair tree is the algorithm used to calculate the interim priority, before > applying weight, but I think after the halflife decay. > > > To make it simple - fifo without fairshare would assign priority based > only on submission time. With faishare, that naive priority is adjusted > based on prior usage by the applicable entities (users/departments - > accounts). > > > Backfill will let you utilize your resources better, since it will allow > "inserting" low priority jobs before higher priority jobs, provided all > jobs have defined wall times, and any inserted job doesn't affect in any > way the start time of a higher priority job, thus allowing utilization of > "holes" when the scheduler waits for resources to free up, in order to > insert some large job. > > > Suppose the system is at 60% utilization of cores, and the next fifo job > requires 42% - it will wait until 2% are free so it can begin, meanwhile > not allowing any job to start, even if it would tke only 30% of the > resources (whic are currently free) and would finish before the 2% are free > anyway. > > Backfill would allow such job to start, as long as it's wall time ensures > it would finish before the 42% job would've started. > > > Fairtree in either case (fifo or backfill) calculates the priority for > each job the same - if the account had used more resources recently (the > halflife decay factor) it would get a lower priority even though it was > submitted earlier than a job from an account that didn't use any resources > recently. > > > As can be expected, backtree has to loop over all jobs in the queue, in > order to see if any job can fit out of order. In very busy/active systems, > that can lead to poor response times, unless tuned correctly in slurm conf > - look at SchedulerParameters, all params starting with bf_ and in > particular bf_max_job_test= ,bf_max_time= and bf_continue (but bf_window= > can also have some impact if set too high). > > see the man page at > https://slurm.schedmd.com/slurm.conf.html#OPT_SchedulerParameters > > > PriorityType=priority/multifactor > PriorityDecayHalfLife=2 > PriorityUsageResetPeriod=DAILY > PriorityWeightFairshare=50 > PriorityFlags=FAIR_TREE > > Regards > Navin. > > > > On Mon, Apr 27, 2020 at 9:37 PM Daniel Letai wrote: > >> Are you sure there are enough resources available? The node is in mixed >> state, so it's configured for both partitions - it's possible that earlier >> lower priority jobs are already running thus blocking the later jobs, >> especially since it's fifo. >> >> >> It would really help if you pasted the results of: >> >> squeue >> >> sinfo >> >> >> As well as the exact sbatch line, so we can see how many resources per >> node are requested. >> >> >> On 26/04/2020 12:00:06, navin srivastava wrote: >> >> Thanks Brian, >> >> As suggested i gone through document and what i understood that the fair >> tree leads to the Fairshare mechanism and based on that the job should be >> scheduling. >> >> so it mean job scheduling will be based on FIFO but priority will be >> decided on the Fairshare. i am not sure if both conflicts here.if i see the >> normal jobs priority is lower than the GPUsmall priority. so resources are >> available with gpusmall partition then it should go. there is no job pend >> due to gpu resources. the gpu resources itself not asked with the job. >> >> is there any article where i can see how the fairshare works and which >> are setting should not be conflict with this. >> According to document it never says that if fair-share is applied then >> FIFO should be disabled. >> >> Regards >> Navin. >> >> >> >> >> >> On Sat, Apr 25, 2020 at 12:47 AM Brian W. Johanson >> wrote: >> >>> >>> If you haven't looked at the man page for slurm.conf, it will answer >>> most if not all your questions. >>> https://slurm.schedmd.com/slurm.conf.html but I would depend on the the >>> manual version that was distributed with the version you have installed as >>> options do change. >>> >>> There is a ton of information that is tedious to get through but reading >>> through it multiple times opens many doors. >>> >>> DefaultTime is listed in there as a Partition option. >>> If you are scheduling gres/gpu resources, it's quite possible there are >>> cores available with no corresponding gpus avail. >>> >>> -b >>> >>> On 4/24/20 2:49 PM, navin srivastava wrote: >>> >>> Thanks Brian. >>> >>> I need to check the jobs
[slurm-users] slurm problem with GrpTres
Hello, I have a problem with GrpTres, I specify the limits with sacctmgr --immediate modify user where user= set GrpTres=cpu=144,node=4 but when the user send serial jobs, for example 5 jobs , the user only can execute 4, and the rest of the jobs are PD with the reason=AssocGrpNodeLimit. I could understand this if the jobs were in differents nodes, but all of them are running in the same node JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 887783 cluster mut PD 0:00 1 (AssocGrpNodeLimit) 887784 cluster mut PD 0:00 1 (AssocGrpNodeLimit) 887785 cluster mut PD 0:00 1 (AssocGrpNodeLimit) 887780 cluster mut R0:02 1 xula1301 887781 cluster mut R0:02 1 xula1301 887782 cluster mut R0:02 1 xula1301 887779 cluster mut R0:05 1 xula1301 I want that the users could use until 4 nodes or/and 144 cores. With parallel jobs it works fine and if the user send a job with 144 serial jobs inside it then it works too. The problem is when the user send serial jobs, then the limit of the node=4 works like job=4, and that isn´t my intention. Any help, please? Thanks in advance Angelines Alberto Morillas Unidad de Arquitectura Informática Despacho: 22.1.32 Telf.: +34 91 346 6119 Fax: +34 91 346 6537 skype: angelines.alberto CIEMAT Avenida Complutense, 40 28040 MADRID
