[slurm-dev] Re: Account preferation
Hello Manuel, thank you very much for your insight. I have read about FairShare too, but I am not sure if/how I can use it for this specific example. But I will consider it! Or, can anyone confirm FairShare should work in this case? Just for clarification: Can't I use backfill and multifactor priority together? First goes backfill, depending on the job length and then it is sorted by priority? Atleast thats were my first thoughts about it :D Best regards, Dennis Am 08.08.2017 um 14:53 schrieb Manuel Rodríguez Pascual: > Hi Dennis, > > If I undersand you correctly, what you need is to use the Multifactor Plugin > https://slurm.schedmd.com/priority_multifactor.html > > In particular, I guess this is relevant for your installation: > > Note: Computing the fair-share factor requires the installation and > operation of the Slurm Accounting Database to provide the assigned > shares and the consumed, computing resources described below. > > and > > PriorityDecayHalfLife This determines the contribution of historical > usage on the composite usage value. The larger the number, the longer > past usage affects fair-share. If set to 0 no decay will be applied. > This is helpful if you want to enforce hard time limits per > association. If set to 0 PriorityUsageResetPeriod must be set to some > interval. The unit is a time string (i.e. min, hr:min:00, > days-hr:min:00, or days-hr). The default value is 7-0 (7 days). > > > Also, note that backfill and multifactor are two different things: > Backfill tries to run small jobs if this does not affect to other ones > waiting on the queue; Multifactor determines the order of the queue. > They are set with different parameters in slurm.conf > > SchedulerType=sched/backfill > PriorityType=priority/multifactor > > > I am not a slurm expert so I may probably be wrong on some > information, but I hope this can serve you as a small help on your > problem. > > Best, > > > Manuel > > 2017-08-08 14:37 GMT+02:00 Dennis Tants <dennis.ta...@zarm.uni-bremen.de>: >> Hey guys, >> >> I was asked to implement a walltime of 24 hours in our cluster (we only >> have 16 nodes, so no need until now). >> Furthermore, I need to prefer single accounts based on compute time. The >> example I was given is further below. >> >> Btw., I'm upgrading to SLURM 17.02.4. >> >> Example: >> If the partition is full, we should start to prefer people who haven't >> used much computing time the last month. >> >> I don't know where I should start with this request to be honest and if >> it is even possible (walltime is not the problem). >> >> First of, I would need to also implement accounting I guess (also, no >> need for until now^^) >> But how to continue? I wanted to implement backfill instead of FIFO, but >> would I also need multifactor priority? >> >> Here are my thoughts of being able to achieve this: >> >> 1. Could I do something like: >> A. People are getting points every month (which they can't actually >> empty within one month) >> B. Compare the points (last month) of all people when a queue appears. >> >> 2. Or something like (same but time based): >> A. Account the whole computing time for two months (and then start to >> delete old entries) >> B. While queue, compare computing time (last month) of all waiting users? >> >> I hope I made myself clear and that you can help me. Every hint is >> appreciated. >> >> Best regards, >> Dennis >> >> -- >> Dennis Tants >> Auszubildender: Fachinformatiker für Systemintegration >> >> ZARM - Zentrum für angewandte Raumfahrttechnologie und Mikrogravitation >> ZARM - Center of Applied Space Technology and Microgravity >> >> Universität Bremen >> Am Fallturm >> 28359 Bremen, Germany >> >> Telefon: 0421 218 57940 >> E-Mail: ta...@zarm.uni-bremen.de >> >> www.zarm.uni-bremen.de >> -- Dennis Tants Auszubildender: Fachinformatiker für Systemintegration ZARM - Zentrum für angewandte Raumfahrttechnologie und Mikrogravitation ZARM - Center of Applied Space Technology and Microgravity Universität Bremen Am Fallturm 28359 Bremen, Germany Telefon: 0421 218 57940 E-Mail: ta...@zarm.uni-bremen.de www.zarm.uni-bremen.de
[slurm-dev] Re: slurm + openmpi + suspend problem
Hello Eugene,
it is just a wild guess, but could you try "srun --mpi=pmi2"(you said
you built OMPI with pmi support) instead of "mpirun".
srun is build-in and I think the preferred way of running parallel
processes. Maybe scontrol is able to suspend it this way.
Regards,
Dennis
Am 10.07.2017 um 22:20 schrieb Eugene Dedits:
> Hello SLURM-DEV
>
>
> I have a problem with slurm, openmpi, and “scontrol suspend”.
>
> My setup is:
> 96-node cluster with IB, running rhel 6.8
> slurm 17.02.1
> openmpi 2.0.0 (built using Intel 2016 compiler)
>
>
> I am running some application (hpl in this particular case) using batch
> script similar to:
> -
> #!/bin/bash
> #SBATCH —partiotion=standard
> #SBATCH -N 10
> #SBATCH —ntasks-per-node=16
>
> mpirun -np 160 xhpl | tee LOG
> -
>
> So I am running it on 160 cores, 2 nodes.
>
> Once job is submitted to the queue and is running I suspend it using
> ~# scontrol suspend JOBID
>
> I see that indeed my job stopped producing output. I go to each of the 10
> nodes that were assigned for my job and see if the xhpl processes are running
> there with :
>
> ~# for i in {10..19}; do ssh node$i “top -b -n | head -n 50 | grep xhpl | wc
> -l”; done
>
> I expect this little script to return 0 from every node (because suspend sent
> the
> SIGSTOP and they shouldn’t show up in top). However I see that processes
> are reliable suspended only on node10. I get:
> 0
> 16
> 16
> …
> 16
>
> So 9 out of 10 nodes still have 16 MPI threads of my xhpl application running
> at 100%.
>
> If I run “scontrol resume JOBID” and then suspend it again I see that
> (sometimes) more
> nodes have “xhpl” processes properly suspended. Every time I resume and
> suspend the
> job, I see different nodes returning 0 in my “ssh-run-top” script.
>
> So all together it looks like the suspend mechanism doesn’t properly work in
> SLURM with
> OpenMPI. I’ve tried compiling OpenMPI with “—with-slurm
> —with-pmi=/path/to/my/slurm”.
> I’ve observed the same behavior.
>
> I would appreciate any help.
>
>
> Thanks,
> Eugene.
>
>
>
>
--
Dennis Tants
Auszubildender: Fachinformatiker für Systemintegration
ZARM - Zentrum für angewandte Raumfahrttechnologie und Mikrogravitation
ZARM - Center of Applied Space Technology and Microgravity
Universität Bremen
Am Fallturm
28359 Bremen, Germany
Telefon: 0421 218 57940
E-Mail: ta...@zarm.uni-bremen.de
www.zarm.uni-bremen.de
[slurm-dev] Re: srun can't use variables in a batch script after upgrade
Hello list, thank you for your thoughts and help. I found my problem to be caused by myself (as always). Using srun to copy my files, I also copied the output file which somehow resulted in SLURM not being able to log further to it, except the end. It switched copy with rsync and excluded my .sbatch .out files. Now running the script worked like a charm. Thank you guys. Best regards, Dennis Am 10.07.2017 um 10:13 schrieb Carlos Fenoy: > Re: [slurm-dev] Re: srun can't use variables in a batch script after > upgrade > Hi, > > any idea why the output of your job is not complete? There is nothing > after "Copying files...". Does the /work/tants directory exists in all > the nodes? The variable $SLURM_JOB_NAME is interpreted by bash so srun > only sees "srun -N2 -n2 rm -rf /work/tants/mpicopytest" > > Regards, > Carlos > > On Mon, Jul 10, 2017 at 10:02 AM, Dennis Tants > <dennis.ta...@zarm.uni-bremen.de > <mailto:dennis.ta...@zarm.uni-bremen.de>> wrote: > > > Hello Loris, > > Am 10.07.2017 um 07:39 schrieb Loris Bennett: > > Hi Dennis, > > > > Dennis Tants <dennis.ta...@zarm.uni-bremen.de > <mailto:dennis.ta...@zarm.uni-bremen.de>> writes: > > > >> Hi list, > >> > >> I am a little bit lost right now and would appreciate your help. > >> We have a little cluster with 16 nodes running with SLURM and it is > >> doing everything we want, except a few > >> little things I want to improve. > >> > >> So that is why I wanted to upgrade our old SLURM 15.X (don't > know the > >> exact version) to 17.02.4 on my test machine. > >> I just deleted the old version completely with 'yum erase slurm-*' > >> (CentOS 7 btw.) and build the new version with rpmbuild. > >> Everything went fine so I started configuring a new > slurm[dbd].conf. > >> This time I also wanted to integrate backfill instead of FIFO > >> and also use accounting (just to know which person uses the most > >> resources). Because we had no databases yet I started > >> slurmdbd and slurmctld without problems. > >> > >> Everything seemed fine with a simple mpi hello world test on > one and two > >> nodes. > >> Now I wanted to enhance the script a bit more and include > working in the > >> local directory of the nodes which is /work. > >> To get everything up and running I used the script which I > attached for > >> you (it also includes the output after running the script). > >> It should basically just copy all data to > /work/tants/$SLURM_JOB_NAME > >> before doing the mpi hello world. > >> But it seems that srun does not know $SLURM_JOB_NAME even > though it is > >> there. > >> /work/tants belongs to the correct user and has rwx permissions. > >> > >> So did I just configure something wrong or what happened here? > Nearly > >> the same example is working on our cluster with > >> 15.X. The script is only for testing purposes, thats why there > are so > >> many echo commands in there. > >> If you see any mistake or can recommend better configurations I > would > >> glady hear them. > >> Should you need any more information I will provide them. > >> Thank you for your time! > > Shouldn't the variable be $SBATCH_JOB_NAME? > > > > Cheers, > > > > Loris > > > > when I use "echo $SLURM_JOB_NAME" it will tell me the name I specified > with #SBATCH -J. > It is not working with srun in this version (it was working in 15.x). > > However, when I now use "echo $SBATCH_JOB_NAME" it is just a blank > variable. As told by someone from the list, > I used the command "env" to verify which variables are available. This > list includes SLURM_JOB_NAME > with the name I specified. So $SLURM_JOB_NAME shouldn't be a problem. > > Thank you for your suggestion though. > Any other hints? > > Best regards, > Dennis > > -- > Dennis Tants > Auszubildender: Fachinformatiker für Systemintegration > > ZARM - Zentrum für angewandte Raumfahrttechnologie und > Mikrogravitation > ZARM - Center of Applied Space Technology and Microgravity > > Universität Bremen > Am Fallturm > 28359 Bremen, Germany > > Telefon: 0421 218 57940 > E-Mail: ta...@zarm.uni-bremen.de <mailto:ta...@zarm.uni-bremen.de> > > www.zarm.uni-bremen.de <http://www.zarm.uni-bremen.de> > > > > > -- > -- > Carles Fenoy -- Dennis Tants Auszubildender: Fachinformatiker für Systemintegration ZARM - Zentrum für angewandte Raumfahrttechnologie und Mikrogravitation ZARM - Center of Applied Space Technology and Microgravity Universität Bremen Am Fallturm 28359 Bremen, Germany Telefon: 0421 218 57940 E-Mail: ta...@zarm.uni-bremen.de www.zarm.uni-bremen.de
[slurm-dev] Re: srun can't use variables in a batch script after upgrade
Hello Loris, Am 10.07.2017 um 07:39 schrieb Loris Bennett: > Hi Dennis, > > Dennis Tants <dennis.ta...@zarm.uni-bremen.de> writes: > >> Hi list, >> >> I am a little bit lost right now and would appreciate your help. >> We have a little cluster with 16 nodes running with SLURM and it is >> doing everything we want, except a few >> little things I want to improve. >> >> So that is why I wanted to upgrade our old SLURM 15.X (don't know the >> exact version) to 17.02.4 on my test machine. >> I just deleted the old version completely with 'yum erase slurm-*' >> (CentOS 7 btw.) and build the new version with rpmbuild. >> Everything went fine so I started configuring a new slurm[dbd].conf. >> This time I also wanted to integrate backfill instead of FIFO >> and also use accounting (just to know which person uses the most >> resources). Because we had no databases yet I started >> slurmdbd and slurmctld without problems. >> >> Everything seemed fine with a simple mpi hello world test on one and two >> nodes. >> Now I wanted to enhance the script a bit more and include working in the >> local directory of the nodes which is /work. >> To get everything up and running I used the script which I attached for >> you (it also includes the output after running the script). >> It should basically just copy all data to /work/tants/$SLURM_JOB_NAME >> before doing the mpi hello world. >> But it seems that srun does not know $SLURM_JOB_NAME even though it is >> there. >> /work/tants belongs to the correct user and has rwx permissions. >> >> So did I just configure something wrong or what happened here? Nearly >> the same example is working on our cluster with >> 15.X. The script is only for testing purposes, thats why there are so >> many echo commands in there. >> If you see any mistake or can recommend better configurations I would >> glady hear them. >> Should you need any more information I will provide them. >> Thank you for your time! > Shouldn't the variable be $SBATCH_JOB_NAME? > > Cheers, > > Loris > when I use "echo $SLURM_JOB_NAME" it will tell me the name I specified with #SBATCH -J. It is not working with srun in this version (it was working in 15.x). However, when I now use "echo $SBATCH_JOB_NAME" it is just a blank variable. As told by someone from the list, I used the command "env" to verify which variables are available. This list includes SLURM_JOB_NAME with the name I specified. So $SLURM_JOB_NAME shouldn't be a problem. Thank you for your suggestion though. Any other hints? Best regards, Dennis -- Dennis Tants Auszubildender: Fachinformatiker für Systemintegration ZARM - Zentrum für angewandte Raumfahrttechnologie und Mikrogravitation ZARM - Center of Applied Space Technology and Microgravity Universität Bremen Am Fallturm 28359 Bremen, Germany Telefon: 0421 218 57940 E-Mail: ta...@zarm.uni-bremen.de www.zarm.uni-bremen.de
[slurm-dev] srun can't use variables in a batch script after upgrade
Hi list, I am a little bit lost right now and would appreciate your help. We have a little cluster with 16 nodes running with SLURM and it is doing everything we want, except a few little things I want to improve. So that is why I wanted to upgrade our old SLURM 15.X (don't know the exact version) to 17.02.4 on my test machine. I just deleted the old version completely with 'yum erase slurm-*' (CentOS 7 btw.) and build the new version with rpmbuild. Everything went fine so I started configuring a new slurm[dbd].conf. This time I also wanted to integrate backfill instead of FIFO and also use accounting (just to know which person uses the most resources). Because we had no databases yet I started slurmdbd and slurmctld without problems. Everything seemed fine with a simple mpi hello world test on one and two nodes. Now I wanted to enhance the script a bit more and include working in the local directory of the nodes which is /work. To get everything up and running I used the script which I attached for you (it also includes the output after running the script). It should basically just copy all data to /work/tants/$SLURM_JOB_NAME before doing the mpi hello world. But it seems that srun does not know $SLURM_JOB_NAME even though it is there. /work/tants belongs to the correct user and has rwx permissions. So did I just configure something wrong or what happened here? Nearly the same example is working on our cluster with 15.X. The script is only for testing purposes, thats why there are so many echo commands in there. If you see any mistake or can recommend better configurations I would glady hear them. Should you need any more information I will provide them. Thank you for your time! Best regards, Dennis -- Dennis Tants Auszubildender: Fachinformatiker für Systemintegration ZARM - Zentrum für angewandte Raumfahrttechnologie und Mikrogravitation ZARM - Center of Applied Space Technology and Microgravity Universität Bremen Am Fallturm 28359 Bremen, Germany Telefon: 0421 218 57940 E-Mail: ta...@zarm.uni-bremen.de www.zarm.uni-bremen.de #!/bin/bash #SBATCH -J mpicopytest #SBATCH -t 01:00:00 #SBATCH -N 2 -n 8 #SBATCH -o copy-test001-%j.out # Vorbereiten echo "Script start" echo "Job name: $SLURM_JOB_NAME with ID: $SLURM_JOB_ID" echo "Allocated nodes: $SLURM_JOB_NODELIST" echo "Amount of tasks to run: $SLURM_NTASKS" echo "Changing to submit dir: $SLURM_SUBMIT_DIR" cd $SLURM_SUBMIT_DIR echo "Preparing data..." echo "Copying files to /work on the nodes" srun -N2 -n2 mkdir /work/tants/$SLURM_JOB_NAME srun -N2 -n2 cp -r * /work/tants/$SLURM_JOB_NAME echo "Finished prepation" # Start der eigentlichen Aufgaben echo "Starting computation" module load OpenMPI/1.10.2 cd /work/tants/$SLURM_JOB_NAME time mpirun -np $SLURM_NTASKS ./mpi-hello-world time srun -N2 -n8 ./mpi-hello-world echo "Ending computation" # Aufräumen echo "Starting cleanup process, copying back to the submit dir and deleting files in /work" srun -N2 -n2 cp -r * $SLURM_SUBMIT_DIR srun -N2 -n2 rm -rf /work/tants/$SLURM_JOB_NAME echo "Script end" # Actual output of the output file I specified Output of copy-test001-%j.out Script start Job name: mpicopytest with ID: 54 Allocated nodes: node[1-2] Amount of tasks to run: 8 Changing to submit dir: /home/tants/pbs Preparing data... Copying files to /work on the nodes Script end # slurm.conf file generated by configurator.html. # Put this file on all nodes of your cluster. # See the slurm.conf man page for more information. # ControlMachine=headnode ControlAddr=192.168.80.254 #BackupController= #BackupAddr= # AuthType=auth/munge #CheckpointType=checkpoint/none CryptoType=crypto/munge #DisableRootJobs=NO EnforcePartLimits=YES #Epilog= #EpilogSlurmctld= #FirstJobId=1 #MaxJobId=99 #GresTypes= #GroupUpdateForce=0 #GroupUpdateTime=600 #JobCheckpointDir=/var/slurm/checkpoint #JobCredentialPrivateKey= #JobCredentialPublicCertificate= JobFileAppend=1 JobRequeue=0 #JobSubmitPlugins=1 #KillOnBadExit=0 LaunchType=launch/slurm #Licenses=foo*4,bar MailProg=/bin/mail MaxJobCount=5000 MaxStepCount=4 #MaxTasksPerNode=128 MpiDefault=none MpiParams=ports=12000-12999 #PluginDir= #PlugStackConfig= PrivateData=jobs ProctrackType=proctrack/cgroup #Prolog= #PrologFlags= #PrologSlurmctld= PropagatePrioProcess=0 #PropagateResourceLimits= #PropagateResourceLimitsExcept= #RebootProgram= ReturnToService=1 #SallocDefaultCommand= SlurmctldPidFile=/var/run/slurmctld.pid SlurmctldPort=6817 SlurmdPidFile=/var/run/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmUser=slurm #SlurmdUser=root #SrunEpilog= #SrunProlog= StateSaveLocation=/var/spool/slurmctld SwitchType=switch/none #TaskEpilog= TaskPlugin=task/none #TaskPluginParam= #TaskProlog= TopologyPlugin=topology/none TmpFS=/tmp #TrackWCKey=no TreeWidth=8 #UnkillableStepProgram= UsePAM=0 # # # TIMERS BatchStartTimeout=10 CompleteWait=8 #Ep
