Re: [slurm-users] Is there any public scientific-workflow example that can be run through Slurm?

[Bernstein, Noam CIV USN NRL (6393) Washington DC (USA)]

I'm the lead developer of another workflow system, wfl 
(github.com/libAtoms/workflow), which 
works with slurm using an abstraction layer we also developed, ExPyRe 
(github.com/libAtoms/Expyre), but in writing 
a recent paper about it we looked at other systems, and ones that we know of 
that use queuing systems include:
ASR atomic simulation recipes (which uses MyQueue)
Atomate/Fireworks
PyIron/Pysqua
AiiDA
icolos (https://github.com/MolecularAI/Icolos)
qmpy (part of OQMD)

Note that I'm not promising that they currently support slurm, but it's a list 
to start your research from.

Noam

Re: [slurm-users] Is there any public scientific-workflow example that can be run through Slurm?

[Ole Holm Nielsen]

Hi Alper,

On 18-08-2023 18:39, Alper Alimoglu wrote:
In slurm we can build pipelines using [slurm dependencies][1], which 
allows us to run workflows.


In my work, I have stuck in a point regarding finding a workflow that I 
can run using Slurm.


As an example, I have to use a workflow benchmark like in here 
https://pegasus.isi.edu/workflow_gallery/ 
 , but all of them are 
implemented inside Pegasus.


I was wondering is there any public scientific-workflow examples that 
can be run through Slurm?


Any suggestion is highly appreciated.

   [1]: 
https://www.hpc.caltech.edu/documentation/faq/dependencies-and-pipelines 


My colleagues at Technical University of Denmark are heavy users of 
workflows through Slurm on our cluster, and this accounts for most of 
our usage.  They have developed an Open Source workflow system:



MyQueue is a frontend for SLURM/PBS/LSF that makes handling of tasks easy. It 
has a command-line interface called mq with a number of Sub-commands and a 
Python interface for managing Workflows. Simple to set up: no system 
administrator or database required.


See https://myqueue.readthedocs.io/en/latest/

I'm personally not involved in MyQueue, but you might take a look to see 
if it's useful in your environment.


Best regards,
Ole

[slurm-users] Is there any public scientific-workflow example that can be run through Slurm?

[Alper Alimoglu]

In slurm we can build pipelines using [slurm dependencies][1], which allows
us to run workflows.

In my work, I have stuck in a point regarding finding a workflow that I can
run using Slurm.

As an example, I have to use a workflow benchmark like in here
https://pegasus.isi.edu/workflow_gallery/ , but all of them are implemented
inside Pegasus.

I was wondering is there any public scientific-workflow examples that can
be run through Slurm?

Any suggestion is highly appreciated.

  [1]:
https://www.hpc.caltech.edu/documentation/faq/dependencies-and-pipelines

-Alper

Re: [slurm-users] extended list of nodes allocated to a job

[Ryan Novosielski]

I didn’t know that one! Thank you.

Sent from my iPhone

On Aug 17, 2023, at 09:50, Alain O' Miniussi  wrote:


Hi Sean,

A colleague pointed to me the following commands:

#scontrol show hostname x[1000,1009,1029-1031]
x1000
x1009
x1029
x1030
x1031
#scontrol show hostlist x[1000,1009,1029,1030,1031]
x[1000,1009,1029-1031]
#


Alain Miniussi
DSI, Pôles Calcul et Genie Log.
Observatoire de la Côte d'Azur
Tél. : +33609650665


- On Aug 17, 2023, at 3:12 PM, Sean Mc Grath  wrote:
Hi Alain,

I don't know if slurm can do that natively. python-hostlist, 
https://www.nsc.liu.se/~kent/python-hostlist/, may provide the functionality 
you need. I have used it in the past to generate a list of hosts that can be 
looped over.

Hope that helps.

Sean

---
Sean McGrath
Senior Systems Administrator, IT Services


From: slurm-users  on behalf of Alain O' 
Miniussi 
Sent: Thursday 17 August 2023 13:44
To: Slurm User Community List 
Subject: [slurm-users] extended list of nodes allocated to a job

Hi,

I'm looking for a way to get the list of nodes where a given job is running in 
a uncompressed way.
That is, I'd like to have node1,node2,node3 instead of node1-node3.
Is there way to achieve that ?
I need the information outside the script.

Thanks


Alain Miniussi
DSI, Pôles Calcul et Genie Log.
Observatoire de la Côte d'Azur
Tél. : +33609650665