Hi all!I've just started trying EasyBuild and I'm finding it pretty attractive and interesting. In our center we are testing EasyBuild for evaluation now and they asked me to prepare a talk on how to use it and include it in our workflows.
We manage several different clusters, some younger and others older, each one with its own version of Operating System and update level. Each cluster has its own Job Scheduler, mainly Slurm but also LSF. I was very interested in the --job option of EasyBuild to allow for distributed builds in a fire and forget manner.
When trying the job support of EasyBuild I found it only works with PBS, which is obviously a problem for us. As we have different schedulers available I would prefer a generalistic approach instead of integrating with a list of schedulers. The idea would be to set some kind of abstractions about the interaction between EasyBuild and the scheduler and allow for command-line invocation of scheduler commands, e.g.:
eb myapp.eb --robot --job=Slurm
which internally would set:
build_command=build_send_job("Slurm")
os.system("build_command")
And for LSF
eb myapp.eb --robot --job=LSF
would set:
build_command=build_send_job("LSF")
os.system("build_command")
There would be some config options or config files describing the
interface of the scheduler, like:
[Slurm]
scheduler_root=""
sender="/bin/sbatch"
params[processes]= '-np'
...
[LSF]
scheduler_root="/opt/LSF"
sender="/bin/bsub"
params[processes]= '-n'
...
I'm not sure if all or at least the most common schedulers' CLI could be
abstracted in this manner as I've only worked with Slurm and LSF. Either
way, would the community be interested in this kind of abstraction?
Also, has someone worked on something similar or a port to Slurm or LSF
we could extend or reuse? I know of Jordi Blasco's efforts even though I
heven't seen his Slurm porting yet.
Regards, Miguel -- Miguel Bernabeu ([email protected]) BSC - Centro Nacional de Supercomputacion C/ Jordi Girona 31 WWW: http://www.bsc.es 08034 Barcelona, Spain Tel: +34-93-405 42 28 e-mail: [email protected] Fax: +34-93-413 77 21
smime.p7s
Description: S/MIME Cryptographic Signature
