Hi Kenneth, all, On 8 August 2014 18:08, Kenneth Hoste <[email protected]> wrote: > > On 08/08/14 17:48, Riccardo Murri wrote: >> >> Actually, if I am allowed a shameless self-plug, we already have a >> Python framework that can submit and manage jobs on different >> batch-queuing systems, see http://gc3pie.googlecode.com/ > > That sounds interesting! > > Let me pick up a crazy project idea we wrote up some time ago: > https://gist.github.com/boegel/9225891 . > > How does gc3pie relate to that?
It's basically that, +- some features. >> I am not familiar with EasyBuild internals, but GC3Pie's job control >> reduces to a few lines that should be relatively quick to plug in: >> >> from gc3libs import Application >> from gc3libs.core import Engine >> >> task = Application(['some', '-unix', '+command', 'here'], ...) >> engine = Engine(...) >> engine.add(task) >> # run task and wait for it to finish >> engine.progress() >> >> If there is interest, I can look at the sources and try to estimate >> how much work it would be to integrate GC3Pie and EasyBuild. > > The first step should be to abstract the current support for --job into a > generic class, and make what's there now derive from that (probably naming > it PbsPython). > > Then, SLURM & LSF could be just another version of that, and so can gc3pie > and DRMAA. > > Unless gc3pie solves all our problems, that would even be better. ;-) Well, it probably won't solve all your problems ;-) but it *is* a single unified interface to several batch systems. So, in theory a GC3Pie backend would allow submission to PBS/Torque, LSF, GridEngine, and SLURM (plus OpenStack, EC2, and good old fork()/exec() but that's less interesting). I'll see if I can spare some time this week to have a look at the EasyBuild source code and estimate how much effort would it be to adapt the current PBS support to use GC3Pie instead. Ciao, R -- Riccardo Murri http://www.s3it.uzh.ch/about/team/ S3IT: Services and Support for Science IT University of Zurich Winterthurerstrasse 190, CH-8057 Zürich (Switzerland) Tel: +41 44 635 4222 Fax: +41 44 635 6888
