On 07/10/2015 11:08, Pablo Cingolani wrote:
It looks like you are looking for something like BDS
http://pcingola.github.io/BigDataScript/
It has the additional advantage that you can port your scripts seamlessly
between Mesos and other cluster systems (SGE, PBS, Torque, etc.).
Yes, that looks very interesting, thank you! It seems to perform the
same role as HTCondor Dagman, but with pluggable backends and a much
more expressive language.
At http://pcingola.github.io/BigDataScript/bigDataScript_manual.html
under "Resource consumption and task options", I don't see any option
for declaring the memory used by a task. Is that a wishlist feature? In
fact, mesos allows arbitrary resources, so it would be good to be able
to specify resource requirements of
any particular resource.
I note that BDS allows a task to specify it runs on one particular
cluster node. In my application it would also be helpful to be able to
specify a particular class of node. (When submitting a job to HTCondor
this could be expanded to a requirements expression)
Regards,
Brian.
