The OP might also be interested in Stolos: https://github.com/sailthru/stolos
combined with Relay: https://github.com/sailthru/relay On Wed, Oct 7, 2015 at 8:15 AM, Clarke, Trevor <tcla...@ball.com> wrote: > I'm currently working on this sort of framework. Unfortunately, source is > not currently available but there is a plan to open source in the next > couple of months. I'm not sure if your need is immediate or if it can wait > for a bit. The framework handles jobs in docker containers with pre and > post steps (copy data into the node, products out, etc.) Individual jobs > can be strung together in a DAG for complex processing. Directories can be > watched for new data and jobs can be started in response to this data. > > ________________________________________ > From: Brian Candler [b.cand...@pobox.com] > Sent: Wednesday, October 07, 2015 3:56 AM > To: user@mesos.apache.org > Subject: Batch/queue frameworks? > > Are there any open-source job queue/batch systems which run under Mesos? > I am thinking of things like HTCondor, Torque etc. > > The requirement is to be able to: > - define an overall job as a set of sub-tasks (could be many thousands) > - put sub-tasks into a queue; execute tasks from the queue > - dependencies: don't add a sub-task into the queue until its precursors > have completed successfully > - restart: after an error, be able to restart the job but skipping those > sub-tasks which completed successfully > - preferably handle short-lived tasks efficiently (of order of 10 > seconds duration) > > Clearly it's possible to write a framework to do this, but I don't want > to re-invent the wheel if it has been done already. > > Thanks, > > Brian. > > P.S. I found Chronos, but it doesn't seem a good match. As far as I can > see, it's intended for applications where you pre-define a bunch of > tasks (via GUI? via REST?) and then trigger them periodically. > > > > This message and any enclosures are intended only for the addressee. > Please > notify the sender by email if you are not the intended recipient. If you > are > not the intended recipient, you may not use, copy, disclose, or distribute > this > message or its contents or enclosures to any other person and any such > actions > may be unlawful. Ball reserves the right to monitor and review all > messages > and enclosures sent to or from this email address. > -- James DeFelice 585.241.9488 (voice) 650.649.6071 (fax)
