I'm currently working on this sort of framework. Unfortunately, source is not currently available but there is a plan to open source in the next couple of months. I'm not sure if your need is immediate or if it can wait for a bit. The framework handles jobs in docker containers with pre and post steps (copy data into the node, products out, etc.) Individual jobs can be strung together in a DAG for complex processing. Directories can be watched for new data and jobs can be started in response to this data.
________________________________________ From: Brian Candler [[email protected]] Sent: Wednesday, October 07, 2015 3:56 AM To: [email protected] Subject: Batch/queue frameworks? Are there any open-source job queue/batch systems which run under Mesos? I am thinking of things like HTCondor, Torque etc. The requirement is to be able to: - define an overall job as a set of sub-tasks (could be many thousands) - put sub-tasks into a queue; execute tasks from the queue - dependencies: don't add a sub-task into the queue until its precursors have completed successfully - restart: after an error, be able to restart the job but skipping those sub-tasks which completed successfully - preferably handle short-lived tasks efficiently (of order of 10 seconds duration) Clearly it's possible to write a framework to do this, but I don't want to re-invent the wheel if it has been done already. Thanks, Brian. P.S. I found Chronos, but it doesn't seem a good match. As far as I can see, it's intended for applications where you pre-define a bunch of tasks (via GUI? via REST?) and then trigger them periodically. This message and any enclosures are intended only for the addressee. Please notify the sender by email if you are not the intended recipient. If you are not the intended recipient, you may not use, copy, disclose, or distribute this message or its contents or enclosures to any other person and any such actions may be unlawful. Ball reserves the right to monitor and review all messages and enclosures sent to or from this email address.
