@Tim We're running a combination of an in-house DAG scheduler we wrote a few years back (kind of like https://github.com/spotify/luigi) in combination with Jenkins. I'm not aware of a fully blown DAG scheduler that exists as a Mesos framework but Jenkins might be a way to go, it works well for us.
We also looked into https://github.com/Yelp/Tron many years back but didn't end up using it, perhaps that's a route you could take. Maybe there's a gap in the market for a DAG scheduler that can run tasks on Mesos ;-) -- Tom Arnfeld Developer // DueDil On Wednesday, May 13, 2015 at 9:15 pm, Tim St Clair <[email protected]>, wrote: Hi Alex, Have you by chance integrated with any of the tradition batch DAG systems? http://pegasus.isi.edu/ , http://ccl.cse.nd.edu/software/makeflow/ I keep longing for folks with decades of experience in HTC&HPC to chime in "on-list". Subtle nudge ;-) Tim From: "Alex Gaudio" <[email protected]> To: [email protected] Sent: Wednesday, May 13, 2015 3:04:20 PM Subject: Re: Batch Scheduler with dependency support Hi Tim (and everyone else!), I am the primary author of Stolos. We use Stolos to run all of our batch jobs on Mesos. The batch jobs are scripts we can run from the command-line. Scripts range from bash scripts, Spark jobs and R scripts. It's a great tool for us because, unlike Chronos, it lets us define a script as stage in a dependency chain, where the script can run with different parameters for different dependency contexts. (The closest usage of this would be to have many Chronos servers, though this does not work in all cases). The tool is a critical component of Sailthru's data science infrastructure, but I believe we are the only people who use the tool right now. If you are interested in learning more, I'm happy to invest time to talk more about Stolos, what it does and how we use it! Alex On Wed, May 13, 2015 at 2:02 PM Tim Chen <[email protected]> wrote: How are you running your batch jobs? Is the batch job script/executable an in-house app? Tim On Wed, May 13, 2015 at 9:46 AM, Andras Kerekes <[email protected]> wrote: You might want to have a look at stolos too: https://github.com/sailthru/stolos Andras From: Aaron Carey [mailto:[email protected]] Sent: Wednesday, May 13, 2015 11:54 AM To: [email protected] Subject: RE: Batch Scheduler with dependency support Thanks! I hadn't come across that one before :) From: [email protected] [[email protected]] on behalf of Jeff Schroeder [[email protected]] Sent: 13 May 2015 16:39 To: [email protected] Subject: Re: Batch Scheduler with dependency support Lookup Hubspot's Singularity On Wednesday, May 13, 2015, Aaron Carey <[email protected]> wrote: Thanks Jeff, Any other options around as well? From: [email protected] [[email protected]] on behalf of Jeff Schroeder [[email protected]] Sent: 13 May 2015 14:12 To: [email protected] Subject: Batch Scheduler with dependency support It does both just as well, along with cron-like functionality. It is harder to install and takes a bit more understanding however. The official tutorial is a process that loops 100 times and then exits. http://aurora.apache.org/documentation/latest/tutorial/#the-script Aurora is pretty much a superset of most other generic frameworks sans maybe hubspot's singularity. On Wednesday, May 13, 2015, Aaron Carey <[email protected]> wrote: I was under the impression Aurora was for long running services? Is it suitable for scheduling one of batch processes too? thanks, Aaron From: [email protected] [[email protected]] on behalf of Jeff Schroeder [[email protected]] Sent: 13 May 2015 13:12 To: [email protected] Subject: Re: Batch Scheduler with dependency support Apache Aurora does this and you can be explicit about the ordering On Wednesday, May 13, 2015, Aaron Carey <[email protected]> wrote: Hi All, I was just wondering if anyone out there knew of a good mesos batch scheduler which supports dependencies between tasks? (ie Task B cannot run until Task A is complete) Thanks, Aaron -- Text by Jeff, typos by iPhone -- Text by Jeff, typos by iPhone -- Text by Jeff, typos by iPhone -- Cheers, Timothy St. Clair Red Hat Inc.
