Hi Alex, Have you by chance integrated with any of the tradition batch DAG systems?
http://pegasus.isi.edu/ , http://ccl.cse.nd.edu/software/makeflow/ I keep longing for folks with decades of experience in HTC&HPC to chime in "on-list". Subtle nudge ;-) Tim ----- Original Message ----- > From: "Alex Gaudio" <[email protected]> > To: [email protected] > Sent: Wednesday, May 13, 2015 3:04:20 PM > Subject: Re: Batch Scheduler with dependency support > Hi Tim (and everyone else!), > I am the primary author of Stolos. We use Stolos to run all of our batch jobs > on Mesos. The batch jobs are scripts we can run from the command-line. > Scripts range from bash scripts, Spark jobs and R scripts. > It's a great tool for us because, unlike Chronos, it lets us define a script > as stage in a dependency chain, where the script can run with different > parameters for different dependency contexts. (The closest usage of this > would be to have many Chronos servers, though this does not work in all > cases). > The tool is a critical component of Sailthru's data science infrastructure, > but I believe we are the only people who use the tool right now. > If you are interested in learning more, I'm happy to invest time to talk more > about Stolos, what it does and how we use it! > Alex > On Wed, May 13, 2015 at 2:02 PM Tim Chen < [email protected] > wrote: > > How are you running your batch jobs? Is the batch job script/executable an > > in-house app? > > > Tim > > > On Wed, May 13, 2015 at 9:46 AM, Andras Kerekes < > > [email protected] > wrote: > > > > You might want to have a look at stolos too: > > > > > > https://github.com/sailthru/stolos > > > > > > Andras > > > > > > From: Aaron Carey [mailto: [email protected] ] > > > > > > Sent: Wednesday, May 13, 2015 11:54 AM > > > > > > To: [email protected] > > > > > > Subject: RE: Batch Scheduler with dependency support > > > > > > Thanks! I hadn't come across that one before :) > > > > > > From: [email protected] [ [email protected] ] on behalf of Jeff > > > Schroeder [ [email protected] ] > > > > > > Sent: 13 May 2015 16:39 > > > > > > To: [email protected] > > > > > > Subject: Re: Batch Scheduler with dependency support > > > > > > Lookup Hubspot's Singularity > > > > > > On Wednesday, May 13, 2015, Aaron Carey < [email protected] > wrote: > > > > > > Thanks Jeff, > > > > > > Any other options around as well? > > > > > > From: [email protected] [ [email protected] ] on behalf of Jeff > > > Schroeder [ [email protected] ] > > > > > > Sent: 13 May 2015 14:12 > > > > > > To: [email protected] > > > > > > Subject: Batch Scheduler with dependency support > > > > > > It does both just as well, along with cron-like functionality. It is > > > harder > > > to install and takes a bit more understanding however. The official > > > tutorial > > > is a process that loops 100 times and then exits. > > > > > > http://aurora.apache.org/documentation/latest/tutorial/#the-script > > > > > > Aurora is pretty much a superset of most other generic frameworks sans > > > maybe > > > hubspot's singularity. > > > > > > On Wednesday, May 13, 2015, Aaron Carey < [email protected] > wrote: > > > > > > I was under the impression Aurora was for long running services? Is it > > > suitable for scheduling one of batch processes too? > > > > > > thanks, > > > > > > Aaron > > > > > > From: [email protected] [ [email protected] ] on behalf of Jeff > > > Schroeder [ [email protected] ] > > > > > > Sent: 13 May 2015 13:12 > > > > > > To: [email protected] > > > > > > Subject: Re: Batch Scheduler with dependency support > > > > > > Apache Aurora does this and you can be explicit about the ordering > > > > > > On Wednesday, May 13, 2015, Aaron Carey < [email protected] > wrote: > > > > > > Hi All, > > > > > > I was just wondering if anyone out there knew of a good mesos batch > > > scheduler > > > which supports dependencies between tasks? (ie Task B cannot run until > > > Task > > > A is complete) > > > > > > Thanks, > > > > > > Aaron > > > > > > -- > > > > > > Text by Jeff, typos by iPhone > > > > > > -- > > > > > > Text by Jeff, typos by iPhone > > > > > > -- > > > > > > Text by Jeff, typos by iPhone > > > -- Cheers, Timothy St. Clair Red Hat Inc.
