Hi Tim (and everyone else!), I am the primary author of Stolos. We use Stolos to run all of our batch jobs on Mesos. The batch jobs are scripts we can run from the command-line. Scripts range from bash scripts, Spark jobs and R scripts.
It's a great tool for us because, unlike Chronos, it lets us define a script as stage in a dependency chain, where the script can run with different parameters for different dependency contexts. (The closest usage of this would be to have many Chronos servers, though this does not work in all cases). The tool is a critical component of Sailthru's data science infrastructure, but I believe we are the only people who use the tool right now. If you are interested in learning more, I'm happy to invest time to talk more about Stolos, what it does and how we use it! Alex On Wed, May 13, 2015 at 2:02 PM Tim Chen <[email protected]> wrote: > How are you running your batch jobs? Is the batch job script/executable an > in-house app? > > Tim > > On Wed, May 13, 2015 at 9:46 AM, Andras Kerekes < > [email protected]> wrote: > >> You might want to have a look at stolos too: >> >> >> >> https://github.com/sailthru/stolos >> >> >> >> Andras >> >> >> >> >> >> *From:* Aaron Carey [mailto:[email protected]] >> *Sent:* Wednesday, May 13, 2015 11:54 AM >> *To:* [email protected] >> *Subject:* RE: Batch Scheduler with dependency support >> >> >> >> Thanks! I hadn't come across that one before :) >> ------------------------------ >> >> *From:* [email protected] [[email protected]] on behalf of Jeff >> Schroeder [[email protected]] >> *Sent:* 13 May 2015 16:39 >> *To:* [email protected] >> *Subject:* Re: Batch Scheduler with dependency support >> >> Lookup Hubspot's Singularity >> >> On Wednesday, May 13, 2015, Aaron Carey <[email protected]> wrote: >> >> Thanks Jeff, >> >> Any other options around as well? >> ------------------------------ >> >> *From:* [email protected] <http://UrlBlockedError.aspx> [ >> [email protected] <http://UrlBlockedError.aspx>] on behalf of Jeff >> Schroeder [[email protected] <http://UrlBlockedError.aspx>] >> *Sent:* 13 May 2015 14:12 >> *To:* [email protected] <http://UrlBlockedError.aspx> >> *Subject:* Batch Scheduler with dependency support >> >> It does both just as well, along with cron-like functionality. It is >> harder to install and takes a bit more understanding however. The official >> tutorial is a process that loops 100 times and then exits. >> >> >> >> http://aurora.apache.org/documentation/latest/tutorial/#the-script >> >> Aurora is pretty much a superset of most other generic frameworks sans >> maybe hubspot's singularity. >> >> >> On Wednesday, May 13, 2015, Aaron Carey <[email protected] >> <http://UrlBlockedError.aspx>> wrote: >> >> I was under the impression Aurora was for long running services? Is it >> suitable for scheduling one of batch processes too? >> >> thanks, >> Aaron >> ------------------------------ >> >> *From:* [email protected] [[email protected]] on behalf of Jeff >> Schroeder [[email protected]] >> *Sent:* 13 May 2015 13:12 >> *To:* [email protected] >> *Subject:* Re: Batch Scheduler with dependency support >> >> Apache Aurora does this and you can be explicit about the ordering >> >> On Wednesday, May 13, 2015, Aaron Carey <[email protected]> wrote: >> >> Hi All, >> >> I was just wondering if anyone out there knew of a good mesos batch >> scheduler which supports dependencies between tasks? (ie Task B cannot run >> until Task A is complete) >> >> Thanks, >> Aaron >> >> >> >> -- >> Text by Jeff, typos by iPhone >> >> >> >> -- >> Text by Jeff, typos by iPhone >> >> >> >> -- >> Text by Jeff, typos by iPhone >> > >
