Hi Alex, Thanks for replying and looks like a really interesting framework! I was orginally aiming the question back to Aaron as he stated he's looking for a batch job scheduler.
Do you have some rough stats how many jobs or data you guys are computing with Stolos? Tim On Wed, May 13, 2015 at 1:04 PM, Alex Gaudio <[email protected]> wrote: > Hi Tim (and everyone else!), > > I am the primary author of Stolos. We use Stolos to run all of our batch > jobs on Mesos. The batch jobs are scripts we can run from the > command-line. Scripts range from bash scripts, Spark jobs and R scripts. > > It's a great tool for us because, unlike Chronos, it lets us define a > script as stage in a dependency chain, where the script can run with > different parameters for different dependency contexts. (The closest usage > of this would be to have many Chronos servers, though this does not work in > all cases). > > The tool is a critical component of Sailthru's data science > infrastructure, but I believe we are the only people who use the tool right > now. > > If you are interested in learning more, I'm happy to invest time to talk > more about Stolos, what it does and how we use it! > > Alex > > On Wed, May 13, 2015 at 2:02 PM Tim Chen <[email protected]> wrote: > >> How are you running your batch jobs? Is the batch job script/executable >> an in-house app? >> >> Tim >> >> On Wed, May 13, 2015 at 9:46 AM, Andras Kerekes < >> [email protected]> wrote: >> >>> You might want to have a look at stolos too: >>> >>> >>> >>> https://github.com/sailthru/stolos >>> >>> >>> >>> Andras >>> >>> >>> >>> >>> >>> *From:* Aaron Carey [mailto:[email protected]] >>> *Sent:* Wednesday, May 13, 2015 11:54 AM >>> *To:* [email protected] >>> *Subject:* RE: Batch Scheduler with dependency support >>> >>> >>> >>> Thanks! I hadn't come across that one before :) >>> ------------------------------ >>> >>> *From:* [email protected] [[email protected]] on behalf of Jeff >>> Schroeder [[email protected]] >>> *Sent:* 13 May 2015 16:39 >>> *To:* [email protected] >>> *Subject:* Re: Batch Scheduler with dependency support >>> >>> Lookup Hubspot's Singularity >>> >>> On Wednesday, May 13, 2015, Aaron Carey <[email protected]> wrote: >>> >>> Thanks Jeff, >>> >>> Any other options around as well? >>> ------------------------------ >>> >>> *From:* [email protected] <http://UrlBlockedError.aspx> [ >>> [email protected] <http://UrlBlockedError.aspx>] on behalf of Jeff >>> Schroeder [[email protected] <http://UrlBlockedError.aspx>] >>> *Sent:* 13 May 2015 14:12 >>> *To:* [email protected] <http://UrlBlockedError.aspx> >>> *Subject:* Batch Scheduler with dependency support >>> >>> It does both just as well, along with cron-like functionality. It is >>> harder to install and takes a bit more understanding however. The official >>> tutorial is a process that loops 100 times and then exits. >>> >>> >>> >>> http://aurora.apache.org/documentation/latest/tutorial/#the-script >>> >>> Aurora is pretty much a superset of most other generic frameworks sans >>> maybe hubspot's singularity. >>> >>> >>> On Wednesday, May 13, 2015, Aaron Carey <[email protected] >>> <http://UrlBlockedError.aspx>> wrote: >>> >>> I was under the impression Aurora was for long running services? Is it >>> suitable for scheduling one of batch processes too? >>> >>> thanks, >>> Aaron >>> ------------------------------ >>> >>> *From:* [email protected] [[email protected]] on behalf of Jeff >>> Schroeder [[email protected]] >>> *Sent:* 13 May 2015 13:12 >>> *To:* [email protected] >>> *Subject:* Re: Batch Scheduler with dependency support >>> >>> Apache Aurora does this and you can be explicit about the ordering >>> >>> On Wednesday, May 13, 2015, Aaron Carey <[email protected]> wrote: >>> >>> Hi All, >>> >>> I was just wondering if anyone out there knew of a good mesos batch >>> scheduler which supports dependencies between tasks? (ie Task B cannot run >>> until Task A is complete) >>> >>> Thanks, >>> Aaron >>> >>> >>> >>> -- >>> Text by Jeff, typos by iPhone >>> >>> >>> >>> -- >>> Text by Jeff, typos by iPhone >>> >>> >>> >>> -- >>> Text by Jeff, typos by iPhone >>> >> >>
