Hi Aaron, yeah, an in-house one.
On Thu, May 14, 2015 at 1:09 AM, Aaron Carey <[email protected]> wrote: > Hi Sharma, > > This sounds eerily familiar! Was this an in-house system you were working > on, or a commercial product? > > Thanks, > Aaron > > ------------------------------ > *From:* Sharma Podila [[email protected]] > *Sent:* 13 May 2015 23:49 > *To:* [email protected] > *Cc:* Douglas Thain; Brian Bockelman > *Subject:* [Junk released by User action] Re: Batch Scheduler with > dependency support > > I keep longing for folks with decades of experience in HTC&HPC to >> chime in "on-list". > > > FWIW, I come from that background, but, am not in that space at this > time. My prior life was in developing a (not open source) distributed job > scheduler and management system for batch and interactive jobs that handled > dependencies, deadlines, preemptions, advance reservation of resources, > etc. with multi-level priority and share tree hierarchy based allocation. > Typically, dependencies and deadlines are handled outside of schedulers and > fed into schedulers as task submission after dependencies have been met. We > found it more optimal to have the scheduler resolve dependencies and > deadlines inherently. This way, a high priority job dependent on another > low priority job can induce higher priority on that dependent job. > Similarly, a job with a deadline depending on another job's completion can > induce an earlier launch of the latter job in order to meet it's deadline. > Also, a dependent job can reserve its resources in advance, knowing the > expected completion time of its dependent jobs. This was important because > in that environment we always had more jobs to run than can run on > available resources. It wasn't unusual to have 10s of 1000s of jobs waiting > in queue to run during the day. > > Not sure if this helps the original question in this thread in any way. > But, I am glad to share my learning, if that helps. > > Sharma > > > On Wed, May 13, 2015 at 1:12 PM, Tim St Clair <[email protected]> wrote: > >> Hi Alex, >> >> Have you by chance integrated with any of the tradition batch DAG >> systems? >> >> http://pegasus.isi.edu/ , http://ccl.cse.nd.edu/software/makeflow/ >> >> >> I keep longing for folks with decades of experience in HTC&HPC to chime >> in "on-list". >> >> Subtle nudge ;-) >> Tim >> >> ------------------------------ >> >> *From: *"Alex Gaudio" <[email protected]> >> *To: *[email protected] >> *Sent: *Wednesday, May 13, 2015 3:04:20 PM >> >> *Subject: *Re: Batch Scheduler with dependency support >> >> Hi Tim (and everyone else!), >> >> I am the primary author of Stolos. We use Stolos to run all of our >> batch jobs on Mesos. The batch jobs are scripts we can run from the >> command-line. Scripts range from bash scripts, Spark jobs and R scripts. >> >> It's a great tool for us because, unlike Chronos, it lets us define a >> script as stage in a dependency chain, where the script can run with >> different parameters for different dependency contexts. (The closest usage >> of this would be to have many Chronos servers, though this does not work in >> all cases). >> >> The tool is a critical component of Sailthru's data science >> infrastructure, but I believe we are the only people who use the tool right >> now. >> >> If you are interested in learning more, I'm happy to invest time to >> talk more about Stolos, what it does and how we use it! >> >> Alex >> >> On Wed, May 13, 2015 at 2:02 PM Tim Chen <[email protected]> wrote: >> >>> How are you running your batch jobs? Is the batch job script/executable >>> an in-house app? >>> >>> Tim >>> >>> On Wed, May 13, 2015 at 9:46 AM, Andras Kerekes < >>> [email protected]> wrote: >>> >>>> You might want to have a look at stolos too: >>>> >>>> >>>> >>>> https://github.com/sailthru/stolos >>>> >>>> >>>> >>>> Andras >>>> >>>> >>>> >>>> >>>> >>>> *From:* Aaron Carey [mailto:[email protected]] >>>> *Sent:* Wednesday, May 13, 2015 11:54 AM >>>> *To:* [email protected] >>>> *Subject:* RE: Batch Scheduler with dependency support >>>> >>>> >>>> >>>> Thanks! I hadn't come across that one before :) >>>> ------------------------------ >>>> >>>> *From:* [email protected] [[email protected]] on behalf of >>>> Jeff Schroeder [[email protected]] >>>> *Sent:* 13 May 2015 16:39 >>>> *To:* [email protected] >>>> *Subject:* Re: Batch Scheduler with dependency support >>>> >>>> Lookup Hubspot's Singularity >>>> >>>> On Wednesday, May 13, 2015, Aaron Carey <[email protected]> wrote: >>>> >>>> Thanks Jeff, >>>> >>>> Any other options around as well? >>>> ------------------------------ >>>> >>>> *From:* [email protected] <http://UrlBlockedError.aspx> [ >>>> [email protected] <http://UrlBlockedError.aspx>] on behalf of Jeff >>>> Schroeder [[email protected] <http://UrlBlockedError.aspx>] >>>> *Sent:* 13 May 2015 14:12 >>>> *To:* [email protected] <http://UrlBlockedError.aspx> >>>> *Subject:* Batch Scheduler with dependency support >>>> >>>> It does both just as well, along with cron-like functionality. It is >>>> harder to install and takes a bit more understanding however. The official >>>> tutorial is a process that loops 100 times and then exits. >>>> >>>> >>>> >>>> http://aurora.apache.org/documentation/latest/tutorial/#the-script >>>> >>>> Aurora is pretty much a superset of most other generic frameworks >>>> sans maybe hubspot's singularity. >>>> >>>> >>>> On Wednesday, May 13, 2015, Aaron Carey <[email protected] >>>> <http://UrlBlockedError.aspx>> wrote: >>>> >>>> I was under the impression Aurora was for long running services? Is it >>>> suitable for scheduling one of batch processes too? >>>> >>>> thanks, >>>> Aaron >>>> ------------------------------ >>>> >>>> *From:* [email protected] [[email protected]] on behalf of >>>> Jeff Schroeder [[email protected]] >>>> *Sent:* 13 May 2015 13:12 >>>> *To:* [email protected] >>>> *Subject:* Re: Batch Scheduler with dependency support >>>> >>>> Apache Aurora does this and you can be explicit about the ordering >>>> >>>> On Wednesday, May 13, 2015, Aaron Carey <[email protected]> wrote: >>>> >>>> Hi All, >>>> >>>> I was just wondering if anyone out there knew of a good mesos batch >>>> scheduler which supports dependencies between tasks? (ie Task B cannot run >>>> until Task A is complete) >>>> >>>> Thanks, >>>> Aaron >>>> >>>> >>>> >>>> -- >>>> Text by Jeff, typos by iPhone >>>> >>>> >>>> >>>> -- >>>> Text by Jeff, typos by iPhone >>>> >>>> >>>> >>>> -- >>>> Text by Jeff, typos by iPhone >>>> >>> >>> >> >> >> -- >> Cheers, >> Timothy St. Clair >> Red Hat Inc. >> > >
