@some point I'd hope the litany of existing DAG generators that exist for legacy batch systems would make it's way to support this ecosystem.
/me coughs Makeflow, pegasus ... | for that matter, one might redux a high throughput systems in a (Docker) world where NP-hard matching no longer makes any sense, b/c it's all cattle. Cheers, Tim ----- Original Message ----- > From: "Adam Bordelon" <[email protected]> > To: [email protected] > Sent: Wednesday, January 21, 2015 3:41:40 AM > Subject: Re: Architecture question > You should also look into Chronos for workflow dependency management of batch > jobs (also supports cron-like scheduling). > On Fri, Jan 9, 2015 at 2:12 PM, Srinimurthy < [email protected] > wrote: > > Tim, > > > This is a SAAS environment where the jobs running on each of these nodes > > are > > varying depending on the workflow run by each company, resources (JVMs) are > > allocated per size and need of the job involved > > > Srinivas > > > On Jan 9, 2015, at 1:59 PM, Tim Chen < [email protected] > wrote: > > > > Hi Srinivas, > > > > > > Can you elaborate more about what does maintaining a dynamic count of > > > executors? > > > > > > You can always write a custom framework that provides the scheduling, > > > similiar to what Marathon or Aurora is doing if it doesn't fit your need. > > > > > > Tim > > > > > > On Fri, Jan 9, 2015 at 1:18 PM, Srinivas Murthy < [email protected] > > > > wrote: > > > > > > > Thanks Vinod. I need to deal with a very conservative management that > > > > needs > > > > a > > > > lot of selling for each additional open source framework. I have > > > > glossed > > > > over Marathon so far. I was hoping to hear there's some way I could > > > > override > > > > the Scheduler and work with what I have, but I hear you say that isn't > > > > the > > > > route I should be pursuing :-) > > > > > > > > > > On Fri, Jan 9, 2015 at 11:43 AM, Vinod Kone < [email protected] > > > > > wrote: > > > > > > > > > > > Have you looked at Aurora or Marathon? They have some (most?) of the > > > > > features > > > > > you are looking for. > > > > > > > > > > > > > > > On Fri, Jan 9, 2015 at 10:59 AM, Srinivas Murthy < > > > > > [email protected] > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > We have a legacy system with home brewn workflows defined in XPDL, > > > > > > running > > > > > > across multiple dozens of nodes. Resources are mapped in XML > > > > > > definition > > > > > > files, and availability of resource to a given task at hand managed > > > > > > by > > > > > > a > > > > > > custom written job scheduler. Jobs communicate status with > > > > > > callback/JMS > > > > > > messages. Job completion decides steps in the workflow. > > > > > > > > > > > > > > > > > > > > > To this eco system now comes some Hadoop/Spark jobs. > > > > > > > > > > > > > > > > > > > > > I am tentatively exploring Mesos to manage this disparate set of > > > > > > clusters. > > > > > > > > > > > > > > > > > > > > > How can I maintain a dynamic count of Executors, how can I provide > > > > > > dynamic > > > > > > workflow orchestration to pull off above architecture in the Mesos > > > > > > world? > > > > > > Sorry for the noob question! > > > > > > > > > > > > > > > -- Cheers, Timothy St. Clair Red Hat Inc.
