You should also look into Chronos for workflow dependency management of batch jobs (also supports cron-like scheduling).
On Fri, Jan 9, 2015 at 2:12 PM, Srinimurthy <[email protected]> wrote: > Tim, > This is a SAAS environment where the jobs running on each of these nodes > are varying depending on the workflow run by each company, resources (JVMs) > are allocated per size and need of the job involved > > Srinivas > > On Jan 9, 2015, at 1:59 PM, Tim Chen <[email protected]> wrote: > > Hi Srinivas, > > Can you elaborate more about what does maintaining a dynamic count of > executors? > > You can always write a custom framework that provides the scheduling, > similiar to what Marathon or Aurora is doing if it doesn't fit your need. > > Tim > > On Fri, Jan 9, 2015 at 1:18 PM, Srinivas Murthy <[email protected]> > wrote: > >> Thanks Vinod. I need to deal with a very conservative management that >> needs a lot of selling for each additional open source framework. I have >> glossed over Marathon so far. I was hoping to hear there's some way I could >> override the Scheduler and work with what I have, but I hear you say that >> isn't the route I should be pursuing :-) >> >> >> On Fri, Jan 9, 2015 at 11:43 AM, Vinod Kone <[email protected]> wrote: >> >>> Have you looked at Aurora or Marathon? They have some (most?) of the >>> features you are looking for. >>> >>> On Fri, Jan 9, 2015 at 10:59 AM, Srinivas Murthy <[email protected]> >>> wrote: >>> >>>> We have a legacy system with home brewn workflows defined in XPDL, >>>> running across multiple dozens of nodes. Resources are mapped in XML >>>> definition files, and availability of resource to a given task at hand >>>> managed by a custom written job scheduler. Jobs communicate status with >>>> callback/JMS messages. Job completion decides steps in the workflow. >>>> >>>> To this eco system now comes some Hadoop/Spark jobs. >>>> I am tentatively exploring Mesos to manage this disparate set of >>>> clusters. >>>> How can I maintain a dynamic count of Executors, how can I provide >>>> dynamic workflow orchestration to pull off above architecture in the Mesos >>>> world? Sorry for the noob question! >>>> >>> >>> >> >
