Tim, This is a SAAS environment where the jobs running on each of these nodes are varying depending on the workflow run by each company, resources (JVMs) are allocated per size and need of the job involved
Srinivas > On Jan 9, 2015, at 1:59 PM, Tim Chen <[email protected]> wrote: > > Hi Srinivas, > > Can you elaborate more about what does maintaining a dynamic count of > executors? > > You can always write a custom framework that provides the scheduling, > similiar to what Marathon or Aurora is doing if it doesn't fit your need. > > Tim > >> On Fri, Jan 9, 2015 at 1:18 PM, Srinivas Murthy <[email protected]> >> wrote: >> Thanks Vinod. I need to deal with a very conservative management that needs >> a lot of selling for each additional open source framework. I have glossed >> over Marathon so far. I was hoping to hear there's some way I could override >> the Scheduler and work with what I have, but I hear you say that isn't the >> route I should be pursuing :-) >> >> >>> On Fri, Jan 9, 2015 at 11:43 AM, Vinod Kone <[email protected]> wrote: >>> Have you looked at Aurora or Marathon? They have some (most?) of the >>> features you are looking for. >>> >>>> On Fri, Jan 9, 2015 at 10:59 AM, Srinivas Murthy <[email protected]> >>>> wrote: >>>> We have a legacy system with home brewn workflows defined in XPDL, running >>>> across multiple dozens of nodes. Resources are mapped in XML definition >>>> files, and availability of resource to a given task at hand managed by a >>>> custom written job scheduler. Jobs communicate status with callback/JMS >>>> messages. Job completion decides steps in the workflow. >>>> >>>> To this eco system now comes some Hadoop/Spark jobs. >>>> I am tentatively exploring Mesos to manage this disparate set of clusters. >>>> How can I maintain a dynamic count of Executors, how can I provide dynamic >>>> workflow orchestration to pull off above architecture in the Mesos world? >>>> Sorry for the noob question! >
