I've replied inline below, also cc'ed some of the Aurora / Thermos developers to better answer your questions.
On Fri, Sep 27, 2013 at 9:00 AM, Dan Colish <[email protected]>wrote: > I have been working on an internal project for executing a large number of > jobs across a cluster for the past couple of months and I am currently > doing a spike on using mesos for some of the cluster management tasks. The > clear prior art winners are Aurora and Marathon, but in both cases they > fall short of what I need. > > In aurora's case, the software is clearly very early in the open sourcing > process and as a result it missing significant pieces. The biggest missing > piece is the actual execution framework, Thermos. [That is what I assume > thermos does. I have no internal knowledge to verify that assumption] > Additionally, Aurora is heavily optimized for a high user count and large > number of incoming jobs. My use case is much simpler. There is only one > effective user and we have a small known set of jobs which need to run. > > On the other hand, Marathon is not designed for job execution if job is > defined to be a smaller unit of work. Instead, Marathon self-describes as a > meta-framework for deploying frameworks to a mesos cluster. A job to > marathon is the framework that runs. I do not think Marathon would be a > good fit for managing the my task execution and retry logic. It is designed > to run at on as a sub-layer of the cluster's resource allocation scheduler > and its abstractions follow suit. > > For my needs Aurora does appear to be a much closer fit than Marathon, but > neither is ideal. Since that is the case, I find myself left with a rough > choice. I am not thrilled with the prospect of yet another framework for > Mesos, but there is a lot of work which I have already completed for my > internal project that would need to reworked to fit with Aurora. Currently > my project can support the following features. > > * Distributed job locking - jobs cannot overlap > Can you elaborate on the ways in which jobs cannot overlap? Aurora may provide what you need here. > * Job execution delay queue - jobs can be run immediately or after a delay > * Job preemption > Aurora has the concept of pre-emption when there are insufficient resources for "production" jobs to run. I will defer to Aurora devs on elaborating here. > * Job success/failure tracking > What kind of tracking? > * Garbage collection of dead jobs > This is present in Aurora. Eventually, a completed job will be purged from the entire system. What kind of garbage collection are you referring to? > * Job execution failover - job is retried on a new executor > In Aurora, jobs are restarted if they fail. > * Executor warming - min # of executors idle > * Executor limits - max # of executors available > > My plan for integration with mesos is to adapt the job manager into a > mesos scheduler and my execution slaves into a mesos executor. At that > point, my framework will be able to run on the mesos cluster, but I have a > few concerns about how to allocated and release resources that the > executors will use over the lifetime of the cluster. I am not sure whether > it is better to be greedy early on in the frameworks life-cycle or to > decline resources initially and scale the framework's slaves when jobs > start coming in. > The better design would be to use resources as you need them (rather than greedily holding onto offers). Is there any motivation for the greedy approach? > Additionally, the relationship between the executor and its associated > driver are not immediately clear to me. If I am reading the code correctly, > they do not provide a way to stop a task in progress short of killing the > executor process. > You can kill tasks in progress, the executor process receives the request to kill the task. See SchedulerDriver::killTask and Executor::killTask (scheduler.hpp and executor.hpp). > I think that mesos will be a nice feature to add to my project and I would > really appreciate any feedback from the community. I will provide progress > updates as I continue work on my experiments. >
