On Sat, Sep 28, 2013 at 1:16 PM, Dan Colish <[email protected]>wrote:
> On Fri, Sep 27, 2013 at 5:21 PM, Benjamin Mahler < > [email protected]> wrote: > >> I've replied inline below, also cc'ed some of the Aurora / Thermos >> developers to better answer your questions. >> > > Thank you very much! I'll respond inline as well. > > >> >> On Fri, Sep 27, 2013 at 9:00 AM, Dan Colish <[email protected]>wrote: >> >>> I have been working on an internal project for executing a large number >>> of jobs across a cluster for the past couple of months and I am currently >>> doing a spike on using mesos for some of the cluster management tasks. The >>> clear prior art winners are Aurora and Marathon, but in both cases they >>> fall short of what I need. >>> >>> In aurora's case, the software is clearly very early in the open >>> sourcing process and as a result it missing significant pieces. The biggest >>> missing piece is the actual execution framework, Thermos. [That is what I >>> assume thermos does. I have no internal knowledge to verify that >>> assumption] Additionally, Aurora is heavily optimized for a high user count >>> and large number of incoming jobs. My use case is much simpler. There is >>> only one effective user and we have a small known set of jobs which need to >>> run. >>> >>> On the other hand, Marathon is not designed for job execution if job is >>> defined to be a smaller unit of work. Instead, Marathon self-describes as a >>> meta-framework for deploying frameworks to a mesos cluster. A job to >>> marathon is the framework that runs. I do not think Marathon would be a >>> good fit for managing the my task execution and retry logic. It is designed >>> to run at on as a sub-layer of the cluster's resource allocation scheduler >>> and its abstractions follow suit. >>> >>> For my needs Aurora does appear to be a much closer fit than Marathon, >>> but neither is ideal. Since that is the case, I find myself left with a >>> rough choice. I am not thrilled with the prospect of yet another framework >>> for Mesos, but there is a lot of work which I have already completed for my >>> internal project that would need to reworked to fit with Aurora. Currently >>> my project can support the following features. >>> >>> * Distributed job locking - jobs cannot overlap >>> >> >> Can you elaborate on the ways in which jobs cannot overlap? Aurora may >> provide what you need here. >> > > My use case requires the guarantee that only one instance of a job can be > run at a given time. In my current implementation, I am using a distributed > pessimistic offline lock to provide this guarantee. My lock implementation > also guarantees modifications are atomic. Looking at Aurora, it follows a > similar approach. Unfortunately, the MemStorarge is not appropriate for > production usage and that is the only Storage implementation in Aurora > which can provide the consistency I need for the locking strategy to work. > It would be possible to include my lock in a persistent Storage > implementation within Aurora, but I suspect that one already exists and > could be made available eventually. If that is not the case, it might be > worth talking more about how my implementation works and if there is any > opportunity for collaboration. > Aurora currently errs on the side of availability (keeping at least N instances up, converging towards N). If we are signaled that a task is LOST, we will aggressively replace it rather than (possibly hopelessly) waiting for an indication that the task is confirmed dead. So far we've suggested that services with stricter mutual exclusion requirements be defensive and build it into the app (e.g. a lock in a database or ZooKeeper). Don't be misled by MemStorage, it's just the in-memory portion of the storage system. All state required for a failover is persisted (see LogStorage). > >> >>> * Job execution delay queue - jobs can be run immediately or after a >>> delay >>> * Job preemption >>> >> >> Aurora has the concept of pre-emption when there are insufficient >> resources for "production" jobs to run. I will defer to Aurora devs on >> elaborating here. >> > > Excellent, my job preemption is actually focused on freeing up jobs which > are running too slow or otherwise failing to make progress. It also > provides part of the general job canceling mechanism. I've looked over the > preemption code in Aurora and it also looks very similar. > > >> >> >>> * Job success/failure tracking >>> >> >> What kind of tracking? >> > > I provide an interface for taking actions upon a job's completion. This > callback can by a job designer for any number of uses. Tracking which jobs > succeed and fail is the only use case I have for it now. There were some > future ideas to use it for kicking of dependent jobs or taking other > administrative actions. > > >> >> >>> * Garbage collection of dead jobs >>> >> >> This is present in Aurora. Eventually, a completed job will be purged >> from the entire system. What kind of garbage collection are you referring >> to? >> > > That is exactly the kind of garbage collection I was speaking of. My > system will remove jobs when they complete and clean up resources such as > the locks taken. It will also preempt and remove jobs which are no longer > making progress. > > >> >> >>> * Job execution failover - job is retried on a new executor >>> >> >> In Aurora, jobs are restarted if they fail. >> > > Unfortunately, my implementation of the feature is quite crude. It will > just retry N times and give up if the job continues to fail. Does Aurora > make any smarter determination about retrying? The job state machine in > aurora provides a few more states than mine so it might have enough > information to provide a richer retry framework. I would need to read a bit > more to figure that out. > Aurora has a few 'modes' to handle this. Anything with service=true is restarted forever. Otherwise, maxTaskFailures indicates how many times a task may fail before we give up on it. >> >>> * Executor warming - min # of executors idle >>> * Executor limits - max # of executors available >>> >>> My plan for integration with mesos is to adapt the job manager into a >>> mesos scheduler and my execution slaves into a mesos executor. At that >>> point, my framework will be able to run on the mesos cluster, but I have a >>> few concerns about how to allocated and release resources that the >>> executors will use over the lifetime of the cluster. I am not sure whether >>> it is better to be greedy early on in the frameworks life-cycle or to >>> decline resources initially and scale the framework's slaves when jobs >>> start coming in. >>> >> >> The better design would be to use resources as you need them (rather than >> greedily holding onto offers). Is there any motivation for the greedy >> approach? >> > > My consideration for the greedy approach was based on what I thought > aurora was doing internally. It looks like the OfferQueue adds resources > and then matches those resources to job tasks from the TaskStore as they > come in. I took that to mean that Aurora would hold on to resources for as > long as possible. I think I am misunderstanding how often offers will come > into a scheduler. A sequence diagram for Aurora's resource to job matching > would be extremely helpful. Its a little tricky to trace control flow > within Aurora since so many actions are asynchronous. My current belief of > what happens is this > > [Scheduler receives offer] -> [Match a pending task to offer immediately] > -> [Matched] -> [begin exec] > > -> [No tasks or not large enough] -> [Add to > OfferQueue] > > [TaskGroup:Monitor] -> [TaskScheduler] -> [Load task from store] -> > [Attempt to match with OfferQueue] -> [Matched] -> [Exec] > > > -> [Unmatched] -> [Return control to Monitor] -> [Monitor backs off or > cancels group] > > > Given these scenarios, I have yet to find a place where Offers will be > cancelled aside from the Scheudler::offerRescinded callback in the > framework scheduler. This is what lead me to believe that Aurora is greedy > about Offers and only gives them up when they are either used to launch a > task or rescinded. I do agree that a fairer system is to only accept offers > that you need and cancel the rest. Exactly how that would be implemented I > will need to think more on. > You're correct that Aurora is greedy with offers. It's my understanding that mesos will soon be more aggressive about rescinding offers, which * should* make this behavior more multi-framework-friendly in the future. We originally responded to offers synchronously when handling resourceOffers, but changed to asynchronously accepting offers so we could have deterministic responsiveness to the callback (since mesos callbacks are effectively singly-threaded, and holding up resourceOffers stalls important things like statusUpdate). Your comprehension of the offer matching in Aurora is correct, though there is one nuance. We have two types of tasks — user tasks and system tasks (e.g. garbage collection to reconcile potential scheduler/slave state differences). System tasks are always accepted synchronously, and user tasks are always accepted asynchronously, via OfferQueue. We do return offers on two conditions: when we're holding two offers for the same slave (returned for coalescing), and after a timeout (defaulting to 5 minutes). > > >> >>> Additionally, the relationship between the executor and its associated >>> driver are not immediately clear to me. If I am reading the code correctly, >>> they do not provide a way to stop a task in progress short of killing the >>> executor process. >>> >> >> You can kill tasks in progress, the executor process receives the request >> to kill the task. See SchedulerDriver::killTask and Executor::killTask >> (scheduler.hpp and executor.hpp). >> >> > > Ah yes, I missed that. It was very clear once I traced the message flow > from Scheduler -> Master -> Slave -> Executor. I was probably confusing the > comment in Executor::launchTask about no other callbacks getting called > before that one completes with the task actually blocking the callbacks. I > suppose if you wrote an executor which did not run the task asynchronously > and launchTask was blocked from returning then you would be in some trouble > with regard to task control. > > >> I think that mesos will be a nice feature to add to my project and I >>> would really appreciate any feedback from the community. I will provide >>> progress updates as I continue work on my experiments. >>> >> >> >
