One other thing, do you use an executor to run many tasks? Or are you using a command task?
On Fri, Jul 17, 2015 at 1:54 PM, Benjamin Mahler <[email protected]> wrote: > Currently, recovered resources are not immediately re-offered as you > noticed, and the default allocation interval is 1 second. I'd recommend > lowering that (e.g. --allocation_interval=50ms), that should improve the > second bullet you listed. Although, in your case it would be better to > immediately re-offer recovered resources (feel free to file a ticket for > supporting that). > > For the first bullet, mind providing some more information? E.g. master > flags, slave flags, scheduler logs, master logs, slave logs, executor logs? > We would need to trace through a task launch to see where the latency is > being introduced. > > On Fri, Jul 17, 2015 at 12:26 PM, Philip Weaver <[email protected]> > wrote: > >> I'm trying to understand the behavior of mesos, and if what I am >> observing is typical or if I'm doing something wrong, and what options I >> have for improving the performance of how offers are made and how tasks are >> executed for my particular use case. >> >> I have written a Scheduler that has a queue of very small tasks (for >> testing, they are "echo hello world", but in production many of them won't >> be much more expensive than that). Each task is configured to use 1 cpu >> resource. When resourceOffers is called, I launch as many tasks as I can in >> the given offers; that is, one call to driver.launchTasks for each offer, >> with a list of tasks that has one task for each cpu in that offer. >> >> On a cluster of 3 nodes and 4 cores each (12 total cores), it takes 120s >> to execute 1000 tasks out of the queue. We are evaluting mesos because we >> want to use it to replace our current homegrown cluster controller, which >> can execute 1000 tasks in way less than 120s. >> >> I am seeing two things that concern me: >> >> - The time between driver.launchTasks and receiving a callback to >> statusUpdate when the task completes is typically 200-500ms, and sometimes >> even as high as 1000-2000ms. >> - The time between when a task completes and when I get an offer for >> the newly freed resource is another 500ms or so. >> >> These latencies explain why I can only execute tasks at a rate of about >> 8/s. >> >> It looks like my offers always include all 4 cores on each machine, which >> would indicate that mesos doesn't like to send an offer as soon as a single >> resource is avaiable, and prefers to delay and send an offer with more >> resources in it. Is this true? >> >> Thanks in advance for any advice you can offer! >> >> - Phllip >> >> >
