Can you share the whole log of master? I'll be helpful :). ---- Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer Platform OpenSource Technology, STG, IBM GCG +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
On Thu, Jan 21, 2016 at 11:57 PM, Tom Arnfeld <t...@duedil.com> wrote: > Guangya - Nope, there's no outstanding offers for any frameworks, the ones > that are getting offers are responding properly. > > Klaus - This was just a sample of logs for a single agent, the cluster has > at least ~40 agents at any one time. > > On 21 January 2016 at 15:20, Guangya Liu <gyliu...@gmail.com> wrote: > >> Can you please help check if some outstanding offers in cluster which >> does not accept by any framework? You can check this via the endpoint of >> /master/state.json endpoint. >> >> If there are some outstanding offers, you can start the master with a >> offer_timeout flag to let master rescind some offers if those offers are >> not accepted by framework. >> >> Cited from >> https://github.com/apache/mesos/blob/master/docs/configuration.md >> >> --offer_timeout=VALUE Duration of time before an offer is rescinded from >> a framework. >> >> This helps fairness when running frameworks that hold on to offers, or >> frameworks that accidentally drop offers. >> >> Thanks, >> >> Guangya >> >> On Thu, Jan 21, 2016 at 9:44 PM, Tom Arnfeld <t...@duedil.com> wrote: >> >>> Hi Klaus, >>> >>> Sorry I think I explained this badly, these are the logs for one slave >>> (that's empty) and we can see that it is making offers to some frameworks. >>> In this instance, the Hadoop framework (and others) are not among those >>> getting any offers, they get offered nothing. The allocator is deciding to >>> send offers in a loop to a certain set of frameworks, starving others. >>> >>> On 21 January 2016 at 13:17, Klaus Ma <klaus1982...@gmail.com> wrote: >>> >>>> Yes, it seems Hadoop framework did not consume all offered resources: >>>> if framework launch task (1 CPUs) on offer (10 CPUs), the other 9 CPUs will >>>> return back to master (recoverResouces). >>>> >>>> ---- >>>> Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer >>>> Platform OpenSource Technology, STG, IBM GCG >>>> +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me >>>> >>>> On Thu, Jan 21, 2016 at 6:46 PM, Tom Arnfeld <t...@duedil.com> wrote: >>>> >>>>> Thanks everyone! >>>>> >>>>> Stephan - There's a couple of useful points there, will definitely >>>>> give it a read. >>>>> >>>>> Klaus - Thanks, we're running a bunch of different frameworks, in that >>>>> list there's Hadoop MRv1, Apache Spark, Marathon and a couple of home >>>>> grown >>>>> frameworks we have. In this particular case the Hadoop framework is the >>>>> major concern, as it's designed to continually accept offers until it has >>>>> enough slots it needs. With the example I gave above, we observe that the >>>>> master is never sending any sizeable offers to some of these frameworks >>>>> (the ones with the larger shares), which is where my confusion stems from. >>>>> >>>>> I've attached a snippet of our active master logs which show the >>>>> activity for a single slave (which has no active executors). We can see >>>>> that it's cycling though sending and recovering declined offers from a >>>>> selection of different frameworks (in order) but I can say that not all of >>>>> the frameworks are receiving these offers, in this case that's the Hadoop >>>>> framework. >>>>> >>>>> >>>>> On 21 January 2016 at 00:26, Klaus Ma <klaus1982...@gmail.com> wrote: >>>>> >>>>>> Hi Tom, >>>>>> >>>>>> Which framework are you using, e.g. Swarm, Marathon or something >>>>>> else? and which language package are you using? >>>>>> >>>>>> DRF will sort role/framework by allocation ratio, and offer all >>>>>> "available" resources by slave; but if the resources it too small (< >>>>>> 0.1CPU) or the resources was reject/declined by framework, the resources >>>>>> will not offer it until filter timeout. For example, in Swarm 1.0, the >>>>>> default filter timeout 5s (because of go scheduler API); so here is case >>>>>> that may impact the utilisation: the Swarm got one slave with 16 CPUS, >>>>>> but >>>>>> only launch one container with 1 CPUS; the other 15 CPUS will return back >>>>>> to master and did not re-offer until filter timeout (5s). >>>>>> I had pull a request to make Swarm's parameters configurable, refer >>>>>> to https://github.com/docker/swarm/pull/1585. I think you can check >>>>>> this case by master log. >>>>>> >>>>>> If any comments, please let me know. >>>>>> >>>>>> ---- >>>>>> Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer >>>>>> Platform OpenSource Technology, STG, IBM GCG >>>>>> +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me >>>>>> >>>>>> On Thu, Jan 21, 2016 at 2:19 AM, Tom Arnfeld <t...@duedil.com> wrote: >>>>>> >>>>>>> Hey, >>>>>>> >>>>>>> I've noticed some interesting behaviour recently when we have lots >>>>>>> of different frameworks connected to our Mesos cluster at once, all >>>>>>> using a >>>>>>> variety of different shares. Some of the frameworks don't get offered >>>>>>> more >>>>>>> resources (for long periods of time, hours even) leaving the cluster >>>>>>> under >>>>>>> utilised. >>>>>>> >>>>>>> Here's an example state where we see this happen.. >>>>>>> >>>>>>> Framework 1 - 13% (user A) >>>>>>> Framework 2 - 22% (user B) >>>>>>> Framework 3 - 4% (user C) >>>>>>> Framework 4 - 0.5% (user C) >>>>>>> Framework 5 - 1% (user C) >>>>>>> Framework 6 - 1% (user C) >>>>>>> Framework 7 - 1% (user C) >>>>>>> Framework 8 - 0.8% (user C) >>>>>>> Framework 9 - 11% (user D) >>>>>>> Framework 10 - 7% (user C) >>>>>>> Framework 11 - 1% (user C) >>>>>>> Framework 12 - 1% (user C) >>>>>>> Framework 13 - 6% (user E) >>>>>>> >>>>>>> In this example, there's another ~30% of the cluster that is >>>>>>> unallocated, and it stays like this for a significant amount of time >>>>>>> until >>>>>>> something changes, perhaps another user joins and allocates the rest.... >>>>>>> chunks of this spare resource is offered to some of the frameworks, but >>>>>>> not >>>>>>> all of them. >>>>>>> >>>>>>> I had always assumed that when lots of frameworks were involved, >>>>>>> eventually the frameworks that would keep accepting resources >>>>>>> indefinitely >>>>>>> would consume the remaining resource, as every other framework had >>>>>>> rejected >>>>>>> the offers. >>>>>>> >>>>>>> Could someone elaborate a little on how the DRF allocator / sorter >>>>>>> handles this situation, is this likely to be related to the different >>>>>>> users >>>>>>> being used? Is there a way to mitigate this? >>>>>>> >>>>>>> We're running version 0.23.1. >>>>>>> >>>>>>> Cheers, >>>>>>> >>>>>>> Tom. >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >
