Hi guys, I have to model as well as possible the capacity scheduler
behavior and I have some questions, hope someone can help me with this
(in the following I will consider all container to be equal for simplicity):
1) Let's say I have queues A and B. A is configured to get 20% (20
containers) of the total cluster capacity (100 containers), B gets 80%
(80 containers). Capacity scheduler gives available resources firstly to
the most under-served queue.
In case A is using 10 containers and B is using 20, who is going to get
the first available container? A is already using 50% of it's assigned
capacity, B just 25%, but A has less containers than B... who is
considered to be more under-served?
2) Does the previous question make sense at all? Because I have a doubt
that when I have free containers I will just serve requests as they
arrive, possibly over-provisioning a queue (that is: if I get a
container request for an app in A, I will give it a container since I
don't know that after a few (milli)seconds I will get a new request from
B, or vice versa). The previous question may have sense if there was
some sort of buffer that is filled with incoming requests, due to the
difficulty of serving them in real time, thus making the scheduler able
to choose the request from the most under-served queue. Is this what
happens?
3) Now let's consider a new configuration:
We have a cluster hosting a total of 40 containers. We have 3 queues: A
is configured to get 39% of cluster capacity, B also gets 39% and C gets
22%. The number of clusters is going to be 15.6, 15.6 and 8.8 for A, B
an C. Since we can't split a container, how does the Capacity scheduler
round these values in a real case? Who gets the two contended
containers? I may think they are considered as extra containers, thus
shared upon need among the three queues. Is this correct?
4) According to the example presented in "Apache Hadoop YARN: Moving
beyond MapReduce and Batch Processing with Apache Hadoop 2" about the
resource allocation with the capacity scheduler, what I understood is
that the chance for a leaf queue to get resources above it's assigned
capacity is always upper-limited by the fraction of cluster capacity
assigned to its first/closer parent queue. That is: if I am a leaf queue
A1, I can only get at most the resources dedicated to my parent A, while
I can't get the ones from B, sibling of A, even if it doesn't have any
running application. Actually at first I thought this over-provisioning
was not limited, and regardless of the queue configuration a single
application could get the whole cluster (excluding per-application
limits). Did I misunderstood the example?
Thanks a lot
Fabio
