[
https://issues.apache.org/jira/browse/YARN-7774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16332708#comment-16332708
]
Arun Suresh commented on YARN-7774:
-----------------------------------
Thanks for the comments [~kkaranasos] and [~cheersyang].
bq. CircularIterator looks to be general enough and deserves its own class with
a generic type, it can be moved to common package..
Will do - i had initially done that. Only issue was we need the element to
implement equals, which the SchedulerNode does not (I had to explicitly get the
nodeId from the schedulerNode and compare it :)) - But yeah, Ill move to a
different class, and that way I can put in some unit tests.
bq. it mean the second allocation can only be made after it iterates all nodes
again? This algorithm doesn't seem to be affinity friendly.
Yup, I did consider that. My thought process was: Given that anti-affinity will
probably be used more often, and even if affinity is used - we've come across
more use cases for RACK and label affinity, than node affinity - Both these
cases are served better with the CircularIterator.
bq. Do we really need the CircularIterator? It seems to me that you can have a
normal iterator initialized outside the for loop and then each time
hasNext()=false, you can re-initialize it. But maybe I am missing something
Yeah, its mostly equivalant (we also need to record the starting element), but
I did not want to clutter the the main loop of the code with additional
variables etc., abstracting out to a different class looked cleaner.
Regarding using the minCardinality > 0 check anti affinity:
Instead of looking at the mincardinalty, I was thinking the following scheme
would be efficient for both node-affinity as well as anti-affinity. If a
SchedulingRequest has been placed on a node, and if there are more Requests,
then instead of initializing the circularIterator from the next node, we
initialize it starting from the CURRENT node. That way, if the next request (or
even if the previous req has numAllocations > 1) the just-placed-node will be
the first candidate - It will also not severely impact anti-affinity placement,
since only a single previously considered node will be re-considered before
moving on to the rest of the cluster. Makes sense ?
bq. Do we clean up the black list for each tag? It seems that black-listing can
change based on the allocations that have been done so far, so we might need to
use it carefully.
The blackists are not persistent - they only hang around time the max retries
are completed - after which both the requests and the blacklists are discarded.
> Miscellaneous fixes to the PlacementProcessor
> ---------------------------------------------
>
> Key: YARN-7774
> URL: https://issues.apache.org/jira/browse/YARN-7774
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Arun Suresh
> Assignee: Arun Suresh
> Priority: Blocker
> Attachments: YARN-7774-YARN-6592.001.patch,
> YARN-7774-YARN-6592.002.patch, YARN-7774-YARN-6592.003.patch,
> YARN-7774-YARN-6592.004.patch
>
>
> JIRA to track the following minor changes:
> * Scheduler must normalize requests that are made using the
> {{attemptAllocationOnNode}} method.
> * Currently, the placement algorithm resets the node iterator for each
> request. The Placement Algorithm should either shuffle the node iterator OR
> use a circular iterator - to ensure a) more nodes are looked at and b) bias
> against placing too many containers on the same node
> * Add a placement retry loop for rejected requests - since there are cases
> especially, when Constraints will be satisfied after a subsequent request has
> been placed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
