[
https://issues.apache.org/jira/browse/YARN-7783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16336636#comment-16336636
]
Weiwei Yang commented on YARN-7783:
-----------------------------------
Hi [~asuresh]
I am +0 on the patch. It resolves the issue that you mentioned in the comment,
but not seems to be general enough to me. The problem here is a circular
dependency,
Like in your example
1) "foo", anti-affinity with "foo"
+Implies node constraint:+ on each node, it cannot have more than 1 foo tags
2) "bar", anti-affinity with "foo"
+Implies node constraint:+ on each node, it cannot have both "bar" and "foo"
tags
I don't think we need an extra validation phase here, instead we could optimize
the place logic. By that I mean when allocate a container to a node, we also
bind a *node constraint* to this node. Then if the algorithm wants to place any
new container to this node, except for checking if it satisfies the placement
constraint, it also check if it satisfies the node constraint. Back to your
example
* req2 is placed on any of nodes, e.g n2, +a node constraint[1] is added to n2
that constrains this node cannot have both "bar" and "foo" tags+
* when the algorithm wants to place req1 on n2, it checks if its placement
constraint is satisfied. It should be as there is no foo container on this node
yet.
* Then the algorithm checks if the node constraint is satisfied. It is not
because it violate node constraint [1].
Would that work?
> Add validation step to ensure constraints are not violated due to order in
> which a request is processed
> -------------------------------------------------------------------------------------------------------
>
> Key: YARN-7783
> URL: https://issues.apache.org/jira/browse/YARN-7783
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Arun Suresh
> Assignee: Arun Suresh
> Priority: Blocker
> Attachments: YARN-7783-YARN-6592.001.patch,
> YARN-7783-YARN-6592.002.patch, YARN-7783-YARN-6592.003.patch,
> YARN-7783-YARN-6592.004.patch
>
>
> When the algorithm has placed a container on a node, allocation tags are
> added to the node if the constraint is satisfied, But depending on the order
> in which the algorithm sees the request, it is possible that a constraint
> that happen to be valid during placement of an earlier-seen request, might
> not be valid after all subsequent requests have been placed.
> For eg:
> Assume nodes n1, n2, n3, n4 and n5
> Consider the 2 constraints:
> # *foo* -> anti-affinity with *foo*
> # *bar* -> anti-affinity with *foo*
> And 2 requests
> # req1: NumAllocations = 4, allocTags = [foo]
> # req2: NumAllocations = 1, allocTags = [bar]
> If *req1* is seen first, the algorithm can place the 4 containers in n1, n2,
> n3 and n4. And when it gets to *req2*, it will see that 4 nodes have the
> *foo* tag and will place it on n5. But if *req2* is seen first, then *bar*
> tag will be placed on any node, since no node will at that point have *foo*,
> and then when it gets to *req1*, since *foo* has no anti-affinity with *bar*,
> the algorithm can end up placing *foo* on a node with *bar* violating the
> second constraint.
> To prevent the above, we need a validation step: after the placements for a
> batch of requests are made, then for each req, we remove its tags from the
> node and try to see of constraints are still satisfied if the tag were to be
> added back on the node.
> When applied to the example above, after the algorithm has run through *req2*
> and then *req1*, we remove the *bar* tag from the node and try to add it back
> on the node. This time, constraint satisfaction will fail, since there is now
> a *foo* tag on the node and *bar* cannot be added. The algorithm will then
> retry placing *req2* on another node.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
