[
https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314876#comment-16314876
]
Wangda Tan commented on YARN-6599:
----------------------------------
Thanks [~asuresh],
bq. Wangda Tan, apologize, but I think we really should split this part.
Let me explain a bit, it may not be as hard as you thought about.
Following changes are related to partition:
1) Added nodePartition to PlacementConstraints
2) Renamed acceptNodePartition to precheckNode so we can blacklist nodes in
AppPlaementAllocator.
3) Check node partition inside new AppPlaementAllocator implementation.
4) Added partition check in PlacementConstraintsUtil.
5) Unit test related changes.
#1/#3 is very straightforward. #2 is needed anyway since we want to plug logics
to check SchedulingRequest while allocating/validating container allocation,
regardless of splitting logic. To me, #3 is also not hard. Please let me know
your thoughts on this. It may take me several hours to split the logic, I'm not
sure if it worths since we're pushing the branch merge soon.
bq. It just means source tag == target expression tag.
This looks not correct, first of all, it is not a new type, it's more like a
syntactic sugar which should be built inside PlacementConstraints, and I'm
doubt if it is really needed. We can discuss it in a separate JIRA.
bq. As I mentioned in the previous comment. There is is this issue of
application priority
To me, this is not different from affinity/anti-affinity to different tags. For
example a Tensorflow "worker" affinity to "parameter server" which is in
different priority. A wrong priority setting could still cause deadlock issue.
And in order to specify intra-app affinity/anti-affinity, we have to include
application id inside allocation tags since by default allocation tag query
should include all apps.
> Support rich placement constraints in scheduler
> -----------------------------------------------
>
> Key: YARN-6599
> URL: https://issues.apache.org/jira/browse/YARN-6599
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Wangda Tan
> Assignee: Wangda Tan
> Attachments: YARN-6599-YARN-6592.003.patch,
> YARN-6599-YARN-6592.004.patch, YARN-6599-YARN-6592.005.patch,
> YARN-6599-YARN-6592.006.patch, YARN-6599-YARN-6592.007.patch,
> YARN-6599-YARN-6592.008.patch, YARN-6599-YARN-6592.wip.002.patch,
> YARN-6599.poc.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
