[
https://issues.apache.org/jira/browse/YARN-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15896978#comment-15896978
]
Varun Saxena edited comment on YARN-6148 at 3/6/17 10:00 AM:
-------------------------------------------------------------
Offline, we (me, [~Naganarasimha] and [~bibinchundatt]) had discussed some more
scenarios which need to be handled.
So following is the plan regarding what will be done.
# We will send a label to node count map containing label to active NM count
mapping for requested labels.
# If labels are disabled, we will report cluster count as earlier and not send
anything in label to active NM count mapping. AM can consider only the cluster
node count reported earlier in such cases. This will also help us in handling
the case for rolling downgrades.
# Non Exclusive labels will not be reported in the label to node count unless
explicitly requested by AM.
# Mapreduce AM would consider both map and reduce label expression separately,
if they are different, while deciding on ignoring AM blacklisting. We will
check label to node count mapping to determine the applicable node counts for
both map and reduce.
# Queue can have a default label too. This needs to be reported to AM as well.
Also needs to be reported on move. Can be handled in YARN-6209. Active NM count
may depend on it and AM currently does not know anything about default label of
queue.
# Also we can report AM node label from Application Submission context in
Register AM response. This is because RM can consider label in ASC if none is
specified in AM ask.
# Container info sent while reporting allocated containers would also contain
label on which container was allocated on. Based on this info we will determine
if the node assigned is a default label of queue, map/reduce partition or non
exclusive label. We will not count non exclusive label towards ignoring
blacklisting.
Thoughts?
was (Author: varun_saxena):
Offline, we (me, [~Naganarasimha] and [~bibinchundatt]) had discussed some more
scenarios which need to be handled.
So following is the plan regarding what will be done.
# We will send a label to node count map containing label to active NM count
mapping for requested labels.
# If labels are disabled, we will report cluster count as earlier and not send
anything in label to active NM count mapping. AM can consider only the cluster
node count reported earlier in such cases. This will also help us in handling
the case for rolling downgrades.
# Non Exclusive labels will not be reported in the label to node count unless
explicitly requested by AM.
# Mapreduce AM would consider both map and reduce label expression separately,
if they are different, while deciding on ignoring AM blacklisting. We will
check label to node count mapping to determine the applicable node counts for
both map and reduce.
# Queue can have a default label too. This needs to be reported to AM as well.
Also needs to be reported on move. Can be handled in YARN-6209. Active NM count
may depend on it and AM currently does not know anything about default label of
queue.
# Container info sent while reporting allocated containers would also contain
label on which container was allocated on. Based on this info we will determine
if the node assigned is a default label of queue, map/reduce partition or non
exclusive label. We will not count non exclusive label towards ignoring
blacklisting.
Thoughts?
> NM node count reported to AM in Allocate Response should consider requested
> node label partitions.
> --------------------------------------------------------------------------------------------------
>
> Key: YARN-6148
> URL: https://issues.apache.org/jira/browse/YARN-6148
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Varun Saxena
> Assignee: Varun Saxena
> Attachments: YARN-6148.01.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
