[
https://issues.apache.org/jira/browse/YARN-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15272901#comment-15272901
]
Daniel Zhi commented on YARN-4676:
----------------------------------
To clarify before I make code changes:
1. HostsFileReader currently allows multiple hosts per line. When hosts are
pure digits, there will be ambiguity with timeout during interpretation. Likely
allowing pure digit would requires pure-digit-host starts with a new line.
2. -1 means infinite timeout (wait forever until ready). null means no
overwrite, use the default timeout.
3. there could be large number of hosts to be decommissioned so the single line
could be huge. grep a particular host would return a huge line in that case. A
mix could be log in a single line for less than N host but otherwise multiple
line. That said, I am ok to change to single line.
4. simple after 1)
5. same as 2
6. ok
7. How about DEFAULT_NM_EXIT_WAIT_MS = 0? So that it could be customized in
cases the delay is preferred.
8. The grace period is to give RM server-side a chance to DECOMMISSION the node
should timeout reaches. A much smaller period like 2 seconds most likely would
be sufficient as NodeManager heartbeat every second during which
DECOMMISSIONING node will be re-evaluated and decommissioned if ready or
timeout.
9. "yarn rmadmin -refreshNodes -g -1" waits forever until the node is ready.
"yarn rmadmin -refreshNodes -g" uses default timeout as specified by the
configuration key.
10. same as 2)
11. ok
12. see 7)
13. ok
14. Here is an example of the tabular logging. Keeping DECOMMISSIONED node a
little longer prevent it from suddenly disappeared from the list after
DECOMMISSIONed.
2015-08-14 20:31:00,797 INFO
org.apache.hadoop.yarn.server.resourcemanager.DecommissioningNodesWatcher (IPC
Server handler 14 on 9023): Decommissioning Nodes:
ip-10-45-166-151.ec2.internal 20s fresh: 0s containers:14
WAIT_CONTAINER timeout:1779s
application_1439334429355_0004 RUNNING MAPREDUCE 7.50% 55s
ip-10-170-95-251.ec2.internal 20s fresh: 0s containers:14
WAIT_CONTAINER timeout:1779s
application_1439334429355_0004 RUNNING MAPREDUCE 7.50% 55s
ip-10-29-137-237.ec2.internal 19s fresh: 0s containers:14
WAIT_CONTAINER timeout:1780s
application_1439334429355_0004 RUNNING MAPREDUCE 7.50% 55s
ip-10-157-4-26.ec2.internal 19s fresh: 0s containers:14
WAIT_CONTAINER timeout:1780s
application_1439334429355_0004 RUNNING MAPREDUCE 7.50% 55s
15. I agree that getDecommissioningStatus suggest the call is read-only. Since
completed apps need to be take into account when evaluate readiness of the
node, getDecommissioningStatus is actually a private method used internally so
it could be changed into private checkDecommissioningStatus(nodeId).
16. readDecommissioningTimeout is to pick up new value without restart RM. It
was requested by EMR customers and I do see the user scenarios. It is only
invoked when there are DECOMMISSIONED nodes and will only be invoked once every
20 seconds (poll period). I have to maintain private patch or consider other
option if remove the feature.
17. ok
18. The method return number of seconds to timeout. I don't mind changing the
name to getTimeoutTimestampInSec() but don't see the reason behind.
19. see the example in 14. This is once every 20 seconds and was very useful
during my development and testing of the work. I see more valuable to leave it
as INFO but as the code become mature and stable, maybe ok to turn into DEBUG.
20. ok
21. The isValidNode() && isNodeInDecommissioning() condition is just a very
quick shallow check --- for a DECOMMISSIONING node, although nodesListManager
would return false for isValidNode() as the node appear in excluded host list,
such node will be allowed to continue as it is in the middle of
DECOMMISSIONING. During the process of the heart beat, decommissioningWatcher
is updated with the latest container status of the node; Later
decomWatcher.checkReadyToBeDecommissioned(rmNode.getNodeID()) evaluates its
readiness and DECOMMISSION the node if ready (include timeout).
22. the call simply returns if within 20 seconds of last call. Currently it
lives inside ResourceTrackerService and uses rmContext. Alternatively
DecommissioningNodesWatcher could be constructed with rmContext and internally
has its own polling thread. Other than not sure yet the code pattern to use for
such internal thread, it appears as valid alternative to me.
23. ok
24. ok
25. Instead of disallow and exit, an alternative way is to allow the graceful
decommission as usual. There will be no difference if no RM restart during the
session. In case RM restart, currently all excluded nodes decommissioned right
away, an enhanced support in future will resume it.
> Automatic and Asynchronous Decommissioning Nodes Status Tracking
> ----------------------------------------------------------------
>
> Key: YARN-4676
> URL: https://issues.apache.org/jira/browse/YARN-4676
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
> Affects Versions: 2.8.0
> Reporter: Daniel Zhi
> Assignee: Daniel Zhi
> Labels: features
> Attachments: GracefulDecommissionYarnNode.pdf,
> GracefulDecommissionYarnNode.pdf, YARN-4676.004.patch, YARN-4676.005.patch,
> YARN-4676.006.patch, YARN-4676.007.patch, YARN-4676.008.patch,
> YARN-4676.009.patch, YARN-4676.010.patch, YARN-4676.011.patch,
> YARN-4676.012.patch, YARN-4676.013.patch
>
>
> YARN-4676 implements an automatic, asynchronous and flexible mechanism to
> graceful decommission
> YARN nodes. After user issues the refreshNodes request, ResourceManager
> automatically evaluates
> status of all affected nodes to kicks out decommission or recommission
> actions. RM asynchronously
> tracks container and application status related to DECOMMISSIONING nodes to
> decommission the
> nodes immediately after there are ready to be decommissioned. Decommissioning
> timeout at individual
> nodes granularity is supported and could be dynamically updated. The
> mechanism naturally supports multiple
> independent graceful decommissioning “sessions” where each one involves
> different sets of nodes with
> different timeout settings. Such support is ideal and necessary for graceful
> decommission request issued
> by external cluster management software instead of human.
> DecommissioningNodeWatcher inside ResourceTrackingService tracks
> DECOMMISSIONING nodes status automatically and asynchronously after
> client/admin made the graceful decommission request. It tracks
> DECOMMISSIONING nodes status to decide when, after all running containers on
> the node have completed, will be transitioned into DECOMMISSIONED state.
> NodesListManager detect and handle include and exclude list changes to kick
> out decommission or recommission as necessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
