[
https://issues.apache.org/jira/browse/MAPREDUCE-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13160166#comment-13160166
]
Jonathan Eagles commented on MAPREDUCE-3272:
--------------------------------------------
Ramya, I could use your help trying to reproduce this.
Here is my setup.
* Not using ephemeral ports for NM
* Single node cluster setup
* Node manager time out is set to 10 minutes
# First the cluster is up and running
$ jps
1761 ResourceManager
4369 Jps
1641 DataNode
1560 NameNode
4029 NodeManager
1840 JobHistoryServer
11996 Launcher
# Next I kill the node manager
$ kill 4029
# I verify the node manager is killed
$ jps
4385 Jps
1761 ResourceManager
1641 DataNode
1560 NameNode
1840 JobHistoryServer
11996 Launcher
# I wait the full 10 minutes for the node manager to time out
2011-11-30 10:45:26,464 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Processing
machine.example.com:45454 of type EXPIRE
2011-11-30 10:45:26,465 INFO
org.apache.hadoop.yarn.util.AbstractLivelinessMonitor:
Expired:machine.example.com:45454 Timed out after 600 secs
2011-11-30 10:45:26,504 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Removed Node
machine.example.com:45454
2011-11-30 10:45:26,505 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl:
machine.example.com:45454 Node Transitioned from RUNNING to LOST
# Total nodes on the RM is decremented by one. Lost nodes on the RM is
incremented by one
# Then I restart the nodemanager
$ yarn-daemon.sh start nodemanager
# Verify the nodemanager connects successfully in the logs
2011-11-30 11:13:50,315 INFO org.apache.hadoop.yarn.util.RackResolver: Resolved
machine.example.com to /default-rack
2011-11-30 11:13:50,328 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Processing
machine.example.com:45454 of type STARTED
2011-11-30 11:13:50,328 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl:
machine.example.com:45454 Node Transitioned from NEW to RUNNING
2011-11-30 11:13:50,328 INFO
org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService:
NodeManager from node machine.example.com(cmPort: 45454 httpPort: 9999)
registered with capability: 8192, assigned nodeId machine.example.com:45454
# I notice the total nodes on the RM is incremented by one. Lost nodes is NOT
decremented by one MAPREDUCE-3271.
# Node manager is again available for use. Expected behavior.
> Lost NMs fail to rejoin
> -----------------------
>
> Key: MAPREDUCE-3272
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3272
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Jonathan Eagles
> Priority: Critical
> Fix For: 0.23.1
>
>
> Lost nodemanagers fail to join back.
> When the NM is lost, RM log reads
> {noformat}
> INFO org.apache.hadoop.yarn.util.AbstractLivelinessMonitor:
> Expired:<host:port> Timed out after 600 secs
> INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl:
> Processing <host:port> of type EXPIRE
> INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Removed
> Node <host:port>
> INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl:
> <host:port> Node Transitioned from RUNNING to LOST
> {noformat}
> When the NM joins back, RM log reads
> {noformat}
> INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService:
> Node not found rebooting <host:port>
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
