[
https://issues.apache.org/jira/browse/YARN-7453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16242855#comment-16242855
]
Arun Suresh commented on YARN-7453:
-----------------------------------
Verified it myself on a fresh cluster running 2.9.0 RC.
Manual mode:
{code}
2017-11-07 15:39:14,420 INFO
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned to
active state
2017-11-07 15:39:14,420 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop
IP=10.200.91.22 OPERATION=transitionToActive TARGET=RM RESULT=SUCCESS
....
2017-11-07 15:40:02,791 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop
IP=10.200.91.22 OPERATION=transitionToStandby TARGET=RM RESULT=SUCCESS
2017-11-07 15:41:08,378 INFO org.apache.hadoop.conf.Configuration: found
resource yarn-site.xml at file:/home/hadoop/cisl-linux-021/conf/yarn-site.xml
....
2017-11-07 15:41:08,624 INFO
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned to
active state
2017-11-07 15:41:08,624 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop
IP=10.200.91.22 OPERATION=transitionToActive TARGET=RM RESULT=SUCCESS
{code}
Commands:
{code}
[hadoop@cisl-linux-021 cisl-linux-021]$ yarn rmadmin -transitionToStandby rm1
[hadoop@cisl-linux-021 cisl-linux-021]$ yarn rmadmin -getAllServiceState
cisl-linux-021:8033 standby
cisl-linux-024:8033 standby
[hadoop@cisl-linux-021 cisl-linux-021]$ yarn rmadmin -transitionToActive rm2
[hadoop@cisl-linux-021 cisl-linux-021]$ yarn rmadmin -getAllServiceState
cisl-linux-021:8033 standby
cisl-linux-024:8033 active
[hadoop@cisl-linux-021 cisl-linux-021]$ yarn rmadmin -transitionToStandby rm2
[hadoop@cisl-linux-021 cisl-linux-021]$ yarn rmadmin -getAllServiceState
cisl-linux-021:8033 standby
cisl-linux-024:8033 standby
[hadoop@cisl-linux-021 cisl-linux-021]$ yarn rmadmin -transitionToActive rm1
[hadoop@cisl-linux-021 cisl-linux-021]$ yarn rmadmin -getAllServiceState
cisl-linux-021:8033 active
cisl-linux-024:8033 standby
{code}
Automatic mode:
{code}
[hadoop@cisl-linux-021 cisl-linux-021]$ yarn rmadmin -getAllServiceState
cisl-linux-021:8033 active
cisl-linux-024:8033 standby
[hadoop@cisl-linux-021 cisl-linux-021]$ yarn rmadmin -getAllServiceState
cisl-linux-021:8033 active
cisl-linux-024:8033 standby
[hadoop@cisl-linux-021 cisl-linux-021]$ yarn-daemon.sh stop resourcemanager
stopping resourcemanager
[hadoop@cisl-linux-021 cisl-linux-021]$ yarn rmadmin -getAllServiceState
17/11/07 15:49:33 INFO ipc.Client: Retrying connect to server:
cisl-linux-021/10.200.91.22:8033. Already tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
cisl-linux-021:8033 Failed to connect: Call From
cisl-linux-021/10.200.91.22 to cisl-linux-021:8033 failed on connection
exception: java.net.ConnectException: Connection refused; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused
cisl-linux-024:8033 active
[hadoop@cisl-linux-021 cisl-linux-021]$ yarn-daemon.sh start resourcemanager
starting resourcemanager, logging to
/home/hadoop/cisl-linux-021/yarn-log/yarn-hadoop-resourcemanager-cisl-linux-021.out
[hadoop@cisl-linux-021 cisl-linux-021]$ yarn rmadmin -getAllServiceState
cisl-linux-021:8033 standby
cisl-linux-024:8033 active
...
[Went to other RM and killed it]
...
[hadoop@cisl-linux-021 cisl-linux-021]$ yarn rmadmin -getAllServiceState
cisl-linux-021:8033 active
17/11/07 15:50:28 INFO ipc.Client: Retrying connect to server:
cisl-linux-024/10.200.91.25:8033. Already tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
cisl-linux-024:8033 Failed to connect: Call From
cisl-linux-021/10.200.91.22 to cisl-linux-024:8033 failed on connection
exception: java.net.ConnectException: Connection refused; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused
...
[Restarted the other RM]
...
[hadoop@cisl-linux-021 cisl-linux-021]$ yarn rmadmin -getAllServiceState
cisl-linux-021:8033 active
cisl-linux-024:8033 standby
{code}
> RM fail to switch to active after first successful start
> --------------------------------------------------------
>
> Key: YARN-7453
> URL: https://issues.apache.org/jira/browse/YARN-7453
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 2.9.0, 3.1.0
> Reporter: Rohith Sharma K S
> Priority: Blocker
>
> It is observed that RM fail to switch to ACTIVE after first successful start!
> The below exception throws when RM is switching from ACTIVE->STANDBY->ACTIVE.
> This continues in loop!
> {noformat}
> 2017-11-07 15:08:11,664 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioning
> to active state
> 2017-11-07 15:08:11,669 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Recovery
> started
> 2017-11-07 15:08:11,669 INFO
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Loaded
> RM state version info 1.5
> 2017-11-07 15:08:11,670 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Failed to
> load/recover state
> org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
> at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:1006)
> at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:910)
> at
> org.apache.curator.framework.imps.CuratorTransactionImpl.doOperation(CuratorTransactionImpl.java:159)
> at
> org.apache.curator.framework.imps.CuratorTransactionImpl.access$200(CuratorTransactionImpl.java:44)
> at
> org.apache.curator.framework.imps.CuratorTransactionImpl$2.call(CuratorTransactionImpl.java:129)
> at
> org.apache.curator.framework.imps.CuratorTransactionImpl$2.call(CuratorTransactionImpl.java:125)
> at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
> at
> org.apache.curator.framework.imps.CuratorTransactionImpl.commit(CuratorTransactionImpl.java:122)
> at
> org.apache.hadoop.util.curator.ZKCuratorManager$SafeTransaction.commit(ZKCuratorManager.java:403)
> at
> org.apache.hadoop.util.curator.ZKCuratorManager.safeSetData(ZKCuratorManager.java:372)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.getAndIncrementEpoch(ZKRMStateStore.java:493)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:771)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:1162)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1202)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1198)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1198)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:320)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:144)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:894)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:473)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:607)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
