Tsuyoshi Ozawa created YARN-4348:
------------------------------------
Summary: ZKRMStateStore.syncInternal should wait for
zkResyncWaitTime instead of zkSessionTimeout
Key: YARN-4348
URL: https://issues.apache.org/jira/browse/YARN-4348
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.7.2
Reporter: Tsuyoshi Ozawa
Assignee: Tsuyoshi Ozawa
The current internal ZK configuration of ZKRMStateStore can cause a following
situation:
1. syncInternal timeouts,
2. but sync succeeded later on.
{quote}
2015-11-11 11:54:05,728 INFO recovery.ZKRMStateStore
(ZKRMStateStore.java:runWithRetries(1241)) - Failed to sync with ZK new
connection. -<--- sync failed
2015-11-11 11:54:05,728 INFO recovery.ZKRMStateStore
(ZKRMStateStore.java:runWithRetries(1244)) - Maxed out ZK retries. Giving up!
2015-11-11 11:54:05,728 ERROR recovery.RMStateStore
(RMStateStore.java:transition(292)) - Error updating appAttempt:
appattempt_1447242474882_0002_000001
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode =
Session expired for
/rmstore/ZKRMStateRoot/RMAppRoot/application_1447242474882_0002/appattempt_1447242474882_0002_000001
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1073)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$9.run(ZKRMStateStore.java:1082)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$9.run(ZKRMStateStore.java:1079)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithCheck(ZKRMStateStore.java:1164)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithRetries(ZKRMStateStore.java:1197)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.existsWithRetries(ZKRMStateStore.java:1079)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.updateApplicationAttemptStateInternal(ZKRMStateStore.java:716)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$UpdateAppAttemptTransition.transition(RMStateStore.java:286)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$UpdateAppAttemptTransition.transition(RMStateStore.java:269)
at
org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.handleStoreEvent(RMStateStore.java:1006)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:1075)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:1070)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:182)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108)
at java.lang.Thread.run(Thread.java:745)
2015-11-11 11:54:05,729 ERROR recovery.RMStateStore
(RMStateStore.java:notifyStoreOperationFailed(1027)) - State store operation
failed
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode =
Session expired for
/rmstore/ZKRMStateRoot/RMAppRoot/application_1447242474882_0002/appattempt_1447242474882_0002_000001
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1073)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$9.run(ZKRMStateStore.java:1082)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$9.run(ZKRMStateStore.java:1079)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithCheck(ZKRMStateStore.java:1164)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithRetries(ZKRMStateStore.java:1197)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.existsWithRetries(ZKRMStateStore.java:1079)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.updateApplicationAttemptStateInternal(ZKRMStateStore.java:716)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$UpdateAppAttemptTransition.transition(RMStateStore.java:286)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$UpdateAppAttemptTransition.transition(RMStateStore.java:269)
at
org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.handleStoreEvent(RMStateStore.java:1006)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:1075)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:1070)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:182)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108)
at java.lang.Thread.run(Thread.java:745)
2015-11-11 11:54:05,730 WARN recovery.RMStateStore
(RMStateStore.java:notifyStoreOperationFailed(1041)) - Skip the state-store
error.
2015-11-11 11:54:05,730 INFO recovery.RMStateStore
(RMStateStore.java:transition(206)) - Updating info for app:
application_1447242474882_0002
2015-11-11 11:54:06,018 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(726)) -
application_1447242474882_0002 State change from FINAL_SAVING to FINISHING
2015-11-11 11:54:06,018 INFO recovery.ZKRMStateStore
(ZKRMStateStore.java:processWatchEvent(913)) - Watcher event type: None with
state:SyncConnected for path:null for Service
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore in state
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: STARTED
2015-11-11 11:54:06,019 INFO recovery.ZKRMStateStore
(ZKRMStateStore.java:processWatchEvent(921)) - ZKRMStateStore Session connected
2015-11-11 11:54:06,019 INFO recovery.ZKRMStateStore
(ZKRMStateStore.java:processResult(123)) - ZooKeeper sync operation succeeded.
path:
/rmstore/ZKRMStateRoot/RMAppRoot/application_1447242474882_0002/appattempt_1447242474882_0002_000001
<- here, sync succeeded.
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
