[ 
https://issues.apache.org/jira/browse/YARN-3385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14528139#comment-14528139
 ] 

zhihai xu commented on YARN-3385:
---------------------------------

[~vinodkv], thanks for the thorough review.
bq. Some commands have multiple delete commands and only one of them may fail 
......
Based on the document about ZooKeeper#multi: {{Executes multiple ZooKeeper 
operations or none of them.}}, for multiple delete commands, either all delete 
operations are done or none of them are done. So I think if any one of them 
fails, the remaining ops won't be done.

I uploaded a new patch YARN-3385.002.patch, which addressed all your comments. 
Please review it.

> Race condition: KeeperException$NoNodeException will cause RM shutdown during 
> ZK node deletion.
> -----------------------------------------------------------------------------------------------
>
>                 Key: YARN-3385
>                 URL: https://issues.apache.org/jira/browse/YARN-3385
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>            Reporter: zhihai xu
>            Assignee: zhihai xu
>            Priority: Critical
>         Attachments: YARN-3385.000.patch, YARN-3385.001.patch, 
> YARN-3385.002.patch
>
>
> Race condition: KeeperException$NoNodeException will cause RM shutdown during 
> ZK node deletion(Op.delete).
> The race condition is similar as YARN-3023.
> since the race condition exists for ZK node creation, it should also exist 
> for  ZK node deletion.
> We see this issue with the following stack trace:
> {code}
> 2015-03-17 19:18:58,958 FATAL 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Received a 
> org.apache.hadoop.yarn.server.resourcemanager.RMFatalEvent of type 
> STATE_STORE_OP_FAILED. Cause:
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
>       at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
>       at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:945)
>       at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:911)
>       at 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$4.run(ZKRMStateStore.java:857)
>       at 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$4.run(ZKRMStateStore.java:854)
>       at 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithCheck(ZKRMStateStore.java:973)
>       at 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithRetries(ZKRMStateStore.java:992)
>       at 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.doMultiWithRetries(ZKRMStateStore.java:854)
>       at 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.removeApplicationStateInternal(ZKRMStateStore.java:647)
>       at 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.handleStoreEvent(RMStateStore.java:691)
>       at 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:766)
>       at 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:761)
>       at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
>       at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
>       at java.lang.Thread.run(Thread.java:745)
> 2015-03-17 19:18:58,959 INFO org.apache.hadoop.util.ExitUtil: Exiting with 
> status 1
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to