[
https://issues.apache.org/jira/browse/HBASE-13217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14358388#comment-14358388
]
ramkrishna.s.vasudevan commented on HBASE-13217:
------------------------------------------------
The stack trace that we get is
{code}
ble-proc
2015-03-12 20:41:07,914 INFO
[(stobdtserver3,16040,1426172670959)-proc-coordinator-pool3-thread-1]
procedure.Procedure: Starting procedure 'TestTable'
2015-03-12 20:41:07,916 INFO [main-EventThread]
procedure.ZKProcedureMemberRpcs: Received procedure start children changed
event: /hbase/flush-table-proc/acquired
2015-03-12 20:41:07,917 INFO
[(stobdtserver3,16040,1426172670959)-proc-coordinator-pool3-thread-1]
procedure.Procedure: Procedure 'TestTable' execution completed
2015-03-12 20:41:07,917 INFO
[(stobdtserver3,16040,1426172670959)-proc-coordinator-pool3-thread-1]
procedure.ZKProcedureUtil: Clearing all znodes for procedure TestTableincluding
nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached
/hbase/flush-table-proc/abort
2015-03-12 20:41:07,920 INFO [main-EventThread]
procedure.ZKProcedureMemberRpcs: Received procedure start children changed
event: /hbase/flush-table-proc/acquired
2015-03-12 20:41:07,922 INFO
[B.defaultRpcServer.handler=15,queue=0,port=16040]
flush.MasterFlushTableProcedureManager: Done waiting - exec procedure
flush-table-proc for 'TestTable'
2015-03-12 20:41:07,930 INFO
[B.defaultRpcServer.handler=15,queue=0,port=16040]
flush.MasterFlushTableProcedureManager: Master flush table procedure is
successful!
2015-03-12 20:41:07,939 ERROR [member: 'stobdtserver3,16040,1426172670959'
subprocedure-pool4-thread-1] procedure.ProcedureMember: Failed to acquire
barrier for procedure: TestTable and member: stobdtserver3,16040,1426172670959
java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for
/hbase/flush-table-proc/acquired/TestTable/stobdtserver3,16040,1426172670959
at
org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.sendMemberAcquired(ZKProcedureMemberRpcs.java:254)
at
org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:166)
at
org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:52)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for
/hbase/flush-table-proc/acquired/TestTable/stobdtserver3,16040,1426172670959
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:512)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:491)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:1257)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:1246)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:1229)
at
org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.sendMemberAcquired(ZKProcedureMemberRpcs.java:245)
... 6 more
2015-03-12 20:41:07,941 ERROR [member: 'stobdtserver3,16040,1426172670959'
subprocedure-pool4-thread-1] procedure.Subprocedure: Failed to acquire barrier
for procedure: TestTable and member: stobdtserver3,16040,1426172670959
java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for
/hbase/flush-table-proc/acquired/TestTable/stobdtserver3,16040,1426172670959
at
org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.sendMemberAcquired(ZKProcedureMemberRpcs.java:254)
at
org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:166)
at
org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:52)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for
/hbase/flush-table-proc/acquired/TestTable/stobdtserver3,16040,1426172670959
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:512)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:491)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:1257)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:1246)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:1229)
at
org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.sendMemberAcquired(ZKProcedureMemberRpcs.java:245)
... 6 more
2015-03-12 20:41:07,952 INFO [main-EventThread]
procedure.ZKProcedureMemberRpcs: Received created
event:/hbase/flush-table-proc/abort/TestTable
2015-03-12 20:41:07,953 ERROR [member: 'stobdtserver3,16040,1426172670959'
subprocedure-pool4-thread-1] procedure.Subprocedure: Subprocedure 'TestTable'
aborting due to a ForeignException!
java.io.IOException via stobdtserver3,16040,1426172670959:java.io.IOException:
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
for /hbase/flush-table-proc/acquired/TestTable/stobdtserver3,16040,1426172670959
at
org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83)
at
org.apache.hadoop.hbase.procedure.Procedure.waitForLatch(Procedure.java:397)
at
org.apache.hadoop.hbase.procedure.Subprocedure.waitForReachedGlobalBarrier(Subprocedure.java:299)
at
org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:171)
at
org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:52)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException:
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
for /hbase/flush-table-proc/acquired/TestTable/stobdtserver3,16040,1426172670959
at
org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.sendMemberAcquired(ZKProcedureMemberRpcs.java:254)
at
org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:166)
... 5 more
Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for
/hbase/flush-table-proc/acquired/TestTable/stobdtserver3,16040,1426172670959
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:512)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:491)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:1257)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:1246)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:1229)
at
org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.sendMemberAcquired(ZKProcedureMemberRpcs.java:245)
... 6 more
2015-03-12 20:41:07,953 ERROR [main-EventThread] procedure.ProcedureMember:
Propagating foreign exception to subprocedure TestTable
org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via
stobdtserver3,16040,1426172670959:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable:
java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for
/hbase/flush-table-proc/acquired/TestTable/stobdtserver3,16040,1426172670959
at
org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171)
at
org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:329)
at
org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeCreated(ZKProcedureMemberRpcs.java:94)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:405)
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
Caused by:
org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable:
java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for
/hbase/flush-table-proc/acquired/TestTable/stobdtserver3,16040,1426172670959
at
org.apache.hadoop.hbase.procedure.Subprocedure.cancel(Subprocedure.java:273)
at
org.apache.hadoop.hbase.procedure.ProcedureMember.controllerConnectionFailure(ProcedureMember.java:225)
at
org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.sendMemberAcquired(ZKProcedureMemberRpcs.java:254)
at
org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:166)
at
org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:52)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2015-03-12 20:41:07,960 INFO [member: 'stobdtserver3,16040,1426172670959'
subprocedure-pool4-thread-1] flush.FlushTableSubprocedure: Aborting all flush
region subprocedure task threads for 'TestTable' due to error
java.io.IOException via stobdtserver3,16040,1426172670959:java.io.IOException:
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
for /hbase/flush-table-proc/acquired/TestTable/stobdtserver3,16040,1426172670959
at
org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83)
at
org.apache.hadoop.hbase.procedure.Procedure.waitForLatch(Procedure.java:397)
at
org.apache.hadoop.hbase.procedure.Subprocedure.waitForReachedGlobalBarrier(Subprocedure.java:299)
at
org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:171)
at
org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:52)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException:
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
for /hbase/flush-table-proc/acquired/TestTable/stobdtserver3,16040,1426172670959
at
org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.sendMemberAcquired(ZKProcedureMemberRpcs.java:254)
at
org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:166)
... 5 more
Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for
/hbase/flush-table-proc/acquired/TestTable/stobdtserver3,16040,1426172670959
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:512)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:491)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:1257)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:1246)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:1229)
at
org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.sendMemberAcquired(ZKProcedureMemberRpcs.java:245)
... 6 more
2015-03-12 20:41:07,961 ERROR [main-EventThread] procedure.Subprocedure:
Propagating foreign exception to subprocedure TestTable
org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via
stobdtserver3,16040,1426172670959:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable:
java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for
/hbase/flush-table-proc/acquired/TestTable/stobdtserver3,16040,1426172670959
at
org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171)
at
org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:329)
at
org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeCreated(ZKProcedureMemberRpcs.java:94)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:405)
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
Caused by:
org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable:
java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for
/hbase/flush-table-proc/acquired/TestTable/stobdtserver3,16040,1426172670959
at
org.apache.hadoop.hbase.procedure.Subprocedure.cancel(Subprocedure.java:273)
at
org.apache.hadoop.hbase.procedure.ProcedureMember.controllerConnectionFailure(ProcedureMember.java:225)
at
org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.sendMemberAcquired(ZKProcedureMemberRpcs.java:254)
at
org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:166)
at
org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:52)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2015-03-12 20:41:07,962 INFO [main-EventThread]
procedure.ZKProcedureMemberRpcs: Received procedure abort children changed
event: /hbase/flush-table-proc/abort
2015-03-12 20:41:07,963 ERROR [main-EventThread] procedure.ProcedureMember:
Propagating foreign exception to subprocedure TestTable
org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via
stobdtserver3,16040,1426172670959:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable:
java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for
/hbase/flush-table-proc/acquired/TestTable/stobdtserver3,16040,1426172670959
at
org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171)
at
org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:329)
at
org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.watchForAbortedProcedures(ZKProcedureMemberRpcs.java:142)
at
org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.access$200(ZKProcedureMemberRpcs.java:55)
at
org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeChildrenChanged(ZKProcedureMemberRpcs.java:110)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:426)
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
Caused by:
org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable:
java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for
/hbase/flush-table-proc/acquired/TestTable/stobdtserver3,16040,1426172670959
at
org.apache.hadoop.hbase.procedure.Subprocedure.cancel(Subprocedure.java:273)
at
org.apache.hadoop.hbase.procedure.ProcedureMember.controllerConnectionFailure(ProcedureMember.java:225)
at
org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.sendMemberAcquired(ZKProcedureMemberRpcs.java:254)
at
org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:166)
at
org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:52)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2015-03-12 20:41:07,964 ERROR [main-EventThread] procedure.Subprocedure:
Propagating foreign exception to subprocedure TestTable
org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via
stobdtserver3,16040,1426172670959:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable:
java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for
/hbase/flush-table-proc/acquired/TestTable/stobdtserver3,16040,1426172670959
at
org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171)
at
org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:329)
at
org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.watchForAbortedProcedures(ZKProcedureMemberRpcs.java:142)
at
org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.access$200(ZKProcedureMemberRpcs.java:55)
at
org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeChildrenChanged(ZKProcedureMemberRpcs.java:110)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:426)
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
Caused by:
org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable:
java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for
/hbase/flush-table-proc/acquired/TestTable/stobdtserver3,16040,1426172670959
at
org.apache.hadoop.hbase.procedure.Subprocedure.cancel(Subprocedure.java:273)
at
org.apache.hadoop.hbase.procedure.ProcedureMember.controllerConnectionFailure(ProcedureMember.java:225)
at
org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.sendMemberAcquired(ZKProcedureMemberRpcs.java:254)
at
org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:166)
at
org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:52)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
> Flush procedure fails in trunk due to ZK issue
> ----------------------------------------------
>
> Key: HBASE-13217
> URL: https://issues.apache.org/jira/browse/HBASE-13217
> Project: HBase
> Issue Type: Bug
> Reporter: ramkrishna.s.vasudevan
>
> When ever I try to flush explicitly in the trunk code the flush procedure
> fails due to ZK issue
> {code}
> ERROR: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable
> via
> stobdtserver3,16040,1426172670959:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable:
> java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for
> /hbase/flush-table-proc/acquired/TestTable/stobdtserver3,16040,1426172670959
> at
> org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83)
> at
> org.apache.hadoop.hbase.procedure.Procedure.isCompleted(Procedure.java:368)
> at
> org.apache.hadoop.hbase.procedure.flush.MasterFlushTableProcedureManager.isProcedureDone(MasterFlushTableProcedureManager.java:196)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.isProcedureDone(MasterRpcServices.java:905)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:47019)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2073)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> Caused by:
> org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable:
> java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for
> /hbase/flush-table-proc/acquired/TestTable/stobdtserver3,16040,1426172670959
> at
> org.apache.hadoop.hbase.procedure.Subprocedure.cancel(Subprocedure.java:273)
> at
> org.apache.hadoop.hbase.procedure.ProcedureMember.controllerConnectionFailure(ProcedureMember.java:225)
> at
> org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.sendMemberAcquired(ZKProcedureMemberRpcs.java:254)
> at
> org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:166)
> at
> org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:52)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ... 1 more
> {code}
> Once this occurs, even on restart of the RS the RS becomes unusable. I have
> verified that the ZK remains intact and there is no problem with it. a bit
> older version of trunk ( 3months) does not have this problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
