[
https://issues.apache.org/jira/browse/HBASE-9458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13760909#comment-13760909
]
Ted Yu commented on HBASE-9458:
-------------------------------
{code}
2013-09-06 22:08:38,534 DEBUG
[rs(hemera.apache.org,52928,1378505286552)-snapshot-pool8-thread-1]
regionserver.StoreFileInfo(291): reference
'hdfs://localhost:53227/user/jenkins/hbase/data/default/test/58e49bd9f8d751f378928495d69dd50d/fam/30b951996ef34885a2f5d64e4acb2467.6d1ed72bf95759cb606e8a6efdc6908e'
to region=6d1ed72bf95759cb606e8a6efdc6908e
hfile=30b951996ef34885a2f5d64e4acb2467
2013-09-06 22:08:38,534 DEBUG
[rs(hemera.apache.org,52928,1378505286552)-snapshot-pool8-thread-1]
regionserver.StoreFileInfo(291): reference
'hdfs://localhost:53227/user/jenkins/hbase/data/default/test/58e49bd9f8d751f378928495d69dd50d/fam/3dbcb2680c5a4d57ac0c4c9096fd54ad.4467222cc50c8e7d4f8da1196b8060a9'
to region=4467222cc50c8e7d4f8da1196b8060a9
hfile=3dbcb2680c5a4d57ac0c4c9096fd54ad
2013-09-06 22:08:38,534 DEBUG
[rs(hemera.apache.org,52928,1378505286552)-snapshot-pool8-thread-1]
regionserver.HRegion(2482): Adding snapshot references for
[hdfs://localhost:53227/user/jenkins/hbase/data/default/test/58e49bd9f8d751f378928495d69dd50d/fam/30b951996ef34885a2f5d64e4acb2467.6d1ed72bf95759cb606e8a6efdc6908e-hdfs://localhost:53227/user/jenkins/hbase/data/default/test/6d1ed72bf95759cb606e8a6efdc6908e/fam/30b951996ef34885a2f5d64e4acb2467-top,
hdfs://localhost:53227/user/jenkins/hbase/data/default/test/58e49bd9f8d751f378928495d69dd50d/fam/3dbcb2680c5a4d57ac0c4c9096fd54ad.4467222cc50c8e7d4f8da1196b8060a9-hdfs://localhost:53227/user/jenkins/hbase/data/default/test/4467222cc50c8e7d4f8da1196b8060a9/fam/3dbcb2680c5a4d57ac0c4c9096fd54ad-top]
hfiles
2013-09-06 22:08:38,534 DEBUG
[rs(hemera.apache.org,52928,1378505286552)-snapshot-pool8-thread-1]
regionserver.HRegion(2494): Creating reference for file (1/2) :
hdfs://localhost:53227/user/jenkins/hbase/data/default/test/58e49bd9f8d751f378928495d69dd50d/fam/30b951996ef34885a2f5d64e4acb2467.6d1ed72bf95759cb606e8a6efdc6908e
2013-09-06 22:08:38,535 DEBUG
[RS:0;hemera:52928-smallCompactions-1378505317956] regionserver.HStore(1383):
Removing store files after compaction...
2013-09-06 22:08:38,540 DEBUG
[RS:0;hemera:52928-smallCompactions-1378505317956] backup.HFileArchiver(438):
Finished archiving from class
org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile,
file:hdfs://localhost:53227/user/jenkins/hbase/data/default/test/58e49bd9f8d751f378928495d69dd50d/fam/30b951996ef34885a2f5d64e4acb2467.6d1ed72bf95759cb606e8a6efdc6908e,
to
hdfs://localhost:53227/user/jenkins/hbase/archive/data/default/test/58e49bd9f8d751f378928495d69dd50d/fam/30b951996ef34885a2f5d64e4acb2467.6d1ed72bf95759cb606e8a6efdc6908e
2013-09-06 22:08:38,542 DEBUG
[RS:0;hemera:52928-smallCompactions-1378505317956] backup.HFileArchiver(438):
Finished archiving from class
org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile,
...
2013-09-06 22:08:43,672 DEBUG [MASTER_TABLE_OPERATIONS-hemera:59551-0]
regionserver.StoreFileInfo(291): reference
'hdfs://localhost:53227/user/jenkins/hbase/.hbase-snapshot/.tmp/snapshotAfterMerge/58e49bd9f8d751f378928495d69dd50d/fam/30b951996ef34885a2f5d64e4acb2467.6d1ed72bf95759cb606e8a6efdc6908e'
to region=6d1ed72bf95759cb606e8a6efdc6908e
hfile=30b951996ef34885a2f5d64e4acb2467
2013-09-06 22:08:43,673 ERROR [MASTER_TABLE_OPERATIONS-hemera:59551-0]
snapshot.TakeSnapshotHandler(201): Failed taking snapshot {
ss=snapshotAfterMerge table=test type=FLUSH } due to exception:Missing parent
hfile for: 30b951996ef34885a2f5d64e4acb2467.6d1ed72bf95759cb606e8a6efdc6908e
org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Missing parent
hfile for: 30b951996ef34885a2f5d64e4acb2467.6d1ed72bf95759cb606e8a6efdc6908e
at
org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifyStoreFile(MasterSnapshotVerifier.java:224)
at
org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.access$000(MasterSnapshotVerifier.java:82)
at
org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier$1.storeFile(MasterSnapshotVerifier.java:210)
at
org.apache.hadoop.hbase.util.FSVisitor.visitRegionStoreFiles(FSVisitor.java:115)
at
org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitRegionStoreFiles(SnapshotReferenceUtil.java:123)
at
org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifyRegion(MasterSnapshotVerifier.java:207)
at
org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifyRegions(MasterSnapshotVerifier.java:175)
{code}
Looks like HFileArchiver#resolveAndArchiveFile() moved the referredTo file
> Intermittent TestFlushSnapshotFromClient#testTakeSnapshotAfterMerge failure
> ---------------------------------------------------------------------------
>
> Key: HBASE-9458
> URL: https://issues.apache.org/jira/browse/HBASE-9458
> Project: HBase
> Issue Type: Test
> Reporter: Ted Yu
>
> From
> https://builds.apache.org/job/HBase-0.96/20/testReport/org.apache.hadoop.hbase.snapshot/TestFlushSnapshotFromClient/testTakeSnapshotAfterMerge/
> :
> {code}
> org.apache.hadoop.hbase.snapshot.HBaseSnapshotException:
> org.apache.hadoop.hbase.snapshot.HBaseSnapshotException: Snapshot {
> ss=snapshotAfterMerge table=test type=FLUSH } had an error. Procedure
> snapshotAfterMerge { waiting=[] done=[] }
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:210)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:221)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:125)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:96)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3120)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.snapshot(HBaseAdmin.java:2672)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.snapshot(HBaseAdmin.java:2605)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.snapshot(HBaseAdmin.java:2612)
> at
> org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient.testTakeSnapshotAfterMerge(TestFlushSnapshotFromClient.java:336)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException:
> org.apache.hadoop.hbase.snapshot.HBaseSnapshotException: Snapshot {
> ss=snapshotAfterMerge table=test type=FLUSH } had an error. Procedure
> snapshotAfterMerge { waiting=[] done=[] }
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:365)
> at
> org.apache.hadoop.hbase.master.HMaster.isSnapshotDone(HMaster.java:2947)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterAdminProtos$MasterAdminService$2.callBlockingMethod(MasterAdminProtos.java:32890)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
> Caused by: org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException via
> Failed taking snapshot { ss=snapshotAfterMerge table=test type=FLUSH } due to
> exception:Missing parent hfile for:
> 30b951996ef34885a2f5d64e4acb2467.6d1ed72bf95759cb606e8a6efdc6908e:org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException:
> Missing parent hfile for:
> 30b951996ef34885a2f5d64e4acb2467.6d1ed72bf95759cb606e8a6efdc6908e
> at
> org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:85)
> at
> org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.rethrowExceptionIfFailed(TakeSnapshotHandler.java:318)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:355)
> ... 4 more
> Caused by: org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException:
> Missing parent hfile for:
> 30b951996ef34885a2f5d64e4acb2467.6d1ed72bf95759cb606e8a6efdc6908e
> at
> org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifyStoreFile(MasterSnapshotVerifier.java:224)
> at
> org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.access$000(MasterSnapshotVerifier.java:82)
> at
> org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier$1.storeFile(MasterSnapshotVerifier.java:210)
> at
> org.apache.hadoop.hbase.util.FSVisitor.visitRegionStoreFiles(FSVisitor.java:115)
> at
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitRegionStoreFiles(SnapshotReferenceUtil.java:123)
> at
> org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifyRegion(MasterSnapshotVerifier.java:207)
> at
> org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifyRegions(MasterSnapshotVerifier.java:175)
> at
> org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifySnapshot(MasterSnapshotVerifier.java:120)
> at
> org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.process(TakeSnapshotHandler.java:189)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:131)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1425)
> at
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1629)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1686)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterAdminProtos$MasterAdminService$BlockingStub.isSnapshotDone(MasterAdminProtos.java:34923)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$5.isSnapshotDone(HConnectionManager.java:2123)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin$24.call(HBaseAdmin.java:2675)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin$24.call(HBaseAdmin.java:2672)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:118)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
