[
https://issues.apache.org/jira/browse/PHOENIX-6274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17252714#comment-17252714
]
Viraj Jasani commented on PHOENIX-6274:
---------------------------------------
[~stoty] I also just tried to run balancer forcefully around the same time when
snapshot gets created and ran the test multiple times in loop locally but not
able to repro corrupted snapshot issue. However, disabled balancer as it's not
harmful anyways in latest patch. Is it fine with you?
> Flaky test TableSnapshotReadsMapReduceIT.testMapReduceSnapshotsMultiRegion
> --------------------------------------------------------------------------
>
> Key: PHOENIX-6274
> URL: https://issues.apache.org/jira/browse/PHOENIX-6274
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 5.0.0, 4.15.0
> Reporter: Viraj Jasani
> Assignee: Viraj Jasani
> Priority: Major
> Fix For: 5.1.0, 4.16.0
>
> Attachments: PHOENIX-6274.4.x.000.patch, PHOENIX-6274.4.x.001.patch,
> PHOENIX-6274.master.000.patch, PHOENIX-6274.master.001.patch
>
>
> Test TableSnapshotReadsMapReduceIT.testMapReduceSnapshotsMultiRegion
> sometimes fails to delete snapshot due to snapshot corruption.
> {code:java}
> ERROR [MASTER_SNAPSHOT_OPERATIONS-master/e09b7a102100:0-0]
> org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier(187): Regions
> moved during the snapshot '{ ss=FOO table=N000006 type=FLUSH }'. expected=2
> snapshotted=3.ERROR [MASTER_SNAPSHOT_OPERATIONS-master/e09b7a102100:0-0]
> org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier(187): Regions
> moved during the snapshot '{ ss=FOO table=N000006 type=FLUSH }'. expected=2
> snapshotted=3.2020-12-18 01:18:45,532 ERROR
> [MASTER_SNAPSHOT_OPERATIONS-master/e09b7a102100:0-0]
> org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler(222): Failed
> taking snapshot { ss=FOO table=N000006 type=FLUSH } due to exception:Regions
> moved during the snapshot '{ ss=FOO table=N000006 type=FLUSH }'. expected=2
> snapshotted=3.org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException:
> Regions moved during the snapshot '{ ss=FOO table=N000006 type=FLUSH }'.
> expected=2 snapshotted=3. at
> org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifyRegions(MasterSnapshotVerifier.java:205)
> at
> org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifySnapshot(MasterSnapshotVerifier.java:119)
> at
> org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.process(TakeSnapshotHandler.java:209)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> Due to this, deletion of snapshot fails:
> {code:java}
> org.apache.hadoop.hbase.snapshot.SnapshotDoesNotExistException: Snapshot
> 'FOO' doesn't exist on the filesystem
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.deleteSnapshot(SnapshotManager.java:312)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.deleteSnapshot(MasterRpcServices.java:694)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> {code}
> Moreover, since the test is all about dealing with multi regions, which are
> generated by splitting, as per test logs, apparently table is not getting
> split correctly:
> {code:java}
> org.apache.hadoop.hbase.regionserver.StoreUtils(123): cannot split
> hdfs://localhost:37149/user/jenkins/test-data/b2a954c8-abb9-5528-bca3-27fcaaab735c/data/default/N000006/903675aa1a12e58bb247916dd7d492dd/0/b4bdabddce254ea5a9fa3ca432e2bca3
> because midkey is the same as first or last row
> {code}
> e.g
> https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-mulitbranch/job/master/157/testReport/org.apache.phoenix.end2end/TableSnapshotReadsMapReduceIT/MatrixBuild___Matrix___HBASE_PROFILE____2_1____BuildAndTest___testMapReduceSnapshotsMultiRegion/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
