[
https://issues.apache.org/jira/browse/PHOENIX-4715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16786192#comment-16786192
]
William Shen commented on PHOENIX-4715:
---------------------------------------
[~apurtell], the "Timed out waiting for lock for row" message might be red
herring, I just spent some time looking at PartialIndexRebuilderIT because
testConcurrentUpsertsWithRebuild was failing for me with
"PartialIndexRebuilderIT.testConcurrentUpsertsWithRebuild:220 Ran out of time"
and i saw the same "Timed out waiting for lock for row: 80 00 00 01 80 00 00
00" messages in the log. However, when I ran the single test case by itself, it
passed and yet "Timed out waiting for lock for row: 80 00 00 01 80 00 00 00"
messages still appeared in the log.
I was running the test on 4.13-HBase-1.2
> PartialIndexRebuilderIT tests fail after switching master to HBase 1.4
> ----------------------------------------------------------------------
>
> Key: PHOENIX-4715
> URL: https://issues.apache.org/jira/browse/PHOENIX-4715
> Project: Phoenix
> Issue Type: Bug
> Reporter: Thomas D'Silva
> Priority: Major
>
> I think the 3 test failures in PartialIndexRebuilderIT started happening
> after we switched master to HBase 1.4 as part of PHOENIX-4076.
> Maybe [~lhofhansl] or [~apurtell] might have some insight
> {code:java}
> [ERROR] Failures:
> [ERROR] PartialIndexRebuilderIT.testConcurrentUpsertsWithRebuild:230 Expected
> equality for V1, but null!=11
> [ERROR] PartialIndexRebuilderIT.testDeleteAndUpsertAfterFailure:347 Expected
> equality for V2, but null!=1
> [ERROR] PartialIndexRebuilderIT.testWriteWhileRebuilding:396 Expected
> equality for V2, but null!=2
> {code}
> testDeleteAndUpsertAfterFailure and testWriteWhileRebuilding pass for me
> locally just before PHOENIX-4076 was committed.
> testConcurrentUpsertsWithRebuild fails with the following exception at the
> commit before PHOENIX-4076 .
> {code:java}
> 2018-04-27 16:14:48,049 ERROR
> [RpcServer.FifoWFPBQ.default.handler=1,queue=0,port=26069]
> org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver(1089):
> IOException during rebuilding:
> org.apache.hadoop.hbase.exceptions.TimeoutIOException: Timed out waiting for
> lock for row: 80 00 00 01 80 00 00 00
> at
> org.apache.phoenix.hbase.index.LockManager.lockRow(LockManager.java:96)
> at
> org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:421)
> at
> org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:370)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$35.call(RegionCoprocessorHost.java:1007)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1749)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1003)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3190)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2976)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2918)
> at
> org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver.rebuildIndices(UngroupedAggregateRegionObserver.java:1074)
> at
> org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver.doPostScannerOpen(UngroupedAggregateRegionObserver.java:369)
> at
> org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.overrideDelegate(BaseScannerRegionObserver.java:245)
> at
> org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:293)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2629)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2833)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34950)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
> 2018-04-27 16:14:48,051 DEBUG
> [RpcServer.FifoWFPBQ.default.handler=1,queue=0,port=26069]
> org.apache.hadoop.hbase.ipc.CallRunner(126):
> RpcServer.FifoWFPBQ.default.handler=1,queue=0,port=26069: callId: 1941
> service: ClientService methodName: Scan size: 40 connection: 127.0.0.1:14017
> org.apache.hadoop.hbase.UnknownScannerException: Throwing
> UnknownScannerException to reset the client scanner state for clients older
> than 1.3.
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2893)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34950)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
> Caused by: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Timed out
> waiting for lock for row: 80 00 00 01 80 00 00 00
> at
> org.apache.phoenix.hbase.index.LockManager.lockRow(LockManager.java:96)
> at
> org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:421)
> at
> org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:370)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$35.call(RegionCoprocessorHost.java:1007)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1749)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1003)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3190)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2976)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2918)
> at
> org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver.rebuildIndices(UngroupedAggregateRegionObserver.java:1074)
> at
> org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver.doPostScannerOpen(UngroupedAggregateRegionObserver.java:369)
> at
> org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.overrideDelegate(BaseScannerRegionObserver.java:245)
> at
> org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:293)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2629)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2833)
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
