[
https://issues.apache.org/jira/browse/HBASE-21621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nihal Jain updated HBASE-21621:
-------------------------------
Description:
*Steps to reproduce*
# Create a table and put some data into it (data should be big enough, say N
rows)
# Flush the table
# Scan the table with reversed set to true
*Expected Result*
N rows should be retrieved in reversed order
*Actual Result*
Less than expected number of rows is retrieved with following error in logs
{noformat}
2018-12-19 21:55:32,944 DEBUG
[RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39007]
regionserver.StoreScanner(1000): Switch to stream read (scanned=262214 bytes)
of cf
2018-12-19 21:55:32,955 ERROR
[RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39007] ipc.RpcServer(471):
Unexpected throwable object
java.lang.AssertionError: Key
\x00\x00\x00\x00\x00\x00\x00\x09/cf:a/1545236714675/Put/vlen=131072/seqid=4
followed by a error order key
\x00\x00\x00\x00\x00\x00\x00\x0F/cf:a/1545236715545/Put/vlen=131072/seqid=8 in
cf cf in reversed scan
at
org.apache.hadoop.hbase.regionserver.ReversedStoreScanner.checkScanOrder(ReversedStoreScanner.java:105)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:568)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:153)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:6598)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6762)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:6535)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3252)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3501)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
2018-12-19 21:55:32,955 DEBUG
[RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39007] ipc.CallRunner(142):
callId: 508 service: ClientService methodName: Scan size: 47 connection:
127.0.0.1:48328 deadline: 1545236792955, exception=java.io.IOException: Key
\x00\x00\x00\x00\x00\x00\x00\x09/cf:a/1545236714675/Put/vlen=131072/seqid=4
followed by a error order key
\x00\x00\x00\x00\x00\x00\x00\x0F/cf:a/1545236715545/Put/vlen=131072/seqid=8 in
cf cf in reversed scan
2018-12-19 21:55:33,060 DEBUG
[RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39007] ipc.CallRunner(142):
callId: 511 service: ClientService methodName: Scan size: 47 connection:
127.0.0.1:48328 deadline: 1545236792955,
exception=org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException:
Expected nextCallSeq: 1 But the nextCallSeq got from client: 0;
request=scanner_id: 2421102592655360183 number_of_rows: 2147483647
close_scanner: false next_call_seq: 0 client_handles_partials: true
client_handles_heartbeats: true track_scan_metrics: false renew: false
2018-12-19 21:55:33,060 DEBUG [Time-limited test]
client.ScannerCallableWithReplicas(200): Scan with primary region returns
org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException:
org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected
nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id:
2421102592655360183 number_of_rows: 2147483647 close_scanner: false
next_call_seq: 0 client_handles_partials: true client_handles_heartbeats: true
track_scan_metrics: false renew: false
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.checkScanNextCallSeq(RSRpcServices.java:3122)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3455)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
{noformat}
*Analysis/Issue*
>From initial analysis it seems problem occurs when we switch read type
> Reversed scan does not return expected number of rows
> ------------------------------------------------------
>
> Key: HBASE-21621
> URL: https://issues.apache.org/jira/browse/HBASE-21621
> Project: HBase
> Issue Type: Bug
> Components: scan
> Affects Versions: 3.0.0, 2.1.1
> Reporter: Nihal Jain
> Priority: Critical
> Attachments: HBASE-21621.master.UT.patch
>
>
> *Steps to reproduce*
> # Create a table and put some data into it (data should be big enough, say N
> rows)
> # Flush the table
> # Scan the table with reversed set to true
> *Expected Result*
> N rows should be retrieved in reversed order
> *Actual Result*
> Less than expected number of rows is retrieved with following error in logs
> {noformat}
> 2018-12-19 21:55:32,944 DEBUG
> [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39007]
> regionserver.StoreScanner(1000): Switch to stream read (scanned=262214 bytes)
> of cf
> 2018-12-19 21:55:32,955 ERROR
> [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39007]
> ipc.RpcServer(471): Unexpected throwable object
> java.lang.AssertionError: Key
> \x00\x00\x00\x00\x00\x00\x00\x09/cf:a/1545236714675/Put/vlen=131072/seqid=4
> followed by a error order key
> \x00\x00\x00\x00\x00\x00\x00\x0F/cf:a/1545236715545/Put/vlen=131072/seqid=8
> in cf cf in reversed scan
> at
> org.apache.hadoop.hbase.regionserver.ReversedStoreScanner.checkScanOrder(ReversedStoreScanner.java:105)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:568)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:153)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:6598)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6762)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:6535)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3252)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3501)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> 2018-12-19 21:55:32,955 DEBUG
> [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39007]
> ipc.CallRunner(142): callId: 508 service: ClientService methodName: Scan
> size: 47 connection: 127.0.0.1:48328 deadline: 1545236792955,
> exception=java.io.IOException: Key
> \x00\x00\x00\x00\x00\x00\x00\x09/cf:a/1545236714675/Put/vlen=131072/seqid=4
> followed by a error order key
> \x00\x00\x00\x00\x00\x00\x00\x0F/cf:a/1545236715545/Put/vlen=131072/seqid=8
> in cf cf in reversed scan
> 2018-12-19 21:55:33,060 DEBUG
> [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39007]
> ipc.CallRunner(142): callId: 511 service: ClientService methodName: Scan
> size: 47 connection: 127.0.0.1:48328 deadline: 1545236792955,
> exception=org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException:
> Expected nextCallSeq: 1 But the nextCallSeq got from client: 0;
> request=scanner_id: 2421102592655360183 number_of_rows: 2147483647
> close_scanner: false next_call_seq: 0 client_handles_partials: true
> client_handles_heartbeats: true track_scan_metrics: false renew: false
> 2018-12-19 21:55:33,060 DEBUG [Time-limited test]
> client.ScannerCallableWithReplicas(200): Scan with primary region returns
> org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException:
> org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected
> nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id:
> 2421102592655360183 number_of_rows: 2147483647 close_scanner: false
> next_call_seq: 0 client_handles_partials: true client_handles_heartbeats:
> true track_scan_metrics: false renew: false
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.checkScanNextCallSeq(RSRpcServices.java:3122)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3455)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> {noformat}
> *Analysis/Issue*
> From initial analysis it seems problem occurs when we switch read type
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
