[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820864#comment-13820864
 ] 

Hadoop QA commented on HBASE-9818:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12613446/9818-combined-2.txt
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 
1.0 profile.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 1 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

{color:red}-1 site{color}.  The patch appears to cause mvn site goal to 
fail.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7827//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7827//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7827//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7827//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7827//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7827//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7827//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7827//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7827//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7827//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7827//console

This message is automatically generated.

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Ted Yu
 Attachments: 9818-combined-2.txt, 9818-combined.txt, 9818-trial.txt, 
 9818-trial2.txt, 9818-v2.txt, 9818-v3.txt, 9818-v4.txt, 9818-v5.txt, 
 trunk-9818.patch, trunk-9818_v1.1.patch


 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13819442#comment-13819442
 ] 

stack commented on HBASE-9818:
--

[~jxiang] Did you open another issue to fix this issue?  If so, what was it?

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Ted Yu
 Attachments: 9818-combined.txt, 9818-trial.txt, 9818-trial2.txt, 
 9818-v2.txt, 9818-v3.txt, 9818-v4.txt, 9818-v5.txt, trunk-9818.patch, 
 trunk-9818_v1.1.patch


 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:361)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:336)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:293)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:258)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:603)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:476)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3546)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3616)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3494)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3485)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3079)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 2013-10-22 10:58:43,665 ERROR [RpcServer.handler=23,port=36020] 
 regionserver.HRegionServer:
 org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected 
 nextCallSeq: 53438 But the nextCallSeq got from client: 53437; 
 request=scanner_id: 1252577470624375060 number_of_rows: 100 close_scanner: 
 false next_call_seq: 53437
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3030)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at 

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Jimmy Xiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13819465#comment-13819465
 ] 

Jimmy Xiang commented on HBASE-9818:


Yes, I fixed the main issue in HBASE-9926. The stream is null because the file 
is closed. There are also some minor issue/enhancement with HFileBlock (more 
related to this call, not scanner) that can be fixed with this jira.

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Ted Yu
 Attachments: 9818-combined.txt, 9818-trial.txt, 9818-trial2.txt, 
 9818-v2.txt, 9818-v3.txt, 9818-v4.txt, 9818-v5.txt, trunk-9818.patch, 
 trunk-9818_v1.1.patch


 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:361)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:336)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:293)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:258)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:603)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:476)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3546)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3616)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3494)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3485)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3079)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 2013-10-22 10:58:43,665 ERROR [RpcServer.handler=23,port=36020] 
 regionserver.HRegionServer:
 org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected 
 nextCallSeq: 53438 But the nextCallSeq got from client: 53437; 
 request=scanner_id: 1252577470624375060 number_of_rows: 100 close_scanner: 
 false next_call_seq: 53437
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3030)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13819477#comment-13819477
 ] 

stack commented on HBASE-9818:
--

Maybe we should close this issue then and do the other fixup in a new one 
dedicated to HFileBlock cleanup?

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Ted Yu
 Attachments: 9818-combined.txt, 9818-trial.txt, 9818-trial2.txt, 
 9818-v2.txt, 9818-v3.txt, 9818-v4.txt, 9818-v5.txt, trunk-9818.patch, 
 trunk-9818_v1.1.patch


 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:361)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:336)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:293)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:258)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:603)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:476)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3546)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3616)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3494)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3485)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3079)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 2013-10-22 10:58:43,665 ERROR [RpcServer.handler=23,port=36020] 
 regionserver.HRegionServer:
 org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected 
 nextCallSeq: 53438 But the nextCallSeq got from client: 53437; 
 request=scanner_id: 1252577470624375060 number_of_rows: 100 close_scanner: 
 false next_call_seq: 53437
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3030)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Jimmy Xiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13819483#comment-13819483
 ] 

Jimmy Xiang commented on HBASE-9818:


That's fine with me.

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Ted Yu
 Attachments: 9818-combined.txt, 9818-trial.txt, 9818-trial2.txt, 
 9818-v2.txt, 9818-v3.txt, 9818-v4.txt, 9818-v5.txt, trunk-9818.patch, 
 trunk-9818_v1.1.patch


 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:361)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:336)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:293)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:258)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:603)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:476)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3546)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3616)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3494)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3485)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3079)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 2013-10-22 10:58:43,665 ERROR [RpcServer.handler=23,port=36020] 
 regionserver.HRegionServer:
 org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected 
 nextCallSeq: 53438 But the nextCallSeq got from client: 53437; 
 request=scanner_id: 1252577470624375060 number_of_rows: 100 close_scanner: 
 false next_call_seq: 53437
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3030)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Jimmy Xiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13819560#comment-13819560
 ] 

Jimmy Xiang commented on HBASE-9818:


[~yuzhih...@gmail.com], with HBASE-9926, I haven't seen such NPE in my test any 
more. Can we close this issue? Thanks for digging into it.

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Ted Yu
 Attachments: 9818-combined.txt, 9818-trial.txt, 9818-trial2.txt, 
 9818-v2.txt, 9818-v3.txt, 9818-v4.txt, 9818-v5.txt, trunk-9818.patch, 
 trunk-9818_v1.1.patch


 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:361)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:336)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:293)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:258)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:603)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:476)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3546)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3616)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3494)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3485)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3079)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 2013-10-22 10:58:43,665 ERROR [RpcServer.handler=23,port=36020] 
 regionserver.HRegionServer:
 org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected 
 nextCallSeq: 53438 But the nextCallSeq got from client: 53437; 
 request=scanner_id: 1252577470624375060 number_of_rows: 100 close_scanner: 
 false next_call_seq: 53437
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3030)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Jimmy Xiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13818195#comment-13818195
 ] 

Jimmy Xiang commented on HBASE-9818:


[~yuzhih...@gmail.com], what do you think about my patch 1.1?

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Ted Yu
 Attachments: 9818-trial.txt, 9818-v2.txt, 9818-v3.txt, 9818-v4.txt, 
 9818-v5.txt, trunk-9818.patch, trunk-9818_v1.1.patch


 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:361)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:336)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:293)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:258)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:603)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:476)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3546)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3616)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3494)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3485)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3079)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 2013-10-22 10:58:43,665 ERROR [RpcServer.handler=23,port=36020] 
 regionserver.HRegionServer:
 org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected 
 nextCallSeq: 53438 But the nextCallSeq got from client: 53437; 
 request=scanner_id: 1252577470624375060 number_of_rows: 100 close_scanner: 
 false next_call_seq: 53437
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3030)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13818310#comment-13818310
 ] 

Hadoop QA commented on HBASE-9818:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12613014/9818-combined.txt
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 
1.0 profile.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

{color:red}-1 site{color}.  The patch appears to cause mvn site goal to 
fail.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7807//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7807//console

This message is automatically generated.

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Ted Yu
 Attachments: 9818-combined.txt, 9818-trial.txt, 9818-trial2.txt, 
 9818-v2.txt, 9818-v3.txt, 9818-v4.txt, 9818-v5.txt, trunk-9818.patch, 
 trunk-9818_v1.1.patch


 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13818037#comment-13818037
 ] 

Ted Yu commented on HBASE-9818:
---

Another round of 200 iterations with 9818-trial.txt passed.
Will continue digging.

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Ted Yu
 Attachments: 9818-trial.txt, 9818-v2.txt, 9818-v3.txt, 9818-v4.txt, 
 9818-v5.txt, trunk-9818.patch, trunk-9818_v1.1.patch


 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:361)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:336)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:293)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:258)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:603)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:476)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3546)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3616)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3494)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3485)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3079)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 2013-10-22 10:58:43,665 ERROR [RpcServer.handler=23,port=36020] 
 regionserver.HRegionServer:
 org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected 
 nextCallSeq: 53438 But the nextCallSeq got from client: 53437; 
 request=scanner_id: 1252577470624375060 number_of_rows: 100 close_scanner: 
 false next_call_seq: 53437
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3030)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at 

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13816126#comment-13816126
 ] 

Ted Yu commented on HBASE-9818:
---

Unfortunately the change in the trial patch seemed to affect timing.
After 200 iterations, there was still no failure:
{code}
$ grep 'rver.TestHRegion' 9818.out | wc
200 400   11400
{code}

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Ted Yu
 Attachments: 9818-trial.txt, 9818-v2.txt, 9818-v3.txt, 9818-v4.txt, 
 9818-v5.txt


 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:361)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:336)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:293)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:258)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:603)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:476)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3546)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3616)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3494)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3485)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3079)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 2013-10-22 10:58:43,665 ERROR [RpcServer.handler=23,port=36020] 
 regionserver.HRegionServer:
 org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected 
 nextCallSeq: 53438 But the nextCallSeq got from client: 53437; 
 request=scanner_id: 1252577470624375060 number_of_rows: 100 close_scanner: 
 false next_call_seq: 53437
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3030)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13816191#comment-13816191
 ] 

Ted Yu commented on HBASE-9818:
---

TestHFileBlock#testConcurrentReading shows how multiple FSReader's use the same 
FSDataInputStreamWrapper

Another solution I can think of is to introduce ref counting for the streams 
FSDataInputStreamWrapper wraps. The FSDataInputStreamWrapper#close() would 
decrement the count and would close the underlying stream if count reaches 0.

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Ted Yu
 Attachments: 9818-trial.txt, 9818-v2.txt, 9818-v3.txt, 9818-v4.txt, 
 9818-v5.txt


 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:361)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:336)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:293)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:258)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:603)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:476)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3546)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3616)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3494)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3485)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3079)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 2013-10-22 10:58:43,665 ERROR [RpcServer.handler=23,port=36020] 
 regionserver.HRegionServer:
 org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected 
 nextCallSeq: 53438 But the nextCallSeq got from client: 53437; 
 request=scanner_id: 1252577470624375060 number_of_rows: 100 close_scanner: 
 false next_call_seq: 53437
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3030)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Jimmy Xiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13816983#comment-13816983
 ] 

Jimmy Xiang commented on HBASE-9818:


Assigned back to Ted since he is still looking into it.

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Ted Yu
 Attachments: 9818-trial.txt, 9818-v2.txt, 9818-v3.txt, 9818-v4.txt, 
 9818-v5.txt, trunk-9818.patch, trunk-9818_v1.1.patch


 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:361)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:336)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:293)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:258)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:603)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:476)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3546)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3616)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3494)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3485)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3079)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 2013-10-22 10:58:43,665 ERROR [RpcServer.handler=23,port=36020] 
 regionserver.HRegionServer:
 org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected 
 nextCallSeq: 53438 But the nextCallSeq got from client: 53437; 
 request=scanner_id: 1252577470624375060 number_of_rows: 100 close_scanner: 
 false next_call_seq: 53437
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3030)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 {noformat}

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13814940#comment-13814940
 ] 

Ted Yu commented on HBASE-9818:
---

200 iterations of TestHRegion and TestAtomicOperation passed.


 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Ted Yu
 Attachments: 9818-v2.txt, 9818-v3.txt, 9818-v4.txt


 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:361)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:336)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:293)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:258)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:603)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:476)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3546)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3616)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3494)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3485)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3079)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 2013-10-22 10:58:43,665 ERROR [RpcServer.handler=23,port=36020] 
 regionserver.HRegionServer:
 org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected 
 nextCallSeq: 53438 But the nextCallSeq got from client: 53437; 
 request=scanner_id: 1252577470624375060 number_of_rows: 100 close_scanner: 
 false next_call_seq: 53437
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3030)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13815176#comment-13815176
 ] 

Hadoop QA commented on HBASE-9818:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12612405/9818-v5.txt
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 
1.0 profile.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 1 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:red}-1 findbugs{color}.  The patch appears to introduce 4 new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

{color:red}-1 site{color}.  The patch appears to cause mvn site goal to 
fail.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
 

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7756//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7756//console

This message is automatically generated.

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Ted Yu
 Attachments: 9818-v2.txt, 9818-v3.txt, 9818-v4.txt, 9818-v5.txt


 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13815184#comment-13815184
 ] 

Sergey Shelukhin commented on HBASE-9818:
-

{code}
+return useHBaseChecksum ?
+new PairFSDataInputStream, Boolean(this.streamNoFsChecksum, 
useHBaseChecksum) : 
+  new PairFSDataInputStream, Boolean(this.stream, useHBaseChecksum);
{code}
the boolean should be saved to local variable before checking, the value could 
change and result in no checksum (or both) being used.
Otherwise +1 from me... hopefully someone else can also take a look.
I am not 100% sure if this is complete fix or just narrows the window quite a 
bit

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Ted Yu
 Attachments: 9818-v2.txt, 9818-v3.txt, 9818-v4.txt, 9818-v5.txt


 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:361)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:336)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:293)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:258)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:603)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:476)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3546)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3616)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3494)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3485)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3079)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 2013-10-22 10:58:43,665 ERROR [RpcServer.handler=23,port=36020] 
 regionserver.HRegionServer:
 org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected 
 nextCallSeq: 53438 But the nextCallSeq got from client: 53437; 
 request=scanner_id: 1252577470624375060 number_of_rows: 100 close_scanner: 
 false next_call_seq: 53437
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3030)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at 

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Jimmy Xiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13815204#comment-13815204
 ] 

Jimmy Xiang commented on HBASE-9818:


I am ok with the change.  But I don't think it fixes the issue.  If the stream 
is really closed, instead of NPE, we will get some IOException instead.

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Ted Yu
 Attachments: 9818-v2.txt, 9818-v3.txt, 9818-v4.txt, 9818-v5.txt


 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:361)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:336)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:293)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:258)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:603)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:476)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3546)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3616)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3494)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3485)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3079)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 2013-10-22 10:58:43,665 ERROR [RpcServer.handler=23,port=36020] 
 regionserver.HRegionServer:
 org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected 
 nextCallSeq: 53438 But the nextCallSeq got from client: 53437; 
 request=scanner_id: 1252577470624375060 number_of_rows: 100 close_scanner: 
 false next_call_seq: 53437
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3030)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at 

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13815407#comment-13815407
 ] 

Hadoop QA commented on HBASE-9818:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12612439/9818-v5.txt
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 
1.0 profile.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 1 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:red}-1 findbugs{color}.  The patch appears to introduce 3 new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

{color:red}-1 site{color}.  The patch appears to cause mvn site goal to 
fail.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
 

 {color:red}-1 core zombie tests{color}.  There are 1 zombie test(s):   
at 
org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:488)

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7760//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7760//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7760//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7760//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7760//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7760//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7760//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7760//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7760//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7760//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7760//console

This message is automatically generated.

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Ted Yu
 Attachments: 9818-v2.txt, 9818-v3.txt, 9818-v4.txt, 9818-v5.txt


 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13815520#comment-13815520
 ] 

Hadoop QA commented on HBASE-9818:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12612473/9818-v5.txt
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 
1.0 profile.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 1 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:red}-1 findbugs{color}.  The patch appears to introduce 4 new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

{color:red}-1 site{color}.  The patch appears to cause mvn site goal to 
fail.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
 

 {color:red}-1 core zombie tests{color}.  There are 1 zombie test(s):   
at 
org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:488)

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7764//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7764//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7764//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7764//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7764//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7764//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7764//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7764//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7764//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7764//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7764//console

This message is automatically generated.

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Ted Yu
 Attachments: 9818-v2.txt, 9818-v3.txt, 9818-v4.txt, 9818-v5.txt


 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13813737#comment-13813737
 ] 

Hadoop QA commented on HBASE-9818:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12612102/9818-v1.txt
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 
1.0 profile.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:red}-1 findbugs{color}.  The patch appears to introduce 2 new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

{color:red}-1 site{color}.  The patch appears to cause mvn site goal to 
fail.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7734//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7734//console

This message is automatically generated.

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
 Attachments: 9818-v1.txt


 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13814071#comment-13814071
 ] 

Ted Yu commented on HBASE-9818:
---

I am looping TestHRegion and TestAtomicOperation 200 times, respectively.
Previously TestAtomicOperation failed at iteration #7. Now the tests reach 
iteration #17 and are running.

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
 Attachments: 9818-v2.txt


 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:361)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:336)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:293)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:258)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:603)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:476)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3546)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3616)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3494)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3485)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3079)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 2013-10-22 10:58:43,665 ERROR [RpcServer.handler=23,port=36020] 
 regionserver.HRegionServer:
 org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected 
 nextCallSeq: 53438 But the nextCallSeq got from client: 53437; 
 request=scanner_id: 1252577470624375060 number_of_rows: 100 close_scanner: 
 false next_call_seq: 53437
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3030)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 {noformat}



--
This 

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13814111#comment-13814111
 ] 

Sergey Shelukhin commented on HBASE-9818:
-

From logs and code it seems like compactions are closing the stream via 
wrapper (via long chain of classes)
Beforehand, they are supposed to notify all scanners to rebuild heap, which is 
a synchronized method (on StoreScanner) and only then close (next/etc are also 
synchronized so it should all be properly sequenced). But somehow it's not 
happening I suspect.
Also in some cases stacks are during some initialization, not next(), so I 
haven't looked/not sure how that was supposed to be synched.
 I didn't have a lot of time, so just looked at the code. Looped 
testWritesWhileGetting 100 times and it never failed locally, which is sad.
Is there any chance/tool to find out when approx these failures started

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Ted Yu
 Attachments: 9818-v2.txt


 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:361)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:336)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:293)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:258)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:603)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:476)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3546)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3616)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3494)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3485)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3079)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 2013-10-22 10:58:43,665 ERROR [RpcServer.handler=23,port=36020] 
 regionserver.HRegionServer:
 org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected 
 nextCallSeq: 53438 But the nextCallSeq got from client: 53437; 
 request=scanner_id: 1252577470624375060 number_of_rows: 100 close_scanner: 
 false next_call_seq: 53437
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3030)
 at 
 

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13814118#comment-13814118
 ] 

Sergey Shelukhin commented on HBASE-9818:
-

But yeah maybe wrapper method needs to be encapsulated with getting stream 
together. Maybe there's no real sync issue, and it just needs to not throw, and 
then on next read it will rebuild the heap. Although it does seem pretty 
suspect, it has a stream and someone closes it in parallel.

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Ted Yu
 Attachments: 9818-v2.txt


 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:361)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:336)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:293)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:258)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:603)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:476)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3546)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3616)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3494)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3485)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3079)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 2013-10-22 10:58:43,665 ERROR [RpcServer.handler=23,port=36020] 
 regionserver.HRegionServer:
 org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected 
 nextCallSeq: 53438 But the nextCallSeq got from client: 53437; 
 request=scanner_id: 1252577470624375060 number_of_rows: 100 close_scanner: 
 false next_call_seq: 53437
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3030)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13814126#comment-13814126
 ] 

Hadoop QA commented on HBASE-9818:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12612207/9818-v2.txt
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 
1.0 profile.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:red}-1 findbugs{color}.  The patch appears to introduce 2 new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

{color:red}-1 site{color}.  The patch appears to cause mvn site goal to 
fail.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   org.apache.hadoop.hbase.regionserver.wal.TestHLog

 {color:red}-1 core zombie tests{color}.  There are 1 zombie test(s):   
at 
org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:486)

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7739//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7739//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7739//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7739//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7739//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7739//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7739//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7739//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7739//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7739//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7739//console

This message is automatically generated.

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Ted Yu
 Attachments: 9818-v2.txt


 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13814142#comment-13814142
 ] 

Ted Yu commented on HBASE-9818:
---

From 
https://builds.apache.org/job/PreCommit-HBASE-Build/7739/testReport/junit/org.apache.hadoop.hbase.regionserver.wal/TestHLog/testAppendClose/
 :
{code}
Stacktrace

java.lang.OutOfMemoryError: Java heap space
at 
org.apache.hadoop.hdfs.util.LightWeightGSet.init(LightWeightGSet.java:81)
at 
org.apache.hadoop.hdfs.server.namenode.BlocksMap.init(BlocksMap.java:320)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.init(FSNamesystem.java:223)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:299)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:569)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1479)
at org.apache.hadoop.hdfs.MiniDFSCluster.init(MiniDFSCluster.java:278)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSClusterForTestHLog(HBaseTestingUtility.java:563)
at 
org.apache.hadoop.hbase.regionserver.wal.TestHLog.testAppendClose(TestHLog.java:434)
{code}
Looks like an environment issue.
'stream closed' exception, added in patch, didn't show up.

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Ted Yu
 Attachments: 9818-v2.txt


 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:361)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:336)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:293)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:258)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:603)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:476)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3546)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3616)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3494)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3485)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3079)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 2013-10-22 10:58:43,665 ERROR [RpcServer.handler=23,port=36020] 
 regionserver.HRegionServer:
 

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13814416#comment-13814416
 ] 

Sergey Shelukhin commented on HBASE-9818:
-

is retrying intentional? probably we should find root cause and not just retry.

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Ted Yu
 Attachments: 9818-v2.txt, 9818-v3.txt


 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:361)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:336)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:293)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:258)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:603)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:476)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3546)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3616)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3494)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3485)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3079)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 2013-10-22 10:58:43,665 ERROR [RpcServer.handler=23,port=36020] 
 regionserver.HRegionServer:
 org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected 
 nextCallSeq: 53438 But the nextCallSeq got from client: 53437; 
 request=scanner_id: 1252577470624375060 number_of_rows: 100 close_scanner: 
 false next_call_seq: 53437
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3030)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 {noformat}



--
This message was sent by Atlassian JIRA

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13814417#comment-13814417
 ] 

Sergey Shelukhin commented on HBASE-9818:
-

logging lgtm. What does it say when it fails

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Ted Yu
 Attachments: 9818-v2.txt, 9818-v3.txt


 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:361)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:336)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:293)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:258)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:603)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:476)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3546)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3616)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3494)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3485)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3079)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 2013-10-22 10:58:43,665 ERROR [RpcServer.handler=23,port=36020] 
 regionserver.HRegionServer:
 org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected 
 nextCallSeq: 53438 But the nextCallSeq got from client: 53437; 
 request=scanner_id: 1252577470624375060 number_of_rows: 100 close_scanner: 
 false next_call_seq: 53437
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3030)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13814428#comment-13814428
 ] 

Sergey Shelukhin commented on HBASE-9818:
-

actually, I looked at it, encapsulating is not such a good idea (returning 
boolean and stream together), it will not remove the race with close unless a 
common lock is added. So root cause might be elsewhere...

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Ted Yu
 Attachments: 9818-v2.txt, 9818-v3.txt


 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:361)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:336)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:293)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:258)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:603)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:476)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3546)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3616)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3494)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3485)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3079)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 2013-10-22 10:58:43,665 ERROR [RpcServer.handler=23,port=36020] 
 regionserver.HRegionServer:
 org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected 
 nextCallSeq: 53438 But the nextCallSeq got from client: 53437; 
 request=scanner_id: 1252577470624375060 number_of_rows: 100 close_scanner: 
 false next_call_seq: 53437
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3030)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Jimmy Xiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13814586#comment-13814586
 ] 

Jimmy Xiang commented on HBASE-9818:


I was wondering how v4 solves the issue. With the patch, if the stream is 
closed somewhere, instead of a NPE, we may get an IOException saying the stream 
is closed. If the stream is not closed somewhere, why is it null? Never 
initialized?

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Ted Yu
 Attachments: 9818-v2.txt, 9818-v3.txt, 9818-v4.txt


 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:361)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:336)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:293)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:258)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:603)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:476)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3546)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3616)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3494)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3485)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3079)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 2013-10-22 10:58:43,665 ERROR [RpcServer.handler=23,port=36020] 
 regionserver.HRegionServer:
 org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected 
 nextCallSeq: 53438 But the nextCallSeq got from client: 53437; 
 request=scanner_id: 1252577470624375060 number_of_rows: 100 close_scanner: 
 false next_call_seq: 53437
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3030)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13814611#comment-13814611
 ] 

Hadoop QA commented on HBASE-9818:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12612315/9818-v4.txt
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 
1.0 profile.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 1 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:red}-1 findbugs{color}.  The patch appears to introduce 4 new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

{color:red}-1 site{color}.  The patch appears to cause mvn site goal to 
fail.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7743//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7743//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7743//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7743//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7743//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7743//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7743//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7743//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7743//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7743//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7743//console

This message is automatically generated.

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Ted Yu
 Attachments: 9818-v2.txt, 9818-v3.txt, 9818-v4.txt


 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13812981#comment-13812981
 ] 

ramkrishna.s.vasudevan commented on HBASE-9818:
---

I got this in 
https://builds.apache.org/job/PreCommit-HBASE-Build/7722//testReport/org.apache.hadoop.hbase.regionserver/TestHRegion/testWritesWhileGetting/.
But not able to reproduce next time.

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang

 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:361)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:336)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:293)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:258)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:603)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:476)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3546)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3616)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3494)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3485)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3079)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 2013-10-22 10:58:43,665 ERROR [RpcServer.handler=23,port=36020] 
 regionserver.HRegionServer:
 org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected 
 nextCallSeq: 53438 But the nextCallSeq got from client: 53437; 
 request=scanner_id: 1252577470624375060 number_of_rows: 100 close_scanner: 
 false next_call_seq: 53437
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3030)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 {noformat}

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13813512#comment-13813512
 ] 

Ted Yu commented on HBASE-9818:
---

From 
https://builds.apache.org/job/PreCommit-HBASE-Build/7730//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/
 :
{code}
2013-11-05 00:29:54,062 DEBUG [Thread-106] regionserver.HStore(1430): Removing 
store files after compaction...
Exception in thread Thread-108 java.lang.NullPointerException at 
org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
at 
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1487)
at 
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
at 
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:360)
at 
org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:776)
at 
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:234)
at 
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:149)
at 
org.apache.hadoop.hbase.regionserver.StoreScanner.init(StoreScanner.java:172)
at 
org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1688)
at 
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.init(HRegion.java:3543)
at 
org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1749)
at 
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1741)
at 
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1718)
at 
org.apache.hadoop.hbase.regionserver.TestAtomicOperation$3.run(TestAtomicOperation.java:449)
{code}

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang

 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:361)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:336)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:293)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:258)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:603)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:476)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3546)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3616)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3494)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3485)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3079)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13813518#comment-13813518
 ] 

Sergey Shelukhin commented on HBASE-9818:
-

Seems to coincide with compaction file replacement... probably the close is 
indeed called and the stream is closed in the wrapper.
I wonder why it didn't happen before

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang

 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:361)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:336)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:293)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:258)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:603)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:476)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3546)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3616)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3494)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3485)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3079)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 2013-10-22 10:58:43,665 ERROR [RpcServer.handler=23,port=36020] 
 regionserver.HRegionServer:
 org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected 
 nextCallSeq: 53438 But the nextCallSeq got from client: 53437; 
 request=scanner_id: 1252577470624375060 number_of_rows: 100 close_scanner: 
 false next_call_seq: 53437
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3030)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 {noformat}



--
This message was sent by Atlassian 

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13813640#comment-13813640
 ] 

Ted Yu commented on HBASE-9818:
---

Encountered NPE in the else branch (with patch v1):
{code}
2013-11-05 04:23:17,524 DEBUG [Thread-776] 
compactions.ExploringCompactionPolicy(122): Exploring compaction algorithm has 
selected 3 files of size 3318 starting at candidate #24 after considering 1884 
permutations with 1884 in ratio
2013-11-05 04:23:17,524 DEBUG [Thread-776] regionserver.HStore(1359): 
fa933591407f5591c2410908ea5065f9 - colfamily11: Initiating minor compaction
2013-11-05 04:23:17,524 INFO  [Thread-776] regionserver.HRegion(1288): Starting 
compaction on colfamily11 in region 
testRowMutationMultiThreads,,1383625393150.fa933591407f5591c2410908ea5065f9.
2013-11-05 04:23:17,524 INFO  [Thread-776] regionserver.HStore(1001): Starting 
compaction of 3 file(s) in colfamily11 of 
testRowMutationMultiThreads,,1383625393150.fa933591407f5591c2410908ea5065f9. 
into 
tmpdir=/grid/0/dev/ty/trunk/hbase-server/target/test-data/1a023b65-addc-4c5d-acaf-e346220aff31/data/default/testRowMutationMultiThreads/fa933591407f5591c2410908ea5065f9/.tmp,
 totalSize=3.2 K
2013-11-05 04:23:17,525 DEBUG [Thread-776] compactions.Compactor(147): 
Compacting 
file:/grid/0/dev/ty/trunk/hbase-server/target/test-data/1a023b65-addc-4c5d-acaf-e346220aff31/data/default/testRowMutationMultiThreads/fa933591407f5591c2410908ea5065f9/colfamily11/94f3260116304004853721be928701f9,
 keycount=1, bloomtype=ROW, size=1.1 K, encoding=NONE, seqNum=1331
2013-11-05 04:23:17,525 DEBUG [Thread-776] compactions.Compactor(147): 
Compacting 
file:/grid/0/dev/ty/trunk/hbase-server/target/test-data/1a023b65-addc-4c5d-acaf-e346220aff31/data/default/testRowMutationMultiThreads/fa933591407f5591c2410908ea5065f9/colfamily11/795449ca77094e769ac390749c89f92b,
 keycount=1, bloomtype=ROW, size=1.1 K, encoding=NONE, seqNum=1342
2013-11-05 04:23:17,525 DEBUG [Thread-776] compactions.Compactor(147): 
Compacting 
file:/grid/0/dev/ty/trunk/hbase-server/target/test-data/1a023b65-addc-4c5d-acaf-e346220aff31/data/default/testRowMutationMultiThreads/fa933591407f5591c2410908ea5065f9/colfamily11/9e2cc96151754cedbc0d461653cbea63,
 keycount=1, bloomtype=ROW, size=1.1 K, encoding=NONE, seqNum=1353
2013-11-05 04:23:17,531 DEBUG [Thread-776] regionserver.HRegionFileSystem(338): 
Committing store file 
/grid/0/dev/ty/trunk/hbase-server/target/test-data/1a023b65-addc-4c5d-acaf-e346220aff31/data/default/testRowMutationMultiThreads/fa933591407f5591c2410908ea5065f9/.tmp/3ac822791e534965b35b0b3d5022881a
 as 
/grid/0/dev/ty/trunk/hbase-server/target/test-data/1a023b65-addc-4c5d-acaf-e346220aff31/data/default/testRowMutationMultiThreads/fa933591407f5591c2410908ea5065f9/colfamily11/3ac822791e534965b35b0b3d5022881a
2013-11-05 04:23:17,532 DEBUG [Thread-776] regionserver.HStore(1430): Removing 
store files after compaction...
Exception in thread Thread-777 java.lang.NullPointerException
at 
org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1230)
at 
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1493)
at 
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1324)
at 
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:360)
2013-11-05 04:23:17,533 DEBUG [Thread-776] backup.HFileArchiver(438): Finished 
archiving from class 
org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, 
file:file:/grid/0/dev/ty/trunk/hbase-server/target/test-data/1a023b65-addc-4c5d-acaf-e346220aff31/data/default/testRowMutationMultiThreads/fa933591407f5591c2410908ea5065f9/colfamily11/94f3260116304004853721be928701f9,
 to 
file:/grid/0/dev/ty/trunk/hbase-server/target/test-data/1a023b65-addc-4c5d-acaf-e346220aff31/archive/data/default/testRowMutationMultiThreads/fa933591407f5591c2410908ea5065f9/colfamily11/94f3260116304004853721be928701f9
at 
org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:776)
at 
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:234)
at 
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:149)
at 
org.apache.hadoop.hbase.regionserver.StoreScanner.init(StoreScanner.java:172)
at 
org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1688)
at 
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.init(HRegion.java:3427)
at 
org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1746)
at 
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1738)
2013-11-05 04:23:17,534 DEBUG [Thread-776] backup.HFileArchiver(438): Finished 
archiving from class 

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13806929#comment-13806929
 ] 

ramkrishna.s.vasudevan commented on HBASE-9818:
---

Which version does this happen Jimmy?

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Sergey Shelukhin

 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:361)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:336)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:293)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:258)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:603)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:476)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3546)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3616)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3494)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3485)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3079)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 2013-10-22 10:58:43,665 ERROR [RpcServer.handler=23,port=36020] 
 regionserver.HRegionServer:
 org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected 
 nextCallSeq: 53438 But the nextCallSeq got from client: 53437; 
 request=scanner_id: 1252577470624375060 number_of_rows: 100 close_scanner: 
 false next_call_seq: 53437
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3030)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Jimmy Xiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13806935#comment-13806935
 ] 

Jimmy Xiang commented on HBASE-9818:


Trunk.

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Sergey Shelukhin

 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:361)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:336)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:293)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:258)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:603)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:476)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3546)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3616)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3494)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3485)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3079)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 2013-10-22 10:58:43,665 ERROR [RpcServer.handler=23,port=36020] 
 regionserver.HRegionServer:
 org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected 
 nextCallSeq: 53438 But the nextCallSeq got from client: 53437; 
 request=scanner_id: 1252577470624375060 number_of_rows: 100 close_scanner: 
 false next_call_seq: 53437
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3030)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13806974#comment-13806974
 ] 

Sergey Shelukhin commented on HBASE-9818:
-

Hmm... I looked at the code a little bit and so far I am in this shouldn't 
happen stage.
Are you sure close is not called in parallel? Could be from region close due 
to schema change. Can you provide surrounding logs.

prepare on the wrapper is called in ctor so I am assuming it couldn't be 
prepare-read race.
Otherwise (outside prepare) the boolean is never set to a value until the 
corresponding stream is not null, all things are volatile, and streams are 
never null-ed until close.


 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Sergey Shelukhin

 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:361)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:336)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:293)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:258)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:603)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:476)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3546)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3616)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3494)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3485)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3079)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 2013-10-22 10:58:43,665 ERROR [RpcServer.handler=23,port=36020] 
 regionserver.HRegionServer:
 org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected 
 nextCallSeq: 53438 But the nextCallSeq got from client: 53437; 
 request=scanner_id: 1252577470624375060 number_of_rows: 100 close_scanner: 
 false next_call_seq: 53437
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3030)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13806992#comment-13806992
 ] 

Sergey Shelukhin commented on HBASE-9818:
-

unassign for now... I will still look if theres more info :)

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang

 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:361)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:336)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:293)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:258)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:603)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:476)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3546)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3616)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3494)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3485)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3079)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 2013-10-22 10:58:43,665 ERROR [RpcServer.handler=23,port=36020] 
 regionserver.HRegionServer:
 org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected 
 nextCallSeq: 53438 But the nextCallSeq got from client: 53437; 
 request=scanner_id: 1252577470624375060 number_of_rows: 100 close_scanner: 
 false next_call_seq: 53437
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3030)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9818) NPE in HFileBlock#AbstractFSReader#readAtOffset

[Jimmy Xiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13802157#comment-13802157
 ] 

Jimmy Xiang commented on HBASE-9818:


These happened at least three times in one run.  It happened before too.

 NPE in HFileBlock#AbstractFSReader#readAtOffset
 ---

 Key: HBASE-9818
 URL: https://issues.apache.org/jira/browse/HBASE-9818
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang

 HFileBlock#istream seems to be null.  I was wondering should we hide 
 FSDataInputStreamWrapper#useHBaseChecksum.
 By the way, this happened when online schema change is enabled (encoding)
 {noformat}
 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] 
 regionserver.HRegionServer:
 java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:361)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:336)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:293)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:258)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:603)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:476)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3546)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3616)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3494)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3485)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3079)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 2013-10-22 10:58:43,665 ERROR [RpcServer.handler=23,port=36020] 
 regionserver.HRegionServer:
 org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected 
 nextCallSeq: 53438 But the nextCallSeq got from client: 53437; 
 request=scanner_id: 1252577470624375060 number_of_rows: 100 close_scanner: 
 false next_call_seq: 53437
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3030)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
 at 
 org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
 at java.lang.Thread.run(Thread.java:724)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)