subject:"\[jira\] \[Commented\] $HBASE\-5864$ Error while reading from hfile in 0.94"


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262400#comment-13262400
 ] 

Lars Hofhansl commented on HBASE-5864:
--

@Ram: You are right it is a small change.
Just wondering whether we actually need the part that changes public 
DataInputStream nextBlockAsStream(BlockType blockType) to public HFileBlock 
nextBlockWithBlockType(BlockType blockType).


 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch, 
 HBASE-5864_3.patch, HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262401#comment-13262401
 ] 

Lars Hofhansl commented on HBASE-5864:
--

Ah OK. Never mind, you need the Block to the checksumBytes.

 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch, 
 HBASE-5864_3.patch, HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262403#comment-13262403
 ] 

Lars Hofhansl commented on HBASE-5864:
--

I think I get the change now. +1

 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch, 
 HBASE-5864_3.patch, HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262812#comment-13262812
 ] 

Lars Hofhansl commented on HBASE-5864:
--

Going to commit in a few unless there're objections.

 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch, 
 HBASE-5864_3.patch, HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263100#comment-13263100
 ] 

Hudson commented on HBASE-5864:
---

Integrated in HBase-0.94 #151 (See 
[https://builds.apache.org/job/HBase-0.94/151/])
HBASE-5864 Error while reading from hfile in 0.94 (Ram) (Revision 1331057)

 Result = ABORTED
larsh : 
Files : 
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java


 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch, 
 HBASE-5864_3.patch, HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263136#comment-13263136
 ] 

Hudson commented on HBASE-5864:
---

Integrated in HBase-0.94-security #22 (See 
[https://builds.apache.org/job/HBase-0.94-security/22/])
HBASE-5864 Error while reading from hfile in 0.94 (Ram) (Revision 1331057)

 Result = FAILURE
larsh : 
Files : 
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java


 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch, 
 HBASE-5864_3.patch, HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263185#comment-13263185
 ] 

Hudson commented on HBASE-5864:
---

Integrated in HBase-TRUNK #2819 (See 
[https://builds.apache.org/job/HBase-TRUNK/2819/])
HBASE-5864 Error while reading from hfile in 0.94 (Ram) (Revision 1331058)

 Result = SUCCESS
larsh : 
Files : 
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java


 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch, 
 HBASE-5864_3.patch, HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263373#comment-13263373
 ] 

Hudson commented on HBASE-5864:
---

Integrated in HBase-TRUNK-security #186 (See 
[https://builds.apache.org/job/HBase-TRUNK-security/186/])
HBASE-5864 Error while reading from hfile in 0.94 (Ram) (Revision 1331058)

 Result = SUCCESS
larsh : 
Files : 
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java


 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch, 
 HBASE-5864_3.patch, HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94

2012-04-25 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13261654#comment-13261654
 ] 

ramkrishna.s.vasudevan commented on HBASE-5864:
---

Updated the patch.  All test cases passed.  Verified a scenario where i had 
multiple level indexes and was able to get the correct midkey.
Please review.

 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch, 
 HBASE-5864_3.patch, HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94

2012-04-25 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13261686#comment-13261686
]

Hadoop QA commented on HBASE-5864:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12524272/HBASE-5864_3.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 6 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 5 new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.TestRegionRebalancing

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1644//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1644//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1644//console

This message is automatically generated.

Error while reading from hfile in 0.94
--

Key: HBASE-5864
URL: https://issues.apache.org/jira/browse/HBASE-5864
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
Fix For: 0.94.0

Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch,
HBASE-5864_3.patch, HBASE-5864_test.patch

Got the following stacktrace during region split.
{noformat}
2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store:
Failed getting store size for value
java.io.IOException: Requested block is out of range: 2906737606134037404,
lastDataBlockOffset: 84764558
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
at
org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
at
org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
at
org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
at
org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
{noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94

2012-04-25 Thread dhruba borthakur (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13261831#comment-13261831
 ] 

dhruba borthakur commented on HBASE-5864:
-

Thanks for finding this ramkrishna. I can see how the bug is occuring, very 
good analysis and thanks for finding it. I m trying to digest the fix you are 
providing.

 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch, 
 HBASE-5864_3.patch, HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94

2012-04-25 Thread Lars Hofhansl (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13261917#comment-13261917
 ] 

Lars Hofhansl commented on HBASE-5864:
--

TestRegionRebalancing is unrelated (HBASE-5848)

 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch, 
 HBASE-5864_3.patch, HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94

2012-04-25 Thread Zhihong Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262266#comment-13262266
 ] 

Zhihong Yu commented on HBASE-5864:
---

Latest patch looks good. Minor comments:
{code}
+  // after reading the root index the check sum bytes has to
{code}
'check sum bytes has to' - 'checksum bytes have to'
{code}
+  // be subracted to know if the mid key exists.
{code}
'subracted' - 'subtracted'


 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch, 
 HBASE-5864_3.patch, HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94

2012-04-25 Thread Lars Hofhansl (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262363#comment-13262363
 ] 

Lars Hofhansl commented on HBASE-5864:
--

This is the only issue in the way of the next RC attempt for 0.94.0.
I don't feel I can +1 this until a lot of study of the implication.

Was hoping Dhruba would be able to grok it. :)


 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch, 
 HBASE-5864_3.patch, HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94

2012-04-25 Thread dhruba borthakur (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262383#comment-13262383
 ] 

dhruba borthakur commented on HBASE-5864:
-

The meat of the change is in readMultiLevelIndexRoot() in which, instead of 
using in.available() it uses the in.available() - sizeofchecksum to determine 
if mid-key metadat is available, now I understand it. 
+1 on the fix.

 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch, 
 HBASE-5864_3.patch, HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94

2012-04-25 Thread dhruba borthakur (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262385#comment-13262385
 ] 

dhruba borthakur commented on HBASE-5864:
-

I also grepped thru the code to see if there is any other place where 
readRootIndex().available() is used but could not find any.

This is a good bug to catch. Ramkrishna: what was the symptom that triggered 
you to look for this bug?

 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch, 
 HBASE-5864_3.patch, HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94

2012-04-25 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262389#comment-13262389
 ] 

ramkrishna.s.vasudevan commented on HBASE-5864:
---

@Dhruba
Thanks for looking into the patch

The testcase that we carried out to get this problem was,

-Created a table with no split keys.
-Start pumping data to this region using parallel threads.
-Allow a couple of flush/compaction.
-Then try to split the region. It did not split saying midkey offset is not in 
range. 


Then we tried to reproduce this with different log messages added and found 
that it is while reading the root level index we get this problem.
Gopi (who saw this bug) and me spent 2 full days to come up with this.


 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch, 
 HBASE-5864_3.patch, HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94

2012-04-25 Thread Lars Hofhansl (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262391#comment-13262391
 ] 

Lars Hofhansl commented on HBASE-5864:
--

bq. The meat of the change is in readMultiLevelIndexRoot() in which, instead of 
using in.available() it uses the in.available() - sizeofchecksum

So it seems we could have a smaller change that just does that (plus the tests)?
I agree that this is a great catch!

 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch, 
 HBASE-5864_3.patch, HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94

2012-04-25 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262393#comment-13262393
 ] 

ramkrishna.s.vasudevan commented on HBASE-5864:
---

@Lars
I still feel the change is very small :).  Just to get the checkSumBytes 
instead of getting the stream am now getting the block from which i can get the 
stream again.

 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch, 
 HBASE-5864_3.patch, HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13260456#comment-13260456
 ] 

ramkrishna.s.vasudevan commented on HBASE-5864:
---

Please correct me if am wrong.  
Am working on a patch, which i can upload for verification.  Maynot be the 
final one.

 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Critical
 Fix For: 0.94.0


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13260455#comment-13260455
 ] 

ramkrishna.s.vasudevan commented on HBASE-5864:
---

The problem is because of using checksums.  This problem will be coming based 
on the block size and the kv size that are added in HFile.
When we try to read a block, Consider the following code 
In BlockIterator.blockRange
{code}
@Override
public HFileBlock nextBlock() throws IOException {
  if (offset = endOffset)
return null;
  HFileBlock b = readBlockData(offset, -1, -1, false);
  offset += b.getOnDiskSizeWithHeader();
  return b;
}
{code}
In case of noncompressed algo in 
{code}
  private HFileBlock readBlockDataInternal(FSDataInputStream is, long offset, 
long onDiskSizeWithHeaderL,
int uncompressedSize, boolean pread, boolean verifyChecksum) 
{code}
{code}
 b = new HFileBlock(headerBuf, getMinorVersion());

// This will also allocate enough room for the next block's header.
b.allocateBuffer(true);

{code}
Inside allocateBuffer
{code}
int cksumBytes = totalChecksumBytes();
int capacityNeeded = headerSize() + uncompressedSizeWithoutHeader +
cksumBytes +
(extraBytes ? headerSize() : 0);

ByteBuffer newBuf = ByteBuffer.allocate(capacityNeeded);
{code}
This byte allocated for this block is having the checksumbytes also added to it.
After fetching the block and its corresponding stream, we try to get the root 
block.
{code}
// Data index. We also read statistics about the block index written after
// the root level.
dataBlockIndexReader.readMultiLevelIndexRoot(
blockIter.nextBlockAsStream(BlockType.ROOT_INDEX),
trailer.getDataIndexCount());

{code}
Inside readMultiLevelIndexRoot
{code}
readRootIndex(in, numEntries);
  if (in.available()  MID_KEY_METADATA_SIZE) {
// No mid-key metadata available.
return;
  }
{code}
Here the InputStream 'in' is formed from the blk.getByteStream().
Here the inputstream actually will have its available length including the 
checksum bytes.
While doing readRootIndex
{code}
public void readRootIndex(DataInput in, final int numEntries)
throws IOException {
{code}
We read only the data and the remaining checksum bytse allocated are still 
available with the input stream.
We know that for every 16k data size we have 4 byte check sum.
Now if my data size is ~66k i will have 20bytes check sum.  So the 
{code}
   if (in.available()  MID_KEY_METADATA_SIZE) {
// No mid-key metadata available.
return;
  }
{code}
will now be greater than MID_KEY_METADATA_SIZE(16 bytes) and thus giving us 
some invalid 'midLeafBlockOffset'.
So when we try to read the file we get some abnormal values.  The same can be 
reproduced using the testcase.



 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Priority: Critical
 Fix For: 0.94.0


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13260634#comment-13260634
 ] 

ramkrishna.s.vasudevan commented on HBASE-5864:
---

I have attached a sample testcase.  Not a regular test case.  If you apply the 
test case and in the logs search for the string 
The midLeafBlockOffset is  .
We should be getting 
{code}
2012-04-24 20:26:21,140 DEBUG [main] 
hfile.HFileBlockIndex$BlockIndexReader(553): The midLeafBlockOffset is  
7767124539785491489
{code}

 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Critical
 Fix For: 0.94.0

 Attachments: HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94

2012-04-24 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13260645#comment-13260645
 ] 

stack commented on HBASE-5864:
--

Good find lads.  I'm not sure I follow.  Is it fixable?

 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Critical
 Fix For: 0.94.0

 Attachments: HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13260665#comment-13260665
 ] 

ramkrishna.s.vasudevan commented on HBASE-5864:
---

The attached patch seems to fix the problem that we encountered.  Needs more 
testing and also with files with old format.
Will find out if something better is there.

 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Critical
 Fix For: 0.94.0

 Attachments: HBASE-5864_1.patch, HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94

2012-04-24 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13260672#comment-13260672
 ] 

stack commented on HBASE-5864:
--

I'm not sure I follow what your patch is doing Ram.  And maybe we need a test 
around split of hfile?

What is this doing:

{code}
-final int ENTRY_COUNT = 1;
+final int ENTRY_COUNT = 5;
{code}

This is asking for too many entries?

Good stuff.

 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Critical
 Fix For: 0.94.0

 Attachments: HBASE-5864_1.patch, HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94

2012-04-24 Thread Zhihong Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13260697#comment-13260697
 ] 

Zhihong Yu commented on HBASE-5864:
---

{code}
-DataInputStream nextBlockAsStream(BlockType blockType) throws IOException;
+HFileBlock nextBlockAsStream(BlockType blockType) throws IOException;
{code}
The method should be named nextBlock() because stream isn't returned.
{code}
+ * Read in the root-level index from the given input stream. Must match
{code}
'input stream' is no longer the input. HFileBlock is.
Please add @return to the javadoc.
For TestHFileWriterV2.java:
{code}
-final Compression.Algorithm COMPRESS_ALGO = Compression.Algorithm.GZ;
+final Compression.Algorithm COMPRESS_ALGO = Compression.Algorithm.NONE;
{code}
We should exercise both compression algorithms. Refactoring is needed.

 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Critical
 Fix For: 0.94.0

 Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch, 
 HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13260700#comment-13260700
 ] 

ramkrishna.s.vasudevan commented on HBASE-5864:
---

Just added a test case to show how midKey() will throw IOException.
Stack what the patch does is,
For the checksum we add some additional bytes while forming the inputstream to 
read the block.
After reading the block we just check if the bytes available in stream is less 
than the MID_KEY_METADATA_SIZE.
But in this case (5 is many? but i get only one root level index), there is 
always some remaining bytes added as part of check sum and it is always more 
than MID_KEY_METADATA_SIZE.

The testcase that we carried out to get this problem was,
Created a table with no split keys.
Start pumping data to this region using parallel threads.
Allow a couple of flush/compaction.  
Then try to split the region.  We got this problem. 
I may be wrong, pls do correct me and feel free to update the patch also.



 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Critical
 Fix For: 0.94.0

 Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch, 
 HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13260703#comment-13260703
 ] 

ramkrishna.s.vasudevan commented on HBASE-5864:
---

@Ted
Sorry just now saw your comments. Will update based on further reviews. Thanks. 
:)

 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Critical
 Fix For: 0.94.0

 Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch, 
 HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94

2012-04-24 Thread Zhihong Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13260709#comment-13260709
 ] 

Zhihong Yu commented on HBASE-5864:
---

Patch v2 passes the new test.
{code}
+  private void writeDataAndReadFromHFile(Path hfilePath,
+  Algorithm COMPRESS_ALGO, int ENTRY_COUNT, boolean findMidKey) throws 
IOException {
{code}
Please don't use all upper case parameter names.

Please refactor the new readRootIndex() to re-use the existing method.

 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Critical
 Fix For: 0.94.0

 Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch, 
 HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94

2012-04-24 Thread Zhihong Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13260743#comment-13260743
 ] 

Zhihong Yu commented on HBASE-5864:
---

The following computation assumes checksum is on:
{code}
+  int numBytes = (int) ChecksumUtil.numBytes(blk
+  .getOnDiskDataSizeWithHeader(), blk.getBytesPerChecksum());
{code}
If checksum is off, we would get 'divide by 0' exception.

I suggest using HFileBlock.totalChecksumBytes() in place of the above.

 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Critical
 Fix For: 0.94.0

 Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch, 
 HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94

2012-04-24 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13260744#comment-13260744
]

Hadoop QA commented on HBASE-5864:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12524007/HBASE-5864_2.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 5 new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.TestRegionRebalancing
org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1627//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1627//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1627//console

This message is automatically generated.

Error while reading from hfile in 0.94
--

Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch,
HBASE-5864_test.patch

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94

2012-04-24 Thread Lars Hofhansl (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13261183#comment-13261183
 ] 

Lars Hofhansl commented on HBASE-5864:
--

Dhruba should have a look this too.

 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Critical
 Fix For: 0.94.0

 Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch, 
 HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13261276#comment-13261276
 ] 

ramkrishna.s.vasudevan commented on HBASE-5864:
---

bq.Dhruba should have a look this too.
+1 on Dhruba seeing this.

 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Critical
 Fix For: 0.94.0

 Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch, 
 HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13261307#comment-13261307
 ] 

ramkrishna.s.vasudevan commented on HBASE-5864:
---

bq.I suggest using HFileBlock.totalChecksumBytes() in place of the above.
Yes Ted this should be used.  It's scope is default so we can use it.  

 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Critical
 Fix For: 0.94.0

 Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch, 
 HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94

2012-04-24 Thread Lars Hofhansl (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13261315#comment-13261315
 ] 

Lars Hofhansl commented on HBASE-5864:
--

I also don't quite follow the patch or problem.
How is it that HBase during normal operation (scanning, etc) can read HFiles 
correctly?

 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Critical
 Fix For: 0.94.0

 Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch, 
 HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94