[jira] [Commented] (HBASE-6043) Add Increment Coalescing in thrift.
[ https://issues.apache.org/jira/browse/HBASE-6043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13943712#comment-13943712 ] Phabricator commented on HBASE-6043: eclark has abandoned the revision HBASE-6043 [jira] Add Increment Coalescing in thrift.. REVISION DETAIL https://reviews.facebook.net/D3315 To: JIRA, eclark Cc: jdcryans Add Increment Coalescing in thrift. --- Key: HBASE-6043 URL: https://issues.apache.org/jira/browse/HBASE-6043 Project: HBase Issue Type: Improvement Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.95.0 Attachments: HBASE-6043-0.patch, HBASE-6043-1.patch, HBASE-6043-10.patch, HBASE-6043-2.patch, HBASE-6043-3.patch, HBASE-6043-4.patch, HBASE-6043-5.patch, HBASE-6043-6.patch, HBASE-6043-7.patch, HBASE-6043-8.patch, HBASE-6043-9.patch, HBASE-6043-ADD.patch Since the thrift server uses the client api reducing the number of rpc's greatly speeds up increments. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13943711#comment-13943711 ] Phabricator commented on HBASE-5959: eclark has abandoned the revision HBASE-5959 [jira] Add other load balancers. REVISION DETAIL https://reviews.facebook.net/D3189 To: JIRA, eclark Cc: tedyu, lhofhansl Add other load balancers Key: HBASE-5959 URL: https://issues.apache.org/jira/browse/HBASE-5959 Project: HBase Issue Type: New Feature Components: master Affects Versions: 0.95.2 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.95.2 Attachments: ASF.LICENSE.NOT.GRANTED--HBASE-5959.D3189.1.patch, ASF.LICENSE.NOT.GRANTED--HBASE-5959.D3189.2.patch, ASF.LICENSE.NOT.GRANTED--HBASE-5959.D3189.3.patch, ASF.LICENSE.NOT.GRANTED--HBASE-5959.D3189.4.patch, ASF.LICENSE.NOT.GRANTED--HBASE-5959.D3189.5.patch, ASF.LICENSE.NOT.GRANTED--HBASE-5959.D3189.6.patch, ASF.LICENSE.NOT.GRANTED--HBASE-5959.D3189.7.patch, HBASE-5959-0.patch, HBASE-5959-1.patch, HBASE-5959-11.patch, HBASE-5959-12.patch, HBASE-5959-13.patch, HBASE-5959-14.patch, HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, HBASE-5959-7.patch, HBASE-5959-8.patch, HBASE-5959-9.patch Now that balancers are pluggable we should give some options. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-7946) Support VLong counters
[
https://issues.apache.org/jira/browse/HBASE-7946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HBASE-7946:
---
Attachment: D8955.1.patch
mbautin requested code review of [jira] [HBASE-7946] Support VLong counters.
Reviewers: lhofhansl, stack, JIRA
Currently increment and incrementColumnValue operations only support 64-bit
counters. Supporting VLong counters would be useful in cases when the majority
of counters would fit in less than 8 bytes.
TEST PLAN
Unit tests
REVISION DETAIL
https://reviews.facebook.net/D8955
AFFECTED FILES
hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
MANAGE HERALD RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/21699/
To: lhofhansl, stack, JIRA, mbautin
Support VLong counters
--
Key: HBASE-7946
URL: https://issues.apache.org/jira/browse/HBASE-7946
Project: HBase
Issue Type: Improvement
Reporter: Mikhail Bautin
Attachments: D8955.1.patch
Currently {{increment}} and {{incrementColumnValue}} operations only support
64-bit counters. Supporting {{VLong}} counters would be useful in cases when
the majority of counters would fit in less than 8 bytes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7031) Support startRow/endRow in TableInputFormat
[ https://issues.apache.org/jira/browse/HBASE-7031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13490817#comment-13490817 ] Phabricator commented on HBASE-7031: Liyin has closed the revision [jira] [HBASE-7031] [89-fb] Add startRow/endRow to TableInputFormat. CHANGED PRIOR TO COMMIT https://reviews.facebook.net/D6129?vs=20805id=21129#differential-review-toc REVISION DETAIL https://reviews.facebook.net/D6129 COMMIT https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1405915 To: Kannan, Karthik, JIRA, mbautin, aaiyer, Liyin, pritamdamania Cc: tjackson Support startRow/endRow in TableInputFormat --- Key: HBASE-7031 URL: https://issues.apache.org/jira/browse/HBASE-7031 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Priority: Minor Attachments: D6129.3.patch, D6129.4.patch We are still using org.apache.hadoop.hbase.mapred.TableInputFormat (as opposed to org.apache.hadoop.hbase.mapreduce.TableInputFormat) for Hadoop Streaming integration. We need to add startRow/endRow support to TableInputFormat due to a product requirement. However, the two TableInputFormat implementations have diverged over time. We need to make mapred.TableInputFormat reuse some of the newer code in mapreduce.TableInputFormat, and add startRow/endRow support to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7031) Support startRow/endRow in TableInputFormat
[ https://issues.apache.org/jira/browse/HBASE-7031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13488277#comment-13488277 ] Phabricator commented on HBASE-7031: tjackson has commented on the revision [jira] [HBASE-7031] [89-fb] Add startRow/endRow to TableInputFormat. A few little things, but otherwise this looks good. Could you grab someone more familiar with HBase/MR to accept? INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/mapred/TableInputFormatBase.java:132 ultranit: space after comma. src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java:73 I'd make the field the concrete type, since it's private. This will let Java see what methods are final and avoid the virtual invocations costs. src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java:227 Can't you just declare the method ##synchronized##? REVISION DETAIL https://reviews.facebook.net/D6129 To: Kannan, Karthik, JIRA, mbautin, pritamdamania Cc: tjackson Support startRow/endRow in TableInputFormat --- Key: HBASE-7031 URL: https://issues.apache.org/jira/browse/HBASE-7031 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Priority: Minor Attachments: D6129.3.patch We are still using org.apache.hadoop.hbase.mapred.TableInputFormat (as opposed to org.apache.hadoop.hbase.mapreduce.TableInputFormat) for Hadoop Streaming integration. We need to add startRow/endRow support to TableInputFormat due to a product requirement. However, the two TableInputFormat implementations have diverged over time. We need to make mapred.TableInputFormat reuse some of the newer code in mapreduce.TableInputFormat, and add startRow/endRow support to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7031) Support startRow/endRow in TableInputFormat
[ https://issues.apache.org/jira/browse/HBASE-7031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-7031: --- Attachment: D6129.4.patch pritamdamania updated the revision [jira] [HBASE-7031] [89-fb] Add startRow/endRow to TableInputFormat. Reviewers: Kannan, Karthik, JIRA, mbautin Addressing comments. REVISION DETAIL https://reviews.facebook.net/D6129 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/mapred/TableInputFormat.java src/main/java/org/apache/hadoop/hbase/mapred/TableInputFormatBase.java src/main/java/org/apache/hadoop/hbase/mapred/TableRecordReader.java src/main/java/org/apache/hadoop/hbase/mapred/TableRecordReaderImpl.java src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormat.java src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java To: Kannan, Karthik, JIRA, mbautin, pritamdamania Cc: tjackson Support startRow/endRow in TableInputFormat --- Key: HBASE-7031 URL: https://issues.apache.org/jira/browse/HBASE-7031 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Priority: Minor Attachments: D6129.3.patch, D6129.4.patch We are still using org.apache.hadoop.hbase.mapred.TableInputFormat (as opposed to org.apache.hadoop.hbase.mapreduce.TableInputFormat) for Hadoop Streaming integration. We need to add startRow/endRow support to TableInputFormat due to a product requirement. However, the two TableInputFormat implementations have diverged over time. We need to make mapred.TableInputFormat reuse some of the newer code in mapreduce.TableInputFormat, and add startRow/endRow support to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7031) Support startRow/endRow in TableInputFormat
[ https://issues.apache.org/jira/browse/HBASE-7031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13488412#comment-13488412 ] Phabricator commented on HBASE-7031: pritamdamania has added reviewers to the revision [jira] [HBASE-7031] [89-fb] Add startRow/endRow to TableInputFormat. Added Reviewers: aaiyer, Liyin Could someone take a look at this diff ? REVISION DETAIL https://reviews.facebook.net/D6129 To: Kannan, Karthik, JIRA, mbautin, aaiyer, Liyin, pritamdamania Cc: tjackson Support startRow/endRow in TableInputFormat --- Key: HBASE-7031 URL: https://issues.apache.org/jira/browse/HBASE-7031 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Priority: Minor Attachments: D6129.3.patch, D6129.4.patch We are still using org.apache.hadoop.hbase.mapred.TableInputFormat (as opposed to org.apache.hadoop.hbase.mapreduce.TableInputFormat) for Hadoop Streaming integration. We need to add startRow/endRow support to TableInputFormat due to a product requirement. However, the two TableInputFormat implementations have diverged over time. We need to make mapred.TableInputFormat reuse some of the newer code in mapreduce.TableInputFormat, and add startRow/endRow support to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7031) Support startRow/endRow in TableInputFormat
[ https://issues.apache.org/jira/browse/HBASE-7031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13488427#comment-13488427 ] Phabricator commented on HBASE-7031: Liyin has accepted the revision [jira] [HBASE-7031] [89-fb] Add startRow/endRow to TableInputFormat. The diff looks good to me. BTW, why does hadoop streaming still depend on the mapred package ? If we can deprecated the mapred package, it helps to remove the duplicated code path. REVISION DETAIL https://reviews.facebook.net/D6129 BRANCH arcpatch-D6129 To: Kannan, Karthik, JIRA, mbautin, aaiyer, Liyin, pritamdamania Cc: tjackson Support startRow/endRow in TableInputFormat --- Key: HBASE-7031 URL: https://issues.apache.org/jira/browse/HBASE-7031 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Priority: Minor Attachments: D6129.3.patch, D6129.4.patch We are still using org.apache.hadoop.hbase.mapred.TableInputFormat (as opposed to org.apache.hadoop.hbase.mapreduce.TableInputFormat) for Hadoop Streaming integration. We need to add startRow/endRow support to TableInputFormat due to a product requirement. However, the two TableInputFormat implementations have diverged over time. We need to make mapred.TableInputFormat reuse some of the newer code in mapreduce.TableInputFormat, and add startRow/endRow support to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7031) Support startRow/endRow in TableInputFormat
[ https://issues.apache.org/jira/browse/HBASE-7031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13488434#comment-13488434 ] Phabricator commented on HBASE-7031: pritamdamania has commented on the revision [jira] [HBASE-7031] [89-fb] Add startRow/endRow to TableInputFormat. I'm not sure about the duplicate use of mapred and mapreduce .. REVISION DETAIL https://reviews.facebook.net/D6129 BRANCH arcpatch-D6129 To: Kannan, Karthik, JIRA, mbautin, aaiyer, Liyin, pritamdamania Cc: tjackson Support startRow/endRow in TableInputFormat --- Key: HBASE-7031 URL: https://issues.apache.org/jira/browse/HBASE-7031 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Priority: Minor Attachments: D6129.3.patch, D6129.4.patch We are still using org.apache.hadoop.hbase.mapred.TableInputFormat (as opposed to org.apache.hadoop.hbase.mapreduce.TableInputFormat) for Hadoop Streaming integration. We need to add startRow/endRow support to TableInputFormat due to a product requirement. However, the two TableInputFormat implementations have diverged over time. We need to make mapred.TableInputFormat reuse some of the newer code in mapreduce.TableInputFormat, and add startRow/endRow support to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7031) Support startRow/endRow in TableInputFormat
[ https://issues.apache.org/jira/browse/HBASE-7031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13486615#comment-13486615 ] Phabricator commented on HBASE-7031: pritamdamania has commandeered the revision [jira] [HBASE-7031] [89-fb] Add startRow/endRow to TableInputFormat. Added Reviewers: mbautin REVISION DETAIL https://reviews.facebook.net/D6129 To: pritamdamania, Kannan, Karthik, JIRA, mbautin Support startRow/endRow in TableInputFormat --- Key: HBASE-7031 URL: https://issues.apache.org/jira/browse/HBASE-7031 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Priority: Minor We are still using org.apache.hadoop.hbase.mapred.TableInputFormat (as opposed to org.apache.hadoop.hbase.mapreduce.TableInputFormat) for Hadoop Streaming integration. We need to add startRow/endRow support to TableInputFormat due to a product requirement. However, the two TableInputFormat implementations have diverged over time. We need to make mapred.TableInputFormat reuse some of the newer code in mapreduce.TableInputFormat, and add startRow/endRow support to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7031) Support startRow/endRow in TableInputFormat
[ https://issues.apache.org/jira/browse/HBASE-7031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-7031: --- Attachment: D6129.3.patch pritamdamania updated the revision [jira] [HBASE-7031] [89-fb] Add startRow/endRow to TableInputFormat. Reviewers: Kannan, Karthik, JIRA, mbautin Patched this locally, could someone take a look at this diff ? REVISION DETAIL https://reviews.facebook.net/D6129 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/mapred/TableInputFormat.java src/main/java/org/apache/hadoop/hbase/mapred/TableInputFormatBase.java src/main/java/org/apache/hadoop/hbase/mapred/TableRecordReader.java src/main/java/org/apache/hadoop/hbase/mapred/TableRecordReaderImpl.java src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormat.java src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java To: Kannan, Karthik, JIRA, mbautin, pritamdamania Support startRow/endRow in TableInputFormat --- Key: HBASE-7031 URL: https://issues.apache.org/jira/browse/HBASE-7031 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Priority: Minor Attachments: D6129.3.patch We are still using org.apache.hadoop.hbase.mapred.TableInputFormat (as opposed to org.apache.hadoop.hbase.mapreduce.TableInputFormat) for Hadoop Streaming integration. We need to add startRow/endRow support to TableInputFormat due to a product requirement. However, the two TableInputFormat implementations have diverged over time. We need to make mapred.TableInputFormat reuse some of the newer code in mapreduce.TableInputFormat, and add startRow/endRow support to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6597) Block Encoding Size Estimation
[ https://issues.apache.org/jira/browse/HBASE-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13482543#comment-13482543 ] Phabricator commented on HBASE-6597: mbautin has commented on the revision [jira] [HBASE-6597] [89-fb] Incremental data block encoding. Replying to comments inline. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/io/encoding/CopyKeyDataBlockEncoder.java:29 Updated the comment. src/main/java/org/apache/hadoop/hbase/io/encoding/DiffKeyDeltaEncoder.java:34 Done. src/main/java/org/apache/hadoop/hbase/io/encoding/FastDiffDeltaEncoder.java:35 Done. src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java:132 Updated the comment. length Bytes.SIZEOF_INT is correct (this is the length of the part of the array we are allowed to write into). src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java:445 Done. src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java:446 No. The two last parameters are for assertEquals. REVISION DETAIL https://reviews.facebook.net/D5895 To: Kannan, Karthik, Liyin, aaiyer, avf, JIRA, mbautin Cc: tedyu Block Encoding Size Estimation -- Key: HBASE-6597 URL: https://issues.apache.org/jira/browse/HBASE-6597 Project: HBase Issue Type: Improvement Components: io Affects Versions: 0.89-fb Reporter: Brian Nixon Assignee: Mikhail Bautin Priority: Minor Attachments: D5895.1.patch, D5895.2.patch, D5895.3.patch, D5895.4.patch Blocks boundaries as created by current writers are determined by the size of the unencoded data. However, blocks in memory are kept encoded. By using an estimate for the encoded size of the block, we can get greater consistency in size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6597) Block Encoding Size Estimation
[ https://issues.apache.org/jira/browse/HBASE-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-6597: --- Attachment: D5895.5.patch mbautin updated the revision [jira] [HBASE-6597] [89-fb] Incremental data block encoding. Reviewers: Kannan, Karthik, Liyin, aaiyer, avf, JIRA Addressing Kannan's comments and rebasing. REVISION DETAIL https://reviews.facebook.net/D5895 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/KeyValue.java src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/io/encoding/CopyKeyDataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoding.java src/main/java/org/apache/hadoop/hbase/io/encoding/DiffKeyDeltaEncoder.java src/main/java/org/apache/hadoop/hbase/io/encoding/EncodedDataBlock.java src/main/java/org/apache/hadoop/hbase/io/encoding/EncodingState.java src/main/java/org/apache/hadoop/hbase/io/encoding/CompressionState.java src/main/java/org/apache/hadoop/hbase/io/encoding/FastDiffDeltaEncoder.java src/main/java/org/apache/hadoop/hbase/io/encoding/PrefixKeyDeltaEncoder.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoderImpl.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java src/main/java/org/apache/hadoop/hbase/io/hfile/NoOpDataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java src/test/java/org/apache/hadoop/hbase/io/encoding/TestDataBlockEncoders.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestIncrementalEncoding.java src/test/java/org/apache/hadoop/hbase/util/TestByteBufferUtils.java To: Kannan, Karthik, Liyin, aaiyer, avf, JIRA, mbautin Cc: tedyu Block Encoding Size Estimation -- Key: HBASE-6597 URL: https://issues.apache.org/jira/browse/HBASE-6597 Project: HBase Issue Type: Improvement Components: io Affects Versions: 0.89-fb Reporter: Brian Nixon Assignee: Mikhail Bautin Priority: Minor Attachments: D5895.1.patch, D5895.2.patch, D5895.3.patch, D5895.4.patch, D5895.5.patch Blocks boundaries as created by current writers are determined by the size of the unencoded data. However, blocks in memory are kept encoded. By using an estimate for the encoded size of the block, we can get greater consistency in size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6983) Metric for unencoded size of cached blocks
[ https://issues.apache.org/jira/browse/HBASE-6983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13482566#comment-13482566 ] Phabricator commented on HBASE-6983: Kannan has accepted the revision [jira] [HBASE-6983] [89-fb] Metric for unencoded size of cached blocks. lgtm! REVISION DETAIL https://reviews.facebook.net/D5979 BRANCH unencoded_cache_size_v11 To: Kannan, Karthik, Liyin, aaiyer, mcorgan, JIRA, mbautin Metric for unencoded size of cached blocks -- Key: HBASE-6983 URL: https://issues.apache.org/jira/browse/HBASE-6983 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Priority: Minor Attachments: D5979.1.patch We need to measure the amount of unencoded data in the block cache when data block encoding is enabled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6597) Block Encoding Size Estimation
[ https://issues.apache.org/jira/browse/HBASE-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13482737#comment-13482737 ] Phabricator commented on HBASE-6597: mbautin has commented on the revision [jira] [HBASE-6597] [89-fb] Incremental data block encoding. Verified that the effective in-cache block size for encoded blocks is close enough to the configured size during a load test. Committing. REVISION DETAIL https://reviews.facebook.net/D5895 To: Kannan, Karthik, Liyin, aaiyer, avf, JIRA, mbautin Cc: tedyu Block Encoding Size Estimation -- Key: HBASE-6597 URL: https://issues.apache.org/jira/browse/HBASE-6597 Project: HBase Issue Type: Improvement Components: io Affects Versions: 0.89-fb Reporter: Brian Nixon Assignee: Mikhail Bautin Priority: Minor Attachments: D5895.1.patch, D5895.2.patch, D5895.3.patch, D5895.4.patch, D5895.5.patch Blocks boundaries as created by current writers are determined by the size of the unencoded data. However, blocks in memory are kept encoded. By using an estimate for the encoded size of the block, we can get greater consistency in size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6983) Metric for unencoded size of cached blocks
[ https://issues.apache.org/jira/browse/HBASE-6983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-6983: --- Attachment: D5979.2.patch mbautin updated the revision [jira] [HBASE-6983] [89-fb] Metric for unencoded size of cached blocks. Reviewers: Kannan, Karthik, Liyin, aaiyer, mcorgan, JIRA Rebasing on incremental data block encoding changes. REVISION DETAIL https://reviews.facebook.net/D5979 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/io/encoding/CopyKeyDataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java src/test/java/org/apache/hadoop/hbase/regionserver/EncodedSeekPerformanceTest.java src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java src/test/java/org/apache/hadoop/hbase/regionserver/metrics/TestSchemaMetrics.java To: Kannan, Karthik, Liyin, aaiyer, mcorgan, JIRA, mbautin Metric for unencoded size of cached blocks -- Key: HBASE-6983 URL: https://issues.apache.org/jira/browse/HBASE-6983 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Priority: Minor Attachments: D5979.1.patch, D5979.2.patch We need to measure the amount of unencoded data in the block cache when data block encoding is enabled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6983) Metric for unencoded size of cached blocks
[ https://issues.apache.org/jira/browse/HBASE-6983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-6983: --- Attachment: D5979.3.patch mbautin updated the revision [jira] [HBASE-6983] [89-fb] Metric for unencoded size of cached blocks. Reviewers: Kannan, Karthik, Liyin, aaiyer, mcorgan, JIRA Fixing rebase conflicts in comments REVISION DETAIL https://reviews.facebook.net/D5979 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/io/encoding/CopyKeyDataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java src/test/java/org/apache/hadoop/hbase/regionserver/EncodedSeekPerformanceTest.java src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java src/test/java/org/apache/hadoop/hbase/regionserver/metrics/TestSchemaMetrics.java To: Kannan, Karthik, Liyin, aaiyer, mcorgan, JIRA, mbautin Metric for unencoded size of cached blocks -- Key: HBASE-6983 URL: https://issues.apache.org/jira/browse/HBASE-6983 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Priority: Minor Attachments: D5979.1.patch, D5979.2.patch, D5979.3.patch We need to measure the amount of unencoded data in the block cache when data block encoding is enabled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6983) Metric for unencoded size of cached blocks
[ https://issues.apache.org/jira/browse/HBASE-6983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13482817#comment-13482817 ] Phabricator commented on HBASE-6983: mbautin has closed the revision [jira] [HBASE-6983] [89-fb] Metric for unencoded size of cached blocks. CHANGED PRIOR TO COMMIT https://reviews.facebook.net/D5979?vs=20355id=20367#differential-review-toc REVISION DETAIL https://reviews.facebook.net/D5979 COMMIT https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1401513 To: Kannan, Karthik, Liyin, aaiyer, mcorgan, JIRA, mbautin Metric for unencoded size of cached blocks -- Key: HBASE-6983 URL: https://issues.apache.org/jira/browse/HBASE-6983 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Priority: Minor Attachments: D5979.1.patch, D5979.2.patch, D5979.3.patch We need to measure the amount of unencoded data in the block cache when data block encoding is enabled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6597) Block Encoding Size Estimation
[ https://issues.apache.org/jira/browse/HBASE-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13482818#comment-13482818 ] Phabricator commented on HBASE-6597: mbautin has abandoned the revision [jira] [HBASE-6597] [89-fb] Incremental data block encoding. Committed as rHBASEEIGHTNINEFBBRANCH1401499. REVISION DETAIL https://reviews.facebook.net/D5895 To: Kannan, Karthik, Liyin, aaiyer, avf, JIRA, mbautin Cc: tedyu Block Encoding Size Estimation -- Key: HBASE-6597 URL: https://issues.apache.org/jira/browse/HBASE-6597 Project: HBase Issue Type: Improvement Components: io Affects Versions: 0.89-fb Reporter: Brian Nixon Assignee: Mikhail Bautin Priority: Minor Attachments: D5895.1.patch, D5895.2.patch, D5895.3.patch, D5895.4.patch, D5895.5.patch Blocks boundaries as created by current writers are determined by the size of the unencoded data. However, blocks in memory are kept encoded. By using an estimate for the encoded size of the block, we can get greater consistency in size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6597) Block Encoding Size Estimation
[ https://issues.apache.org/jira/browse/HBASE-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13481530#comment-13481530 ] Phabricator commented on HBASE-6597: Kannan has commented on the revision [jira] [HBASE-6597] [89-fb] Incremental data block encoding. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/io/encoding/CopyKeyDataBlockEncoder.java:29 Regarding the second bullet (in my comment), I suppose it is ok to leave this as is... as it does simplify the calling logic a little bit. We should just add here in comments that * this is used for non-encoded blocks. * and, keeps blocks in old format (without the DBE specific headers). src/main/java/org/apache/hadoop/hbase/io/encoding/FastDiffDeltaEncoder.java:35 use integer compression for key, value and prefix (7-bit encoding) -- use integer compression for key, value and prefix lengths (7-bit encoding) src/main/java/org/apache/hadoop/hbase/io/encoding/DiffKeyDeltaEncoder.java:34 ditto (as other comment) s/prefix/prefix lengths REVISION DETAIL https://reviews.facebook.net/D5895 To: Kannan, Karthik, Liyin, aaiyer, avf, JIRA, mbautin Cc: tedyu Block Encoding Size Estimation -- Key: HBASE-6597 URL: https://issues.apache.org/jira/browse/HBASE-6597 Project: HBase Issue Type: Improvement Components: io Affects Versions: 0.89-fb Reporter: Brian Nixon Assignee: Mikhail Bautin Priority: Minor Attachments: D5895.1.patch, D5895.2.patch, D5895.3.patch, D5895.4.patch Blocks boundaries as created by current writers are determined by the size of the unencoded data. However, blocks in memory are kept encoded. By using an estimate for the encoded size of the block, we can get greater consistency in size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6597) Block Encoding Size Estimation
[ https://issues.apache.org/jira/browse/HBASE-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13481531#comment-13481531 ] Phabricator commented on HBASE-6597: Kannan has commented on the revision [jira] [HBASE-6597] [89-fb] Incremental data block encoding. Mikhail - looks great. Pending comments are very trivial. REVISION DETAIL https://reviews.facebook.net/D5895 To: Kannan, Karthik, Liyin, aaiyer, avf, JIRA, mbautin Cc: tedyu Block Encoding Size Estimation -- Key: HBASE-6597 URL: https://issues.apache.org/jira/browse/HBASE-6597 Project: HBase Issue Type: Improvement Components: io Affects Versions: 0.89-fb Reporter: Brian Nixon Assignee: Mikhail Bautin Priority: Minor Attachments: D5895.1.patch, D5895.2.patch, D5895.3.patch, D5895.4.patch Blocks boundaries as created by current writers are determined by the size of the unencoded data. However, blocks in memory are kept encoded. By using an estimate for the encoded size of the block, we can get greater consistency in size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7031) Support startRow/endRow in TableInputFormat
[ https://issues.apache.org/jira/browse/HBASE-7031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13481926#comment-13481926 ] Phabricator commented on HBASE-7031: mbautin has commented on the revision [jira] [HBASE-7031] [89-fb] Add startRow/endRow to TableInputFormat. Verified that startRow/endRow works successfully on a dev cluster. REVISION DETAIL https://reviews.facebook.net/D6129 To: pritamdamania, Kannan, Karthik, JIRA, mbautin Support startRow/endRow in TableInputFormat --- Key: HBASE-7031 URL: https://issues.apache.org/jira/browse/HBASE-7031 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Priority: Minor We are still using org.apache.hadoop.hbase.mapred.TableInputFormat (as opposed to org.apache.hadoop.hbase.mapreduce.TableInputFormat) for Hadoop Streaming integration. We need to add startRow/endRow support to TableInputFormat due to a product requirement. However, the two TableInputFormat implementations have diverged over time. We need to make mapred.TableInputFormat reuse some of the newer code in mapreduce.TableInputFormat, and add startRow/endRow support to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6463) Support multiple memstore snapshots in order to support small/large flushes of cache.
[ https://issues.apache.org/jira/browse/HBASE-6463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13480663#comment-13480663 ] Phabricator commented on HBASE-6463: Kannan has resigned from the revision [jira] [HBASE-6463] [89-fb] Multiple Snapshot Buffering. REVISION DETAIL https://reviews.facebook.net/D4389 To: mbautin, Liyin, Karthik, JIRA, nixon Cc: FBHBase Support multiple memstore snapshots in order to support small/large flushes of cache. -- Key: HBASE-6463 URL: https://issues.apache.org/jira/browse/HBASE-6463 Project: HBase Issue Type: Improvement Components: regionserver, util Affects Versions: 0.89-fb Reporter: Brian Nixon If cache is underutilized due to log size triggered flushes, should be able to buffer multiple snapshots in memory and flush all together into one file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6597) Block Encoding Size Estimation
[ https://issues.apache.org/jira/browse/HBASE-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13480431#comment-13480431 ] Phabricator commented on HBASE-6597: Kannan has commented on the revision [jira] [HBASE-6597] [89-fb] Incremental data block encoding. comments thus far... INLINE COMMENTS src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java:445 mismath - mismatch src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java:446 don't you need two more %s's. src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java:132 shouldn't this be: if (length - offset Bytes.SIZEOF_INT) src/main/java/org/apache/hadoop/hbase/io/encoding/CopyKeyDataBlockEncoder.java:29 I think this comment is no longer valid. * It gets used for ENCODING = 'NONE' case now correct? * Wondering now, if that was a correct choice... because we seem to be having to jump through some hoops to handle this encoder as a separate case (such as to not write the headers, etc.). REVISION DETAIL https://reviews.facebook.net/D5895 To: Kannan, Karthik, Liyin, aaiyer, avf, JIRA, mbautin Cc: tedyu Block Encoding Size Estimation -- Key: HBASE-6597 URL: https://issues.apache.org/jira/browse/HBASE-6597 Project: HBase Issue Type: Improvement Components: io Affects Versions: 0.89-fb Reporter: Brian Nixon Assignee: Mikhail Bautin Priority: Minor Attachments: D5895.1.patch, D5895.2.patch, D5895.3.patch, D5895.4.patch Blocks boundaries as created by current writers are determined by the size of the unencoded data. However, blocks in memory are kept encoded. By using an estimate for the encoded size of the block, we can get greater consistency in size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6983) Metric for unencoded size of cached blocks
[ https://issues.apache.org/jira/browse/HBASE-6983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-6983: --- Attachment: D5979.1.patch mbautin requested code review of [jira] [HBASE-6983] [89-fb] Metric for unencoded size of cached blocks. Reviewers: Kannan, Karthik, Liyin, aaiyer, mcorgan, JIRA We need to measure the amount of unencoded data in the block cache when data block encoding is enabled. TEST PLAN Unit tests REVISION DETAIL https://reviews.facebook.net/D5979 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/io/encoding/CopyKeyDataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java src/test/java/org/apache/hadoop/hbase/regionserver/EncodedSeekPerformanceTest.java src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java src/test/java/org/apache/hadoop/hbase/regionserver/metrics/TestSchemaMetrics.java To: JIRA Metric for unencoded size of cached blocks -- Key: HBASE-6983 URL: https://issues.apache.org/jira/browse/HBASE-6983 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Priority: Minor Attachments: D5979.1.patch We need to measure the amount of unencoded data in the block cache when data block encoding is enabled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6955) Block read time should be in ms, not in ns
[ https://issues.apache.org/jira/browse/HBASE-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13475598#comment-13475598 ] Phabricator commented on HBASE-6955: mbautin has closed the revision [jira] [HBASE-6955] [89-fb] Block read time should be in ms, not in ns. CHANGED PRIOR TO COMMIT https://reviews.facebook.net/D5901?vs=19917id=19947#differential-review-toc REVISION DETAIL https://reviews.facebook.net/D5901 COMMIT https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1397821 To: Kannan, JIRA, mbautin Block read time should be in ms, not in ns -- Key: HBASE-6955 URL: https://issues.apache.org/jira/browse/HBASE-6955 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Priority: Minor Attachments: D5901.1.patch, D5901.2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6955) Block read time should be in ms, not in ns
[ https://issues.apache.org/jira/browse/HBASE-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-6955: --- Attachment: D5901.2.patch mbautin updated the revision [jira] [HBASE-6955] [89-fb] Block read time should be in ms, not in ns. Reviewers: Kannan, JIRA Addressing Kannan's comments. REVISION DETAIL https://reviews.facebook.net/D5901 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java src/test/java/org/apache/hadoop/hbase/regionserver/HFileReadWriteTest.java To: Kannan, JIRA, mbautin Block read time should be in ms, not in ns -- Key: HBASE-6955 URL: https://issues.apache.org/jira/browse/HBASE-6955 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Priority: Minor Attachments: D5901.1.patch, D5901.2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6955) Block read time should be in ms, not in ns
[ https://issues.apache.org/jira/browse/HBASE-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13475319#comment-13475319 ] Phabricator commented on HBASE-6955: Kannan has accepted the revision [jira] [HBASE-6955] [89-fb] Block read time should be in ms, not in ns. lgtm! REVISION DETAIL https://reviews.facebook.net/D5901 BRANCH fix_fs_read_time_v5 To: Kannan, JIRA, mbautin Block read time should be in ms, not in ns -- Key: HBASE-6955 URL: https://issues.apache.org/jira/browse/HBASE-6955 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Priority: Minor Attachments: D5901.1.patch, D5901.2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6597) Block Encoding Size Estimation
[ https://issues.apache.org/jira/browse/HBASE-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-6597: --- Attachment: D5895.3.patch mbautin updated the revision [jira] [HBASE-6597] [89-fb] Incremental data block encoding. Reviewers: Kannan, Karthik, Liyin, aaiyer, avf, JIRA Critical bugfixes in updating encoder state. REVISION DETAIL https://reviews.facebook.net/D5895 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/KeyValue.java src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/io/encoding/CompressionState.java src/main/java/org/apache/hadoop/hbase/io/encoding/CopyKeyDataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoding.java src/main/java/org/apache/hadoop/hbase/io/encoding/DiffKeyDeltaEncoder.java src/main/java/org/apache/hadoop/hbase/io/encoding/EncodedDataBlock.java src/main/java/org/apache/hadoop/hbase/io/encoding/FastDiffDeltaEncoder.java src/main/java/org/apache/hadoop/hbase/io/encoding/PrefixKeyDeltaEncoder.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoderImpl.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java src/main/java/org/apache/hadoop/hbase/io/hfile/NoOpDataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java src/test/java/org/apache/hadoop/hbase/io/encoding/TestDataBlockEncoders.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestIncrementalEncoding.java src/test/java/org/apache/hadoop/hbase/util/TestByteBufferUtils.java To: Kannan, Karthik, Liyin, aaiyer, avf, JIRA, mbautin Cc: tedyu Block Encoding Size Estimation -- Key: HBASE-6597 URL: https://issues.apache.org/jira/browse/HBASE-6597 Project: HBase Issue Type: Improvement Components: io Affects Versions: 0.89-fb Reporter: Brian Nixon Assignee: Mikhail Bautin Priority: Minor Attachments: D5895.1.patch, D5895.2.patch, D5895.3.patch Blocks boundaries as created by current writers are determined by the size of the unencoded data. However, blocks in memory are kept encoded. By using an estimate for the encoded size of the block, we can get greater consistency in size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6597) Block Encoding Size Estimation
[ https://issues.apache.org/jira/browse/HBASE-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13471766#comment-13471766 ] Phabricator commented on HBASE-6597: mbautin has commented on the revision [jira] [HBASE-6597] [89-fb] Incremental data block encoding. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java:38 Done. src/main/java/org/apache/hadoop/hbase/io/encoding/CopyKeyDataBlockEncoder.java:93 Yes, that's a good catch. src/main/java/org/apache/hadoop/hbase/io/encoding/DiffKeyDeltaEncoder.java:340 Done. src/main/java/org/apache/hadoop/hbase/io/encoding/DiffKeyDeltaEncoder.java:420 Done (here and in other encoded writers). src/main/java/org/apache/hadoop/hbase/io/encoding/DiffKeyDeltaEncoder.java:422 Yes, the comment was incorrect. Moved this field to the encoder state. src/main/java/org/apache/hadoop/hbase/io/encoding/DiffKeyDeltaEncoder.java:497 Done. src/main/java/org/apache/hadoop/hbase/io/encoding/FastDiffDeltaEncoder.java:411 Done. src/main/java/org/apache/hadoop/hbase/io/encoding/PrefixKeyDeltaEncoder.java:173 Done. src/main/java/org/apache/hadoop/hbase/io/encoding/PrefixKeyDeltaEncoder.java:186 Yes, this javadoc was incorrect. Moved this to encoder state. REVISION DETAIL https://reviews.facebook.net/D5895 To: Kannan, Karthik, Liyin, aaiyer, avf, JIRA, mbautin Cc: tedyu Block Encoding Size Estimation -- Key: HBASE-6597 URL: https://issues.apache.org/jira/browse/HBASE-6597 Project: HBase Issue Type: Improvement Components: io Affects Versions: 0.89-fb Reporter: Brian Nixon Assignee: Mikhail Bautin Priority: Minor Attachments: D5895.1.patch, D5895.2.patch, D5895.3.patch Blocks boundaries as created by current writers are determined by the size of the unencoded data. However, blocks in memory are kept encoded. By using an estimate for the encoded size of the block, we can get greater consistency in size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6597) Block Encoding Size Estimation
[ https://issues.apache.org/jira/browse/HBASE-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13471784#comment-13471784 ] Phabricator commented on HBASE-6597: mbautin has commented on the revision [jira] [HBASE-6597] [89-fb] Incremental data block encoding. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java:76 includesMemstoreTS means that we are memstore timestamp is part of both input and output. We don't change that aspect of the data format on data block encoding/decoding. src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java:119 Added an assertion to BufferedEncodedWriter. The code below won't make currentState null if it is not null initially. src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java:447 As you can see, the DataBlockEncoder class does not have a lot of state (unlike the EncodedWriter) so I don't know what else I could include here. src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java:96 Oops. That was for debugging. Good catch! REVISION DETAIL https://reviews.facebook.net/D5895 To: Kannan, Karthik, Liyin, aaiyer, avf, JIRA, mbautin Cc: tedyu Block Encoding Size Estimation -- Key: HBASE-6597 URL: https://issues.apache.org/jira/browse/HBASE-6597 Project: HBase Issue Type: Improvement Components: io Affects Versions: 0.89-fb Reporter: Brian Nixon Assignee: Mikhail Bautin Priority: Minor Attachments: D5895.1.patch, D5895.2.patch, D5895.3.patch Blocks boundaries as created by current writers are determined by the size of the unencoded data. However, blocks in memory are kept encoded. By using an estimate for the encoded size of the block, we can get greater consistency in size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6597) Block Encoding Size Estimation
[ https://issues.apache.org/jira/browse/HBASE-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13471799#comment-13471799 ] Phabricator commented on HBASE-6597: tedyu has commented on the revision [jira] [HBASE-6597] [89-fb] Incremental data block encoding. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java:447 Got you. Then this implementation is fine. REVISION DETAIL https://reviews.facebook.net/D5895 To: Kannan, Karthik, Liyin, aaiyer, avf, JIRA, mbautin Cc: tedyu Block Encoding Size Estimation -- Key: HBASE-6597 URL: https://issues.apache.org/jira/browse/HBASE-6597 Project: HBase Issue Type: Improvement Components: io Affects Versions: 0.89-fb Reporter: Brian Nixon Assignee: Mikhail Bautin Priority: Minor Attachments: D5895.1.patch, D5895.2.patch, D5895.3.patch Blocks boundaries as created by current writers are determined by the size of the unencoded data. However, blocks in memory are kept encoded. By using an estimate for the encoded size of the block, we can get greater consistency in size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6597) Block Encoding Size Estimation
[ https://issues.apache.org/jira/browse/HBASE-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-6597: --- Attachment: D5895.4.patch mbautin updated the revision [jira] [HBASE-6597] [89-fb] Incremental data block encoding. Reviewers: Kannan, Karthik, Liyin, aaiyer, avf, JIRA Addressing Ted's comments. REVISION DETAIL https://reviews.facebook.net/D5895 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/KeyValue.java src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/io/encoding/CopyKeyDataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoding.java src/main/java/org/apache/hadoop/hbase/io/encoding/DiffKeyDeltaEncoder.java src/main/java/org/apache/hadoop/hbase/io/encoding/EncodedDataBlock.java src/main/java/org/apache/hadoop/hbase/io/encoding/EncodingState.java src/main/java/org/apache/hadoop/hbase/io/encoding/CompressionState.java src/main/java/org/apache/hadoop/hbase/io/encoding/FastDiffDeltaEncoder.java src/main/java/org/apache/hadoop/hbase/io/encoding/PrefixKeyDeltaEncoder.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoderImpl.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java src/main/java/org/apache/hadoop/hbase/io/hfile/NoOpDataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java src/test/java/org/apache/hadoop/hbase/io/encoding/TestDataBlockEncoders.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestIncrementalEncoding.java src/test/java/org/apache/hadoop/hbase/util/TestByteBufferUtils.java To: Kannan, Karthik, Liyin, aaiyer, avf, JIRA, mbautin Cc: tedyu Block Encoding Size Estimation -- Key: HBASE-6597 URL: https://issues.apache.org/jira/browse/HBASE-6597 Project: HBase Issue Type: Improvement Components: io Affects Versions: 0.89-fb Reporter: Brian Nixon Assignee: Mikhail Bautin Priority: Minor Attachments: D5895.1.patch, D5895.2.patch, D5895.3.patch, D5895.4.patch Blocks boundaries as created by current writers are determined by the size of the unencoded data. However, blocks in memory are kept encoded. By using an estimate for the encoded size of the block, we can get greater consistency in size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6597) Block Encoding Size Estimation
[ https://issues.apache.org/jira/browse/HBASE-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-6597: --- Attachment: D5895.2.patch mbautin updated the revision [jira] [HBASE-6597] [89-fb] Incremental data block encoding. Reviewers: Kannan, Karthik, Liyin, aaiyer, avf, JIRA Removing code duplication on the encoding codepath. Addressing some of Ted's comments (I will double-check that I have not missed any of them later). REVISION DETAIL https://reviews.facebook.net/D5895 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/KeyValue.java src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/io/encoding/CompressionState.java src/main/java/org/apache/hadoop/hbase/io/encoding/CopyKeyDataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoding.java src/main/java/org/apache/hadoop/hbase/io/encoding/DiffKeyDeltaEncoder.java src/main/java/org/apache/hadoop/hbase/io/encoding/EncodedDataBlock.java src/main/java/org/apache/hadoop/hbase/io/encoding/FastDiffDeltaEncoder.java src/main/java/org/apache/hadoop/hbase/io/encoding/PrefixKeyDeltaEncoder.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoderImpl.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java src/main/java/org/apache/hadoop/hbase/io/hfile/NoOpDataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java src/test/java/org/apache/hadoop/hbase/io/encoding/TestDataBlockEncoders.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestIncrementalEncoding.java src/test/java/org/apache/hadoop/hbase/util/TestByteBufferUtils.java To: Kannan, Karthik, Liyin, aaiyer, avf, JIRA, mbautin Cc: tedyu Block Encoding Size Estimation -- Key: HBASE-6597 URL: https://issues.apache.org/jira/browse/HBASE-6597 Project: HBase Issue Type: Improvement Components: io Affects Versions: 0.89-fb Reporter: Brian Nixon Assignee: Mikhail Bautin Priority: Minor Attachments: D5895.1.patch, D5895.2.patch Blocks boundaries as created by current writers are determined by the size of the unencoded data. However, blocks in memory are kept encoded. By using an estimate for the encoded size of the block, we can get greater consistency in size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6597) Block Encoding Size Estimation
[ https://issues.apache.org/jira/browse/HBASE-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13471330#comment-13471330 ] Phabricator commented on HBASE-6597: tedyu has commented on the revision [jira] [HBASE-6597] [89-fb] Incremental data block encoding. Code is cleaner in patch v2. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/io/encoding/PrefixKeyDeltaEncoder.java:173 This class can be private. src/main/java/org/apache/hadoop/hbase/io/encoding/PrefixKeyDeltaEncoder.java:186 Javadoc and code mismatch. src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java:76 Would memstoreTS always be part of in ? If so, do you need to advance the position inside in when includesMemstoreTS is false ? src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java:119 Consider adding an assertion. src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java:447 Do you want to include more information in String representation ? src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java:96 Why skipping the case of includesMemstoreTS being true ? REVISION DETAIL https://reviews.facebook.net/D5895 To: Kannan, Karthik, Liyin, aaiyer, avf, JIRA, mbautin Cc: tedyu Block Encoding Size Estimation -- Key: HBASE-6597 URL: https://issues.apache.org/jira/browse/HBASE-6597 Project: HBase Issue Type: Improvement Components: io Affects Versions: 0.89-fb Reporter: Brian Nixon Assignee: Mikhail Bautin Priority: Minor Attachments: D5895.1.patch, D5895.2.patch Blocks boundaries as created by current writers are determined by the size of the unencoded data. However, blocks in memory are kept encoded. By using an estimate for the encoded size of the block, we can get greater consistency in size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6597) Block Encoding Size Estimation
[ https://issues.apache.org/jira/browse/HBASE-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13471355#comment-13471355 ] Phabricator commented on HBASE-6597: tedyu has commented on the revision [jira] [HBASE-6597] [89-fb] Incremental data block encoding. My previous comments for PrefixKeyDeltaEncoder.java were for patch v1. Phabricator remembered my unfinished comments and sent them out. FYI REVISION DETAIL https://reviews.facebook.net/D5895 To: Kannan, Karthik, Liyin, aaiyer, avf, JIRA, mbautin Cc: tedyu Block Encoding Size Estimation -- Key: HBASE-6597 URL: https://issues.apache.org/jira/browse/HBASE-6597 Project: HBase Issue Type: Improvement Components: io Affects Versions: 0.89-fb Reporter: Brian Nixon Assignee: Mikhail Bautin Priority: Minor Attachments: D5895.1.patch, D5895.2.patch Blocks boundaries as created by current writers are determined by the size of the unencoded data. However, blocks in memory are kept encoded. By using an estimate for the encoded size of the block, we can get greater consistency in size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6597) Block Encoding Size Estimation
[
https://issues.apache.org/jira/browse/HBASE-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13471358#comment-13471358
]
Phabricator commented on HBASE-6597:
tedyu has commented on the revision [jira] [HBASE-6597] [89-fb] Incremental
data block encoding.
I got some test failures in TestCacheOnWrite:
testStoreFileCacheOnWrite[2](org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite):
expected:{ENCODED_DATA=9[65, LEAF_INDEX=121], BLOOM_CHUNK=9, INT... but
was:{ENCODED_DATA=9[91, LEAF_INDEX=124], BLOOM_CHUNK=9, INT...
testStoreFileCacheOnWrite[5](org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite):
expected:{ENCODED_DATA=9[65, LEAF_INDEX=121], BLOOM_CHUNK=9, INT... but
was:{ENCODED_DATA=9[91, LEAF_INDEX=124], BLOOM_CHUNK=9, INT...
testStoreFileCacheOnWrite[8](org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite):
expected:{ENCODED_DATA=9[65, LEAF_INDEX=121], BLOOM_CHUNK=9, INT... but
was:{ENCODED_DATA=9[91, LEAF_INDEX=124], BLOOM_CHUNK=9, INT...
testStoreFileCacheOnWrite[11](org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite):
expected:{ENCODED_DATA=9[65, LEAF_INDEX=121], BLOOM_CHUNK=9, INT... but
was:{ENCODED_DATA=9[91, LEAF_INDEX=124], BLOOM_CHUNK=9, INT...
testStoreFileCacheOnWrite[14](org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite):
expected:{ENCODED_DATA=9[65, LEAF_INDEX=121], BLOOM_CHUNK=9, INT... but
was:{ENCODED_DATA=9[91, LEAF_INDEX=124], BLOOM_CHUNK=9, INT...
testStoreFileCacheOnWrite[17](org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite):
expected:{ENCODED_DATA=9[65, LEAF_INDEX=121], BLOOM_CHUNK=9, INT... but
was:{ENCODED_DATA=9[91, LEAF_INDEX=124], BLOOM_CHUNK=9, INT...
Here is one of the above:
testStoreFileCacheOnWrite[2](org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite)
Time elapsed: 0.295 sec FAILURE!
org.junit.ComparisonFailure: expected:{ENCODED_DATA=9[65, LEAF_INDEX=121],
BLOOM_CHUNK=9, INT... but was:{ENCODED_DATA=9[91, LEAF_INDEX=124],
BLOOM_CHUNK=9, INT...
at org.junit.Assert.assertEquals(Assert.java:123)
at org.junit.Assert.assertEquals(Assert.java:145)
at
org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite.readStoreFile(TestCacheOnWrite.java:259)
at
org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite.testStoreFileCacheOnWrite(TestCacheOnWrite.java:203)
REVISION DETAIL
https://reviews.facebook.net/D5895
To: Kannan, Karthik, Liyin, aaiyer, avf, JIRA, mbautin
Cc: tedyu
Block Encoding Size Estimation
--
Key: HBASE-6597
URL: https://issues.apache.org/jira/browse/HBASE-6597
Project: HBase
Issue Type: Improvement
Components: io
Affects Versions: 0.89-fb
Reporter: Brian Nixon
Assignee: Mikhail Bautin
Priority: Minor
Attachments: D5895.1.patch, D5895.2.patch
Blocks boundaries as created by current writers are determined by the size of
the unencoded data. However, blocks in memory are kept encoded. By using an
estimate for the encoded size of the block, we can get greater consistency in
size.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6597) Block Encoding Size Estimation
[ https://issues.apache.org/jira/browse/HBASE-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13470475#comment-13470475 ] Phabricator commented on HBASE-6597: tedyu has commented on the revision [jira] [HBASE-6597] [89-fb] Incremental data block encoding. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java:38 Length of prefix is returned. Name this method getCommonPrefixLength ? src/main/java/org/apache/hadoop/hbase/io/encoding/CopyKeyDataBlockEncoder.java:93 Do we need to consider memstoreTS ? src/main/java/org/apache/hadoop/hbase/io/encoding/DiffKeyDeltaEncoder.java:340 Check / assert that skipLastBytes is not negative ? src/main/java/org/apache/hadoop/hbase/io/encoding/DiffKeyDeltaEncoder.java:422 prevKey stores the previous key, right ? src/main/java/org/apache/hadoop/hbase/io/encoding/DiffKeyDeltaEncoder.java:497 Name this variable negativeDiffTimestamp ? src/main/java/org/apache/hadoop/hbase/io/encoding/FastDiffDeltaEncoder.java:411 This class can be private, right ? src/main/java/org/apache/hadoop/hbase/io/encoding/DiffKeyDeltaEncoder.java:420 This class can be private, right ? REVISION DETAIL https://reviews.facebook.net/D5895 To: Kannan, Karthik, Liyin, aaiyer, avf, JIRA, mbautin Cc: tedyu Block Encoding Size Estimation -- Key: HBASE-6597 URL: https://issues.apache.org/jira/browse/HBASE-6597 Project: HBase Issue Type: Improvement Components: io Affects Versions: 0.89-fb Reporter: Brian Nixon Priority: Minor Attachments: D5895.1.patch Blocks boundaries as created by current writers are determined by the size of the unencoded data. However, blocks in memory are kept encoded. By using an estimate for the encoded size of the block, we can get greater consistency in size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6872) Number of records written/read per second on regionserver level
[ https://issues.apache.org/jira/browse/HBASE-6872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13469222#comment-13469222 ] Phabricator commented on HBASE-6872: mbautin has closed the revision [jira] [HBASE-6872] [89-fb] Fix TestRegionServerMetrics.testNumReadsAndWrites. CHANGED PRIOR TO COMMIT https://reviews.facebook.net/D5853?vs=19311id=19359#differential-review-toc REVISION DETAIL https://reviews.facebook.net/D5853 COMMIT https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1393955 To: Kannan, Karthik, JIRA, Liyin, mbautin Cc: adela, Liyin, aaiyer, avf Number of records written/read per second on regionserver level --- Key: HBASE-6872 URL: https://issues.apache.org/jira/browse/HBASE-6872 Project: HBase Issue Type: New Feature Components: regionserver Reporter: Adela Maznikar Priority: Minor Attachments: D5853.1.patch, D5853.2.patch, D5853.3.patch Regionserver level metrics that shows the number of records written/read per second. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6943) [89-fb] Do not catch certain exceptions trying to get an RS connection
[ https://issues.apache.org/jira/browse/HBASE-6943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-6943: --- Attachment: D5877.1.patch mbautin requested code review of [jira] [HBASE-6943] [89-fb] Do not catch certain exceptions trying to get an RS connection. Reviewers: Kannan, Liyin, Karthik, JIRA When getting a regionserver connection in 0.89-fb in HBaseClient, we catch all types of Throwable. I have observed a real case when the client looked stuck. On debugging it turned out that a NoSuchMethodError was thrown and caught, leaving the connection in an inconsistent state (initialized socket but null streams). All following attempts resulted in NPEs that were also caught, and no errors were logged. From the user's perspective the client was just stuck. The root cause was the absence of a required jar (hence the NoSuchMethodError) but it was not reported properly. TEST PLAN Run a client with the same configuration as before and verify it does not get stuck. REVISION DETAIL https://reviews.facebook.net/D5877 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/13929/ To: Kannan, Liyin, Karthik, JIRA, mbautin [89-fb] Do not catch certain exceptions trying to get an RS connection -- Key: HBASE-6943 URL: https://issues.apache.org/jira/browse/HBASE-6943 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Attachments: D5877.1.patch When getting a regionserver connection in 0.89-fb in HBaseClient, we catch all types of Throwable. I have observed a real case when the client looked stuck. On debugging it turned out that a NoSuchMethodError was thrown and caught, leaving the connection in an inconsistent state (initialized socket but null streams). All following attempts resulted in NPEs that were also caught, and no errors were logged. From the user's perspective the client was just stuck. The root cause was the absence of a required jar (hence the NoSuchMethodError) but it was not reported properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6943) [89-fb] Do not catch certain exceptions trying to get an RS connection
[ https://issues.apache.org/jira/browse/HBASE-6943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13469535#comment-13469535 ] Phabricator commented on HBASE-6943: Kannan has added CCs to the revision [jira] [HBASE-6943] [89-fb] Do not catch certain exceptions trying to get an RS connection. Added CCs: avf, adela, pritamdamania, aaiyer, nspiegelberg, amirshim, mycnyc Let's try to cc individually until we can figure out how to more easily email the group automatically. REVISION DETAIL https://reviews.facebook.net/D5877 To: Kannan, Liyin, Karthik, JIRA, mbautin Cc: avf, adela, pritamdamania, aaiyer, nspiegelberg, amirshim, mycnyc [89-fb] Do not catch certain exceptions trying to get an RS connection -- Key: HBASE-6943 URL: https://issues.apache.org/jira/browse/HBASE-6943 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Attachments: D5877.1.patch When getting a regionserver connection in 0.89-fb in HBaseClient, we catch all types of Throwable. I have observed a real case when the client looked stuck. On debugging it turned out that a NoSuchMethodError was thrown and caught, leaving the connection in an inconsistent state (initialized socket but null streams). All following attempts resulted in NPEs that were also caught, and no errors were logged. From the user's perspective the client was just stuck. The root cause was the absence of a required jar (hence the NoSuchMethodError) but it was not reported properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6943) [89-fb] Do not catch certain exceptions trying to get an RS connection
[ https://issues.apache.org/jira/browse/HBASE-6943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13469537#comment-13469537 ] Phabricator commented on HBASE-6943: Kannan has accepted the revision [jira] [HBASE-6943] [89-fb] Do not catch certain exceptions trying to get an RS connection. lgtm! good catch... REVISION DETAIL https://reviews.facebook.net/D5877 BRANCH stuck_client_v4 To: Kannan, Liyin, Karthik, JIRA, mbautin Cc: avf, adela, pritamdamania, aaiyer, nspiegelberg, amirshim, mycnyc [89-fb] Do not catch certain exceptions trying to get an RS connection -- Key: HBASE-6943 URL: https://issues.apache.org/jira/browse/HBASE-6943 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Attachments: D5877.1.patch When getting a regionserver connection in 0.89-fb in HBaseClient, we catch all types of Throwable. I have observed a real case when the client looked stuck. On debugging it turned out that a NoSuchMethodError was thrown and caught, leaving the connection in an inconsistent state (initialized socket but null streams). All following attempts resulted in NPEs that were also caught, and no errors were logged. From the user's perspective the client was just stuck. The root cause was the absence of a required jar (hence the NoSuchMethodError) but it was not reported properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6943) [89-fb] Do not catch certain exceptions trying to get an RS connection
[ https://issues.apache.org/jira/browse/HBASE-6943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13469607#comment-13469607 ] Phabricator commented on HBASE-6943: aaiyer has commented on the revision [jira] [HBASE-6943] [89-fb] Do not catch certain exceptions trying to get an RS connection. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1383 Do we need to do this in other places as well? -- getRegionServer with/without retries? Perhaps a good place to do this is to do it in translateException. If we see that the throwable is one of these bad ones. we can just throw ie again. REVISION DETAIL https://reviews.facebook.net/D5877 BRANCH stuck_client_v4 To: Kannan, Liyin, Karthik, JIRA, mbautin Cc: avf, adela, pritamdamania, aaiyer, nspiegelberg, amirshim, mycnyc [89-fb] Do not catch certain exceptions trying to get an RS connection -- Key: HBASE-6943 URL: https://issues.apache.org/jira/browse/HBASE-6943 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Attachments: D5877.1.patch When getting a regionserver connection in 0.89-fb in HBaseClient, we catch all types of Throwable. I have observed a real case when the client looked stuck. On debugging it turned out that a NoSuchMethodError was thrown and caught, leaving the connection in an inconsistent state (initialized socket but null streams). All following attempts resulted in NPEs that were also caught, and no errors were logged. From the user's perspective the client was just stuck. The root cause was the absence of a required jar (hence the NoSuchMethodError) but it was not reported properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6943) [89-fb] Do not catch certain exceptions trying to get an RS connection
[ https://issues.apache.org/jira/browse/HBASE-6943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-6943: --- Attachment: D5877.2.patch mbautin updated the revision [jira] [HBASE-6943] [89-fb] Do not catch certain exceptions trying to get an RS connection. Reviewers: Kannan, Liyin, Karthik, JIRA Addressing Amit's feedback REVISION DETAIL https://reviews.facebook.net/D5877 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java To: Kannan, Liyin, Karthik, JIRA, mbautin Cc: avf, adela, pritamdamania, aaiyer, nspiegelberg, amirshim, mycnyc [89-fb] Do not catch certain exceptions trying to get an RS connection -- Key: HBASE-6943 URL: https://issues.apache.org/jira/browse/HBASE-6943 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Attachments: D5877.1.patch, D5877.2.patch When getting a regionserver connection in 0.89-fb in HBaseClient, we catch all types of Throwable. I have observed a real case when the client looked stuck. On debugging it turned out that a NoSuchMethodError was thrown and caught, leaving the connection in an inconsistent state (initialized socket but null streams). All following attempts resulted in NPEs that were also caught, and no errors were logged. From the user's perspective the client was just stuck. The root cause was the absence of a required jar (hence the NoSuchMethodError) but it was not reported properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6943) [89-fb] Do not catch certain exceptions trying to get an RS connection
[ https://issues.apache.org/jira/browse/HBASE-6943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-6943: --- Attachment: D5877.3.patch mbautin updated the revision [jira] [HBASE-6943] [89-fb] Do not catch certain exceptions trying to get an RS connection. Reviewers: Kannan, Liyin, Karthik, JIRA Catching an arbitrary throwable and wrapping it with an IOException in setupIOstreams. REVISION DETAIL https://reviews.facebook.net/D5877 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java To: Kannan, Liyin, Karthik, JIRA, mbautin Cc: avf, adela, pritamdamania, aaiyer, nspiegelberg, amirshim, mycnyc [89-fb] Do not catch certain exceptions trying to get an RS connection -- Key: HBASE-6943 URL: https://issues.apache.org/jira/browse/HBASE-6943 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Attachments: D5877.1.patch, D5877.2.patch, D5877.3.patch When getting a regionserver connection in 0.89-fb in HBaseClient, we catch all types of Throwable. I have observed a real case when the client looked stuck. On debugging it turned out that a NoSuchMethodError was thrown and caught, leaving the connection in an inconsistent state (initialized socket but null streams). All following attempts resulted in NPEs that were also caught, and no errors were logged. From the user's perspective the client was just stuck. The root cause was the absence of a required jar (hence the NoSuchMethodError) but it was not reported properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6943) [89-fb] Do not catch certain exceptions trying to get an RS connection
[ https://issues.apache.org/jira/browse/HBASE-6943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-6943: --- Attachment: D5877.4.patch mbautin updated the revision [jira] [HBASE-6943] [89-fb] Do not catch certain exceptions trying to get an RS connection. Reviewers: Kannan, Liyin, Karthik, JIRA Removing some unnecessary changes. REVISION DETAIL https://reviews.facebook.net/D5877 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java To: Kannan, Liyin, Karthik, JIRA, mbautin Cc: avf, adela, pritamdamania, aaiyer, nspiegelberg, amirshim, mycnyc [89-fb] Do not catch certain exceptions trying to get an RS connection -- Key: HBASE-6943 URL: https://issues.apache.org/jira/browse/HBASE-6943 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Attachments: D5877.1.patch, D5877.2.patch, D5877.3.patch, D5877.4.patch When getting a regionserver connection in 0.89-fb in HBaseClient, we catch all types of Throwable. I have observed a real case when the client looked stuck. On debugging it turned out that a NoSuchMethodError was thrown and caught, leaving the connection in an inconsistent state (initialized socket but null streams). All following attempts resulted in NPEs that were also caught, and no errors were logged. From the user's perspective the client was just stuck. The root cause was the absence of a required jar (hence the NoSuchMethodError) but it was not reported properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6943) [89-fb] Do not catch certain exceptions trying to get an RS connection
[ https://issues.apache.org/jira/browse/HBASE-6943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13469799#comment-13469799 ] Phabricator commented on HBASE-6943: mbautin has closed the revision [jira] [HBASE-6943] [89-fb] Do not catch certain exceptions trying to get an RS connection. CHANGED PRIOR TO COMMIT https://reviews.facebook.net/D5877?vs=19413id=19425#differential-review-toc REVISION DETAIL https://reviews.facebook.net/D5877 COMMIT https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1394307 To: Kannan, Liyin, Karthik, JIRA, mbautin Cc: avf, adela, pritamdamania, aaiyer, nspiegelberg, amirshim, mycnyc [89-fb] Do not catch certain exceptions trying to get an RS connection -- Key: HBASE-6943 URL: https://issues.apache.org/jira/browse/HBASE-6943 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Attachments: D5877.1.patch, D5877.2.patch, D5877.3.patch, D5877.4.patch When getting a regionserver connection in 0.89-fb in HBaseClient, we catch all types of Throwable. I have observed a real case when the client looked stuck. On debugging it turned out that a NoSuchMethodError was thrown and caught, leaving the connection in an inconsistent state (initialized socket but null streams). All following attempts resulted in NPEs that were also caught, and no errors were logged. From the user's perspective the client was just stuck. The root cause was the absence of a required jar (hence the NoSuchMethodError) but it was not reported properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6597) Block Encoding Size Estimation
[ https://issues.apache.org/jira/browse/HBASE-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-6597: --- Attachment: D5895.1.patch mbautin requested code review of [jira] [HBASE-6597] [89-fb] Incremental data block encoding. Reviewers: Kannan, Karthik, Liyin, aaiyer, avf, JIRA Instead of accumulating a block of key-values of predetermined size and then delta-encoding it, we encode key-value pairs as we add them and start a new block when the **encoded** size exceeds the configured block size. This work was done by Brian Nixon. I did necessary testing and bug fixes. TEST PLAN Run unit tests REVISION DETAIL https://reviews.facebook.net/D5895 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/KeyValue.java src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/io/encoding/CopyKeyDataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoding.java src/main/java/org/apache/hadoop/hbase/io/encoding/DiffKeyDeltaEncoder.java src/main/java/org/apache/hadoop/hbase/io/encoding/EncodedDataBlock.java src/main/java/org/apache/hadoop/hbase/io/encoding/FastDiffDeltaEncoder.java src/main/java/org/apache/hadoop/hbase/io/encoding/PrefixKeyDeltaEncoder.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoderImpl.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java src/main/java/org/apache/hadoop/hbase/io/hfile/NoOpDataBlockEncoder.java src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java src/test/java/org/apache/hadoop/hbase/io/encoding/TestDataBlockEncoders.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestIncrementalEncoding.java src/test/java/org/apache/hadoop/hbase/util/TestByteBufferUtils.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/13989/ To: Kannan, Karthik, Liyin, aaiyer, avf, JIRA, mbautin Block Encoding Size Estimation -- Key: HBASE-6597 URL: https://issues.apache.org/jira/browse/HBASE-6597 Project: HBase Issue Type: Improvement Components: io Affects Versions: 0.89-fb Reporter: Brian Nixon Priority: Minor Attachments: D5895.1.patch Blocks boundaries as created by current writers are determined by the size of the unencoded data. However, blocks in memory are kept encoded. By using an estimate for the encoded size of the block, we can get greater consistency in size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6597) Block Encoding Size Estimation
[ https://issues.apache.org/jira/browse/HBASE-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13469933#comment-13469933 ] Phabricator commented on HBASE-6597: mbautin has commented on the revision [jira] [HBASE-6597] [89-fb] Incremental data block encoding. The original review (Facebook-internal, just for the record): https://phabricator.fb.com/D554523 REVISION DETAIL https://reviews.facebook.net/D5895 To: Kannan, Karthik, Liyin, aaiyer, avf, JIRA, mbautin Block Encoding Size Estimation -- Key: HBASE-6597 URL: https://issues.apache.org/jira/browse/HBASE-6597 Project: HBase Issue Type: Improvement Components: io Affects Versions: 0.89-fb Reporter: Brian Nixon Priority: Minor Attachments: D5895.1.patch Blocks boundaries as created by current writers are determined by the size of the unencoded data. However, blocks in memory are kept encoded. By using an estimate for the encoded size of the block, we can get greater consistency in size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6955) Block read time should be in ms, not in ns
[ https://issues.apache.org/jira/browse/HBASE-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-6955: --- Attachment: D5901.1.patch mbautin requested code review of [jira] [HBASE-6955] [89-fb] Block read time should be in ms, not in ns. Reviewers: Kannan, JIRA We update block read time in nanoseconds, which is inconsistent with all other latency metrics in the system. TEST PLAN Unit tests REVISION DETAIL https://reviews.facebook.net/D5901 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/13995/ To: Kannan, JIRA, mbautin Block read time should be in ms, not in ns -- Key: HBASE-6955 URL: https://issues.apache.org/jira/browse/HBASE-6955 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Priority: Minor Attachments: D5901.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6955) Block read time should be in ms, not in ns
[ https://issues.apache.org/jira/browse/HBASE-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13469979#comment-13469979 ] Phabricator commented on HBASE-6955: Kannan has commented on the revision [jira] [HBASE-6955] [89-fb] Block read time should be in ms, not in ns. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java:312 In RegionServerMetrics.java, can we also export the pread metrics much like we do for the non-pread case: int ops = HFile.getReadOps(); if (ops != 0) this.fsReadLatency.inc(ops, HFile.getReadTimeMs()); REVISION DETAIL https://reviews.facebook.net/D5901 To: Kannan, JIRA, mbautin Block read time should be in ms, not in ns -- Key: HBASE-6955 URL: https://issues.apache.org/jira/browse/HBASE-6955 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Priority: Minor Attachments: D5901.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3793) HBASE-3468 Broke checkAndPut with null value
[
https://issues.apache.org/jira/browse/HBASE-3793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HBASE-3793:
---
Attachment: D5835.1.patch
mbautin requested code review of [jira] [HBASE-3793] [89-fb] Fix TestHRegion
failure with zero-byte expected array in compare-and-put.
Reviewers: Liyin, Kannan, JIRA
Passing a zero-byte expected value to checkAndPut and similar methods now
means we are expecting to see a zero-byte value, not a non-existent value. This
should have been part of rHBASEEIGHTNINEFBBRANCH1391219.
TEST PLAN
TestHRegion
REVISION DETAIL
https://reviews.facebook.net/D5835
AFFECTED FILES
src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/13821/
To: Liyin, Kannan, JIRA, mbautin
HBASE-3468 Broke checkAndPut with null value
Key: HBASE-3793
URL: https://issues.apache.org/jira/browse/HBASE-3793
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.92.0
Reporter: Lars George
Assignee: Ming Ma
Priority: Blocker
Fix For: 0.92.0
Attachments: D5835.1.patch, HBASE-3793.patch, HBASE-3793-TRUNK.patch
The previous code called Bytes.equal() which does a check for null on the
left or right argument. Now the comparator calls Bytes.compareTo() - which
has no check for null. But this is a valid input and checks for existence. I
actually noticed this running
https://github.com/larsgeorge/hbase-book/blob/master/ch04/src/main/java/client/CheckAndPutExample.java
This used to work, now it throws an NPE
{noformat}
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:854)
at
org.apache.hadoop.hbase.filter.WritableByteArrayComparable.compareTo(WritableByteArrayComparable.java:63)
at
org.apache.hadoop.hbase.regionserver.HRegion.checkAndMutate(HRegion.java:1681)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkAndMutate(HRegionServer.java:1693)
... 6 more
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1026)
at org.apache.hadoop.hbase.client.HTable.checkAndPut(HTable.java:750)
at client.CheckAndPutExample.main(CheckAndPutExample.java:33)
{noformat}
Easy fixable, just needs to handle the null value before even calling
comparator.compareTo().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3793) HBASE-3468 Broke checkAndPut with null value
[
https://issues.apache.org/jira/browse/HBASE-3793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13468879#comment-13468879
]
Phabricator commented on HBASE-3793:
Kannan has accepted the revision [jira] [HBASE-3793] [89-fb] Fix TestHRegion
failure with zero-length expected value in compare-and-put.
REVISION DETAIL
https://reviews.facebook.net/D5835
BRANCH
fix_test_hregion
To: Liyin, Kannan, JIRA, mbautin
HBASE-3468 Broke checkAndPut with null value
Key: HBASE-3793
URL: https://issues.apache.org/jira/browse/HBASE-3793
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.92.0
Reporter: Lars George
Assignee: Ming Ma
Priority: Blocker
Fix For: 0.92.0
Attachments: D5835.1.patch, HBASE-3793.patch, HBASE-3793-TRUNK.patch
The previous code called Bytes.equal() which does a check for null on the
left or right argument. Now the comparator calls Bytes.compareTo() - which
has no check for null. But this is a valid input and checks for existence. I
actually noticed this running
https://github.com/larsgeorge/hbase-book/blob/master/ch04/src/main/java/client/CheckAndPutExample.java
This used to work, now it throws an NPE
{noformat}
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:854)
at
org.apache.hadoop.hbase.filter.WritableByteArrayComparable.compareTo(WritableByteArrayComparable.java:63)
at
org.apache.hadoop.hbase.regionserver.HRegion.checkAndMutate(HRegion.java:1681)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkAndMutate(HRegionServer.java:1693)
... 6 more
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1026)
at org.apache.hadoop.hbase.client.HTable.checkAndPut(HTable.java:750)
at client.CheckAndPutExample.main(CheckAndPutExample.java:33)
{noformat}
Easy fixable, just needs to handle the null value before even calling
comparator.compareTo().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6930) [89-fb] Avoid acquiring the same row lock repeatedly
[ https://issues.apache.org/jira/browse/HBASE-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-6930: --- Attachment: D5841.1.patch mbautin requested code review of [jira] [HBASE-6930] [89-fb] Fix TestThriftServerLegacy: notifyAll should be inside synchronized block. Reviewers: Kannan, Liyin, Karthik, JIRA There were a couple of reasons why TestThriftServerLegacy has been failing recently in the HBase 89-fb branch: - rHBASEEIGHTNINEFBBRANCH1393468 was calling notifyAll outside a synchronized block - rHBASEEIGHTNINEFBBRANCH1391219 changed the meaning of a null expected value passed to checkAndMutate but that was not reflected in the Thrift handler TEST PLAN Run TestThriftServerLegacy REVISION DETAIL https://reviews.facebook.net/D5841 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/13833/ To: Kannan, Liyin, Karthik, JIRA, mbautin [89-fb] Avoid acquiring the same row lock repeatedly Key: HBASE-6930 URL: https://issues.apache.org/jira/browse/HBASE-6930 Project: HBase Issue Type: Bug Reporter: Liyin Tang Attachments: D5841.1.patch, D5841.2.patch When processing the multiPut, multiMutations or multiDelete operations, each IPC handler thread tries to acquire a lock for each row key in these batches. If there are duplicated row keys in these batches, previously the IPC handler thread will repeatedly acquire the same row key again and again. So the optimization is to sort each batch operation based on the row key in the client side, and skip acquiring the same row lock repeatedly in the server side. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6930) [89-fb] Avoid acquiring the same row lock repeatedly
[ https://issues.apache.org/jira/browse/HBASE-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-6930: --- Attachment: D5841.2.patch mbautin updated the revision [jira] [HBASE-6930] [89-fb] Fix TestThriftServerLegacy: notifyAll should be inside synchronized block, and a null checkAndMutate expected value should be handled correctly. Reviewers: Kannan, Liyin, Karthik, JIRA Adding ThriftServerRunner fixes REVISION DETAIL https://reviews.facebook.net/D5841 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServerLegacy.java To: Kannan, Liyin, Karthik, JIRA, mbautin [89-fb] Avoid acquiring the same row lock repeatedly Key: HBASE-6930 URL: https://issues.apache.org/jira/browse/HBASE-6930 Project: HBase Issue Type: Bug Reporter: Liyin Tang Attachments: D5841.1.patch, D5841.2.patch When processing the multiPut, multiMutations or multiDelete operations, each IPC handler thread tries to acquire a lock for each row key in these batches. If there are duplicated row keys in these batches, previously the IPC handler thread will repeatedly acquire the same row key again and again. So the optimization is to sort each batch operation based on the row key in the client side, and skip acquiring the same row lock repeatedly in the server side. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6930) [89-fb] Avoid acquiring the same row lock repeatedly
[ https://issues.apache.org/jira/browse/HBASE-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13468929#comment-13468929 ] Phabricator commented on HBASE-6930: Liyin has accepted the revision [jira] [HBASE-6930] [89-fb] Fix TestThriftServerLegacy: notifyAll should be inside synchronized block, and a null checkAndMutate expected value should be handled correctly. Thanks Mikhail ! REVISION DETAIL https://reviews.facebook.net/D5841 BRANCH fix_locked_rows_v2 To: Kannan, Liyin, Karthik, JIRA, mbautin [89-fb] Avoid acquiring the same row lock repeatedly Key: HBASE-6930 URL: https://issues.apache.org/jira/browse/HBASE-6930 Project: HBase Issue Type: Bug Reporter: Liyin Tang Attachments: D5841.1.patch, D5841.2.patch When processing the multiPut, multiMutations or multiDelete operations, each IPC handler thread tries to acquire a lock for each row key in these batches. If there are duplicated row keys in these batches, previously the IPC handler thread will repeatedly acquire the same row key again and again. So the optimization is to sort each batch operation based on the row key in the client side, and skip acquiring the same row lock repeatedly in the server side. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6872) Number of records written/read per second on regionserver level
[ https://issues.apache.org/jira/browse/HBASE-6872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-6872: --- Attachment: D5853.1.patch mbautin requested code review of [jira] [HBASE-6872] [89-fb] Fix TestRegionServerMetrics.testNumReadsAndWrites. Reviewers: Kannan, Karthik, JIRA rHBASEEIGHTNINEFBBRANCH1389841 introduced an unstable test in TestRegionServerMetrics: testNumReadsAndWrites. Read and write counters should be reset to zero before starting the test. TEST PLAN Run TestRegionServerMetrics REVISION DETAIL https://reviews.facebook.net/D5853 AFFECTED FILES src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/13863/ To: Kannan, Karthik, JIRA, mbautin Number of records written/read per second on regionserver level --- Key: HBASE-6872 URL: https://issues.apache.org/jira/browse/HBASE-6872 Project: HBase Issue Type: New Feature Components: regionserver Reporter: Adela Maznikar Priority: Minor Attachments: D5853.1.patch Regionserver level metrics that shows the number of records written/read per second. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6872) Number of records written/read per second on regionserver level
[ https://issues.apache.org/jira/browse/HBASE-6872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13468995#comment-13468995 ] Phabricator commented on HBASE-6872: Kannan has added CCs to the revision [jira] [HBASE-6872] [89-fb] Fix TestRegionServerMetrics.testNumReadsAndWrites. Added CCs: adela, Liyin, aaiyer, avf REVISION DETAIL https://reviews.facebook.net/D5853 To: Kannan, Karthik, JIRA, mbautin Cc: adela, Liyin, aaiyer, avf Number of records written/read per second on regionserver level --- Key: HBASE-6872 URL: https://issues.apache.org/jira/browse/HBASE-6872 Project: HBase Issue Type: New Feature Components: regionserver Reporter: Adela Maznikar Priority: Minor Attachments: D5853.1.patch Regionserver level metrics that shows the number of records written/read per second. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6872) Number of records written/read per second on regionserver level
[ https://issues.apache.org/jira/browse/HBASE-6872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13468997#comment-13468997 ] Phabricator commented on HBASE-6872: Kannan has commented on the revision [jira] [HBASE-6872] [89-fb] Fix TestRegionServerMetrics.testNumReadsAndWrites. INLINE COMMENTS src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java:230 do you need the 3rd argument? REVISION DETAIL https://reviews.facebook.net/D5853 To: Kannan, Karthik, JIRA, mbautin Cc: adela, Liyin, aaiyer, avf Number of records written/read per second on regionserver level --- Key: HBASE-6872 URL: https://issues.apache.org/jira/browse/HBASE-6872 Project: HBase Issue Type: New Feature Components: regionserver Reporter: Adela Maznikar Priority: Minor Attachments: D5853.1.patch Regionserver level metrics that shows the number of records written/read per second. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3793) HBASE-3468 Broke checkAndPut with null value
[
https://issues.apache.org/jira/browse/HBASE-3793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13469005#comment-13469005
]
Phabricator commented on HBASE-3793:
mbautin has closed the revision [jira] [HBASE-3793] [89-fb] Fix TestHRegion
failure with zero-length expected value in compare-and-put.
CHANGED PRIOR TO COMMIT
https://reviews.facebook.net/D5835?vs=19239id=19287#differential-review-toc
REVISION DETAIL
https://reviews.facebook.net/D5835
COMMIT
https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1393838
To: Liyin, Kannan, JIRA, mbautin
HBASE-3468 Broke checkAndPut with null value
Key: HBASE-3793
URL: https://issues.apache.org/jira/browse/HBASE-3793
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.92.0
Reporter: Lars George
Assignee: Ming Ma
Priority: Blocker
Fix For: 0.92.0
Attachments: D5835.1.patch, HBASE-3793.patch, HBASE-3793-TRUNK.patch
The previous code called Bytes.equal() which does a check for null on the
left or right argument. Now the comparator calls Bytes.compareTo() - which
has no check for null. But this is a valid input and checks for existence. I
actually noticed this running
https://github.com/larsgeorge/hbase-book/blob/master/ch04/src/main/java/client/CheckAndPutExample.java
This used to work, now it throws an NPE
{noformat}
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:854)
at
org.apache.hadoop.hbase.filter.WritableByteArrayComparable.compareTo(WritableByteArrayComparable.java:63)
at
org.apache.hadoop.hbase.regionserver.HRegion.checkAndMutate(HRegion.java:1681)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkAndMutate(HRegionServer.java:1693)
... 6 more
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1026)
at org.apache.hadoop.hbase.client.HTable.checkAndPut(HTable.java:750)
at client.CheckAndPutExample.main(CheckAndPutExample.java:33)
{noformat}
Easy fixable, just needs to handle the null value before even calling
comparator.compareTo().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6930) [89-fb] Avoid acquiring the same row lock repeatedly
[ https://issues.apache.org/jira/browse/HBASE-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13469004#comment-13469004 ] Phabricator commented on HBASE-6930: mbautin has closed the revision [jira] [HBASE-6930] [89-fb] Fix TestThriftServerLegacy: notifyAll should be inside synchronized block, and a null checkAndMutate expected value should be handled correctly. CHANGED PRIOR TO COMMIT https://reviews.facebook.net/D5841?vs=19251id=19293#differential-review-toc REVISION DETAIL https://reviews.facebook.net/D5841 COMMIT https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1393839 To: Kannan, Liyin, Karthik, JIRA, mbautin [89-fb] Avoid acquiring the same row lock repeatedly Key: HBASE-6930 URL: https://issues.apache.org/jira/browse/HBASE-6930 Project: HBase Issue Type: Bug Reporter: Liyin Tang Attachments: D5841.1.patch, D5841.2.patch When processing the multiPut, multiMutations or multiDelete operations, each IPC handler thread tries to acquire a lock for each row key in these batches. If there are duplicated row keys in these batches, previously the IPC handler thread will repeatedly acquire the same row key again and again. So the optimization is to sort each batch operation based on the row key in the client side, and skip acquiring the same row lock repeatedly in the server side. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6872) Number of records written/read per second on regionserver level
[ https://issues.apache.org/jira/browse/HBASE-6872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13469018#comment-13469018 ] Phabricator commented on HBASE-6872: Liyin has commented on the revision [jira] [HBASE-6872] [89-fb] Fix TestRegionServerMetrics.testNumReadsAndWrites. INLINE COMMENTS src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java:218 How about moving the counter reset logic inside of the doMetrics(). https://phabricator.fb.com/differential/diff/1485597/ REVISION DETAIL https://reviews.facebook.net/D5853 To: Kannan, Karthik, JIRA, mbautin Cc: adela, Liyin, aaiyer, avf Number of records written/read per second on regionserver level --- Key: HBASE-6872 URL: https://issues.apache.org/jira/browse/HBASE-6872 Project: HBase Issue Type: New Feature Components: regionserver Reporter: Adela Maznikar Priority: Minor Attachments: D5853.1.patch Regionserver level metrics that shows the number of records written/read per second. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6872) Number of records written/read per second on regionserver level
[ https://issues.apache.org/jira/browse/HBASE-6872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-6872: --- Attachment: D5853.2.patch mbautin updated the revision [jira] [HBASE-6872] [89-fb] Fix TestRegionServerMetrics.testNumReadsAndWrites. Reviewers: Kannan, Karthik, JIRA Addressing review comments REVISION DETAIL https://reviews.facebook.net/D5853 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java To: Kannan, Karthik, JIRA, mbautin Cc: adela, Liyin, aaiyer, avf Number of records written/read per second on regionserver level --- Key: HBASE-6872 URL: https://issues.apache.org/jira/browse/HBASE-6872 Project: HBase Issue Type: New Feature Components: regionserver Reporter: Adela Maznikar Priority: Minor Attachments: D5853.1.patch, D5853.2.patch Regionserver level metrics that shows the number of records written/read per second. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6872) Number of records written/read per second on regionserver level
[ https://issues.apache.org/jira/browse/HBASE-6872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-6872: --- Attachment: D5853.3.patch mbautin updated the revision [jira] [HBASE-6872] [89-fb] Fix TestRegionServerMetrics.testNumReadsAndWrites. Reviewers: Kannan, Karthik, JIRA Moving the reset of read/write counters to metrics() REVISION DETAIL https://reviews.facebook.net/D5853 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java To: Kannan, Karthik, JIRA, mbautin Cc: adela, Liyin, aaiyer, avf Number of records written/read per second on regionserver level --- Key: HBASE-6872 URL: https://issues.apache.org/jira/browse/HBASE-6872 Project: HBase Issue Type: New Feature Components: regionserver Reporter: Adela Maznikar Priority: Minor Attachments: D5853.1.patch, D5853.2.patch, D5853.3.patch Regionserver level metrics that shows the number of records written/read per second. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6872) Number of records written/read per second on regionserver level
[ https://issues.apache.org/jira/browse/HBASE-6872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13469046#comment-13469046 ] Phabricator commented on HBASE-6872: Liyin has accepted the revision [jira] [HBASE-6872] [89-fb] Fix TestRegionServerMetrics.testNumReadsAndWrites. Thanks Mikhail ! REVISION DETAIL https://reviews.facebook.net/D5853 BRANCH fix_region_server_metrics_v3 To: Kannan, Karthik, JIRA, Liyin, mbautin Cc: adela, Liyin, aaiyer, avf Number of records written/read per second on regionserver level --- Key: HBASE-6872 URL: https://issues.apache.org/jira/browse/HBASE-6872 Project: HBase Issue Type: New Feature Components: regionserver Reporter: Adela Maznikar Priority: Minor Attachments: D5853.1.patch, D5853.2.patch, D5853.3.patch Regionserver level metrics that shows the number of records written/read per second. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6882) Thrift IOError should include exception class
[ https://issues.apache.org/jira/browse/HBASE-6882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465103#comment-13465103 ] Phabricator commented on HBASE-6882: mbautin has closed the revision [jira] [HBASE-6882] [89-fb] Thrift IOError should include exception class. CHANGED PRIOR TO COMMIT https://reviews.facebook.net/D5679?vs=18669id=18843#differential-review-toc REVISION DETAIL https://reviews.facebook.net/D5679 COMMIT https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1391221 To: Liyin, Karthik, aaiyer, chip, JIRA, mbautin Thrift IOError should include exception class - Key: HBASE-6882 URL: https://issues.apache.org/jira/browse/HBASE-6882 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D5679.1.patch Return exception class as part of IOError thrown from the Thrift proxy or the embedded Thrift server in the regionserver. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6871) HFileBlockIndex Write Error BlockIndex in HFile V2
[ https://issues.apache.org/jira/browse/HBASE-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-6871: --- Attachment: D5703.1.patch mbautin requested code review of [jira] [HBASE-6871] [89-fb] Test case to reproduce block index corruption. Reviewers: lhofhansl, Kannan, Liyin, stack, JIRA A small test case to reproduce an incorrect HFile generated when an inline index chunk is promoted to root chunk under some circumstances. TEST PLAN Run test, ensure that it fails without the fix REVISION DETAIL https://reviews.facebook.net/D5703 AFFECTED FILES src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileInlineToRootChunkConversion.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/13389/ To: lhofhansl, Kannan, Liyin, stack, JIRA, mbautin HFileBlockIndex Write Error BlockIndex in HFile V2 -- Key: HBASE-6871 URL: https://issues.apache.org/jira/browse/HBASE-6871 Project: HBase Issue Type: Bug Components: HFile Affects Versions: 0.94.1 Environment: redhat 5u4 Reporter: Fenng Wang Priority: Critical Attachments: 428a400628ae412ca45d39fce15241fd.hfile, 787179746cc347ce9bb36f1989d17419.hfile, 960a026ca370464f84903ea58114bc75.hfile, d0026fa8d59b4df291718f59dd145aad.hfile, D5703.1.patch, hbase-6871-0.94.patch, ImportHFile.java, test_hfile_block_index.sh After writing some data, compaction and scan operation both failure, the exception message is below: 2012-09-18 06:32:26,227 ERROR org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: Compaction failed regionName=hfile_test,,1347778722498.d220df43fb9d8af4633bd7f547613f9e., storeName=page_info, fileCount=7, fileSize=1.3m (188.0k, 188.0k, 188.0k, 188.0k, 188.0k, 185.8k, 223.3k), priority=9, time=45826250816757428java.io.IOException: Could not reseek StoreFileScanner[HFileScanner for reader reader=hdfs://hadoopdev1.cm6:9000/hbase/hfile_test/d220df43fb9d8af4633bd7f547613f9e/page_info/b0f6118f58de47ad9d87cac438ee0895, compression=lzo, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], firstKey=http://com.truereligionbrandjeans.www/Womens_Dresses/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/4010.html/page_info:anchor_sig/1347764439449/DeleteColumn, lastKey=http://com.trura.www//page_info:page_type/1347763395089/Put, avgKeyLen=776, avgValueLen=4, entries=12853, length=228611, cur=http://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl/c/4970.html/page_info:is_deleted/1347764003865/Put/vlen=1/ts=0] to key http://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl/c/4970.html/page_info:is_deleted/OLDEST_TIMESTAMP/Minimum/vlen=0/ts=0 at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:178) at org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:54) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:299) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:244) at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:521) at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:402) at org.apache.hadoop.hbase.regionserver.Store.compactStore(Store.java:1570) at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:997) at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1216) at org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest.run(CompactionRequest.java:250) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at
[jira] [Commented] (HBASE-6871) HFileBlockIndex Write Error BlockIndex in HFile V2
[ https://issues.apache.org/jira/browse/HBASE-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13464150#comment-13464150 ] Phabricator commented on HBASE-6871: Kannan has accepted the revision [jira] [HBASE-6871] [89-fb] Test case to reproduce block index corruption. nice test! REVISION DETAIL https://reviews.facebook.net/D5703 BRANCH repro_interm_index_bug To: lhofhansl, Kannan, Liyin, stack, JIRA, mbautin HFileBlockIndex Write Error BlockIndex in HFile V2 -- Key: HBASE-6871 URL: https://issues.apache.org/jira/browse/HBASE-6871 Project: HBase Issue Type: Bug Components: HFile Affects Versions: 0.94.1 Environment: redhat 5u4 Reporter: Fenng Wang Priority: Critical Attachments: 428a400628ae412ca45d39fce15241fd.hfile, 787179746cc347ce9bb36f1989d17419.hfile, 960a026ca370464f84903ea58114bc75.hfile, d0026fa8d59b4df291718f59dd145aad.hfile, D5703.1.patch, hbase-6871-0.94.patch, ImportHFile.java, test_hfile_block_index.sh After writing some data, compaction and scan operation both failure, the exception message is below: 2012-09-18 06:32:26,227 ERROR org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: Compaction failed regionName=hfile_test,,1347778722498.d220df43fb9d8af4633bd7f547613f9e., storeName=page_info, fileCount=7, fileSize=1.3m (188.0k, 188.0k, 188.0k, 188.0k, 188.0k, 185.8k, 223.3k), priority=9, time=45826250816757428java.io.IOException: Could not reseek StoreFileScanner[HFileScanner for reader reader=hdfs://hadoopdev1.cm6:9000/hbase/hfile_test/d220df43fb9d8af4633bd7f547613f9e/page_info/b0f6118f58de47ad9d87cac438ee0895, compression=lzo, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], firstKey=http://com.truereligionbrandjeans.www/Womens_Dresses/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/4010.html/page_info:anchor_sig/1347764439449/DeleteColumn, lastKey=http://com.trura.www//page_info:page_type/1347763395089/Put, avgKeyLen=776, avgValueLen=4, entries=12853, length=228611, cur=http://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl/c/4970.html/page_info:is_deleted/1347764003865/Put/vlen=1/ts=0] to key http://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl/c/4970.html/page_info:is_deleted/OLDEST_TIMESTAMP/Minimum/vlen=0/ts=0 at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:178) at org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:54) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:299) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:244) at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:521) at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:402) at org.apache.hadoop.hbase.regionserver.Store.compactStore(Store.java:1570) at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:997) at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1216) at org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest.run(CompactionRequest.java:250) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: Expected block type LEAF_INDEX, but got INTERMEDIATE_INDEX: blockType=INTERMEDIATE_INDEX, onDiskSizeWithoutHeader=8514, uncompressedSizeWithoutHeader=131837, prevBlockOffset=-1, dataBeginsWith=\x00\x00\x00\x9B\x00\x00\x00\x00\x00\x00\x03#\x00\x00\x050\x00\x00\x08\xB7\x00\x00\x0Cr\x00\x00\x0F\xFA\x00\x00\x120, fileOffset=218942at
[jira] [Commented] (HBASE-6871) HFileBlockIndex Write Error BlockIndex in HFile V2
[ https://issues.apache.org/jira/browse/HBASE-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13464208#comment-13464208 ] Phabricator commented on HBASE-6871: stack has commented on the revision [jira] [HBASE-6871] [89-fb] Test case to reproduce block index corruption. Nice test Mikhail. You might consider explaining in a comment why 7 kvs of the type in the test bring on the issue (unless you think it fine just pointing at the issue). REVISION DETAIL https://reviews.facebook.net/D5703 BRANCH repro_interm_index_bug To: lhofhansl, Kannan, Liyin, stack, JIRA, mbautin HFileBlockIndex Write Error BlockIndex in HFile V2 -- Key: HBASE-6871 URL: https://issues.apache.org/jira/browse/HBASE-6871 Project: HBase Issue Type: Bug Components: HFile Affects Versions: 0.94.1 Environment: redhat 5u4 Reporter: Fenng Wang Priority: Critical Fix For: 0.94.3, 0.96.0 Attachments: 428a400628ae412ca45d39fce15241fd.hfile, 787179746cc347ce9bb36f1989d17419.hfile, 960a026ca370464f84903ea58114bc75.hfile, d0026fa8d59b4df291718f59dd145aad.hfile, D5703.1.patch, hbase-6871-0.94.patch, ImportHFile.java, test_hfile_block_index.sh After writing some data, compaction and scan operation both failure, the exception message is below: 2012-09-18 06:32:26,227 ERROR org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: Compaction failed regionName=hfile_test,,1347778722498.d220df43fb9d8af4633bd7f547613f9e., storeName=page_info, fileCount=7, fileSize=1.3m (188.0k, 188.0k, 188.0k, 188.0k, 188.0k, 185.8k, 223.3k), priority=9, time=45826250816757428java.io.IOException: Could not reseek StoreFileScanner[HFileScanner for reader reader=hdfs://hadoopdev1.cm6:9000/hbase/hfile_test/d220df43fb9d8af4633bd7f547613f9e/page_info/b0f6118f58de47ad9d87cac438ee0895, compression=lzo, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], firstKey=http://com.truereligionbrandjeans.www/Womens_Dresses/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/4010.html/page_info:anchor_sig/1347764439449/DeleteColumn, lastKey=http://com.trura.www//page_info:page_type/1347763395089/Put, avgKeyLen=776, avgValueLen=4, entries=12853, length=228611, cur=http://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl/c/4970.html/page_info:is_deleted/1347764003865/Put/vlen=1/ts=0] to key http://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl/c/4970.html/page_info:is_deleted/OLDEST_TIMESTAMP/Minimum/vlen=0/ts=0 at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:178) at org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:54) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:299) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:244) at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:521) at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:402) at org.apache.hadoop.hbase.regionserver.Store.compactStore(Store.java:1570) at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:997) at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1216) at org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest.run(CompactionRequest.java:250) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: Expected block type LEAF_INDEX, but got INTERMEDIATE_INDEX: blockType=INTERMEDIATE_INDEX, onDiskSizeWithoutHeader=8514,
[jira] [Updated] (HBASE-6871) HFileBlockIndex Write Error BlockIndex in HFile V2
[ https://issues.apache.org/jira/browse/HBASE-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-6871: --- Attachment: D5703.2.patch mbautin updated the revision [jira] [HBASE-6871] [89-fb] Test case to reproduce block index corruption. Reviewers: lhofhansl, Kannan, Liyin, stack, JIRA Addressing Michael's comments. REVISION DETAIL https://reviews.facebook.net/D5703 AFFECTED FILES src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileInlineToRootChunkConversion.java To: lhofhansl, Kannan, Liyin, stack, JIRA, mbautin HFileBlockIndex Write Error BlockIndex in HFile V2 -- Key: HBASE-6871 URL: https://issues.apache.org/jira/browse/HBASE-6871 Project: HBase Issue Type: Bug Components: HFile Affects Versions: 0.94.1 Environment: redhat 5u4 Reporter: Fenng Wang Priority: Critical Fix For: 0.94.3, 0.96.0 Attachments: 428a400628ae412ca45d39fce15241fd.hfile, 787179746cc347ce9bb36f1989d17419.hfile, 960a026ca370464f84903ea58114bc75.hfile, d0026fa8d59b4df291718f59dd145aad.hfile, D5703.1.patch, D5703.2.patch, hbase-6871-0.94.patch, ImportHFile.java, test_hfile_block_index.sh After writing some data, compaction and scan operation both failure, the exception message is below: 2012-09-18 06:32:26,227 ERROR org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: Compaction failed regionName=hfile_test,,1347778722498.d220df43fb9d8af4633bd7f547613f9e., storeName=page_info, fileCount=7, fileSize=1.3m (188.0k, 188.0k, 188.0k, 188.0k, 188.0k, 185.8k, 223.3k), priority=9, time=45826250816757428java.io.IOException: Could not reseek StoreFileScanner[HFileScanner for reader reader=hdfs://hadoopdev1.cm6:9000/hbase/hfile_test/d220df43fb9d8af4633bd7f547613f9e/page_info/b0f6118f58de47ad9d87cac438ee0895, compression=lzo, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], firstKey=http://com.truereligionbrandjeans.www/Womens_Dresses/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/4010.html/page_info:anchor_sig/1347764439449/DeleteColumn, lastKey=http://com.trura.www//page_info:page_type/1347763395089/Put, avgKeyLen=776, avgValueLen=4, entries=12853, length=228611, cur=http://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl/c/4970.html/page_info:is_deleted/1347764003865/Put/vlen=1/ts=0] to key http://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl/c/4970.html/page_info:is_deleted/OLDEST_TIMESTAMP/Minimum/vlen=0/ts=0 at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:178) at org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:54) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:299) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:244) at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:521) at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:402) at org.apache.hadoop.hbase.regionserver.Store.compactStore(Store.java:1570) at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:997) at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1216) at org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest.run(CompactionRequest.java:250) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: Expected block type LEAF_INDEX, but got INTERMEDIATE_INDEX: blockType=INTERMEDIATE_INDEX, onDiskSizeWithoutHeader=8514, uncompressedSizeWithoutHeader=131837, prevBlockOffset=-1,
[jira] [Updated] (HBASE-6871) HFileBlockIndex Write Error BlockIndex in HFile V2
[ https://issues.apache.org/jira/browse/HBASE-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-6871: --- Attachment: D5703.3.patch mbautin updated the revision [jira] [HBASE-6871] [89-fb] Test case to reproduce block index corruption. Reviewers: lhofhansl, Kannan, Liyin, stack, JIRA Fix the comment REVISION DETAIL https://reviews.facebook.net/D5703 AFFECTED FILES src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileInlineToRootChunkConversion.java To: lhofhansl, Kannan, Liyin, stack, JIRA, mbautin HFileBlockIndex Write Error BlockIndex in HFile V2 -- Key: HBASE-6871 URL: https://issues.apache.org/jira/browse/HBASE-6871 Project: HBase Issue Type: Bug Components: HFile Affects Versions: 0.94.1 Environment: redhat 5u4 Reporter: Fenng Wang Priority: Critical Fix For: 0.94.3, 0.96.0 Attachments: 428a400628ae412ca45d39fce15241fd.hfile, 787179746cc347ce9bb36f1989d17419.hfile, 960a026ca370464f84903ea58114bc75.hfile, d0026fa8d59b4df291718f59dd145aad.hfile, D5703.1.patch, D5703.2.patch, D5703.3.patch, hbase-6871-0.94.patch, ImportHFile.java, test_hfile_block_index.sh After writing some data, compaction and scan operation both failure, the exception message is below: 2012-09-18 06:32:26,227 ERROR org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: Compaction failed regionName=hfile_test,,1347778722498.d220df43fb9d8af4633bd7f547613f9e., storeName=page_info, fileCount=7, fileSize=1.3m (188.0k, 188.0k, 188.0k, 188.0k, 188.0k, 185.8k, 223.3k), priority=9, time=45826250816757428java.io.IOException: Could not reseek StoreFileScanner[HFileScanner for reader reader=hdfs://hadoopdev1.cm6:9000/hbase/hfile_test/d220df43fb9d8af4633bd7f547613f9e/page_info/b0f6118f58de47ad9d87cac438ee0895, compression=lzo, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], firstKey=http://com.truereligionbrandjeans.www/Womens_Dresses/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/4010.html/page_info:anchor_sig/1347764439449/DeleteColumn, lastKey=http://com.trura.www//page_info:page_type/1347763395089/Put, avgKeyLen=776, avgValueLen=4, entries=12853, length=228611, cur=http://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl/c/4970.html/page_info:is_deleted/1347764003865/Put/vlen=1/ts=0] to key http://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl/c/4970.html/page_info:is_deleted/OLDEST_TIMESTAMP/Minimum/vlen=0/ts=0 at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:178) at org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:54) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:299) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:244) at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:521) at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:402) at org.apache.hadoop.hbase.regionserver.Store.compactStore(Store.java:1570) at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:997) at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1216) at org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest.run(CompactionRequest.java:250) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: Expected block type LEAF_INDEX, but got INTERMEDIATE_INDEX: blockType=INTERMEDIATE_INDEX, onDiskSizeWithoutHeader=8514, uncompressedSizeWithoutHeader=131837, prevBlockOffset=-1,
[jira] [Updated] (HBASE-6871) HFileBlockIndex Write Error BlockIndex in HFile V2
[ https://issues.apache.org/jira/browse/HBASE-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-6871: --- Attachment: D5703.4.patch mbautin updated the revision [jira] [HBASE-6871] [89-fb] Test case to reproduce block index corruption. Reviewers: lhofhansl, Kannan, Liyin, stack, JIRA Fixing confusing loop REVISION DETAIL https://reviews.facebook.net/D5703 AFFECTED FILES src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileInlineToRootChunkConversion.java To: lhofhansl, Kannan, Liyin, stack, JIRA, mbautin HFileBlockIndex Write Error BlockIndex in HFile V2 -- Key: HBASE-6871 URL: https://issues.apache.org/jira/browse/HBASE-6871 Project: HBase Issue Type: Bug Components: HFile Affects Versions: 0.94.1 Environment: redhat 5u4 Reporter: Fenng Wang Priority: Critical Fix For: 0.94.3, 0.96.0 Attachments: 428a400628ae412ca45d39fce15241fd.hfile, 787179746cc347ce9bb36f1989d17419.hfile, 960a026ca370464f84903ea58114bc75.hfile, d0026fa8d59b4df291718f59dd145aad.hfile, D5703.1.patch, D5703.2.patch, D5703.3.patch, D5703.4.patch, hbase-6871-0.94.patch, ImportHFile.java, test_hfile_block_index.sh After writing some data, compaction and scan operation both failure, the exception message is below: 2012-09-18 06:32:26,227 ERROR org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: Compaction failed regionName=hfile_test,,1347778722498.d220df43fb9d8af4633bd7f547613f9e., storeName=page_info, fileCount=7, fileSize=1.3m (188.0k, 188.0k, 188.0k, 188.0k, 188.0k, 185.8k, 223.3k), priority=9, time=45826250816757428java.io.IOException: Could not reseek StoreFileScanner[HFileScanner for reader reader=hdfs://hadoopdev1.cm6:9000/hbase/hfile_test/d220df43fb9d8af4633bd7f547613f9e/page_info/b0f6118f58de47ad9d87cac438ee0895, compression=lzo, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], firstKey=http://com.truereligionbrandjeans.www/Womens_Dresses/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/4010.html/page_info:anchor_sig/1347764439449/DeleteColumn, lastKey=http://com.trura.www//page_info:page_type/1347763395089/Put, avgKeyLen=776, avgValueLen=4, entries=12853, length=228611, cur=http://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl/c/4970.html/page_info:is_deleted/1347764003865/Put/vlen=1/ts=0] to key http://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl/c/4970.html/page_info:is_deleted/OLDEST_TIMESTAMP/Minimum/vlen=0/ts=0 at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:178) at org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:54) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:299) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:244) at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:521) at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:402) at org.apache.hadoop.hbase.regionserver.Store.compactStore(Store.java:1570) at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:997) at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1216) at org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest.run(CompactionRequest.java:250) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: Expected block type LEAF_INDEX, but got INTERMEDIATE_INDEX: blockType=INTERMEDIATE_INDEX, onDiskSizeWithoutHeader=8514, uncompressedSizeWithoutHeader=131837,
[jira] [Updated] (HBASE-6871) HFileBlockIndex Write Error BlockIndex in HFile V2
[ https://issues.apache.org/jira/browse/HBASE-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-6871: --- Attachment: D5703.5.patch mbautin updated the revision [jira] [HBASE-6871] [89-fb] Block index corruption test case and fix. Reviewers: lhofhansl, Kannan, Liyin, stack, JIRA Adding the bugfix. REVISION DETAIL https://reviews.facebook.net/D5703 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileInlineToRootChunkConversion.java To: lhofhansl, Kannan, Liyin, stack, JIRA, mbautin HFileBlockIndex Write Error BlockIndex in HFile V2 -- Key: HBASE-6871 URL: https://issues.apache.org/jira/browse/HBASE-6871 Project: HBase Issue Type: Bug Components: HFile Affects Versions: 0.94.1 Environment: redhat 5u4 Reporter: Fenng Wang Priority: Critical Fix For: 0.94.3, 0.96.0 Attachments: 428a400628ae412ca45d39fce15241fd.hfile, 787179746cc347ce9bb36f1989d17419.hfile, 960a026ca370464f84903ea58114bc75.hfile, d0026fa8d59b4df291718f59dd145aad.hfile, D5703.1.patch, D5703.2.patch, D5703.3.patch, D5703.4.patch, D5703.5.patch, hbase-6871-0.94.patch, ImportHFile.java, test_hfile_block_index.sh After writing some data, compaction and scan operation both failure, the exception message is below: 2012-09-18 06:32:26,227 ERROR org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: Compaction failed regionName=hfile_test,,1347778722498.d220df43fb9d8af4633bd7f547613f9e., storeName=page_info, fileCount=7, fileSize=1.3m (188.0k, 188.0k, 188.0k, 188.0k, 188.0k, 185.8k, 223.3k), priority=9, time=45826250816757428java.io.IOException: Could not reseek StoreFileScanner[HFileScanner for reader reader=hdfs://hadoopdev1.cm6:9000/hbase/hfile_test/d220df43fb9d8af4633bd7f547613f9e/page_info/b0f6118f58de47ad9d87cac438ee0895, compression=lzo, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], firstKey=http://com.truereligionbrandjeans.www/Womens_Dresses/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/4010.html/page_info:anchor_sig/1347764439449/DeleteColumn, lastKey=http://com.trura.www//page_info:page_type/1347763395089/Put, avgKeyLen=776, avgValueLen=4, entries=12853, length=228611, cur=http://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl/c/4970.html/page_info:is_deleted/1347764003865/Put/vlen=1/ts=0] to key http://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl/c/4970.html/page_info:is_deleted/OLDEST_TIMESTAMP/Minimum/vlen=0/ts=0 at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:178) at org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:54) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:299) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:244) at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:521) at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:402) at org.apache.hadoop.hbase.regionserver.Store.compactStore(Store.java:1570) at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:997) at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1216) at org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest.run(CompactionRequest.java:250) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: Expected block type LEAF_INDEX, but got INTERMEDIATE_INDEX:
[jira] [Commented] (HBASE-6871) HFileBlockIndex Write Error BlockIndex in HFile V2
[ https://issues.apache.org/jira/browse/HBASE-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13464265#comment-13464265 ] Phabricator commented on HBASE-6871: lhofhansl has commented on the revision [jira] [HBASE-6871] [89-fb] Block index corruption test case and fix. Can't pretend to fully understands what going on in the test, but if it reproduces the problem that's perfect. Thanks for doing this Mikhail. The fix is different from the one proposed on the issue. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java:988 Are these expected exceptional cases? If not, should we have asserts instead? REVISION DETAIL https://reviews.facebook.net/D5703 BRANCH repro_interm_index_bug_v7 To: lhofhansl, Kannan, Liyin, stack, JIRA, mbautin HFileBlockIndex Write Error BlockIndex in HFile V2 -- Key: HBASE-6871 URL: https://issues.apache.org/jira/browse/HBASE-6871 Project: HBase Issue Type: Bug Components: HFile Affects Versions: 0.94.1 Environment: redhat 5u4 Reporter: Fenng Wang Priority: Critical Fix For: 0.94.3, 0.96.0 Attachments: 428a400628ae412ca45d39fce15241fd.hfile, 787179746cc347ce9bb36f1989d17419.hfile, 960a026ca370464f84903ea58114bc75.hfile, d0026fa8d59b4df291718f59dd145aad.hfile, D5703.1.patch, D5703.2.patch, D5703.3.patch, D5703.4.patch, D5703.5.patch, hbase-6871-0.94.patch, ImportHFile.java, test_hfile_block_index.sh After writing some data, compaction and scan operation both failure, the exception message is below: 2012-09-18 06:32:26,227 ERROR org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: Compaction failed regionName=hfile_test,,1347778722498.d220df43fb9d8af4633bd7f547613f9e., storeName=page_info, fileCount=7, fileSize=1.3m (188.0k, 188.0k, 188.0k, 188.0k, 188.0k, 185.8k, 223.3k), priority=9, time=45826250816757428java.io.IOException: Could not reseek StoreFileScanner[HFileScanner for reader reader=hdfs://hadoopdev1.cm6:9000/hbase/hfile_test/d220df43fb9d8af4633bd7f547613f9e/page_info/b0f6118f58de47ad9d87cac438ee0895, compression=lzo, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], firstKey=http://com.truereligionbrandjeans.www/Womens_Dresses/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/4010.html/page_info:anchor_sig/1347764439449/DeleteColumn, lastKey=http://com.trura.www//page_info:page_type/1347763395089/Put, avgKeyLen=776, avgValueLen=4, entries=12853, length=228611, cur=http://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl/c/4970.html/page_info:is_deleted/1347764003865/Put/vlen=1/ts=0] to key http://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl/c/4970.html/page_info:is_deleted/OLDEST_TIMESTAMP/Minimum/vlen=0/ts=0 at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:178) at org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:54) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:299) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:244) at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:521) at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:402) at org.apache.hadoop.hbase.regionserver.Store.compactStore(Store.java:1570) at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:997) at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1216) at org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest.run(CompactionRequest.java:250) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at
[jira] [Commented] (HBASE-6871) HFileBlockIndex Write Error BlockIndex in HFile V2
[ https://issues.apache.org/jira/browse/HBASE-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13464350#comment-13464350 ] Phabricator commented on HBASE-6871: mbautin has commented on the revision [jira] [HBASE-6871] [89-fb] Block index corruption test case and fix. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java:988 I could have thrown AssertionErrors but I think these exceptions are more specific. If the index writer is configured as a single-level-only, then the operation of writing inline blocks is unsupported. If curInlineChunk is nul, that means this function has been called with closing=true and calling it again in this state is illegal. REVISION DETAIL https://reviews.facebook.net/D5703 BRANCH repro_interm_index_bug_v7 To: lhofhansl, Kannan, Liyin, stack, JIRA, mbautin HFileBlockIndex Write Error BlockIndex in HFile V2 -- Key: HBASE-6871 URL: https://issues.apache.org/jira/browse/HBASE-6871 Project: HBase Issue Type: Bug Components: HFile Affects Versions: 0.94.1 Environment: redhat 5u4 Reporter: Fenng Wang Priority: Critical Fix For: 0.94.3, 0.96.0 Attachments: 428a400628ae412ca45d39fce15241fd.hfile, 787179746cc347ce9bb36f1989d17419.hfile, 960a026ca370464f84903ea58114bc75.hfile, d0026fa8d59b4df291718f59dd145aad.hfile, D5703.1.patch, D5703.2.patch, D5703.3.patch, D5703.4.patch, D5703.5.patch, hbase-6871-0.94.patch, ImportHFile.java, test_hfile_block_index.sh After writing some data, compaction and scan operation both failure, the exception message is below: 2012-09-18 06:32:26,227 ERROR org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: Compaction failed regionName=hfile_test,,1347778722498.d220df43fb9d8af4633bd7f547613f9e., storeName=page_info, fileCount=7, fileSize=1.3m (188.0k, 188.0k, 188.0k, 188.0k, 188.0k, 185.8k, 223.3k), priority=9, time=45826250816757428java.io.IOException: Could not reseek StoreFileScanner[HFileScanner for reader reader=hdfs://hadoopdev1.cm6:9000/hbase/hfile_test/d220df43fb9d8af4633bd7f547613f9e/page_info/b0f6118f58de47ad9d87cac438ee0895, compression=lzo, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], firstKey=http://com.truereligionbrandjeans.www/Womens_Dresses/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/4010.html/page_info:anchor_sig/1347764439449/DeleteColumn, lastKey=http://com.trura.www//page_info:page_type/1347763395089/Put, avgKeyLen=776, avgValueLen=4, entries=12853, length=228611, cur=http://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl/c/4970.html/page_info:is_deleted/1347764003865/Put/vlen=1/ts=0] to key http://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl/c/4970.html/page_info:is_deleted/OLDEST_TIMESTAMP/Minimum/vlen=0/ts=0 at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:178) at org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:54) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:299) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:244) at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:521) at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:402) at org.apache.hadoop.hbase.regionserver.Store.compactStore(Store.java:1570) at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:997) at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1216) at org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest.run(CompactionRequest.java:250) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at
[jira] [Commented] (HBASE-6871) HFileBlockIndex Write Error BlockIndex in HFile V2
[ https://issues.apache.org/jira/browse/HBASE-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13464390#comment-13464390 ] Phabricator commented on HBASE-6871: mbautin has closed the revision [jira] [HBASE-6871] [89-fb] Block index corruption test case and fix. CHANGED PRIOR TO COMMIT https://reviews.facebook.net/D5703?vs=18759id=18765#differential-review-toc REVISION DETAIL https://reviews.facebook.net/D5703 COMMIT https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1390819 To: lhofhansl, Kannan, Liyin, stack, JIRA, mbautin HFileBlockIndex Write Error BlockIndex in HFile V2 -- Key: HBASE-6871 URL: https://issues.apache.org/jira/browse/HBASE-6871 Project: HBase Issue Type: Bug Components: HFile Affects Versions: 0.94.1 Environment: redhat 5u4 Reporter: Fenng Wang Priority: Critical Fix For: 0.94.3, 0.96.0 Attachments: 428a400628ae412ca45d39fce15241fd.hfile, 787179746cc347ce9bb36f1989d17419.hfile, 960a026ca370464f84903ea58114bc75.hfile, d0026fa8d59b4df291718f59dd145aad.hfile, D5703.1.patch, D5703.2.patch, D5703.3.patch, D5703.4.patch, D5703.5.patch, hbase-6871-0.94.patch, ImportHFile.java, test_hfile_block_index.sh After writing some data, compaction and scan operation both failure, the exception message is below: 2012-09-18 06:32:26,227 ERROR org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: Compaction failed regionName=hfile_test,,1347778722498.d220df43fb9d8af4633bd7f547613f9e., storeName=page_info, fileCount=7, fileSize=1.3m (188.0k, 188.0k, 188.0k, 188.0k, 188.0k, 185.8k, 223.3k), priority=9, time=45826250816757428java.io.IOException: Could not reseek StoreFileScanner[HFileScanner for reader reader=hdfs://hadoopdev1.cm6:9000/hbase/hfile_test/d220df43fb9d8af4633bd7f547613f9e/page_info/b0f6118f58de47ad9d87cac438ee0895, compression=lzo, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], firstKey=http://com.truereligionbrandjeans.www/Womens_Dresses/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/4010.html/page_info:anchor_sig/1347764439449/DeleteColumn, lastKey=http://com.trura.www//page_info:page_type/1347763395089/Put, avgKeyLen=776, avgValueLen=4, entries=12853, length=228611, cur=http://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl/c/4970.html/page_info:is_deleted/1347764003865/Put/vlen=1/ts=0] to key http://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl/c/4970.html/page_info:is_deleted/OLDEST_TIMESTAMP/Minimum/vlen=0/ts=0 at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:178) at org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:54) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:299) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:244) at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:521) at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:402) at org.apache.hadoop.hbase.regionserver.Store.compactStore(Store.java:1570) at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:997) at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1216) at org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest.run(CompactionRequest.java:250) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: Expected block type LEAF_INDEX, but got INTERMEDIATE_INDEX: blockType=INTERMEDIATE_INDEX, onDiskSizeWithoutHeader=8514,
[jira] [Updated] (HBASE-6882) Thrift IOError should include exception class
[ https://issues.apache.org/jira/browse/HBASE-6882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-6882: --- Attachment: D5679.1.patch mbautin requested code review of [jira] [HBASE-6882] [89-fb] Thrift IOError should include exception class. Reviewers: Liyin, Karthik, aaiyer, chip, JIRA Return exception class as part of IOError thrown from the Thrift proxy or the embedded Thrift server in the regionserver. TEST PLAN Unit tests Test through C++ HBase client REVISION DETAIL https://reviews.facebook.net/D5679 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/RegionException.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegionThriftServer.java src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java src/main/java/org/apache/hadoop/hbase/thrift/generated/IOError.java src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/13341/ To: Liyin, Karthik, aaiyer, chip, JIRA, mbautin Thrift IOError should include exception class - Key: HBASE-6882 URL: https://issues.apache.org/jira/browse/HBASE-6882 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D5679.1.patch Return exception class as part of IOError thrown from the Thrift proxy or the embedded Thrift server in the regionserver. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6882) Thrift IOError should include exception class
[ https://issues.apache.org/jira/browse/HBASE-6882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13463294#comment-13463294 ] Phabricator commented on HBASE-6882: Liyin has accepted the revision [jira] [HBASE-6882] [89-fb] Thrift IOError should include exception class. LGTM ! REVISION DETAIL https://reviews.facebook.net/D5679 BRANCH ioerror_class_name To: Liyin, Karthik, aaiyer, chip, JIRA, mbautin Thrift IOError should include exception class - Key: HBASE-6882 URL: https://issues.apache.org/jira/browse/HBASE-6882 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D5679.1.patch Return exception class as part of IOError thrown from the Thrift proxy or the embedded Thrift server in the regionserver. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6619) Do no unregister and re-register interest ops in RPC
[ https://issues.apache.org/jira/browse/HBASE-6619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13461142#comment-13461142 ] Phabricator commented on HBASE-6619: mbautin has closed the revision [jira] [HBASE-6619] [89-fb] Do no unregister and re-register interest ops in RPC. CHANGED PRIOR TO COMMIT https://reviews.facebook.net/D5283?vs=18369id=18417#differential-review-toc REVISION DETAIL https://reviews.facebook.net/D5283 COMMIT https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1388798 To: Karthik, michalgr Cc: JIRA, Kannan, aaiyer, avf, mbautin, Liyin, gqchen Do no unregister and re-register interest ops in RPC Key: HBASE-6619 URL: https://issues.apache.org/jira/browse/HBASE-6619 Project: HBase Issue Type: Bug Components: IPC/RPC Reporter: Karthik Ranganathan Assignee: Michal Gregorczyk While investigating perf of HBase, Michal noticed that we could cut about 5-40% (depending on number of threads) from the total get time in the RPC on the server side if we eliminated re-registering for interest ops. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13278600#comment-13278600 ] Phabricator commented on HBASE-5776: Kannan has accepted the revision [jira][89-fb][HBASE-5776] HTableMultiplexer. looks good Liyin. Remaining are cosmetic comments, hence accepting! INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1731 space after to src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:174 failed is unused src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:159 failed is unused src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:359 -1 - add space between - and 1. REVISION DETAIL https://reviews.facebook.net/D2775 BRANCH HBASE-5776 To: Kannan, Liyin Cc: JIRA, tedyu HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, D2775.2.patch, D2775.3.patch, D2775.4.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-5776: --- Attachment: D2775.5.patch Liyin updated the revision [jira][89-fb][HBASE-5776] HTableMultiplexer. Reviewers: Kannan Addressed Kannan's comments REVISION DETAIL https://reviews.facebook.net/D2775 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/HConstants.java src/main/java/org/apache/hadoop/hbase/client/HConnection.java src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java src/main/java/org/apache/hadoop/hbase/client/HTable.java src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java src/main/java/org/apache/hadoop/hbase/client/MultiPut.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java To: Kannan, Liyin Cc: JIRA, tedyu HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, D2775.2.patch, D2775.3.patch, D2775.4.patch, D2775.5.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13279084#comment-13279084 ] Phabricator commented on HBASE-5776: Liyin has closed the revision [jira][89-fb][HBASE-5776] HTableMultiplexer. Close this 89-fb revision and will port to apache trunk soon. REVISION DETAIL https://reviews.facebook.net/D2775 To: Kannan, Liyin Cc: JIRA, tedyu HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, D2775.2.patch, D2775.3.patch, D2775.4.patch, D2775.5.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6007) Make getTableRegions return an empty list if the table does not exist
[ https://issues.apache.org/jira/browse/HBASE-6007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13279138#comment-13279138 ] Phabricator commented on HBASE-6007: stack has commented on the revision [jira] [HBASE-6007] [89-fb] Make getTableRegions return an empty list if the table does not exist. +1 REVISION DETAIL https://reviews.facebook.net/D3243 To: Kannan, Liyin, JIRA, mbautin Cc: stack Make getTableRegions return an empty list if the table does not exist - Key: HBASE-6007 URL: https://issues.apache.org/jira/browse/HBASE-6007 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Priority: Minor Attachments: D3243.1.patch Making the getTableRegions Thrift API method handle TableNotFoundException and return an empty list in that case. Without this the behavior is dependent on whether an HTable object is present in the thread-local cache in case a table was deleted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6007) Make getTableRegions return an empty list if the table does not exist
[ https://issues.apache.org/jira/browse/HBASE-6007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13279153#comment-13279153 ] Phabricator commented on HBASE-6007: Liyin has accepted the revision [jira] [HBASE-6007] [89-fb] Make getTableRegions return an empty list if the table does not exist. +1, Thanks Mikhail ! REVISION DETAIL https://reviews.facebook.net/D3243 BRANCH fix_get_table_regions To: Kannan, Liyin, JIRA, mbautin Cc: stack Make getTableRegions return an empty list if the table does not exist - Key: HBASE-6007 URL: https://issues.apache.org/jira/browse/HBASE-6007 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Priority: Minor Attachments: D3243.1.patch Making the getTableRegions Thrift API method handle TableNotFoundException and return an empty list in that case. Without this the behavior is dependent on whether an HTable object is present in the thread-local cache in case a table was deleted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6007) Make getTableRegions return an empty list if the table does not exist
[ https://issues.apache.org/jira/browse/HBASE-6007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13279163#comment-13279163 ] Phabricator commented on HBASE-6007: stack has commented on the revision [jira] [HBASE-6007] [89-fb] Make getTableRegions return an empty list if the table does not exist. Want to apply to trunk to Liyin? REVISION DETAIL https://reviews.facebook.net/D3243 BRANCH fix_get_table_regions To: Kannan, Liyin, JIRA, mbautin Cc: stack Make getTableRegions return an empty list if the table does not exist - Key: HBASE-6007 URL: https://issues.apache.org/jira/browse/HBASE-6007 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Priority: Minor Attachments: D3243.1.patch Making the getTableRegions Thrift API method handle TableNotFoundException and return an empty list in that case. Without this the behavior is dependent on whether an HTable object is present in the thread-local cache in case a table was deleted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6007) Make getTableRegions return an empty list if the table does not exist
[ https://issues.apache.org/jira/browse/HBASE-6007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13279356#comment-13279356 ] Phabricator commented on HBASE-6007: mbautin has closed the revision [jira] [HBASE-6007] [89-fb] Make getTableRegions return an empty list if the table does not exist. REVISION DETAIL https://reviews.facebook.net/D3243 COMMIT https://reviews.facebook.net/rHBASE1340310 To: Kannan, Liyin, JIRA, mbautin Cc: stack Make getTableRegions return an empty list if the table does not exist - Key: HBASE-6007 URL: https://issues.apache.org/jira/browse/HBASE-6007 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Attachments: D3243.1.patch, jira-HBASE-6007-Make-getTableRegions-return-an-empty-2012-05-18_17_02_40.patch Making the getTableRegions Thrift API method handle TableNotFoundException and return an empty list in that case. Without this the behavior is dependent on whether an HTable object is present in the thread-local cache in case a table was deleted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6007) Make getTableRegions return an empty list if the table does not exist
[ https://issues.apache.org/jira/browse/HBASE-6007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13279354#comment-13279354 ] Phabricator commented on HBASE-6007: mbautin has commented on the revision [jira] [HBASE-6007] [89-fb] Make getTableRegions return an empty list if the table does not exist. Michael: I have committed this into trunk. REVISION DETAIL https://reviews.facebook.net/D3243 BRANCH fix_get_table_regions To: Kannan, Liyin, JIRA, mbautin Cc: stack Make getTableRegions return an empty list if the table does not exist - Key: HBASE-6007 URL: https://issues.apache.org/jira/browse/HBASE-6007 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Attachments: D3243.1.patch, jira-HBASE-6007-Make-getTableRegions-return-an-empty-2012-05-18_17_02_40.patch Making the getTableRegions Thrift API method handle TableNotFoundException and return an empty list in that case. Without this the behavior is dependent on whether an HTable object is present in the thread-local cache in case a table was deleted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5987) HFileBlockIndex improvement
[ https://issues.apache.org/jira/browse/HBASE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1327#comment-1327 ] Phabricator commented on HBASE-5987: mbautin has closed the revision [jira][89-fb] [HBASE-5987] HFileBlockIndex improvement. REVISION DETAIL https://reviews.facebook.net/D3237 COMMIT https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1339581 To: Kannan, mbautin, Liyin Cc: JIRA, todd, tedyu HFileBlockIndex improvement --- Key: HBASE-5987 URL: https://issues.apache.org/jira/browse/HBASE-5987 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D3237.1.patch, D3237.2.patch, D3237.3.patch, D3237.4.patch, D3237.5.patch, D3237.6.patch, D3237.7.patch, D3237.8.patch, screen_shot_of_sequential_scan_profiling.png Recently we find out a performance problem that it is quite slow when multiple requests are reading the same block of data or index. From the profiling, one of the causes is the IdLock contention which has been addressed in HBASE-5898. Another issue is that the HFileScanner will keep asking the HFileBlockIndex about the data block location for each target key value during the scan process(reSeekTo), even though the target key value has already been in the current data block. This issue will cause certain index block very HOT, especially when it is a sequential scan. To solve this issue, we propose the following solutions: First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is smaller than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is bigger, then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup. Secondary, we propose to push this idea a little further that the HFileBlockIndex shall index on the last key value of each data block instead of indexing on the start key value. The motivation is to solve the HBASE-4443 issue (avoid seeking to previous block when key you are interested in is the first one of a block) as well as +the defects mentioned above+. For example, if the target key value is smaller than the start key value of the data block N. There is no way for sure the target key value is in the data block N or N-1. So it has to seek from data block N-1. However, if the block index is based on the last key value for each data block and the target key value is beween the last key value of data block N-1 and data block N, then the target key value is supposed be data block N for sure. As long as HBase only supports the forward scan, the last key value makes more sense to be indexed on than the start key value. Thanks Kannan and Mikhail for the insightful discussions and suggestions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-5959: --- Attachment: HBASE-5959.D3189.7.patch eclark updated the revision HBASE-5959 [jira] Add other load balancers. Reviewers: JIRA Rebase to current trunk REVISION DETAIL https://reviews.facebook.net/D3189 AFFECTED FILES pom.xml src/main/java/org/apache/hadoop/hbase/master/HMaster.java src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/RegionPlan.java src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/balancer/ClusterLoadState.java src/main/java/org/apache/hadoop/hbase/master/balancer/DefaultLoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerFactory.java src/main/java/org/apache/hadoop/hbase/master/LoadBalancerFactory.java src/main/java/org/apache/hadoop/hbase/master/balancer/RegionInfoComparator.java src/main/java/org/apache/hadoop/hbase/master/ServerAndLoad.java src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java src/main/java/org/apache/hadoop/hbase/master/balancer/ServerAndLoad.java src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java To: JIRA, eclark Cc: tedyu Add other load balancers Key: HBASE-5959 URL: https://issues.apache.org/jira/browse/HBASE-5959 Project: HBase Issue Type: New Feature Components: master Affects Versions: 0.96.0 Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, HBASE-5959-7.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch, HBASE-5959.D3189.5.patch, HBASE-5959.D3189.6.patch, HBASE-5959.D3189.7.patch Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-5776: --- Attachment: D2775.4.patch Liyin updated the revision [jira][89-fb][HBASE-5776] HTableMultiplexer. Reviewers: Kannan Addressed some offline comments from Kannan about improving the failed put processing. REVISION DETAIL https://reviews.facebook.net/D2775 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/HConstants.java src/main/java/org/apache/hadoop/hbase/client/HConnection.java src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java src/main/java/org/apache/hadoop/hbase/client/HTable.java src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java src/main/java/org/apache/hadoop/hbase/client/MultiPut.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java To: Kannan, Liyin Cc: JIRA, tedyu HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, D2775.2.patch, D2775.3.patch, D2775.4.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13276923#comment-13276923 ] Phabricator commented on HBASE-5959: tedyu has commented on the revision HBASE-5959 [jira] Add other load balancers. More comments to follow. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:251 Do we need to re-fetch these config parameters in each iteration ? src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:44 There're extraneous empty lines such as this one. Please remove them. src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:78 'one cluster' - 'cluster with one server' src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:80 it's - cluster has src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:197 'use' - 'uses' src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:202 Please add javadoc src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:204 Typo: 'pickRandmoRegion' - 'pickRandomRegion' src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:144 Remove empty line. src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:154 'plan' - 'plans' src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:224 Please finish the sentence. src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:226 Specify what is returned. src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:241 'balancer.stochastic' - 'stochastic.balancer' src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:256 I think locality cost should be given higher weight. REVISION DETAIL https://reviews.facebook.net/D3189 To: JIRA, eclark Cc: tedyu Add other load balancers Key: HBASE-5959 URL: https://issues.apache.org/jira/browse/HBASE-5959 Project: HBase Issue Type: New Feature Components: master Affects Versions: 0.96.0 Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, HBASE-5959-7.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13276942#comment-13276942 ] Phabricator commented on HBASE-5959: eclark has commented on the revision HBASE-5959 [jira] Add other load balancers. I'll get the config storing in the next version INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:241 There are already config paramaters with balancer.configName we should keep the hierarchy. src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:256 And I actually think it should be given a lower weight as the table cost is something that represents an on going cost to the cluster and the locality cost is a one time transfer. The impetus for doing this work was that we saw production clusters that were no balancing because the old balancer did not want to move regions around to balance the number of regions per table (Something I've seen on several different production clusters now; a real issue on anything that has big batch jobs). Placing locality's weight above table cost would just mean that the same thing could happen. However I think the middle ground combined with the ability to change per install is enough without testing on production clusters. src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:251 Nope I'll add that in my next version. REVISION DETAIL https://reviews.facebook.net/D3189 To: JIRA, eclark Cc: tedyu Add other load balancers Key: HBASE-5959 URL: https://issues.apache.org/jira/browse/HBASE-5959 Project: HBase Issue Type: New Feature Components: master Affects Versions: 0.96.0 Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, HBASE-5959-7.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5987) HFileBlockIndex improvement
[ https://issues.apache.org/jira/browse/HBASE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13276941#comment-13276941 ] Phabricator commented on HBASE-5987: mbautin has commented on the revision [jira][89-fb] [HBASE-5987] HFileBlockIndex improvement. Looks good! A few minor comments inline. Also please submit the diff with lint (using arc diff --preview instead of arc diff --only)/ INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/HConstants.java:545 Please add a comment that the actual value is irrelevant because this is always compared by reference. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java:437-440 This documentation is still confusing. Is i the ith position, or is the actual key the ith position? I would say i is the position and the returned key is the key at the ith position. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java:413 Clarify the meaning of is equal, i.e. that it must be exactly the same object, not just an equal byte array. src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java:63 This is unnecessary (we don't use compression by default). src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java:77 It is not schemMetricSnapshot, it is schemaMetricSnapshot (schem is not a word). REVISION DETAIL https://reviews.facebook.net/D3237 To: Kannan, mbautin, Liyin Cc: JIRA, todd, tedyu HFileBlockIndex improvement --- Key: HBASE-5987 URL: https://issues.apache.org/jira/browse/HBASE-5987 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D3237.1.patch, D3237.2.patch, screen_shot_of_sequential_scan_profiling.png Recently we find out a performance problem that it is quite slow when multiple requests are reading the same block of data or index. From the profiling, one of the causes is the IdLock contention which has been addressed in HBASE-5898. Another issue is that the HFileScanner will keep asking the HFileBlockIndex about the data block location for each target key value during the scan process(reSeekTo), even though the target key value has already been in the current data block. This issue will cause certain index block very HOT, especially when it is a sequential scan. To solve this issue, we propose the following solutions: First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is smaller than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is bigger, then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup. Secondary, we propose to push this idea a little further that the HFileBlockIndex shall index on the last key value of each data block instead of indexing on the start key value. The motivation is to solve the HBASE-4443 issue (avoid seeking to previous block when key you are interested in is the first one of a block) as well as +the defects mentioned above+. For example, if the target key value is smaller than the start key value of the data block N. There is no way for sure the target key value is in the data block N or N-1. So it has to seek from data block N-1. However, if the block index is based on the last key value for each data block and the target key value is beween the last key value of data block N-1 and data block N, then the target key value is supposed be data block N for sure. As long as HBase only supports the forward scan, the last key value makes more sense to be indexed on than the start key value. Thanks Kannan and Mikhail for the insightful discussions and suggestions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5987) HFileBlockIndex improvement
[ https://issues.apache.org/jira/browse/HBASE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-5987: --- Attachment: D3237.3.patch Liyin updated the revision [jira][89-fb] [HBASE-5987] HFileBlockIndex improvement. Reviewers: Kannan, mbautin Address Mikhail's comments REVISION DETAIL https://reviews.facebook.net/D3237 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/HConstants.java src/main/java/org/apache/hadoop/hbase/io/hfile/BlockWithScanInfo.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java To: Kannan, mbautin, Liyin Cc: JIRA, todd, tedyu HFileBlockIndex improvement --- Key: HBASE-5987 URL: https://issues.apache.org/jira/browse/HBASE-5987 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D3237.1.patch, D3237.2.patch, D3237.3.patch, screen_shot_of_sequential_scan_profiling.png Recently we find out a performance problem that it is quite slow when multiple requests are reading the same block of data or index. From the profiling, one of the causes is the IdLock contention which has been addressed in HBASE-5898. Another issue is that the HFileScanner will keep asking the HFileBlockIndex about the data block location for each target key value during the scan process(reSeekTo), even though the target key value has already been in the current data block. This issue will cause certain index block very HOT, especially when it is a sequential scan. To solve this issue, we propose the following solutions: First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is smaller than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is bigger, then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup. Secondary, we propose to push this idea a little further that the HFileBlockIndex shall index on the last key value of each data block instead of indexing on the start key value. The motivation is to solve the HBASE-4443 issue (avoid seeking to previous block when key you are interested in is the first one of a block) as well as +the defects mentioned above+. For example, if the target key value is smaller than the start key value of the data block N. There is no way for sure the target key value is in the data block N or N-1. So it has to seek from data block N-1. However, if the block index is based on the last key value for each data block and the target key value is beween the last key value of data block N-1 and data block N, then the target key value is supposed be data block N for sure. As long as HBase only supports the forward scan, the last key value makes more sense to be indexed on than the start key value. Thanks Kannan and Mikhail for the insightful discussions and suggestions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5987) HFileBlockIndex improvement
[ https://issues.apache.org/jira/browse/HBASE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277001#comment-13277001 ] Phabricator commented on HBASE-5987: mbautin has accepted the revision [jira][89-fb] [HBASE-5987] HFileBlockIndex improvement. Just one minor comment (please address on commit). INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java:413 HContants - HConstants (missed an s) REVISION DETAIL https://reviews.facebook.net/D3237 BRANCH HBASE-5987-fb To: Kannan, mbautin, Liyin Cc: JIRA, todd, tedyu HFileBlockIndex improvement --- Key: HBASE-5987 URL: https://issues.apache.org/jira/browse/HBASE-5987 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D3237.1.patch, D3237.2.patch, D3237.3.patch, screen_shot_of_sequential_scan_profiling.png Recently we find out a performance problem that it is quite slow when multiple requests are reading the same block of data or index. From the profiling, one of the causes is the IdLock contention which has been addressed in HBASE-5898. Another issue is that the HFileScanner will keep asking the HFileBlockIndex about the data block location for each target key value during the scan process(reSeekTo), even though the target key value has already been in the current data block. This issue will cause certain index block very HOT, especially when it is a sequential scan. To solve this issue, we propose the following solutions: First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is smaller than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is bigger, then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup. Secondary, we propose to push this idea a little further that the HFileBlockIndex shall index on the last key value of each data block instead of indexing on the start key value. The motivation is to solve the HBASE-4443 issue (avoid seeking to previous block when key you are interested in is the first one of a block) as well as +the defects mentioned above+. For example, if the target key value is smaller than the start key value of the data block N. There is no way for sure the target key value is in the data block N or N-1. So it has to seek from data block N-1. However, if the block index is based on the last key value for each data block and the target key value is beween the last key value of data block N-1 and data block N, then the target key value is supposed be data block N for sure. As long as HBase only supports the forward scan, the last key value makes more sense to be indexed on than the start key value. Thanks Kannan and Mikhail for the insightful discussions and suggestions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5987) HFileBlockIndex improvement
[
https://issues.apache.org/jira/browse/HBASE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277017#comment-13277017
]
Phabricator commented on HBASE-5987:
todd has commented on the revision [jira][89-fb] [HBASE-5987] HFileBlockIndex
improvement.
Would be nice to have a simple benchmark - eg load a million rows and time
count 'table', { CACHE = 1000 } from the shell with and without.
INLINE COMMENTS
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockWithScanInfo.java:23
typo: references wrong class name here
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockWithScanInfo.java:28
could do with a short javadoc, eg:
/**
* The first key in the next block following this one in the HFile.
* If this key is unknown, this is reference-equal with
HConstants.NO_NEXT_INDEXED_KEY
*/
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java:526 are you
guaranteed that firstKey.arrayOffset() == 0 here? I would have assumed firstKey
could be an array slice
REVISION DETAIL
https://reviews.facebook.net/D3237
BRANCH
HBASE-5987-fb
To: Kannan, mbautin, Liyin
Cc: JIRA, todd, tedyu
HFileBlockIndex improvement
---
Key: HBASE-5987
URL: https://issues.apache.org/jira/browse/HBASE-5987
Project: HBase
Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
Attachments: D3237.1.patch, D3237.2.patch, D3237.3.patch,
screen_shot_of_sequential_scan_profiling.png
Recently we find out a performance problem that it is quite slow when
multiple requests are reading the same block of data or index.
From the profiling, one of the causes is the IdLock contention which has been
addressed in HBASE-5898.
Another issue is that the HFileScanner will keep asking the HFileBlockIndex
about the data block location for each target key value during the scan
process(reSeekTo), even though the target key value has already been in the
current data block. This issue will cause certain index block very HOT,
especially when it is a sequential scan.
To solve this issue, we propose the following solutions:
First, we propose to lookahead for one more block index so that the
HFileScanner would know the start key value of next data block. So if the
target key value for the scan(reSeekTo) is smaller than that start kv of
next data block, it means the target key value has a very high possibility in
the current data block (if not in current data block, then the start kv of
next data block should be returned. +Indexing on the start key has some
defects here+) and it shall NOT query the HFileBlockIndex in this case. On
the contrary, if the target key value is bigger, then it shall query the
HFileBlockIndex. This improvement shall help to reduce the hotness of
HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block
Cache lookup.
Secondary, we propose to push this idea a little further that the
HFileBlockIndex shall index on the last key value of each data block instead
of indexing on the start key value. The motivation is to solve the HBASE-4443
issue (avoid seeking to previous block when key you are interested in is
the first one of a block) as well as +the defects mentioned above+.
For example, if the target key value is smaller than the start key value of
the data block N. There is no way for sure the target key value is in the
data block N or N-1. So it has to seek from data block N-1. However, if the
block index is based on the last key value for each data block and the target
key value is beween the last key value of data block N-1 and data block N,
then the target key value is supposed be data block N for sure.
As long as HBase only supports the forward scan, the last key value makes
more sense to be indexed on than the start key value.
Thanks Kannan and Mikhail for the insightful discussions and suggestions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5987) HFileBlockIndex improvement
[ https://issues.apache.org/jira/browse/HBASE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-5987: --- Attachment: D3237.4.patch Liyin updated the revision [jira][89-fb] [HBASE-5987] HFileBlockIndex improvement. Reviewers: Kannan, mbautin Good point, Todd ! REVISION DETAIL https://reviews.facebook.net/D3237 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/HConstants.java src/main/java/org/apache/hadoop/hbase/io/hfile/BlockWithScanInfo.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java To: Kannan, mbautin, Liyin Cc: JIRA, todd, tedyu HFileBlockIndex improvement --- Key: HBASE-5987 URL: https://issues.apache.org/jira/browse/HBASE-5987 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D3237.1.patch, D3237.2.patch, D3237.3.patch, D3237.4.patch, screen_shot_of_sequential_scan_profiling.png Recently we find out a performance problem that it is quite slow when multiple requests are reading the same block of data or index. From the profiling, one of the causes is the IdLock contention which has been addressed in HBASE-5898. Another issue is that the HFileScanner will keep asking the HFileBlockIndex about the data block location for each target key value during the scan process(reSeekTo), even though the target key value has already been in the current data block. This issue will cause certain index block very HOT, especially when it is a sequential scan. To solve this issue, we propose the following solutions: First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is smaller than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is bigger, then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup. Secondary, we propose to push this idea a little further that the HFileBlockIndex shall index on the last key value of each data block instead of indexing on the start key value. The motivation is to solve the HBASE-4443 issue (avoid seeking to previous block when key you are interested in is the first one of a block) as well as +the defects mentioned above+. For example, if the target key value is smaller than the start key value of the data block N. There is no way for sure the target key value is in the data block N or N-1. So it has to seek from data block N-1. However, if the block index is based on the last key value for each data block and the target key value is beween the last key value of data block N-1 and data block N, then the target key value is supposed be data block N for sure. As long as HBase only supports the forward scan, the last key value makes more sense to be indexed on than the start key value. Thanks Kannan and Mikhail for the insightful discussions and suggestions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5987) HFileBlockIndex improvement
[ https://issues.apache.org/jira/browse/HBASE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277090#comment-13277090 ] Phabricator commented on HBASE-5987: todd has commented on the revision [jira][89-fb] [HBASE-5987] HFileBlockIndex improvement. Thanks for fixing. I'm surprised the unit tests weren't failing before. Is that because the ByteBuffer usually does have arrayOffset() == 0, so the bug wasn't actually causing a problem? Or do we need more test coverage? REVISION DETAIL https://reviews.facebook.net/D3237 BRANCH HBASE-5987-fb To: Kannan, mbautin, Liyin Cc: JIRA, todd, tedyu HFileBlockIndex improvement --- Key: HBASE-5987 URL: https://issues.apache.org/jira/browse/HBASE-5987 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D3237.1.patch, D3237.2.patch, D3237.3.patch, D3237.4.patch, screen_shot_of_sequential_scan_profiling.png Recently we find out a performance problem that it is quite slow when multiple requests are reading the same block of data or index. From the profiling, one of the causes is the IdLock contention which has been addressed in HBASE-5898. Another issue is that the HFileScanner will keep asking the HFileBlockIndex about the data block location for each target key value during the scan process(reSeekTo), even though the target key value has already been in the current data block. This issue will cause certain index block very HOT, especially when it is a sequential scan. To solve this issue, we propose the following solutions: First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is smaller than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is bigger, then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup. Secondary, we propose to push this idea a little further that the HFileBlockIndex shall index on the last key value of each data block instead of indexing on the start key value. The motivation is to solve the HBASE-4443 issue (avoid seeking to previous block when key you are interested in is the first one of a block) as well as +the defects mentioned above+. For example, if the target key value is smaller than the start key value of the data block N. There is no way for sure the target key value is in the data block N or N-1. So it has to seek from data block N-1. However, if the block index is based on the last key value for each data block and the target key value is beween the last key value of data block N-1 and data block N, then the target key value is supposed be data block N for sure. As long as HBase only supports the forward scan, the last key value makes more sense to be indexed on than the start key value. Thanks Kannan and Mikhail for the insightful discussions and suggestions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5987) HFileBlockIndex improvement
[ https://issues.apache.org/jira/browse/HBASE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-5987: --- Attachment: D3237.5.patch Liyin updated the revision [jira][89-fb] [HBASE-5987] HFileBlockIndex improvement. Reviewers: Kannan, mbautin REVISION DETAIL https://reviews.facebook.net/D3237 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/HConstants.java src/main/java/org/apache/hadoop/hbase/io/hfile/BlockWithScanInfo.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java To: Kannan, mbautin, Liyin Cc: JIRA, todd, tedyu HFileBlockIndex improvement --- Key: HBASE-5987 URL: https://issues.apache.org/jira/browse/HBASE-5987 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D3237.1.patch, D3237.2.patch, D3237.3.patch, D3237.4.patch, D3237.5.patch, screen_shot_of_sequential_scan_profiling.png Recently we find out a performance problem that it is quite slow when multiple requests are reading the same block of data or index. From the profiling, one of the causes is the IdLock contention which has been addressed in HBASE-5898. Another issue is that the HFileScanner will keep asking the HFileBlockIndex about the data block location for each target key value during the scan process(reSeekTo), even though the target key value has already been in the current data block. This issue will cause certain index block very HOT, especially when it is a sequential scan. To solve this issue, we propose the following solutions: First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is smaller than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is bigger, then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup. Secondary, we propose to push this idea a little further that the HFileBlockIndex shall index on the last key value of each data block instead of indexing on the start key value. The motivation is to solve the HBASE-4443 issue (avoid seeking to previous block when key you are interested in is the first one of a block) as well as +the defects mentioned above+. For example, if the target key value is smaller than the start key value of the data block N. There is no way for sure the target key value is in the data block N or N-1. So it has to seek from data block N-1. However, if the block index is based on the last key value for each data block and the target key value is beween the last key value of data block N-1 and data block N, then the target key value is supposed be data block N for sure. As long as HBase only supports the forward scan, the last key value makes more sense to be indexed on than the start key value. Thanks Kannan and Mikhail for the insightful discussions and suggestions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5987) HFileBlockIndex improvement
[ https://issues.apache.org/jira/browse/HBASE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277131#comment-13277131 ] Phabricator commented on HBASE-5987: Liyin has commented on the revision [jira][89-fb] [HBASE-5987] HFileBlockIndex improvement. I think we haven't done a seekBefore to the previous block with a reSeekTo in this previous block together. I shall create a unit test to cover that. REVISION DETAIL https://reviews.facebook.net/D3237 BRANCH HBASE-5987-fb To: Kannan, mbautin, Liyin Cc: JIRA, todd, tedyu HFileBlockIndex improvement --- Key: HBASE-5987 URL: https://issues.apache.org/jira/browse/HBASE-5987 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D3237.1.patch, D3237.2.patch, D3237.3.patch, D3237.4.patch, D3237.5.patch, screen_shot_of_sequential_scan_profiling.png Recently we find out a performance problem that it is quite slow when multiple requests are reading the same block of data or index. From the profiling, one of the causes is the IdLock contention which has been addressed in HBASE-5898. Another issue is that the HFileScanner will keep asking the HFileBlockIndex about the data block location for each target key value during the scan process(reSeekTo), even though the target key value has already been in the current data block. This issue will cause certain index block very HOT, especially when it is a sequential scan. To solve this issue, we propose the following solutions: First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is smaller than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is bigger, then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup. Secondary, we propose to push this idea a little further that the HFileBlockIndex shall index on the last key value of each data block instead of indexing on the start key value. The motivation is to solve the HBASE-4443 issue (avoid seeking to previous block when key you are interested in is the first one of a block) as well as +the defects mentioned above+. For example, if the target key value is smaller than the start key value of the data block N. There is no way for sure the target key value is in the data block N or N-1. So it has to seek from data block N-1. However, if the block index is based on the last key value for each data block and the target key value is beween the last key value of data block N-1 and data block N, then the target key value is supposed be data block N for sure. As long as HBase only supports the forward scan, the last key value makes more sense to be indexed on than the start key value. Thanks Kannan and Mikhail for the insightful discussions and suggestions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
