[jira] [Updated] (HBASE-5387) Reuse compression streams in HFileBlock.Writer
[ https://issues.apache.org/jira/browse/HBASE-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-5387: --- Attachment: D1725.2.patch mbautin updated the revision [jira] [HBASE-5387] [89-fb] Reuse compression streams in HFileBlock.Writer. Reviewers: khemani, Liyin, Kannan, lhofhansl, tedyu, JIRA Rebasing on recent changes in 89-fb (data block encoding). REVISION DETAIL https://reviews.facebook.net/D1725 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/io/hfile/Compression.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java src/main/java/org/apache/hadoop/hbase/io/hfile/ReusableStreamGzipCodec.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java Reuse compression streams in HFileBlock.Writer -- Key: HBASE-5387 URL: https://issues.apache.org/jira/browse/HBASE-5387 Project: HBase Issue Type: Bug Affects Versions: 0.94.0 Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Critical Fix For: 0.94.0 Attachments: 5387.txt, D1719.1.patch, D1719.2.patch, D1719.3.patch, D1719.4.patch, D1719.5.patch, D1725.1.patch, D1725.2.patch, Fix-deflater-leak-2012-02-10_18_48_45.patch, Fix-deflater-leak-2012-02-11_17_13_10.patch, Fix-deflater-leak-2012-02-12_00_37_27.patch We need to to reuse compression streams in HFileBlock.Writer instead of allocating them every time. The motivation is that when using Java's built-in implementation of Gzip, we allocate a new GZIPOutputStream object and an associated native data structure every time we create a compression stream. The native data structure is only deallocated in the finalizer. This is one suspected cause of recent TestHFileBlock failures on Hadoop QA: https://builds.apache.org/job/HBase-TRUNK/2658/testReport/org.apache.hadoop.hbase.io.hfile/TestHFileBlock/testPreviousOffset_1_/. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5387) Reuse compression streams in HFileBlock.Writer
[ https://issues.apache.org/jira/browse/HBASE-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-5387: -- Resolution: Fixed Status: Resolved (was: Patch Available) Reuse compression streams in HFileBlock.Writer -- Key: HBASE-5387 URL: https://issues.apache.org/jira/browse/HBASE-5387 Project: HBase Issue Type: Bug Affects Versions: 0.94.0 Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Critical Fix For: 0.94.0 Attachments: 5387.txt, D1719.1.patch, D1719.2.patch, D1719.3.patch, D1719.4.patch, D1719.5.patch, Fix-deflater-leak-2012-02-10_18_48_45.patch, Fix-deflater-leak-2012-02-11_17_13_10.patch, Fix-deflater-leak-2012-02-12_00_37_27.patch We need to to reuse compression streams in HFileBlock.Writer instead of allocating them every time. The motivation is that when using Java's built-in implementation of Gzip, we allocate a new GZIPOutputStream object and an associated native data structure every time we create a compression stream. The native data structure is only deallocated in the finalizer. This is one suspected cause of recent TestHFileBlock failures on Hadoop QA: https://builds.apache.org/job/HBase-TRUNK/2658/testReport/org.apache.hadoop.hbase.io.hfile/TestHFileBlock/testPreviousOffset_1_/. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5387) Reuse compression streams in HFileBlock.Writer
[ https://issues.apache.org/jira/browse/HBASE-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-5387: --- Attachment: D1725.1.patch mbautin requested code review of [jira] [HBASE-5387] [89-fb] Reuse compression streams in HFileBlock.Writer. Reviewers: khemani, Liyin, Kannan, lhofhansl, tedyu, JIRA Porting the trunk patch https://reviews.facebook.net/D1719 to 89-fb to make Gzip compression streams reusable when no Hadoop native libraries loaded. Also, making the Compression class and HFileBlock in 89-fb more similar to its trunk version. This touches the same code in HFileBlock with the data block encoding patch (https://reviews.facebook.net/D1659), which will have to be rebased on this. TEST PLAN Run unit tests REVISION DETAIL https://reviews.facebook.net/D1725 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/io/hfile/Compression.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java src/main/java/org/apache/hadoop/hbase/io/hfile/ReusableStreamGzipCodec.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/3681/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. Reuse compression streams in HFileBlock.Writer -- Key: HBASE-5387 URL: https://issues.apache.org/jira/browse/HBASE-5387 Project: HBase Issue Type: Bug Affects Versions: 0.94.0 Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Critical Fix For: 0.94.0 Attachments: 5387.txt, D1719.1.patch, D1719.2.patch, D1719.3.patch, D1719.4.patch, D1719.5.patch, D1725.1.patch, Fix-deflater-leak-2012-02-10_18_48_45.patch, Fix-deflater-leak-2012-02-11_17_13_10.patch, Fix-deflater-leak-2012-02-12_00_37_27.patch We need to to reuse compression streams in HFileBlock.Writer instead of allocating them every time. The motivation is that when using Java's built-in implementation of Gzip, we allocate a new GZIPOutputStream object and an associated native data structure every time we create a compression stream. The native data structure is only deallocated in the finalizer. This is one suspected cause of recent TestHFileBlock failures on Hadoop QA: https://builds.apache.org/job/HBase-TRUNK/2658/testReport/org.apache.hadoop.hbase.io.hfile/TestHFileBlock/testPreviousOffset_1_/. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5387) Reuse compression streams in HFileBlock.Writer
[ https://issues.apache.org/jira/browse/HBASE-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Bautin updated HBASE-5387: -- Status: Patch Available (was: Open) Reuse compression streams in HFileBlock.Writer -- Key: HBASE-5387 URL: https://issues.apache.org/jira/browse/HBASE-5387 Project: HBase Issue Type: Bug Affects Versions: 0.94.0 Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Critical Fix For: 0.94.0 Attachments: 5387.txt, D1719.1.patch, D1719.2.patch, D1719.3.patch, D1719.4.patch, D1719.5.patch, Fix-deflater-leak-2012-02-10_18_48_45.patch, Fix-deflater-leak-2012-02-11_17_13_10.patch, Fix-deflater-leak-2012-02-12_00_37_27.patch We need to to reuse compression streams in HFileBlock.Writer instead of allocating them every time. The motivation is that when using Java's built-in implementation of Gzip, we allocate a new GZIPOutputStream object and an associated native data structure every time we create a compression stream. The native data structure is only deallocated in the finalizer. This is one suspected cause of recent TestHFileBlock failures on Hadoop QA: https://builds.apache.org/job/HBase-TRUNK/2658/testReport/org.apache.hadoop.hbase.io.hfile/TestHFileBlock/testPreviousOffset_1_/. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5387) Reuse compression streams in HFileBlock.Writer
[ https://issues.apache.org/jira/browse/HBASE-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Bautin updated HBASE-5387: -- Status: Open (was: Patch Available) Reuse compression streams in HFileBlock.Writer -- Key: HBASE-5387 URL: https://issues.apache.org/jira/browse/HBASE-5387 Project: HBase Issue Type: Bug Affects Versions: 0.94.0 Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Critical Fix For: 0.94.0 Attachments: 5387.txt, D1719.1.patch, D1719.2.patch, D1719.3.patch, D1719.4.patch, D1719.5.patch, Fix-deflater-leak-2012-02-10_18_48_45.patch, Fix-deflater-leak-2012-02-11_17_13_10.patch, Fix-deflater-leak-2012-02-12_00_37_27.patch We need to to reuse compression streams in HFileBlock.Writer instead of allocating them every time. The motivation is that when using Java's built-in implementation of Gzip, we allocate a new GZIPOutputStream object and an associated native data structure every time we create a compression stream. The native data structure is only deallocated in the finalizer. This is one suspected cause of recent TestHFileBlock failures on Hadoop QA: https://builds.apache.org/job/HBase-TRUNK/2658/testReport/org.apache.hadoop.hbase.io.hfile/TestHFileBlock/testPreviousOffset_1_/. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5387) Reuse compression streams in HFileBlock.Writer
[ https://issues.apache.org/jira/browse/HBASE-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Bautin updated HBASE-5387: -- Attachment: Fix-deflater-leak-2012-02-12_00_37_27.patch Reuse compression streams in HFileBlock.Writer -- Key: HBASE-5387 URL: https://issues.apache.org/jira/browse/HBASE-5387 Project: HBase Issue Type: Bug Affects Versions: 0.94.0 Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Critical Fix For: 0.94.0 Attachments: 5387.txt, D1719.1.patch, D1719.2.patch, D1719.3.patch, D1719.4.patch, D1719.5.patch, Fix-deflater-leak-2012-02-10_18_48_45.patch, Fix-deflater-leak-2012-02-11_17_13_10.patch, Fix-deflater-leak-2012-02-12_00_37_27.patch We need to to reuse compression streams in HFileBlock.Writer instead of allocating them every time. The motivation is that when using Java's built-in implementation of Gzip, we allocate a new GZIPOutputStream object and an associated native data structure every time we create a compression stream. The native data structure is only deallocated in the finalizer. This is one suspected cause of recent TestHFileBlock failures on Hadoop QA: https://builds.apache.org/job/HBase-TRUNK/2658/testReport/org.apache.hadoop.hbase.io.hfile/TestHFileBlock/testPreviousOffset_1_/. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5387) Reuse compression streams in HFileBlock.Writer
[ https://issues.apache.org/jira/browse/HBASE-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-5387: --- Attachment: D1719.1.patch mbautin requested code review of [jira] [HBASE-5387] Reuse compression streams in HFileBlock.Writer. Reviewers: tedyu, Liyin, dhruba, JIRA We need to to reuse compression streams in HFileBlock.Writer instead of allocating them every time. The motivation is that when using Java's built-in implementation of Gzip, we allocate a new GZIPOutputStream object and an associated native data structure any time. This is one suspected cause of recent TestHFileBlock failures on Hadoop QA: https://builds.apache.org/job/HBase-TRUNK/2658/testReport/org.apache.hadoop.hbase.io.hfile/TestHFileBlock/testPreviousOffset_1_/. TEST PLAN * Run unit tests * Create a GZIP-compressed CF with new code, load some data, shut down HBase, deploy old code, restart HBase, and scan the table REVISION DETAIL https://reviews.facebook.net/D1719 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/io/hfile/Compression.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java src/main/java/org/apache/hadoop/hbase/io/hfile/ReusableStreamGzipCodec.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java Reuse compression streams in HFileBlock.Writer -- Key: HBASE-5387 URL: https://issues.apache.org/jira/browse/HBASE-5387 Project: HBase Issue Type: Bug Affects Versions: 0.94.0 Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Critical Fix For: 0.94.0 Attachments: D1719.1.patch, Fix-deflater-leak-2012-02-10_18_48_45.patch We need to to reuse compression streams in HFileBlock.Writer instead of allocating them every time. The motivation is that when using Java's built-in implementation of Gzip, we allocate a new GZIPOutputStream object and an associated native data structure every time we create a compression stream. The native data structure is only deallocated in the finalizer. This is one suspected cause of recent TestHFileBlock failures on Hadoop QA: https://builds.apache.org/job/HBase-TRUNK/2658/testReport/org.apache.hadoop.hbase.io.hfile/TestHFileBlock/testPreviousOffset_1_/. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5387) Reuse compression streams in HFileBlock.Writer
[ https://issues.apache.org/jira/browse/HBASE-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-5387: --- Attachment: D1719.2.patch mbautin updated the revision [jira] [HBASE-5387] Reuse compression streams in HFileBlock.Writer. Reviewers: tedyu, Liyin, dhruba, JIRA Addressing review comments. REVISION DETAIL https://reviews.facebook.net/D1719 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/io/hfile/Compression.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java src/main/java/org/apache/hadoop/hbase/io/hfile/ReusableStreamGzipCodec.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java Reuse compression streams in HFileBlock.Writer -- Key: HBASE-5387 URL: https://issues.apache.org/jira/browse/HBASE-5387 Project: HBase Issue Type: Bug Affects Versions: 0.94.0 Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Critical Fix For: 0.94.0 Attachments: D1719.1.patch, D1719.2.patch, Fix-deflater-leak-2012-02-10_18_48_45.patch We need to to reuse compression streams in HFileBlock.Writer instead of allocating them every time. The motivation is that when using Java's built-in implementation of Gzip, we allocate a new GZIPOutputStream object and an associated native data structure every time we create a compression stream. The native data structure is only deallocated in the finalizer. This is one suspected cause of recent TestHFileBlock failures on Hadoop QA: https://builds.apache.org/job/HBase-TRUNK/2658/testReport/org.apache.hadoop.hbase.io.hfile/TestHFileBlock/testPreviousOffset_1_/. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5387) Reuse compression streams in HFileBlock.Writer
[ https://issues.apache.org/jira/browse/HBASE-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Bautin updated HBASE-5387: -- Attachment: Fix-deflater-leak-2012-02-11_17_13_10.patch Reuse compression streams in HFileBlock.Writer -- Key: HBASE-5387 URL: https://issues.apache.org/jira/browse/HBASE-5387 Project: HBase Issue Type: Bug Affects Versions: 0.94.0 Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Critical Fix For: 0.94.0 Attachments: D1719.1.patch, D1719.2.patch, Fix-deflater-leak-2012-02-10_18_48_45.patch, Fix-deflater-leak-2012-02-11_17_13_10.patch We need to to reuse compression streams in HFileBlock.Writer instead of allocating them every time. The motivation is that when using Java's built-in implementation of Gzip, we allocate a new GZIPOutputStream object and an associated native data structure every time we create a compression stream. The native data structure is only deallocated in the finalizer. This is one suspected cause of recent TestHFileBlock failures on Hadoop QA: https://builds.apache.org/job/HBase-TRUNK/2658/testReport/org.apache.hadoop.hbase.io.hfile/TestHFileBlock/testPreviousOffset_1_/. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5387) Reuse compression streams in HFileBlock.Writer
[ https://issues.apache.org/jira/browse/HBASE-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-5387: --- Attachment: D1719.3.patch mbautin updated the revision [jira] [HBASE-5387] Reuse compression streams in HFileBlock.Writer. Reviewers: tedyu, Liyin, dhruba, JIRA Breaking compression stream creation methods into multiple parts. Avoiding using finish-on-flush stream in a couple of places, and deprecating the currently used method that is always called with downstream buffer size of 0, at least within HBase code. The method is still there for backwards compatibility. Also, getting rid of hard-coded buffer size by delegating the native compressor case to superclass in ReusableStreamGzipCodec. REVISION DETAIL https://reviews.facebook.net/D1719 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/io/hfile/Compression.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java src/main/java/org/apache/hadoop/hbase/io/hfile/ReusableStreamGzipCodec.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java src/test/java/org/apache/hadoop/hbase/regionserver/DataBlockEncodingTool.java Reuse compression streams in HFileBlock.Writer -- Key: HBASE-5387 URL: https://issues.apache.org/jira/browse/HBASE-5387 Project: HBase Issue Type: Bug Affects Versions: 0.94.0 Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Critical Fix For: 0.94.0 Attachments: D1719.1.patch, D1719.2.patch, D1719.3.patch, Fix-deflater-leak-2012-02-10_18_48_45.patch, Fix-deflater-leak-2012-02-11_17_13_10.patch We need to to reuse compression streams in HFileBlock.Writer instead of allocating them every time. The motivation is that when using Java's built-in implementation of Gzip, we allocate a new GZIPOutputStream object and an associated native data structure every time we create a compression stream. The native data structure is only deallocated in the finalizer. This is one suspected cause of recent TestHFileBlock failures on Hadoop QA: https://builds.apache.org/job/HBase-TRUNK/2658/testReport/org.apache.hadoop.hbase.io.hfile/TestHFileBlock/testPreviousOffset_1_/. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5387) Reuse compression streams in HFileBlock.Writer
[ https://issues.apache.org/jira/browse/HBASE-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-5387: --- Attachment: D1719.4.patch mbautin updated the revision [jira] [HBASE-5387] Reuse compression streams in HFileBlock.Writer. Reviewers: tedyu, Liyin, dhruba, JIRA, lhofhansl Fixing the bug in the previous version of the patch (broke some tests in case of no compression). createCompressionStream(OutputStream, Compressor, int) remains the primary API, and the new package-private method only used from HFileBlock is createPlainCompressionStream. REVISION DETAIL https://reviews.facebook.net/D1719 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/io/hfile/Compression.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java src/main/java/org/apache/hadoop/hbase/io/hfile/ReusableStreamGzipCodec.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java Reuse compression streams in HFileBlock.Writer -- Key: HBASE-5387 URL: https://issues.apache.org/jira/browse/HBASE-5387 Project: HBase Issue Type: Bug Affects Versions: 0.94.0 Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Critical Fix For: 0.94.0 Attachments: D1719.1.patch, D1719.2.patch, D1719.3.patch, D1719.4.patch, Fix-deflater-leak-2012-02-10_18_48_45.patch, Fix-deflater-leak-2012-02-11_17_13_10.patch We need to to reuse compression streams in HFileBlock.Writer instead of allocating them every time. The motivation is that when using Java's built-in implementation of Gzip, we allocate a new GZIPOutputStream object and an associated native data structure every time we create a compression stream. The native data structure is only deallocated in the finalizer. This is one suspected cause of recent TestHFileBlock failures on Hadoop QA: https://builds.apache.org/job/HBase-TRUNK/2658/testReport/org.apache.hadoop.hbase.io.hfile/TestHFileBlock/testPreviousOffset_1_/. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5387) Reuse compression streams in HFileBlock.Writer
[ https://issues.apache.org/jira/browse/HBASE-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-5387: --- Attachment: D1719.5.patch mbautin updated the revision [jira] [HBASE-5387] Reuse compression streams in HFileBlock.Writer. Reviewers: tedyu, Liyin, dhruba, JIRA, lhofhansl Forgot to actually use ReusableStreamGzipCodec. REVISION DETAIL https://reviews.facebook.net/D1719 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/io/hfile/Compression.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java src/main/java/org/apache/hadoop/hbase/io/hfile/ReusableStreamGzipCodec.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java Reuse compression streams in HFileBlock.Writer -- Key: HBASE-5387 URL: https://issues.apache.org/jira/browse/HBASE-5387 Project: HBase Issue Type: Bug Affects Versions: 0.94.0 Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Critical Fix For: 0.94.0 Attachments: D1719.1.patch, D1719.2.patch, D1719.3.patch, D1719.4.patch, D1719.5.patch, Fix-deflater-leak-2012-02-10_18_48_45.patch, Fix-deflater-leak-2012-02-11_17_13_10.patch We need to to reuse compression streams in HFileBlock.Writer instead of allocating them every time. The motivation is that when using Java's built-in implementation of Gzip, we allocate a new GZIPOutputStream object and an associated native data structure every time we create a compression stream. The native data structure is only deallocated in the finalizer. This is one suspected cause of recent TestHFileBlock failures on Hadoop QA: https://builds.apache.org/job/HBase-TRUNK/2658/testReport/org.apache.hadoop.hbase.io.hfile/TestHFileBlock/testPreviousOffset_1_/. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5387) Reuse compression streams in HFileBlock.Writer
[ https://issues.apache.org/jira/browse/HBASE-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-5387: -- Attachment: 5387.txt Latest patch from Mikhail. Reuse compression streams in HFileBlock.Writer -- Key: HBASE-5387 URL: https://issues.apache.org/jira/browse/HBASE-5387 Project: HBase Issue Type: Bug Affects Versions: 0.94.0 Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Critical Fix For: 0.94.0 Attachments: 5387.txt, D1719.1.patch, D1719.2.patch, D1719.3.patch, D1719.4.patch, D1719.5.patch, Fix-deflater-leak-2012-02-10_18_48_45.patch, Fix-deflater-leak-2012-02-11_17_13_10.patch We need to to reuse compression streams in HFileBlock.Writer instead of allocating them every time. The motivation is that when using Java's built-in implementation of Gzip, we allocate a new GZIPOutputStream object and an associated native data structure every time we create a compression stream. The native data structure is only deallocated in the finalizer. This is one suspected cause of recent TestHFileBlock failures on Hadoop QA: https://builds.apache.org/job/HBase-TRUNK/2658/testReport/org.apache.hadoop.hbase.io.hfile/TestHFileBlock/testPreviousOffset_1_/. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5387) Reuse compression streams in HFileBlock.Writer
[ https://issues.apache.org/jira/browse/HBASE-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Bautin updated HBASE-5387: -- Attachment: Fix-deflater-leak-2012-02-10_18_48_45.patch Reuse compression streams in HFileBlock.Writer -- Key: HBASE-5387 URL: https://issues.apache.org/jira/browse/HBASE-5387 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Attachments: Fix-deflater-leak-2012-02-10_18_48_45.patch We need to to reuse compression streams in HFileBlock.Writer instead of allocating them every time. The motivation is that when using Java's built-in implementation of Gzip, we allocate a new GZIPOutputStream object and an associated native data structure any time. This is one suspected cause of recent TestHFileBlock failures on Hadoop QA: https://builds.apache.org/job/HBase-TRUNK/2658/testReport/org.apache.hadoop.hbase.io.hfile/TestHFileBlock/testPreviousOffset_1_/. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5387) Reuse compression streams in HFileBlock.Writer
[ https://issues.apache.org/jira/browse/HBASE-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-5387: -- Status: Patch Available (was: Open) Reuse compression streams in HFileBlock.Writer -- Key: HBASE-5387 URL: https://issues.apache.org/jira/browse/HBASE-5387 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Attachments: Fix-deflater-leak-2012-02-10_18_48_45.patch We need to to reuse compression streams in HFileBlock.Writer instead of allocating them every time. The motivation is that when using Java's built-in implementation of Gzip, we allocate a new GZIPOutputStream object and an associated native data structure any time. This is one suspected cause of recent TestHFileBlock failures on Hadoop QA: https://builds.apache.org/job/HBase-TRUNK/2658/testReport/org.apache.hadoop.hbase.io.hfile/TestHFileBlock/testPreviousOffset_1_/. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5387) Reuse compression streams in HFileBlock.Writer
[ https://issues.apache.org/jira/browse/HBASE-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-5387: -- Priority: Critical (was: Major) Affects Version/s: 0.94.0 Fix Version/s: 0.94.0 Hadoop Flags: Reviewed Reuse compression streams in HFileBlock.Writer -- Key: HBASE-5387 URL: https://issues.apache.org/jira/browse/HBASE-5387 Project: HBase Issue Type: Bug Affects Versions: 0.94.0 Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Critical Fix For: 0.94.0 Attachments: Fix-deflater-leak-2012-02-10_18_48_45.patch We need to to reuse compression streams in HFileBlock.Writer instead of allocating them every time. The motivation is that when using Java's built-in implementation of Gzip, we allocate a new GZIPOutputStream object and an associated native data structure every time we create a compression stream. The native data structure is only deallocated in the finalizer. This is one suspected cause of recent TestHFileBlock failures on Hadoop QA: https://builds.apache.org/job/HBase-TRUNK/2658/testReport/org.apache.hadoop.hbase.io.hfile/TestHFileBlock/testPreviousOffset_1_/. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
