[
https://issues.apache.org/jira/browse/HBASE-28338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17814702#comment-17814702
]
Hudson commented on HBASE-28338:
--------------------------------
Results for branch branch-2.6
[build #49 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/49/]:
(x) *{color:red}-1 overall{color}*
----
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/49/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/49/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/49/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/49/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 3. [see log for
details|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/49//artifact/output-integration/hadoop-3.log].
(note that this means we didn't check the Hadoop 3 shaded client)
> Bounded leak of FSDataInputStream buffers from checksum switching
> -----------------------------------------------------------------
>
> Key: HBASE-28338
> URL: https://issues.apache.org/jira/browse/HBASE-28338
> Project: HBase
> Issue Type: Bug
> Reporter: Bryan Beaudreault
> Priority: Major
> Labels: pull-request-available
>
> In FSDataInputStreamWrapper, the unbuffer() method caches an unbuffer
> instance the first time it is called. When an FSDataInputStreamWrapper is
> initialized, it has hbase checksum disabled.
> In HFileInfo.initTrailerAndContext we get the stream, read the trailer, then
> call unbuffer. At this point, checksums have not been enabled yet via
> prepareForBlockReader. So the call to unbuffer() caches the current
> non-checksum stream as the unbuffer instance.
> Later, in initMetaAndIndex we do a similar thing. This time,
> prepareForBlockReader has been called, so we are now using hbase checksums.
> When initMetaAndIndex calls unbuffer(), it uses the old unbuffer instance
> which actually has been closed when we switched to hbase checksums. So that
> call does nothing, and the new no-checksum input stream is never unbuffered.
> I haven't seen this cause an issue with normal hdfs replication (though
> haven't gone looking). It's very problematic for Erasure Coding because
> DFSStripedInputStream holds a large buffer (numDataBlocks * cellSize, so 6mb
> for RS-6-3-1024k) that is only used for stream reads NOT pread. The
> FSDataInputStreamWrapper we are talking about here is only used for pread in
> hbase, so those 6mb buffers just hang around totally unused but
> unreclaimable. Since there is an input stream per StoreFile, this can add up
> very quickly on big servers.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
