[
https://issues.apache.org/jira/browse/HDFS-14535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16856285#comment-16856285
]
Zheng Hu commented on HDFS-14535:
---------------------------------
bq. If the total number of blocks being opened by the client is not too large,
then they should all fit in the cache of open FDs, and we wouldn't need to
round-trip to the DN. Maybe for this workload you should consider increasing
dfs.client.read.shortcircuit.streams.cache.size from its default 256 (along
with increasing hbase's ulimit -n)?
Thanks for your suggestion, we've disabled the blockCache of HBase to make all
the read QPS to HDFS directly(because we want to show the heap allocation
difference when reading into pooled DirectByteBuffers rather than an heap byte
array), so there're so many requstShortCircuitFds, that's OK.
BTW, I think this improvement can be applied to almost all of the branches
(If the branch is not EOL) ? Mind to do the cherry-pick ? HBase team may also
need an upgrading from 2.8.5 to the future 2.8.6.
Thanks.
> The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is
> causing lots of heap allocation in HBase when using short-circut read
> ----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-14535
> URL: https://issues.apache.org/jira/browse/HDFS-14535
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Zheng Hu
> Assignee: Zheng Hu
> Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-14535.patch
>
>
> Our HBase team are trying to read the blocks from HDFS into pooled offheap
> ByteBuffers directly (HBASE-21879), and recently we had some benchmark,
> found that almost 45% heap allocation from the DFS client. The heap
> allocation flame graph can be see here:
> https://issues.apache.org/jira/secure/attachment/12970295/async-prof-pid-25042-alloc-2.svg
> After checking the code path, we found that when requesting file descriptors
> from a DomainPeer, we allocated huge 8KB buffer for BufferedOutputStream,
> though the protocal content was quite small and just few bytes.
> It made a heavy GC pressure for HBase when cacheHitRatio < 60%, which
> increased the HBase P999 latency. Actually, we can pre-allocate a small
> buffer for the BufferedOutputStream, such as 512 bytes, it's enough to read
> the short-circuit fd protocal content. we've created a patch like that, and
> the allocation flame graph show that after the patch, the heap allocation
> from DFS client dropped from 45% to 27%, that's a very good thing I think.
> see:
> https://issues.apache.org/jira/secure/attachment/12970475/async-prof-pid-24534-alloc-2.svg
> Hope this attached patch can be merged into HDFS trunk, also Hadoop-2.8.x,
> HBase will benifit a lot from this.
> Thanks.
> For more details, can see here:
> https://issues.apache.org/jira/browse/HBASE-22387?focusedCommentId=16851639&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16851639
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
