[
https://issues.apache.org/jira/browse/HDFS-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16769858#comment-16769858
]
Sahil Takiar commented on HDFS-14111:
-------------------------------------
I can't attach patches to this JIRA since its not assigned to me, so I put the
patch in a gist link:
https://gist.github.com/sahilTakiar/4e8cbd47beb7324501c371e61e1eb8d7
The patch extends the {{StreamCapabilities}} class to support
{{ByteBufferReadable}} and replaces the {{readDirect}} call in {{hdfsOpenFile}}
with a JNI that call to the {{StreamCapabilities#hasCapability}}.
If we are ok with this approach, I would like to wait until HDFS-14267 has been
merged before submitting this patch, so that we can get some test coverage for
this change.
> hdfsOpenFile on HDFS causes unnecessary IO from file offset 0
> -------------------------------------------------------------
>
> Key: HDFS-14111
> URL: https://issues.apache.org/jira/browse/HDFS-14111
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, libhdfs
> Affects Versions: 3.2.0
> Reporter: Todd Lipcon
> Priority: Major
>
> hdfsOpenFile() calls readDirect() with a 0-length argument in order to check
> whether the underlying stream supports bytebuffer reads. With DFSInputStream,
> the read(0) isn't short circuited, and results in the DFSClient opening a
> block reader. In the case of a remote block, the block reader will actually
> issue a read of the whole block, causing the datanode to perform unnecessary
> IO and network transfers in order to fill up the client's TCP buffers. This
> causes performance degradation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
