[
https://issues.apache.org/jira/browse/HDFS-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760737#comment-16760737
]
Steve Loughran commented on HDFS-14111:
---------------------------------------
for the specific case of HDFS & libc, is there a way to delay the check until
the first read() of the stream. The lazy-seek code in the object stores all put
off their GET calls until that, because the common trace of the reads of the
second and later blocks of a splittable file format are:
{code:java}
open()
seek to start of split()
read()
{code}
And for the columnar formats where the file footer lists the position of
columns, one of, in query planning
{code:java}
open()
seek to EOF-offset
read footer
{code}
+ for the reads, a sequence of seek/read or seek/positioned-read. If the footer
info isn't passed down to the workers (Hive does that)., then they also have to
read the feet of all the files
> hdfsOpenFile on HDFS causes unnecessary IO from file offset 0
> -------------------------------------------------------------
>
> Key: HDFS-14111
> URL: https://issues.apache.org/jira/browse/HDFS-14111
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, libhdfs
> Affects Versions: 3.2.0
> Reporter: Todd Lipcon
> Priority: Major
>
> hdfsOpenFile() calls readDirect() with a 0-length argument in order to check
> whether the underlying stream supports bytebuffer reads. With DFSInputStream,
> the read(0) isn't short circuited, and results in the DFSClient opening a
> block reader. In the case of a remote block, the block reader will actually
> issue a read of the whole block, causing the datanode to perform unnecessary
> IO and network transfers in order to fill up the client's TCP buffers. This
> causes performance degradation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
