[
https://issues.apache.org/jira/browse/HADOOP-11867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029389#comment-17029389
]
Owen O'Malley commented on HADOOP-11867:
----------------------------------------
Ok, my first patch adds the API, the default method, and the utilities for
this. I also included the implementation for RawLocalFileSystem and
ChecksumFileSystem because they were easiest to test and let me use the APIs in
non-trivial ways. I also included a benchmark that tests against the local file
system:
{code}
Benchmark (bufferKind) (fileSystemKind) Mode Cnt
Score Error Units
AsyncBenchmark.asyncFileChanArray direct N/A avgt 20
1432.396 ± 232.443 us/op
AsyncBenchmark.asyncFileChanArray array N/A avgt 20
1551.400 ± 65.639 us/op
AsyncBenchmark.asyncRead direct local avgt 20
2514.926 ± 245.603 us/op
AsyncBenchmark.asyncRead direct raw avgt 20
1440.434 ± 207.504 us/op
AsyncBenchmark.asyncRead array local avgt 20
2798.031 ± 135.023 us/op
AsyncBenchmark.asyncRead array raw avgt 20
1524.360 ± 54.462 us/op
AsyncBenchmark.syncRead N/A local avgt 20
9515.604 ± 123.311 us/op
AsyncBenchmark.syncRead N/A raw avgt 20
2402.039 ± 36.620 us/op
{code}
> FS API: Add a high-performance vectored Read to FSDataInputStream API
> ---------------------------------------------------------------------
>
> Key: HADOOP-11867
> URL: https://issues.apache.org/jira/browse/HADOOP-11867
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, fs/azure, fs/s3, hdfs-client
> Affects Versions: 3.0.0
> Reporter: Gopal Vijayaraghavan
> Assignee: Owen O'Malley
> Priority: Major
> Labels: performance
>
> The most significant way to read from a filesystem in an efficient way is to
> let the FileSystem implementation handle the seek behaviour underneath the
> API to be the most efficient as possible.
> A better approach to the seek problem is to provide a sequence of read
> locations as part of a single call, while letting the system schedule/plan
> the reads ahead of time.
> This is exceedingly useful for seek-heavy readers on HDFS, since this allows
> for potentially optimizing away the seek-gaps within the FSDataInputStream
> implementation.
> For seek+read systems with even more latency than locally-attached disks,
> something like a {{readFully(long[] offsets, ByteBuffer[] chunks)}} would
> take of the seeks internally while reading chunk.remaining() bytes into each
> chunk (which may be {{slice()}}ed off a bigger buffer).
> The base implementation can stub in this as a sequence of seeks + read() into
> ByteBuffers, without forcing each FS implementation to override this in any
> way.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
