[
https://issues.apache.org/jira/browse/HADOOP-11867?focusedWorklogId=765944&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-765944
]
ASF GitHub Bot logged work on HADOOP-11867:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 04/May/22 12:18
Start Date: 04/May/22 12:18
Worklog Time Spent: 10m
Work Description: steveloughran closed pull request #3499: HADOOP-11867.
Add a high performance vectored read API to file system.
URL: https://github.com/apache/hadoop/pull/3499
Issue Time Tracking
-------------------
Worklog Id: (was: 765944)
Time Spent: 13h (was: 12h 50m)
> Add a high-performance vectored read API.
> -----------------------------------------
>
> Key: HADOOP-11867
> URL: https://issues.apache.org/jira/browse/HADOOP-11867
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/azure, fs/s3, hdfs-client
> Affects Versions: 3.0.0
> Reporter: Gopal Vijayaraghavan
> Assignee: Mukund Thakur
> Priority: Major
> Labels: performance, pull-request-available
> Time Spent: 13h
> Remaining Estimate: 0h
>
> The most significant way to read from a filesystem in an efficient way is to
> let the FileSystem implementation handle the seek behaviour underneath the
> API to be the most efficient as possible.
> A better approach to the seek problem is to provide a sequence of read
> locations as part of a single call, while letting the system schedule/plan
> the reads ahead of time.
> This is exceedingly useful for seek-heavy readers on HDFS, since this allows
> for potentially optimizing away the seek-gaps within the FSDataInputStream
> implementation.
> For seek+read systems with even more latency than locally-attached disks,
> something like a {{readFully(long[] offsets, ByteBuffer[] chunks)}} would
> take of the seeks internally while reading chunk.remaining() bytes into each
> chunk (which may be {{slice()}}ed off a bigger buffer).
> The base implementation can stub in this as a sequence of seeks + read() into
> ByteBuffers, without forcing each FS implementation to override this in any
> way.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
