[ https://issues.apache.org/jira/browse/HADOOP-11867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349507#comment-16349507 ]
Gopal V commented on HADOOP-11867: ---------------------------------- No, this is not about zero-copy readers or locality (i.e no Located blocks, no MMAP), but about the API common between all the disaggregated storage engines, to reduce total API round-trips to fetch ranges (delegating internal block layouts to the storage layer). HDFS-3051 looks like it is specific optimization to the HDFS implementation, which speeds up the IO when you have locality along with an API like this. > FS API: Add a high-performance vectored Read to FSDataInputStream API > --------------------------------------------------------------------- > > Key: HADOOP-11867 > URL: https://issues.apache.org/jira/browse/HADOOP-11867 > Project: Hadoop Common > Issue Type: New Feature > Affects Versions: 2.8.0 > Reporter: Gopal V > Assignee: Gopal V > Priority: Major > > The most significant way to read from a filesystem in an efficient way is to > let the FileSystem implementation handle the seek behaviour underneath the > API to be the most efficient as possible. > A better approach to the seek problem is to provide a sequence of read > locations as part of a single call, while letting the system schedule/plan > the reads ahead of time. > This is exceedingly useful for seek-heavy readers on HDFS, since this allows > for potentially optimizing away the seek-gaps within the FSDataInputStream > implementation. > For seek+read systems with even more latency than locally-attached disks, > something like a {{readFully(long[] offsets, ByteBuffer[] chunks)}} would > take of the seeks internally while reading chunk.remaining() bytes into each > chunk (which may be {{slice()}}ed off a bigger buffer). > The base implementation can stub in this as a sequence of seeks + read() into > ByteBuffers, without forcing each FS implementation to override this in any > way. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org