[
https://issues.apache.org/jira/browse/HADOOP-15606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16548372#comment-16548372
]
Steve Loughran commented on HADOOP-15606:
-----------------------------------------
Not sure this will suffice; FileInputFormat is doing the partitioning off
listFiles() responses, so it is using the length at that time to partition
things. It will need a way to say "until EOF" so partitions know to query the
stream (or just read() to EOF), Potentially traumatic across tools
> FSDataInputStream/FSInputStream to implement a length()/size() method for
> amount of data
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-15606
> URL: https://issues.apache.org/jira/browse/HADOOP-15606
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/s3
> Affects Versions: 3.1.0
> Reporter: Steve Loughran
> Priority: Major
>
> server-side transcoding, predicate-pushdown filtering in storage, client-side
> decryption can all result in an input stream shorter or longer than the file
> size as measured in getFileStatus/listFiles/listStatus.
> Assuming the length is known once open() returns, the FSDataInputStream can
> return the length of the data, which can then be used for accurate seeks
> within the data.
> * requires the streams to know their length; easy to check (hasCapabilities),
> or make the new method return an Optional<Long>, but caller will need to (a)
> look for this if present and (b) fall back.
> * Could be rolled out for all our own clients/connectors, would take time for
> others
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
