[
https://issues.apache.org/jira/browse/HADOOP-15606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16675115#comment-16675115
]
Steve Loughran commented on HADOOP-15606:
-----------------------------------------
* RawLocalFS can do this with {{FileInputStream.getFileChannel.getSize()}}
* HDFS methods can be relayed to
* unencrypted S3AInputStream can just return the content length the file was
created with. If the file changes during the read (or existing changes become
manifest), this will become inconsistent
* CSE-S3 files will have to return their length differently, but I'm not sure
how well seeking works in their world anyway
> FSDataInputStream/FSInputStream to implement getFileLength()
> ------------------------------------------------------------
>
> Key: HADOOP-15606
> URL: https://issues.apache.org/jira/browse/HADOOP-15606
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/s3
> Affects Versions: 3.1.0
> Reporter: Steve Loughran
> Priority: Major
>
> server-side transcoding, predicate-pushdown filtering in storage, client-side
> decryption can all result in an input stream shorter or longer than the file
> size as measured in getFileStatus/listFiles/listStatus.
> Assuming the length is known once open() returns, the FSDataInputStream can
> return the length of the data, which can then be used for accurate seeks
> within the data.
> * requires the streams to know their length; easy to check (hasCapabilities),
> or make the new method return an Optional<Long>, but caller will need to (a)
> look for this if present and (b) fall back.
> * Could be rolled out for all our own clients/connectors, would take time for
> others
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
