[
https://issues.apache.org/jira/browse/HADOOP-18247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17540194#comment-17540194
]
Ahmar Suhail commented on HADOOP-18247:
---------------------------------------
testOpenFileShorterLength because inputStreamStats need to created before
opening the stream so that the time to prepare to open the file is included.
testOpenFileLongerLength fails because in this [read
method|https://github.com/apache/hadoop/blob/feature-HADOOP-18028-s3a-prefetch/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3Reader.java#L95]
when for eg: file length is 8 and length to read is 18, it swallows the EOF
exception and returns the num of successful bytes read which will be 8. So
readFully doesn't throw an EOF immediately but then tries to read bytes 8-18
again, and then fails. To fix this, readOneBlockWithRetries should check the
value returned
[here|https://github.com/apache/hadoop/blob/feature-HADOOP-18028-s3a-prefetch/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3Reader.java#L101]
and return -1 if there's an EOF is readOneBlock. Similar to behaviour in
S3AInputStream
[here|https://github.com/apache/hadoop/blob/feature-HADOOP-18028-s3a-prefetch/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L497]
Should also maybe add a test for larger files so S3CachingInputStream's
behaviour is tested?
> Tests in ITestS3AOpenCost are failing
> -------------------------------------
>
> Key: HADOOP-18247
> URL: https://issues.apache.org/jira/browse/HADOOP-18247
> Project: Hadoop Common
> Issue Type: Sub-task
> Reporter: Ahmar Suhail
> Assignee: Ahmar Suhail
> Priority: Minor
>
> After rebasing, when prefetching is enabled testOpenFileLongerLength &
> testOpenFileShorterLength fail
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
