[
https://issues.apache.org/jira/browse/HADOOP-11570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14315940#comment-14315940
]
Dan Hecht commented on HADOOP-11570:
------------------------------------
I've run the tests:
{code}
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 805.836 sec -
in org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles
Running org.apache.hadoop.fs.s3a.TestS3AFileSystemContract
Tests run: 43, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 99.061 sec -
in org.apache.hadoop.fs.s3a.TestS3AFileSystemContract
Running org.apache.hadoop.fs.s3a.TestS3AConfiguration
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.339 sec - in
org.apache.hadoop.fs.s3a.TestS3AConfiguration
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.7 sec - in
org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate
Tests run: 6, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 11.965 sec - in
org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.024 sec - in
org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.032 sec - in
org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.334 sec -
in org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.292 sec - in
org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.697 sec - in
org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir
Results :
Tests run: 96, Failures: 0, Errors: 0, Skipped: 3
{code}
I had also verified that the patch does as expected by watching RX bytes using
'iftop' under various seek/read patterns.
> S3AInputStream.close() downloads the remaining bytes of the object from S3
> --------------------------------------------------------------------------
>
> Key: HADOOP-11570
> URL: https://issues.apache.org/jira/browse/HADOOP-11570
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.6.0
> Reporter: Dan Hecht
> Attachments: HADOOP-11570-001.patch
>
>
> Currently, S3AInputStream.close() calls S3Object.close(). But,
> S3Object.close() will read the remaining bytes of the S3 object, potentially
> transferring a lot of bytes from S3 that are discarded. Instead, the wrapped
> stream should be aborted to avoid transferring discarded bytes (unless the
> preceding read() finished at contentLength). For example, reading only the
> first byte of a 1 GB object and then closing the stream will result in all 1
> GB transferred from S3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
