[
https://issues.apache.org/jira/browse/HADOOP-13560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13560:
------------------------------------
Attachment: HADOOP-13560-branch-2-001.patch
Patch 001
# scale test which can create a configurable file; good to try with small (<1
part) uploads as well as medium, and uploads so big they bring your heap down.
# New Scale Test (prefix STest) likes lots of upload bandwidth. It's running
non-parallelized...I wonder if it should be left out of integration entirely.
# they are also good for demonstrating the failure modes of S3A bits
# Note how test methods explicitly ask JUnit to run in a given order. Allows
the tests to isolate their operations yet still work in sequence.
# The upload one does a no-op if the destination file of that size exists. It
was meant to let you skip that bit if it was already there...I think I'll pull
that feature as it only gets in the way.
$ lots of extra monitoring of what is going on inside S3A, including a gauge of
active PUT request counts and bytes pending.
# more troubleshooting docs.
# The fast output stream will retry on errors during request completion and
abort.
* HADOOP-13569 S3AFastOutputStream to take ProgressListener in file create();
used in test runs
* HADOOP-13566 NPE in S3AFastOutputStream.write
* HADOOP-13567 S3AFileSystem to override getStoragetStatistics() and so serve
up its statistics
> S3A to support huge file writes and operations -with tests
> ----------------------------------------------------------
>
> Key: HADOOP-13560
> URL: https://issues.apache.org/jira/browse/HADOOP-13560
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.9.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HADOOP-13560-branch-2-001.patch
>
>
> An AWS SDK [issue|https://github.com/aws/aws-sdk-java/issues/367] highlights
> that metadata isn't copied on large copies.
> 1. Add a test to do that large copy/rname and verify that the copy really
> works
> 1. Verify that metadata makes it over.
> Verifying large file rename is important on its own, as it is needed for very
> large commit operations for committers using rename
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
