[
https://issues.apache.org/jira/browse/HADOOP-13560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15465361#comment-15465361
]
Steve Loughran commented on HADOOP-13560:
-----------------------------------------
Cancelling patch to do the fix for this
# Support multiple back ends for buffering streams (current one for regression
testing, java nio bytebuffer for real use & potentially off-heap storage, and
files fixed to partition size. The latter will have no limit on size other than
HDD capacity & do async upload for incremental writes, rather than the big
delay on close(). This will mean that it should be usable everywhere.
# collect metrics on upload performance: #of writes, number active, delay
between submitting to the executor pool and execution, per-stream upload
bandwidth as inferred from duration of the upload operations, per-stream and
per FS instance. That will let you measure what is going on w.r.t s3a upload
performance in production, including collecting stats of what b/w you get from
different VM instances.
> S3A to support huge file writes and operations -with tests
> ----------------------------------------------------------
>
> Key: HADOOP-13560
> URL: https://issues.apache.org/jira/browse/HADOOP-13560
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.9.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HADOOP-13560-branch-2-001.patch
>
>
> An AWS SDK [issue|https://github.com/aws/aws-sdk-java/issues/367] highlights
> that metadata isn't copied on large copies.
> 1. Add a test to do that large copy/rname and verify that the copy really
> works
> 1. Verify that metadata makes it over.
> Verifying large file rename is important on its own, as it is needed for very
> large commit operations for committers using rename
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
