[
https://issues.apache.org/jira/browse/HADOOP-11525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299875#comment-14299875
]
Steve Loughran commented on HADOOP-11525:
-----------------------------------------
I should add a couple of more points here
# if we do want to expose performance characteristics, there may be other
aspects to publish. Today "blocksize" is the sole value exported. recommended
read buffer and write buffer sizes are another. It's hard to tune though:
buffer policy is a function of the latency and bandwidth to the far end, as
well as the characteristics of the test destination
# individual filesystem instances do need the right to publish different
values. Example: S3 EU has create-consistency, whereas S3 US E doesn't, third
party S3-compatible filesystems may be fully consistent.
> FileSystem should expose some performance characteristics for caller (e.g.,
> FsShell) to choose the right algorithm.
> -------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-11525
> URL: https://issues.apache.org/jira/browse/HADOOP-11525
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools
> Affects Versions: 2.6.0
> Reporter: Lei (Eddy) Xu
> Assignee: Lei (Eddy) Xu
> Attachments: HADOOP-11525.000.patch
>
>
> When running {{hadoop fs -put}}, {{FsShell}} creates a {{._COPYING_.}} file
> on the target directory, and then renames it to target file when the write is
> done. However, for some targeted systems, such as S3, Azure and Swift, a
> partial failure write request (i.e., {{PUT}}) has not side effect, while the
> {{rename}} operation is expensive.
> {{FileSystem}} should expose some characteristics so that the operation such
> as {{CommandWithDestination#copyStreamToTarget()}} can detect and choose the
> right way to do.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
