[
https://issues.apache.org/jira/browse/HADOOP-15292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16387551#comment-16387551
]
Steve Loughran commented on HADOOP-15292:
-----------------------------------------
I am only now becoming aware of how some aspects of DistCp are inefficient when
used against object stores (HADOOP-15281, HADOOP-15209); this looks like
another.
# The pread is potentially most underperforming in object stores which don't do
good seeks on input streams: s3a in the distant past, swift,
# I'd like to see what happens when you test that in hadoop-aws (it's a scale
test, which needs -Dscale and {{ITestAzureNativeContractDistCp}}.
# Stream opening count be measured in {{ITestS3AContractDistCp}}, if a test
for this was wired up & FileSystem.Statistics queried before and after the
operation.
# L305: could add a check for object store clients which seek even when desired
== actual pos by wrapping seek() with {{if (getPos() != position}}
And for extra rigorousness, build up hadoop and try doing some real distcp with
src/dest any of the stores and ftp, which is used as a backup source.
> Distcp's use of pread is slowing it down.
> -----------------------------------------
>
> Key: HADOOP-15292
> URL: https://issues.apache.org/jira/browse/HADOOP-15292
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
> Affects Versions: 3.0.0
> Reporter: Virajith Jalaparti
> Priority: Minor
> Attachments: HADOOP-15292.000.patch
>
>
> Distcp currently uses positioned-reads (in
> RetriableFileCopyCommand#copyBytes) when the source offset is > 0. This
> results in unnecessary overheads (new BlockReader being created on the
> client-side, multiple readBlock() calls to the Datanodes, each of requires
> the creation of a BlockSender and an inputstream to the ReplicaInfo).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
