Steve Loughran commented on HADOOP-15292:

I am only now becoming aware of how some aspects of DistCp are inefficient when 
used against object stores (HADOOP-15281, HADOOP-15209); this looks like 

# The pread is potentially most underperforming in object stores which don't do 
good seeks on input streams: s3a in the distant past, swift, 
# I'd like to see what happens when you test that in hadoop-aws (it's a scale 
test, which needs -Dscale and {{ITestAzureNativeContractDistCp}}. 
# Stream opening count be measured inĀ  {{ITestS3AContractDistCp}}, if a test 
for this was wired up & FileSystem.Statistics queried before and after the 
# L305: could add a check for object store clients which seek even when desired 
== actual pos by wrapping seek() with {{if (getPos() != position}}

And for extra rigorousness, build up hadoop and try doing some real distcp with 
src/dest any of the stores and ftp, which is used as a backup source.

> Distcp's use of pread is slowing it down.
> -----------------------------------------
>                 Key: HADOOP-15292
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15292
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: tools/distcp
>    Affects Versions: 3.0.0
>            Reporter: Virajith Jalaparti
>            Priority: Minor
>         Attachments: HADOOP-15292.000.patch
> Distcp currently uses positioned-reads (in 
> RetriableFileCopyCommand#copyBytes) when the source offset is > 0. This 
> results in unnecessary overheads (new BlockReader being created on the 
> client-side, multiple readBlock() calls to the Datanodes, each of requires 
> the creation of a BlockSender and an inputstream to the ReplicaInfo).

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

Reply via email to