[
https://issues.apache.org/jira/browse/HADOOP-15384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16522708#comment-16522708
]
Steve Loughran commented on HADOOP-15384:
-----------------------------------------
Patch 001: passes thread count down, logs it, and splits time for reporting
listing into source and dest
This has tangible improvements when using object stores as a destination,
though the mimicking of directories trees can still make distcp to some remote
stores (s3, swift) still tangibly awful. This is all you can do short of a
complete rewrite. Which I don't intend to propose for the following reason:
distcp is a complex and critical part of too many people's workflows.
Testing: ran {{ITestS3AContractDistCp}} against S3 Ireland. No new tests, as
the distcp contract tests were already setting num threads: I have set the
thread count to 40 though, that being the maximum.
Here's the output of the relevant phase of
{{testUpdateDeepDirectoryStructureToRemote}}
{code}
2018-06-25 19:35:18,047 [Thread-139] INFO mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(387)) - -delete option is enabled. About to
remove entries from target that are missing in source
2018-06-25 19:35:18,062 [Thread-139] INFO mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(396)) - Source listing completed in
0:00:00.015
2018-06-25 19:35:18,063 [Thread-139] INFO mapred.CopyCommitter
(CopyCommitter.java:listTargetFiles(554)) - Scanning destination directory
s3a://hwdev-steve-ireland-new/test/ITestS3AContractDistCp/testUpdateDeepDirectoryStructureToRemote/remote/DELAY_LISTING_ME/outputDir/inputDir
with thread count: 40
2018-06-25 19:35:19,872 [Thread-139] INFO tools.SimpleCopyListing
(SimpleCopyListing.java:printStats(608)) - Paths (files+dirs) cnt = 11; dirCnt
= 5
2018-06-25 19:35:19,872 [Thread-139] INFO tools.SimpleCopyListing
(SimpleCopyListing.java:doBuildListing(402)) - Build file listing completed.
2018-06-25 19:35:19,886 [Thread-139] INFO tools.DistCp
(CopyListing.java:buildListing(94)) - Number of paths in the copy list: 11
2018-06-25 19:35:19,899 [Thread-139] INFO tools.DistCp
(CopyListing.java:buildListing(94)) - Number of paths in the copy list: 11
2018-06-25 19:35:19,913 [Thread-139] INFO mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(415)) - Destination listing completed in
0:00:01.851
{code}
And for {{ITestAzureNativeContractDistCp}}
{code}
2018-06-25 20:11:44,992 INFO [Thread-147]: mapred.LocalJobRunner
(LocalJobRunner.java:runTasks(486)) - map task executor complete.
2018-06-25 20:11:44,992 INFO [Thread-147]: mapred.CopyCommitter
(CopyCommitter.java:concatFileChunks(210)) - concat file chunks ...
2018-06-25 20:11:45,405 INFO [Thread-147]: mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(387)) - -delete option is enabled. About to
remove entries from target that are missing in source
2018-06-25 20:11:45,418 INFO [Thread-147]: mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(396)) - Source listing completed in
0:00:00.013
2018-06-25 20:11:45,419 INFO [Thread-147]: mapred.CopyCommitter
(CopyCommitter.java:listTargetFiles(554)) - Scanning destination directory
wasb://[email protected]/test/ITestAzureNativeContractDistCp/testUpdateDeepDirectoryStructureToRemote/remote/outputDir/inputDir
with thread count: 40
2018-06-25 20:11:46,338 INFO [Thread-147]: tools.SimpleCopyListing
(SimpleCopyListing.java:printStats(608)) - Paths (files+dirs) cnt = 11; dirCnt
= 5
2018-06-25 20:11:46,338 INFO [Thread-147]: tools.SimpleCopyListing
(SimpleCopyListing.java:doBuildListing(402)) - Build file listing completed.
2018-06-25 20:11:46,351 INFO [Thread-147]: tools.DistCp
(CopyListing.java:buildListing(94)) - Number of paths in the copy list: 11
2018-06-25 20:11:46,365 INFO [Thread-147]: tools.DistCp
(CopyListing.java:buildListing(94)) - Number of paths in the copy list: 11
2018-06-25 20:11:46,377 INFO [Thread-147]: mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(415)) - Destination listing completed in
0:00:00.959
{code}
A small bird fell out the sky, deceased, during the S3A run. It didn't happen
on a rerun —I'm assuming unrelated. If more wild animals die during S3
integration tests then it'd be something to consider a significant regression
in the AWS SDK
+ [~ehiggs], [~fabbri]
> distcp numListstatusThreads option doesn't get to -delete scan
> --------------------------------------------------------------
>
> Key: HADOOP-15384
> URL: https://issues.apache.org/jira/browse/HADOOP-15384
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: tools/distcp
> Affects Versions: 3.1.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Attachments: HADOOP-15384-001.patch
>
>
> The distcp {{numListstatusThreads}} option isn't used when configuring the
> GlobbedCopyListing used in {{CopyComitter.deleteMissing()}}
> This means that for large scans of object stores, performance is
> significantly worse.
> Fix: pass the option down from the task conf
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
