[ 
https://issues.apache.org/jira/browse/HADOOP-15281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383909#comment-16383909
 ] 

Steve Loughran commented on HADOOP-15281:
-----------------------------------------

debug level log of a distcp of one file to s3a. 20 metadata requests get logged 
per file.
{code}
16:51:20,555 INFO  mapred.CopyMapper (CopyMapper.java:map(154)) - Copying 
file:/home/s/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java
 to 
s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java
16:51:20,555 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - op_get_file_status += 1  ->  
8987
16:51:20,555 DEBUG s3a.S3AFileSystem 
(S3AFileSystem.java:innerGetFileStatus(2098)) - Getting path status for 
s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java
  
(distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java)
16:51:20,555 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests += 
1  ->  19643
16:51:20,591 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests += 
1  ->  19644
16:51:20,626 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - object_list_requests += 1  
->  7689
16:51:20,664 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(2245)) 
- Not Found: 
s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java
16:51:20,664 INFO  mapred.RetriableFileCopyCommand 
(RetriableFileCopyCommand.java:getTmpFile(245)) - Creating temp file: 
s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0
16:51:20,664 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - op_create += 1  ->  851
16:51:20,664 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - op_get_file_status += 1  ->  
8988
16:51:20,664 DEBUG s3a.S3AFileSystem 
(S3AFileSystem.java:innerGetFileStatus(2098)) - Getting path status for 
s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0
  
(distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0)
16:51:20,664 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests += 
1  ->  19645
16:51:20,697 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests += 
1  ->  19646
16:51:20,732 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - object_list_requests += 1  
->  7690
16:51:20,770 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(2245)) 
- Not Found: 
s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0
16:51:20,770 DEBUG s3a.S3ABlockOutputStream 
(S3ABlockOutputStream.java:<init>(169)) - Initialized S3ABlockOutputStream for 
distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0 
output to FileBlock{index=1, 
destFile=/tmp/hadoop-stevel/s3a/s3ablock-0001-1407016756609004936.tmp, 
state=Writing, dataSize=0, limit=8388608}
16:51:20,772 DEBUG s3a.S3ABlockOutputStream 
(S3ABlockOutputStream.java:close(349)) - 
S3ABlockOutputStream{WriteOperationHelper {bucket=hwdev-steve-ireland-new}, 
blockSize=8388608, activeBlock=FileBlock{index=1, 
destFile=/tmp/hadoop-stevel/s3a/s3ablock-0001-1407016756609004936.tmp, 
state=Writing, dataSize=16963, limit=8388608}}: Closing block #1: current 
block= FileBlock{index=1, 
destFile=/tmp/hadoop-stevel/s3a/s3ablock-0001-1407016756609004936.tmp, 
state=Writing, dataSize=16963, limit=8388608}
16:51:20,772 DEBUG s3a.S3ABlockOutputStream 
(S3ABlockOutputStream.java:putObject(413)) - Executing regular upload for 
WriteOperationHelper {bucket=hwdev-steve-ireland-new}
16:51:20,772 DEBUG s3a.S3ADataBlocks (S3ADataBlocks.java:startUpload(324)) - 
Start datablock[1] upload
16:51:20,772 DEBUG s3a.S3ADataBlocks (S3ADataBlocks.java:enterState(231)) - 
FileBlock{index=1, 
destFile=/tmp/hadoop-stevel/s3a/s3ablock-0001-1407016756609004936.tmp, 
state=Writing, dataSize=16963, limit=8388608}: entering state Upload
16:51:20,772 DEBUG s3a.S3ABlockOutputStream 
(S3ABlockOutputStream.java:clearActiveBlock(216)) - Clearing active block
16:51:20,772 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3AFileSystem 
(S3AFileSystem.java:putObjectDirect(1520)) - PUT 16963 bytes to 
distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0
16:51:20,772 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3AFileSystem 
(S3AFileSystem.java:incrementPutStartStatistics(1583)) - PUT start 16963 bytes
16:51:20,772 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - object_put_requests += 1  -> 
 1146
16:51:20,969 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3AFileSystem 
(S3AFileSystem.java:incrementPutCompletedStatistics(1600)) - PUT completed 
success=true; 16963 bytes
16:51:20,969 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - object_put_bytes += 16963  
->  42252442
16:51:20,969 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - 
object_put_requests_completed += 1  ->  1146
16:51:20,969 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3AFileSystem 
(S3AFileSystem.java:finishedWrite(2662)) - Finished write to 
distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0, 
len 16963
16:51:20,969 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - object_delete_requests += 1  
->  2846
16:51:21,017 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3ABlockOutputStream 
(S3AUtils.java:closeAll(1297)) - Closing 
org.apache.hadoop.fs.s3a.S3ADataBlocks$BlockUploadData@5dc0fde5
16:51:21,018 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3ABlockOutputStream 
(S3AUtils.java:closeAll(1297)) - Closing FileBlock{index=1, 
destFile=/tmp/hadoop-stevel/s3a/s3ablock-0001-1407016756609004936.tmp, 
state=Upload, dataSize=16963, limit=8388608}
16:51:21,018 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3ADataBlocks 
(S3ADataBlocks.java:enterState(231)) - FileBlock{index=1, 
destFile=/tmp/hadoop-stevel/s3a/s3ablock-0001-1407016756609004936.tmp, 
state=Upload, dataSize=16963, limit=8388608}: entering state Closed
16:51:21,018 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3ADataBlocks 
(S3ADataBlocks.java:close(346)) - Closed FileBlock{index=1, 
destFile=/tmp/hadoop-stevel/s3a/s3ablock-0001-1407016756609004936.tmp, 
state=Closed, dataSize=16963, limit=8388608}
16:51:21,018 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3ADataBlocks 
(S3ADataBlocks.java:innerClose(886)) - Closing FileBlock{index=1, 
destFile=/tmp/hadoop-stevel/s3a/s3ablock-0001-1407016756609004936.tmp, 
state=Closed, dataSize=16963, limit=8388608}
16:51:21,018 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3ADataBlocks 
(S3ADataBlocks.java:closeBlock(940)) - block[1]: closeBlock()
16:51:21,018 DEBUG s3a.S3ABlockOutputStream 
(S3ABlockOutputStream.java:close(389)) - Upload complete to 
distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0 
by WriteOperationHelper {bucket=hwdev-steve-ireland-new}
16:51:21,019 DEBUG s3a.S3ABlockOutputStream (S3AUtils.java:closeAll(1297)) - 
Closing FileBlock{index=1, 
destFile=/tmp/hadoop-stevel/s3a/s3ablock-0001-1407016756609004936.tmp, 
state=Closed, dataSize=16963, limit=8388608}
16:51:21,019 DEBUG s3a.S3ABlockOutputStream (S3AUtils.java:closeAll(1297)) - 
Closing org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory@5bfba522
16:51:21,019 DEBUG s3a.S3ABlockOutputStream 
(S3ABlockOutputStream.java:close(395)) - Statistics: 
OutputStreamStatistics{blocksSubmitted=1, blocksInQueue=1, blocksActive=0, 
blockUploadsCompleted=0, blockUploadsFailed=0, bytesPendingUpload=0, 
bytesUploaded=16963, blocksAllocated=1, blocksReleased=1, 
blocksActivelyAllocated=0, exceptionsInMultipartFinalize=0, transferDuration=0 
ms, queueDuration=0 ms, averageQueueTime=0 ms, totalUploadDuration=0 ms, 
effectiveBandwidth=0.0 bytes/s}
16:51:21,019 DEBUG s3a.S3ABlockOutputStream (S3AUtils.java:closeAll(1297)) - 
Closing OutputStreamStatistics{blocksSubmitted=1, blocksInQueue=1, 
blocksActive=0, blockUploadsCompleted=0, blockUploadsFailed=0, 
bytesPendingUpload=0, bytesUploaded=16963, blocksAllocated=1, blocksReleased=1, 
blocksActivelyAllocated=0, exceptionsInMultipartFinalize=0, transferDuration=0 
ms, queueDuration=0 ms, averageQueueTime=0 ms, totalUploadDuration=0 ms, 
effectiveBandwidth=0.0 bytes/s}
16:51:21,019 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:getFileChecksum(2990)) 
- 
getFileChecksum(s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0)
16:51:21,019 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests += 
1  ->  19647
16:51:21,193 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - op_exists += 1  ->  1702
16:51:21,193 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - op_get_file_status += 1  ->  
8989
16:51:21,193 DEBUG s3a.S3AFileSystem 
(S3AFileSystem.java:innerGetFileStatus(2098)) - Getting path status for 
s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java
  
(distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java)
16:51:21,193 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests += 
1  ->  19648
16:51:21,230 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests += 
1  ->  19649
16:51:21,264 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - object_list_requests += 1  
->  7691
16:51:21,303 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(2245)) 
- Not Found: 
s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java
16:51:21,303 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - op_exists += 1  ->  1703
16:51:21,303 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - op_get_file_status += 1  ->  
8990
16:51:21,303 DEBUG s3a.S3AFileSystem 
(S3AFileSystem.java:innerGetFileStatus(2098)) - Getting path status for 
s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom
  
(distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom)
16:51:21,303 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests += 
1  ->  19650
16:51:21,340 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests += 
1  ->  19651
16:51:21,397 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(2188)) 
- Found file (with /): fake directory
16:51:21,397 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerRename(889)) - 
Rename path 
s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0
 to 
s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java
16:51:21,397 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - op_rename += 1  ->  851
16:51:21,397 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - op_get_file_status += 1  ->  
8991
16:51:21,397 DEBUG s3a.S3AFileSystem 
(S3AFileSystem.java:innerGetFileStatus(2098)) - Getting path status for 
s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0
  
(distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0)
16:51:21,397 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests += 
1  ->  19652
16:51:21,433 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(2166)) 
- Found exact file: normal file
16:51:21,433 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - op_get_file_status += 1  ->  
8992
16:51:21,433 DEBUG s3a.S3AFileSystem 
(S3AFileSystem.java:innerGetFileStatus(2098)) - Getting path status for 
s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java
  
(distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java)
16:51:21,433 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests += 
1  ->  19653
16:51:21,466 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests += 
1  ->  19654
16:51:21,500 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - object_list_requests += 1  
->  7692
16:51:21,537 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(2245)) 
- Not Found: 
s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java
16:51:21,537 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerRename(943)) - 
rename: destination path 
s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java
 not found
16:51:21,537 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - op_get_file_status += 1  ->  
8993
16:51:21,537 DEBUG s3a.S3AFileSystem 
(S3AFileSystem.java:innerGetFileStatus(2098)) - Getting path status for 
s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom
  
(distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom)
16:51:21,537 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests += 
1  ->  19655
16:51:21,571 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests += 
1  ->  19656
16:51:21,606 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(2188)) 
- Found file (with /): fake directory
16:51:21,606 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerRename(973)) - 
rename: renaming file 
s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0
 to 
s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java
16:51:21,606 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:copyFile(2507)) - 
copyFile 
distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0 
-> 
distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java
 
16:51:21,606 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests += 
1  ->  19657
16:51:21,851 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1724)) - 
Delete path 
s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0
 - recursive false
16:51:21,852 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1781)) - 
delete: Path is a file
16:51:21,852 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - object_delete_requests += 1  
->  2847
16:51:21,890 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerRename(1065)) - 
source & dest parents are different; fix up dir markers
16:51:21,890 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - object_delete_requests += 1  
->  2848
16:51:21,990 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests += 
1  ->  19658
16:51:22,143 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests += 
1  ->  19659
16:51:22,180 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - object_list_requests += 1  
->  7693
16:51:22,223 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(2221)) 
- Found path as directory (with /): 0/1
16:51:22,223 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(2225)) 
- Summary: distcp/hadoop-common-project/.DS_Store 6148
16:51:22,223 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - op_delete += 1  ->  851
16:51:22,224 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - op_get_file_status += 1  ->  
8994
16:51:22,224 DEBUG s3a.S3AFileSystem 
(S3AFileSystem.java:innerGetFileStatus(2098)) - Getting path status for 
s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0
  
(distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0)
16:51:22,224 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests += 
1  ->  19660
16:51:22,260 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests += 
1  ->  19661
16:51:22,295 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - object_list_requests += 1  
->  7694
16:51:22,335 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(2245)) 
- Not Found: 
s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0
16:51:22,335 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:delete(1700)) - 
Couldn't delete 
s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0
 - does not exist
16:51:22,335 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - op_get_file_status += 1  ->  
8995
16:51:22,335 DEBUG s3a.S3AFileSystem 
(S3AFileSystem.java:innerGetFileStatus(2098)) - Getting path status for 
s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java
  
(distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java)
16:51:22,335 DEBUG s3a.S3AStorageStatistics 
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests += 
1  ->  19662
16:51:22,378 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(2166)) 
- Found exact file: normal file
{code}

> Distcp to add no-rename copy option
> -----------------------------------
>
>                 Key: HADOOP-15281
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15281
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: tools/distcp
>    Affects Versions: 3.0.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Major
>
> Currently Distcp uploads a file by two strategies
> # append parts
> # copy to temp then rename
> option 2 executes the following swquence in {{promoteTmpToTarget}}
> {code}
>     if ((fs.exists(target) && !fs.delete(target, false))
>         || (!fs.exists(target.getParent()) && !fs.mkdirs(target.getParent()))
>         || !fs.rename(tmpTarget, target)) {
>       throw new IOException("Failed to promote tmp-file:" + tmpTarget
>                               + " to: " + target);
>     }
> {code}
> For any object store, that's a lot of HTTP requests; for S3A you are looking 
> at 12+ requests and an O(data) copy call. 
> This is not a good upload strategy for any store which manifests its output 
> atomically at the end of the write().
> Proposed: add a switch to write direct to the dest path. either a conf option 
> or a CLI option



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

Reply via email to