Virajith Jalaparti commented on HADOOP-15292:

bq.  Probably not worth adding metrics but maybe extend the stream in the unit 
test and track how many times we open it.

[~elgoiri], I extended {{TestCopyMapper#testCopyWithAppend}} to test the number 
of calls to {{readBlock}} on the Datanode (this is captured by the metric 
{{readsFromLocalClient}}), in  [^HADOOP-15292.001.patch]. The buffer size is 
set such that if pread was used, this metric would increase much more than the 
number of files that are being appended to.

[~ste...@apache.org] I tested this over an internal filesystem. This issue came 
up when I was using distcp to copy data from the filesystem using the tiered 
HDFS feature (HDFS-9806). This particular filesystem throttled calls to open() 
beyond a certain QPS. For large enough data, distcp never succeeded as it 
caused way too many calls to open() (pread causes a separate InputStream to be 
opened for the {{ProvidedReplica}} for each 8k chunks of data, which result in 
an open() call to the remote filesystem). With this fix, Distcp runs fine, and 
I don't see the throttling any more. 
I think this goes towards to the "extra rigorousness" you are looking for. I am 
not really setup to test this with s3.

> Distcp's use of pread is slowing it down.
> -----------------------------------------
>                 Key: HADOOP-15292
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15292
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: tools/distcp
>    Affects Versions: 3.0.0
>            Reporter: Virajith Jalaparti
>            Priority: Minor
>         Attachments: HADOOP-15292.000.patch, HADOOP-15292.001.patch
> Distcp currently uses positioned-reads (in 
> RetriableFileCopyCommand#copyBytes) when the source offset is > 0. This 
> results in unnecessary overheads (new BlockReader being created on the 
> client-side, multiple readBlock() calls to the Datanodes, each of which 
> requires the creation of a BlockSender and an inputstream to the ReplicaInfo).

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

Reply via email to