[
https://issues.apache.org/jira/browse/HADOOP-15292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16388928#comment-16388928
]
Virajith Jalaparti commented on HADOOP-15292:
---------------------------------------------
bq. Probably not worth adding metrics but maybe extend the stream in the unit
test and track how many times we open it.
[~elgoiri], I extended {{TestCopyMapper#testCopyWithAppend}} to test the number
of calls to {{readBlock}} on the Datanode (this is captured by the metric
{{readsFromLocalClient}}), in [^HADOOP-15292.001.patch]. The buffer size is
set such that if pread was used, this metric would increase much more than the
number of files that are being appended to.
[[email protected]] I tested this over an internal filesystem. This issue came
up when I was using distcp to copy data from the filesystem using the tiered
HDFS feature (HDFS-9806). This particular filesystem throttled calls to open()
beyond a certain QPS. For large enough data, distcp never succeeded as it
caused way too many calls to open() (pread causes a separate InputStream to be
opened for the {{ProvidedReplica}} for each 8k chunks of data, which result in
an open() call to the remote filesystem). With this fix, Distcp runs fine, and
I don't see the throttling any more.
I think this goes towards to the "extra rigorousness" you are looking for. I am
not really setup to test this with s3.
> Distcp's use of pread is slowing it down.
> -----------------------------------------
>
> Key: HADOOP-15292
> URL: https://issues.apache.org/jira/browse/HADOOP-15292
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
> Affects Versions: 3.0.0
> Reporter: Virajith Jalaparti
> Priority: Minor
> Attachments: HADOOP-15292.000.patch, HADOOP-15292.001.patch
>
>
> Distcp currently uses positioned-reads (in
> RetriableFileCopyCommand#copyBytes) when the source offset is > 0. This
> results in unnecessary overheads (new BlockReader being created on the
> client-side, multiple readBlock() calls to the Datanodes, each of which
> requires the creation of a BlockSender and an inputstream to the ReplicaInfo).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
