Distcp : setup with update is too slow when latency is high
-----------------------------------------------------------
Key: MAPREDUCE-2149
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2149
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp
Affects Versions: 0.21.0, 0.20.2
Reporter: Raghu Angadi
Assignee: Raghu Angadi
If you run distcp with '-update' option, for _each of the files_ present on
source cluster setup invokes a separate RPC to destination cluster to fetch
file info.
Usually this overhead is not very noticeable when both cluster are
geographically close to each other. But if the latency is large, setup could
take couple of orders of magnitude longer.
E.g. : source has 10k directories, each with about 10 files, round trip latency
between source and destination is 75 ms (typical for coast-to-coast clusters).
If we run distcp on source cluster, set up would take about _2.5 hours_
irrespective of whether destination has these files or not. '-lsr' on the same
dest dir from source cluster would take up to 12 min (depending on how many
directories already exist on dest).
* A fairly simple fix to how setup() iterates should bring the set up time to
same as '-lsr'. I will have a patch for this.. (though 12 min is too large).
* A more scalable option is to differ update check to mappers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
