[
https://issues.apache.org/jira/browse/MAPREDUCE-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Liu updated MAPREDUCE-7292:
--------------------------------
Attachment: MAPREDUCE-7292.002.patch
> Use snapshot diff to build file listing when copying to blob storage
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-7292
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7292
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: distcp
> Reporter: John Liu
> Priority: Minor
> Attachments: MAPREDUCE-7292.001.patch, MAPREDUCE-7292.002.patch
>
>
> The DistCp sync option should be extensible for copying to blob storage,
> which is not a distributed filesystem. Clients of DistCp could benefit from
> using the HDFS snapshot diff report to create the file listing in less time.
> A valid use case is to copy new files added to HDFS to a remote blob storage.
> The client ensures all new files are copied over but does not require the
> destination filesystem to be a distributed filesystem or have the previous
> snapshot.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
