John Liu created HDFS-15522:
-------------------------------
Summary: Use snapshot diff to build file listing when copying to
blob storage
Key: HDFS-15522
URL: https://issues.apache.org/jira/browse/HDFS-15522
Project: Hadoop HDFS
Issue Type: Improvement
Components: distcp
Reporter: John Liu
The DistCp sync option should be extensible for copying to blob storage, which
is not a distributed filesystem. Clients of DistCp could benefit from using the
HDFS snapshot diff report to create the file listing in less time. A valid use
case is to copy new files added to HDFS to a remote blob storage. The client
ensures all new files are copied over but does not require the destination
filesystem to be a distributed filesystem or have the previous snapshot.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
