Sayeenath created HADOOP-16770:
----------------------------------
Summary: Compare two directories in HDFS filesystem for every 5
mins interval for same cluster. (smiliar like diff command in linux)
Key: HADOOP-16770
URL: https://issues.apache.org/jira/browse/HADOOP-16770
Project: Hadoop Common
Issue Type: Task
Components: hdfs-client
Affects Versions: 2.10.0
Reporter: Sayeenath
Hi team,
Created two hadoop clusters, one cluster is storing files in new directories
based on TIME based directories are created in Hadoop FileSystem say
/a/b/time/a.txt b.txt..
For every 5 mins, compare this cluster 1 filesytem for two different
directories whether any new directories with list of files are updated or not ,
if its updated in dir 1, then update those files only to be moved to dir 2.
Later those new directories files copied to HDFS cluster 2 file system.
Currently HDFS not supported hdfs dfs -diff command, Any solution for this?
Have tried -copyFromLocal and copyToLocal command, it uses lot of diskspace
while copying local to hdfs & hdfs to local.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
