Sayeenath created HADOOP-16770: ---------------------------------- Summary: Compare two directories in HDFS filesystem for every 5 mins interval for same cluster. (smiliar like diff command in linux) Key: HADOOP-16770 URL: https://issues.apache.org/jira/browse/HADOOP-16770 Project: Hadoop Common Issue Type: Task Components: hdfs-client Affects Versions: 2.10.0 Reporter: Sayeenath
Hi team, Created two hadoop clusters, one cluster is storing files in new directories based on TIME based directories are created in Hadoop FileSystem say /a/b/time/a.txt b.txt.. For every 5 mins, compare this cluster 1 filesytem for two different directories whether any new directories with list of files are updated or not , if its updated in dir 1, then update those files only to be moved to dir 2. Later those new directories files copied to HDFS cluster 2 file system. Currently HDFS not supported hdfs dfs -diff command, Any solution for this? Have tried -copyFromLocal and copyToLocal command, it uses lot of diskspace while copying local to hdfs & hdfs to local. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org