[
https://issues.apache.org/jira/browse/HADOOP-16770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16999551#comment-16999551
]
Kihwal Lee commented on HADOOP-16770:
-------------------------------------
That's not something file systems support. I am sure many users have built a
simple rsync-like tools.
> Compare two directories in HDFS filesystem for every 5 mins interval for same
> cluster. (smiliar like diff command in linux)
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-16770
> URL: https://issues.apache.org/jira/browse/HADOOP-16770
> Project: Hadoop Common
> Issue Type: Task
> Components: hdfs-client
> Affects Versions: 2.10.0
> Reporter: GanGSTR
> Priority: Major
>
> Hi team,
> Created two hadoop clusters, one cluster is storing files in new directories
> based on TIME based directories are created in Hadoop FileSystem say
> /a/b/time/a.txt b.txt..
> For every 5 mins, compare this cluster 1 filesytem for two different
> directories whether any new directories with list of files are updated or not
> , if its updated in dir 1, then update those files only to be moved to dir 2.
> Later those new directories files copied to HDFS cluster 2 file system.
> Currently HDFS not supported hdfs dfs -diff command, Any solution for this?
> Have tried -copyFromLocal and copyToLocal command, it uses lot of diskspace
> while copying local to hdfs & hdfs to local.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
