On Feb 28, 2008, at 2:43 AM, Miles Osborne wrote:
Currently, we have the following setup:
--cluster A, running Nutch: small RAM per node
--cluster B, just running Hadoop: lots of RAM per node
At some point in the future we will want cluster B to talk to
cluster A, and
ideally this should be DFS-to-DFS
Is this possible? Or do we need to do something like:
Cluster A --> Unix filesystem --> Cluster B
via hadoop dfs -cat / -put operations etc
To copy between clusters, there is a tool called distcp. Look at "bin/
hadoop distcp". It runs a map/reduce job that copies a group of
files. It can also be used to copy between versions of hadoop, if the
source file system is hftp, which uses xml to read hdfs.
-- Owen
