It doesn't necessarily have to be on the same machine, but the machine NiFi is on would have to be able to communicate with the name-node and data-nodes in order to push/pull data to/from HDFS. In your example this would mean your local machine would need to be able to access the name-node and data-node on your VM.
In the cluster to cluster scenario... If each cluster is mostly closed off from a networking perspective, you could potentially have edge nodes on each cluster that were able to reach each other. Each of those edge nodes could run a NiFi instance, and the two NiFi instances could talk directly to each other. The first one would use List/FetchHDFS and would have to be able to communicate with cluster #1, it would send data via site-to-site to the second instance which would use PutHDFS and have to be able to communicate with cluster #2. If both clusters can fully talk to each other than you don't need this edge node approach, you could just have a NiFi instance, or cluster, that pulls from one HDFS and pushes to the other. As far as comparing to distcp, keep in mind that distcp launches a map-reduce job to perform a heavily parallelized copy, this would work a little different in NiFi. If you had a NiFi cluster you could scale it so each node in the cluster was pulling data, otherwise with a single instance it would be limited to how much processing that instance can perform. Hope this helps. -Bryan On Wed, Dec 9, 2015 at 12:01 AM, digvijayp <digvijay.pisal1...@gmail.com> wrote: > Thanks for the responce Bryan .... > > I am getting the same error when applying the /root as directory. > So is it necessary to install nifi on the same machine when we install the > hadoop? > Basically i am trying to explore to use nifi to data flow from 1 HDFS > cluster to other HDFS cluster instead of doing manually by distcp. Is nifi > recommended to be used in such scenario? > > Thanks in advance > > > > -- > View this message in context: > http://apache-nifi-developer-list.39713.n7.nabble.com/Facing-Issue-while-connecting-with-HDFS-tp5684p5703.html > Sent from the Apache NiFi Developer List mailing list archive at > Nabble.com. >
