Keaton, Glad you are liking it. Honestly DistCP is purpose built for the case of copying from one cluster to another. I don't think I'd recommend NiFi over DistCP for that very focused use case. DistCP should be by design the best at that.
NiFI definitely is used extensively for data center to data center flows including automating feeds of data showing up in portions of HDFS on one side to a directory on another but that is in an automated dataflow fashion rather than the snapshot/copy that DistCP represents. Thanks Joe On Fri, Feb 12, 2016 at 2:24 PM, Keaton Cleve <[email protected]> wrote: > Hi all, > > I have been evaluating Nifi and it seems like a very good tool. > Congratulations. > > I have different workflows in mind such as File > Kafka, File > HTTP, File > > HDFS, HDFS > HTTP, etc. for which Nifi seem very suited. > > One question I have however is how suited is Nifi for workflows such as HDFS > in data center to HDFS in another data center? It would be nice to have all > the dataflows in the same tool and to benefit from Nifi's data provenance > for those tasks, but how would performance compare to doing a Distcp task? > > What would be other advantages of using Nifi over Distcp? > > Thanks,
