Hi Andre, Yes we use that approach at my company. We have almost 15+ flows running on NiFi-> Kafka-> NiFi in production without any major issues. UUID is a drawback but in our case, it is not an issue so we were good to go with this approach.
Only issue you might face is if you have a lot of flows connecting to Kafka/Zookeeper that might need an increase in connection limit in Zookeeper. For that, if you are using Hortonworks HDP then in zookeeper *"custom zoo.cfg"* add the following property: *maxClientCnxns=*number of connections you might require*. *For example, you can put maxClientCnxns to like 600 or so There should not be any other major issues. Let me know if I can help with anything. Cheers, Raman On Fri, Jan 12, 2018 at 3:22 AM, Andre <[email protected]> wrote: > All, > > Has anyone ever used Kafka as a "replacement" for Site to Site(not really > but sort of)? > > The idea is to take from Mark Payne's comments back in Dec 2016 [1] and > use something like: > > (*.* flow) -> MergeContent (FlowFile Stream v3) -> PublishKafka -> KAFKA > -> ConsumeKafka -> UnpackContent -> (*.* flow) > > > While the technique is possible, has anyone used this in production? > > Any caveats other than the new uuid already highlighted by Mark? > > > Cheers > > > > [1] https://lists.apache.org/thread.html/9bd4a4abf5d94e85d7e > a62a3a2bd8c0079f8ccd827a68ad9d3de0d04@%3Cdev.nifi.apache.org%3E > >
