Hey, guys and gals. I’m pretty new to NiFi so its quite possible, if not likely, that I am missing something. That said, would someone please comment on this observation. conclusion and and questions.
My observation is that FlowFiles get queued in local queues on individual nodes and those flow files and queues are not replicated to other nodes. So its my conclusion that if a node is lost, the files queued to that node are also lost (or at least stuck there until if and when the node is recovered.) While I understand that there might be ways to work around this, by using separate NiFi clusters and remote process groups, and duplicate the flow processing, is there anything on the road map that would address this in NiFi itself? Storing the queues and flow flies on a distributed file system like HDFS, comes to mind, so that any “local” queue is distributed to more than one node. Thanks in advance, Chris
