Hello, I'd like to know if NiFi is suitable for my slightly unusual use case.
Basically, the system should route files/events/etc. from various sources, validate, process them and route somewhere. However, I need to do this in a reliable way. This means, the system needs to have extensive monitoring/reprocessing capabilities. Various errors can occur in the system - an outbound destination gets unavailable, a malformed event doesn't match schema, an exception occurred during processing, etc. In my case, the system has to allow discovering + fixing + reprocessing of any failed event. I've figured that it is possible to route every processor's "failure" state to a dead-letter-queue, fire up a notification (Slack, API call, etc.), reroute the DLQ to the input again and so on. It seems to be a bit complicated with adding a few retry steps before sending to the DLQ everywhere but OK. However, I'm a bit worried about the monitoring side. For instance, is NiFi, in its current state, able/intended to persist Bulletin Board messages? I can imagine that after a operator sees an error on the Bulletin Board he/she traces the flow file, fixes the issue and marks the incident as solved. Currently the BB messages seems to be a bit temporary. I guess the intended way is to create a custom Reporting task in order to handle this. The Data Provenance itself is great, but still I can't imagine it being usable without a major customization. In my case I would like to monitor mostly a set of "inbound oriented" processors. So it would be possible to list eg. all current events on inbound processors in a (pseudo) realtime way. Again, I guess custom API/tasks could handle this. To summarize: is NiFi usable for this particular use case? Thanks for any answers and also for your effort with this project! A.
