Hi Andy, Regarding bulletins, you can use the new SiteToSiteBulletinReportingTask [1] to achieve what you are expecting (in NiFi 1.2.0). For provenance, you can definitely use the REST API but could also be interested by the SiteToSiteProvenanceReportingTask [2] with the addition of NIFI-3859 [3]. You may also be interested by this article [4].
>From a high-level perspective all what you described is perfectly doable. Obviously, it'd be necessary to go into the details of what are the monitoring tools you want to integrate with but, again, NiFi sounds totally suitable to your use case. [1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-site-to-site-reporting-nar/1.2.0/org.apache.nifi.reporting.SiteToSiteBulletinReportingTask/index.html [2] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-site-to-site-reporting-nar/1.2.0/org.apache.nifi.reporting.SiteToSiteProvenanceReportingTask/index.html [3] https://issues.apache.org/jira/browse/NIFI-3859 [4] https://pierrevillard.com/2017/05/13/monitoring-nifi-site2site-reporting-tasks/ Pierre 2017-05-15 19:54 GMT+02:00 Andy Yar <[email protected]>: > Hello, > I'd like to know if NiFi is suitable for my slightly unusual use case. > > Basically, the system should route files/events/etc. from various > sources, validate, process them and route somewhere. However, I need > to do this in a reliable way. This means, the system needs to have > extensive monitoring/reprocessing capabilities. Various errors can > occur in the system - an outbound destination gets unavailable, a > malformed event doesn't match schema, an exception occurred during > processing, etc. In my case, the system has to allow discovering + > fixing + reprocessing of any failed event. > > I've figured that it is possible to route every processor's "failure" > state to a dead-letter-queue, fire up a notification (Slack, API call, > etc.), reroute the DLQ to the input again and so on. It seems to be a > bit complicated with adding a few retry steps before sending to the > DLQ everywhere but OK. > > However, I'm a bit worried about the monitoring side. For instance, is > NiFi, in its current state, able/intended to persist Bulletin Board > messages? I can imagine that after a operator sees an error on the > Bulletin Board he/she traces the flow file, fixes the issue and marks > the incident as solved. Currently the BB messages seems to be a bit > temporary. I guess the intended way is to create a custom Reporting > task in order to handle this. > > The Data Provenance itself is great, but still I can't imagine it > being usable without a major customization. In my case I would like to > monitor mostly a set of "inbound oriented" processors. So it would be > possible to list eg. all current events on inbound processors in a > (pseudo) realtime way. Again, I guess custom API/tasks could handle > this. > > To summarize: is NiFi usable for this particular use case? > > Thanks for any answers and also for your effort with this project! > > A. >
