Tim, The "secondary flow" issue is something I wanted to address as well, so I decoupled the formatting of data from the transmission of that data into a new paradigm/service I'm calling a RecordServiceSink. A ReportingTask can use RecordServiceSink to allow the user to choose where the reporting "records" are to go. I'm doing this work under NIFI-6780 [1] and the PR is currently under review [2], so hopefully it'll be coming soon to a NiFi near you :)
With this capability you could have a ProvenanceRecordReportingTask for example, one that uses a RecordSinkService rather than strictly site-to-site. My PR does include a SiteToSiteReportingRecordSink in order to add feature parity in case such a "ProvenanceRecordReportingTask" was implemented. If you use a SiteToSiteReportingRecordSink with it, I'd expect that to be pretty much the same as using the SiteToSiteProvenanceReportingTask. In your case you'd want to add a RecordSinkService implementation for your own in-house system, but in either case the reporting task itself would be the same. I expect it would probably look similar to the current SiteToSiteReportingTask in terms of code; the changes would be to get the results into a RecordSet and call sendData() on the RecordSinkService. With this we'd have a more generic reporting task that anyone could use to get provenance data to any supported "record sink"/target, without the need for a sub-flow, input port, and the associated extraneous provenance events generated for that sub-flow. I should mention there's also a DatabaseRecordSink (that uses a DBCPConnectionPool controller service) in the PR as well, so there are two initial possible sinks/destinations available. I'll follow those up with a PrometheusRecordSink and perhaps some KafkaRecordSink implementations as well. Happy to discuss this more, and please feel free to look at the PR and comment on it as you like, I'm always looking for feedback! Regards, Matt [1] https://issues.apache.org/jira/browse/NIFI-6780 [2] https://github.com/apache/nifi/pull/3826 On Fri, Oct 18, 2019 at 1:50 PM Tim Dean <[email protected]> wrote: > > Thanks Mike, > > I understand what you are saying but I am really trying to avoid having a > secondary flow in NiFi if I can avoid it. > > It seems like NiFi was designed to allow this kind of custom reporting task > and that I should ideally be able to do this without using the NiFi-provided > S2S implementation if that doesn’t suit my needs. > > - Tim > > Sent from my iPhone > > > On Oct 18, 2019, at 12:36 PM, Mike Thomsen <[email protected]> wrote: > >
