Thanks Matt - and sorry for the delayed response I think this approach would work well for me, so I look forward to trying it out once it has been released.
-Tim > On Oct 18, 2019, at 1:49 PM, Matt Burgess <[email protected]> wrote: > > Tim, > > The "secondary flow" issue is something I wanted to address as well, > so I decoupled the formatting of data from the transmission of that > data into a new paradigm/service I'm calling a RecordServiceSink. A > ReportingTask can use RecordServiceSink to allow the user to choose > where the reporting "records" are to go. I'm doing this work under > NIFI-6780 [1] and the PR is currently under review [2], so hopefully > it'll be coming soon to a NiFi near you :) > > With this capability you could have a ProvenanceRecordReportingTask > for example, one that uses a RecordSinkService rather than strictly > site-to-site. My PR does include a SiteToSiteReportingRecordSink in > order to add feature parity in case such a > "ProvenanceRecordReportingTask" was implemented. If you use a > SiteToSiteReportingRecordSink with it, I'd expect that to be pretty > much the same as using the SiteToSiteProvenanceReportingTask. In your > case you'd want to add a RecordSinkService implementation for your own > in-house system, but in either case the reporting task itself would be > the same. I expect it would probably look similar to the current > SiteToSiteReportingTask in terms of code; the changes would be to get > the results into a RecordSet and call sendData() on the > RecordSinkService. With this we'd have a more generic reporting task > that anyone could use to get provenance data to any supported "record > sink"/target, without the need for a sub-flow, input port, and the > associated extraneous provenance events generated for that sub-flow. > > I should mention there's also a DatabaseRecordSink (that uses a > DBCPConnectionPool controller service) in the PR as well, so there are > two initial possible sinks/destinations available. I'll follow those > up with a PrometheusRecordSink and perhaps some KafkaRecordSink > implementations as well. > > Happy to discuss this more, and please feel free to look at the PR and > comment on it as you like, I'm always looking for feedback! > > Regards, > Matt > > [1] https://issues.apache.org/jira/browse/NIFI-6780 > [2] https://github.com/apache/nifi/pull/3826 > > On Fri, Oct 18, 2019 at 1:50 PM Tim Dean <[email protected]> wrote: >> >> Thanks Mike, >> >> I understand what you are saying but I am really trying to avoid having a >> secondary flow in NiFi if I can avoid it. >> >> It seems like NiFi was designed to allow this kind of custom reporting task >> and that I should ideally be able to do this without using the NiFi-provided >> S2S implementation if that doesn’t suit my needs. >> >> - Tim >> >> Sent from my iPhone >> >>> On Oct 18, 2019, at 12:36 PM, Mike Thomsen <[email protected]> wrote: >>>
