NiFi provenance searches are not a good integration pattern for external systems. I.e. using it to periodicaly fetch history burdens the cluster (those searches can be heavy) and disrupt normal processing SLAs.
Pushing provenance events out to an external system (pitebtially even filtered down to components of interest) is a much more predictable pattern and provides lots of flexibility on how to interpret the events. Andrew On Thu, Feb 14, 2019, 11:26 PM Ali Nazemian <[email protected]> wrote: > Can I expect the Nifi search provenance part do the job for me? > > On Fri, 15 Feb. 2019, 13:21 Mike Thomsen <[email protected] wrote: > >> Ali, >> >> There is a site to site publishing task for provenance that you can add >> as a root controller service that would be great here. It'll just take all >> of your provenance data periodically and ship it off to another NiFi server >> or cluster that can process all of the provenance data as blocks of JSON >> data. A common pattern there is to filter down to the events you want and >> publish to ElasticSearch. >> >> On Thu, Feb 14, 2019 at 7:05 PM Ali Nazemian <[email protected]> >> wrote: >> >>> Hi All, >>> >>> I am investigating to see how Nifi provenance can be used as an event >>> store for a long period of time. Our use case is very burst based and >>> sometimes we may not receive any event for a period of time and sometimes >>> we may get burst traffic. On average we can say maybe around 1000 eps is >>> the expected throughput at this stage. Nifi has a powerful provenance that >>> gives you an ability to also index based on some attributes. I am >>> investigating how reliable is to use Nifi provenance store for a long >>> period of time and enable index for a few extra attributes. Has anybody >>> used Nifi provenance at this scale? Can lots of Lucene indices create other >>> issues within Nifi as provenance uses Lucene for the indexing? >>> >>> P.S: Our use case is pretty light for Nifi as we are not going to have >>> any ETL and Nifi is being used mostly as an Orchestrator of multiple >>> Microservices. >>> >>> Regards, >>> Ali >>> >>
