Hi Pierre Atlas indeed comes into my attention in the beginning, but my production environment does not support hdfs which is the fundamental of Atlas.
The provenance is a out of box feature in NiFi, and we also use NiFi to do simple transformations, therefore I am searching some provenance solution that can be leveraged in NiFi, that would be perfect. Cheers Pierre Villard <[email protected]>于2021年1月11日 周一下午6:42写道： > Hi, > > Just want to also mention that the integration that NiFi provides with > Apache Atlas (through a dedicated reporting task) is supposed to provide > this kind of end-to-end lineage across multiple components. > > Thanks, > Pierre > > Le lun. 11 janv. 2021 à 18:58, Chris Sampson <[email protected]> > a écrit : > >> Might be worth taking a look at the "alternate.identifier" (see >> https://nifi.apache.org/docs/nifi-docs/html/nifi-in-depth.html#associating-disparate-data >> ). >> >> >> --- >> *Chris Sampson* >> IT Consultant >> [email protected] >> <https://www.naimuri.com/> >> >> >> On Mon, 11 Jan 2021 at 14:53, Yi Wang <[email protected]> wrote: >> >>> Hi Nifi team and experts, >>> >>> In such an example scenario: >>> >>> Data —> Nifi —> Kafka —> external system (spark, flink etc) —> Kafka —> >>> Nifi —> s3 >>> >>> How do I fill in the ‘gap’ of data lineage? >>> Once the data leave Nifi, its provenance life is ended (as far as I >>> know), even when the same data is sent back to Nifi later, Nifi treats them >>> as different data. So how can I handle this in order to get a complete data >>> lineage graph? >>> >>> Any idea or suggestions? Thanks in advance. >>> >>> >>> Cheers >>> >>>
