mattyb149 commented on PR #9171: URL: https://github.com/apache/nifi/pull/9171#issuecomment-2282878687
My mistake on the version, I pushed the fix. I believe the discussion took place on Slack but it's too old to retrieve from the Apache NiFi Slack as we don't have the pro version. Basically the idea is to provide a sort of "linked-list adjacency" that you see implemented fully as "index-free adjacency" in graph databases. Currently the lineage computation involves getting every event for a FlowFile, then sorting by timestamp, then repeating for all parent and child FlowFiles. This is too slow for determining an entire provenance graph. Consider GenerateFlowFile -> UpdateAttribute -> UpdateAttribute -> <whatever>, the UUID for the FlowFile remains the same but to tell which UpdateAttribute came first, you have to sort them by time. previousEventIds is used to enable a sort of "on-the-fly" streaming behavior to stream new provenance events into a graph database without having to re-link any of the previous nodes, just adding new nodes and links from the previous event(s) to the new one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
