[
https://issues.apache.org/jira/browse/NIFI-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16235464#comment-16235464
]
Koji Kawamura commented on NIFI-3709:
-------------------------------------
[~ikethecoder] I've added an extension point inspired by your improvement,
would you rebase your branch using this one? Other parts have been changed,
too, so you may need to adjust your code. However, adding FlowFile level
lineage strategy should be easier than before.
https://github.com/ijokarumawak/nifi/tree/nifi-3709-4
If you can extract a commit which has only FlowFile level lineage strategy
additions, it would be very easy for me to cherry-pick it so that your
contribution gets recorded in commit history and available when I create a PR
using my branch.
Also, I've added another strategy, now my branch has 'Simple Path' and
'Complete Path'. I've summarized comparison of these in this Gist. Would you
take a look? I wonder if 'Complete Path' would be useful to your use-case, too.
Please let me know how you think. Thanks!
https://gist.github.com/ijokarumawak/9c3f2b0e44d1f3f2cb1addc9eb53d893
> Export NiFi flow dataset lineage to Apache Atlas
> ------------------------------------------------
>
> Key: NIFI-3709
> URL: https://issues.apache.org/jira/browse/NIFI-3709
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Reporter: Koji Kawamura
> Assignee: Koji Kawamura
> Priority: Major
>
> While Apache NiFi has provenance and event level lineage support within its
> data flow, Apache Atlas also does manage lineage between dataset and process
> those interacting with such data.
> It would be beneficial for users who use both NiFi and Atlas and if they can
> see end-to-end data lineage on Atlas lineage graph, as some type of dataset
> are processed by both NiFi and technologies around Atlas such as Storm,
> Falcon or Sqoop. For example, Kafka topics and Hive tables.
> In order to make this integration happen, I propose a NiFi reporting task
> that analyzes NiFi flow then creates DataSet and Process entities in Atlas.
> The challenge is how to design NiFi flow dataset level lineage within Atlas
> lineage graph.
> If we just add a single NiFi process and connect every DataSet from/to it, it
> would be too ambiguous since it won't be clear which part of a NiFi flow
> actually interact with certain dataset.
> But if we put every NiFi processor as independent process in Atlas, it would
> be too granular, too. Also, we already have detailed event level lineage in
> NiFi, we wouldn't need the same level in Atlas.
> If we can group certain processors in a NiFI flow as a process in Atlas, it
> would be a nice granularity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
