[
https://issues.apache.org/jira/browse/NIFI-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16218156#comment-16218156
]
Koji Kawamura commented on NIFI-3709:
-------------------------------------
[~ikethecoder] Thanks for sharing your work. I'm glad to know that you were
able to utilize the extension point to analyze Provenance event such as for
PutFile. Those part can be merged easily. So let me merge your commit once my
work has been squashed into a final commit. Let's keep in touch!
As for the ByFileLineageStrategy, I am not quite sure if it's going to work
with Atlas well. Because Atlas is designed for tracking DataSet level lineage
rather than event level. That's why I decided to drop granularity to DataSet
and flow_path level when NiFi reporting task reports lineage to Atlas.
If user would like to see a complete and detailed lineage of a FlowFile, NiFi
provenance viewer would be more appropriate I think. By using NiFi provenance
UI, user can see where the FlowFile came from and where it went.
https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#viewing-flowfile-lineage
I think the core benefit of exporting NiFi lineage to Atlas is that it let user
can see a bigger picture not only within NiFi, but also other softwares such as
Apache Storm, Kafka. Also, FlowFile lives inside a NiFi flow and does not have
the same level of persistence like Database record or file on a file system,
IMHO.
However, I may be missing something. Would you elaborate the benefit of
exporting FlowFile level lineage to Atlas? Example Atlas lineage screenshot
would be helpful to understand the use-case. Thanks!
> Export NiFi flow dataset lineage to Apache Atlas
> ------------------------------------------------
>
> Key: NIFI-3709
> URL: https://issues.apache.org/jira/browse/NIFI-3709
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Reporter: Koji Kawamura
> Assignee: Koji Kawamura
>
> While Apache NiFi has provenance and event level lineage support within its
> data flow, Apache Atlas also does manage lineage between dataset and process
> those interacting with such data.
> It would be beneficial for users who use both NiFi and Atlas and if they can
> see end-to-end data lineage on Atlas lineage graph, as some type of dataset
> are processed by both NiFi and technologies around Atlas such as Storm,
> Falcon or Sqoop. For example, Kafka topics and Hive tables.
> In order to make this integration happen, I propose a NiFi reporting task
> that analyzes NiFi flow then creates DataSet and Process entities in Atlas.
> The challenge is how to design NiFi flow dataset level lineage within Atlas
> lineage graph.
> If we just add a single NiFi process and connect every DataSet from/to it, it
> would be too ambiguous since it won't be clear which part of a NiFi flow
> actually interact with certain dataset.
> But if we put every NiFi processor as independent process in Atlas, it would
> be too granular, too. Also, we already have detailed event level lineage in
> NiFi, we wouldn't need the same level in Atlas.
> If we can group certain processors in a NiFI flow as a process in Atlas, it
> would be a nice granularity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
