Thank you very much!
How could I not have seen it before!
Belén
On 6/7/23 10:14, Bart Maertens wrote:
Hi Belén,
You can log to a Neo4j graph database by setting
the NEO4J_LOGGING_CONNECTION variable [1].
Additional metadata and lineage data can be added, but that will
require additional (meta)data parsing.
This should already get you going, but there's always room for
improvement.
Feel free to create feature request tickets in our github repository
[3] for any functionality you'd like to see in a future Apache Hop
release.
[1]
https://hop.apache.org/manual/latest/technology/neo4j/index.html#_perspective_execution_lineage
<https://urldefense.com/v3/__https://hop.apache.org/manual/latest/technology/neo4j/index.html*_perspective_execution_lineage__;Iw!!D9dNQwwGXtA!Ut7uYGMwdWb5652vbkpfcd1kjt164f_SrBP5pWev7rYeJnl42P_0vAwpMPFRghYRWjSFJfO1SIHLhSzg-IPRz33NPQ$>
[2] https://www.know-bi.be/blog/data-lineage-with-neo4j-and-apache-hop
<https://urldefense.com/v3/__https://www.know-bi.be/blog/data-lineage-with-neo4j-and-apache-hop__;!!D9dNQwwGXtA!Ut7uYGMwdWb5652vbkpfcd1kjt164f_SrBP5pWev7rYeJnl42P_0vAwpMPFRghYRWjSFJfO1SIHLhSzg-IOZh6dt-w$>
[3] https://github.com/apache/hop/issues
<https://urldefense.com/v3/__https://github.com/apache/hop/issues__;!!D9dNQwwGXtA!Ut7uYGMwdWb5652vbkpfcd1kjt164f_SrBP5pWev7rYeJnl42P_0vAwpMPFRghYRWjSFJfO1SIHLhSzg-IM1BxM3Pg$>
Regards,
Bart
On Thu, Jul 6, 2023 at 10:05 AM Belén Gracia <[email protected]> wrote:
Hi!
I am just starting to use HOP although I have been using PDI (kettle)
for several years. There is something I have been missing and that is
being able to automate the lineage of tables by parsing the
jobs/transformations or transformations/pipelines.
Is it possible to introduce customizable metadata in the steps? There
are many steps that would natively allow this automation, but getting
information about the tables read by other steps like "Table
input" or
"Database join" is not always that easy.
Do you others have this need, how have you solved it?
I am all ears :)
Best,
Belén
