nodyke opened a new issue #8969:
URL: https://github.com/apache/airflow/issues/8969
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the
following questions.
Don't worry if they're not all applicable; just try to include what you can
:-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
**Description**
As part of data lineage implementation, there is requirement to sending
lineage data from airflow to Apache Atlas. Current module, what represented in
1.10 stable version, has a few problems:
1. Create new Atlas Operator entity for each dagrun
2. Can't control creating missing entities using configs
3. Fail operator if sending lineage data was failed
4. Http timeout can't be configured
5. Current Atlas type definition has a small set of attributes
6. Errors in class wrappers for atlas types
<!-- A short description of your feature -->
**Use case / motivation**
As part of analytic data platform, auto importing lineage data is needed and
the most part of data lineage can be send by airflow in auto mode. Our module
use as base old Atlas backend module, but contains fixes and improvements. What
was fixed:
1. Create airflow operator atlas entity don't use execution date any
more.
2. Add config property for enabled/disabled creating missing inlets,
outlets entities.
3. Add config property for enabled/disabled falling operator if sending
lineage was failed
4. Add atlas timeout config property
5. Add "template_fields" into airflow operator typedef and add
additional config property for setting any additional operator attributes
6. Fix DataSet class wrapper, add abstract types for file and jdbc source
7. Added utils methods for correct generating inlets and outlets objects.
<!-- What do you want to happen?
Rather than telling us how you might implement this solution, try to take a
step back and describe what you are trying to achieve.
-->
**Related Issues**
AIRFLOW-5912
<!-- Is there currently another issue associated with this? -->
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
