Reuben van Ammers created AIRFLOW-3314:
------------------------------------------
Summary: The lineage automatic inlets feature does not work as
described.
Key: AIRFLOW-3314
URL: https://issues.apache.org/jira/browse/AIRFLOW-3314
Project: Apache Airflow
Issue Type: Bug
Affects Versions: 1.10.0
Reporter: Reuben van Ammers
Attachments: test_lineage_broken.py
Currently, this is the description of the arguments to the prepare lineage
wrapper regarding inlets in airflow/lineage/__init__.py:
inlets can be:
"auto" -> picks up any outlets from direct upstream tasks that have outlets
defined, as such that if A -> B -> C and B does not have outlets but A does,
these are provided as inlets.
This implies that non state producing tasks should have no effect on the
behaviour for inlets and outlets, which is desirable to easily add operators
that don't change state of files (such as for tracking/communication).
This is not the current behaviour. This can be seen by changing the test case
in tests/lineage/test_lineage.py to use the auto feature on operation 5 to use
auto. Given the description, one would expect 1 inlet file, the the file
produced from op3. However, as can be seen in the attatched broken test, this
is not the case, and the presence of the non-state affecting operator breaks
the test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
