kacpermuda commented on code in PR #75328:
URL: https://github.com/apache/airflow/pull/75328#discussion_r4390394039
##########
providers/openlineage/src/airflow/providers/openlineage/plugins/listener.py:
##########
@@ -163,6 +163,27 @@ def _emit_manual_state_change_event(adapter_method_name:
str, stats_key: str, **
Stats.gauge(stats_key, len(Serde.to_json(event).encode("utf-8")))
+def _get_task_end_date(task_instance: RuntimeTaskInstance | TaskInstance) ->
datetime:
+ """
+ Return the end date for a terminal event of a task instance that finished
in the current process.
+
+ The Airflow 3 task runner sets ``end_date`` before it notifies listeners
and reports that same value as
+ the stored one, so the event time matches it. In the cases below the time
of the notification is used
+ instead, so the event time is later than the stored ``end_date`` by
however long the task's state change
+ callbacks took:
+
+ - Airflow 2, where the event time is kept as it was before ``end_date``
was used.
+ - Airflow 3.0.0 to 3.0.4, whose task runner never sets ``end_date`` on the
task instance.
+ - Before Airflow 3.4.0, a task that raised ``AirflowSkipException``, or a
``TriggerDagRunOperator`` task
+ skipped or failed because the Dag run it triggers already exists.
+ """
+ if AIRFLOW_V_3_0_PLUS:
Review Comment:
You're right, there's no Airflow 2 case. I checked 2.11.2: the success path
sets `end_date` and calls `set_duration()` before `on_success_callback` and the
listener. `handle_failure` sets `end_date` before it calls
`on_task_instance_failed`. So `end_date` is always set by the time the listener
runs.
I removed the version check. `_get_task_end_date` now uses `ti.end_date`
whenever it's a `datetime`, so the event time and the facet come from one
value. I also added an Airflow 2 test for success and failure with an
`end_date` different from now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
