kacpermuda commented on code in PR #39520:
URL: https://github.com/apache/airflow/pull/39520#discussion_r1595241671
##########
airflow/providers/openlineage/utils/utils.py:
##########
@@ -268,6 +269,91 @@ def get_airflow_run_facet(
}
+def get_dag_structure_job_facet(dag_run: DagRun):
+ dag = dag_run.dag
+ task_groups = {
+ tg_id: {
+ "parent_group": tg.parent_group.group_id,
+ }
+ for tg_id, tg in dag.task_group_dict.items()
+ }
+ tasks = {
+ task.task_id: {
+ "operator": task._task_module + "." + task._task_type,
+ "task_group": task.task_group.group_id if task.task_group else
None,
+ "config_selective_enable": is_selective_lineage_enabled(task),
+ "config_disabled_for_operators":
is_operator_disabled(task._task_module + "." + task._task_type),
+ # `produces_ol_events` will tell the consumer whether they should
expect an event from given task,
+ # no matter the conditions of disablement - it's more futureproof
than checking everything
+ # separately (selective_enable, disabled_for_operators and
possibly more in the future)
+ "produces_ol_events": not is_operator_disabled(task._task_module +
"." + task._task_type)
+ or is_selective_lineage_enabled(task),
+ # not sure if state is good to have in Job type facet, maybe we
should create another facet like
+ # AirflowDagRunStateRunFacet that's below?
+ "state": dag_run.get_task_instance(task.task_id).current_state()
Review Comment:
@mobuchowski @JDarDagran
I'm not sure if we should add the task state to this Job facet - we should
then attach it to both start and complete events, so we would duplicate the
information. Maybe we should create another facet (f.e.
AirflowDagRunStateRunFacet like below) and only attach it to dag complete run
facets?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]