kacpermuda commented on issue #68661: URL: https://github.com/apache/airflow/issues/68661#issuecomment-5033369726
Hey, I think we should approach this differently. In OpenLineage, the dbt structured log processor already wraps individual dbt executions and their events with a top-level invocation event that acts as the parent for a single dbt execution. We should implement a similar pattern in the artifact processor (see [this issue](https://github.com/OpenLineage/OpenLineage/issues/4720)). We'll also need to prepare the artifact processor so it can be invoked before any artifacts are available, allowing it to emit a top level `START` event. This could be called from `get_openlineage_facets_on_start`, with the generated `run_id` stored in the operator for later use. Then from the dbt Cloud operator, we could invoke the artifact processor and pass custom run facets describing the dbt Cloud job. Those facets would be attached to the new top-level invocation event. This means these top-level events would no longer be dbt Cloud-specific—they would be used for any invocation of the dbt artifact processor, with dbt Cloud simply adding extra facets (such as job ID, run ID, etc.) when applicable. So I think most of the work—and the prerequisite for this feature—belongs in the OpenLineage repository. Once that's in place, we can update the operator and complete the integration in the Airflow repository. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
