jaklan commented on issue #34206: URL: https://github.com/apache/airflow/issues/34206#issuecomment-2342574661
@Lee-W but the point is - one dbt Cloud job can be responsible for building e.g. 30 tables (they can be e.g. all tables with a specific tag: `--select tag:foobar`). Each table should be represented as a Dataset in Airflow. Let's assume 25 of them succeeded, 5 failed. Based on job metadata retrieved from dbt Cloud, we would like to send events for the successful ones - so we have to trigger 25 Dataset events. Then you run the same job the next day, this time 20 tables succeeded, 10 failed - in that case we want trigger 20 Dataset events. And so on, and so on... If we can't attach multiple events to one alias (so in the first case we would attach 25 events, in the second one - 20 events), we would need to statically define all 30 aliases as `outlets` - and that's not really acceptable, because the list of tables in the scope of that dbt Cloud job is dynamic: one day you can add 5 new tables with the tag `foobar`, so then the job would trigger 35 tables instead of 30. And in that case we would need to update Airflow DAG manually to define 5 new task outlets. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
