astro-anand opened a new issue, #55475: URL: https://github.com/apache/airflow/issues/55475
### Description Improved error propagation for dynamic DAG files. I propose introducing an exception type that PUTs an import error via the Airflow REST API without causing the dynamic dag loop to exit. To do this, a new API route would be created that allows users to PUT a DAG parsing error. Then we would create an ImportErrorException class that could be used in a try except block in the dynamic DAG code which would propagate the error. The errors would be cleaned up automatically when the dynamic dag is parsed. We could also implement a DELETE endpoint for import errors as persistent errors would be resurfaced by the dag parsing loop. ### Use case/motivation A common pattern in Airflow is using a single file to generate DAGs dynamically from config files. If a user creates an invalid config, there are two general outcome 1. The entire dynamic dag generation process is interrupted and an import error is surfaced for the dynamic dag module. This approach is not ideal because one bad config can disrupt the every dynamic DAG. This can affect shared testing environments significantly. 2. The user has implemented a try/except block in the dynamic dag file and the import error is suppressed, likely with error level logs being emitted in the DAG processor. This approach is suboptimal for data engineers because the logs are emitted by the dag processor instead of surfaced as import errors. Instead of either of these approaches, specific configuration error should be surfaced as import errors. Users should be able to propagate their own, richer errors as ImportErrors ### Related issues _No response_ ### Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
