potiuk commented on issue #45276: URL: https://github.com/apache/airflow/issues/45276#issuecomment-3094469726
> @ephraimbuddy How can one find whether this was due to an import timeout? The actually shown import error indicate real issues, not timeouts. You need to look at the errors. It does not have to be timeouts, it can be **ANYTHING** that changes between one parsing and second parsing. This basically means that parsing of some of your Dags is simply **unstable** and sometimes works, sometimes does not. You need to look at those import errors (also they should be logged by dag file processor I think, and for sure they are stored in the database of Airflow (see ERD schema on our docs) and you can compare the errors when they differ. The actual extra error you see when you have 13 of them is what you should figure out and find out why it might sometimes cause import error and sometimes not. It can be anything and it 100% depends on how your Dags are written - so only you and your team can investigate the root causes and fix them. There might be many, many different reasons - depending on the environmental changes - database timeouts, reading external files that might impact what happens during parsing, or maybe even potentially inconsistent environment your dag file processor(s) run - maybe some of your environment does not have some libraries installed. Also a number of users (completely against our recommendations) dynamically install dependencies in their environment rather than for example bake all dependencies in their images or have a read-only non-modifiable virtual env, and due to racing conditions that might cause all kinds of weirdness when parsing done in one environment (with some deps) might result with different results than the one done in effectively different environment (for example when you install or remove dependencies dynamicall without redeplo ying airflow in a "frozen" environment. So .. all in your hands to analyse and find out what's wrong with *your* Dags and *your* environment rather than look for issue in Airflow. I am converting that one to a discussion, would be nice to hear back what you find out after your investigations - that might help people who will have similar issues like you in the future when they see what mistakes could be avoided were done when deploying airflow or authoring Dags. Also looking at "Best practices" in our docs might help - we explain there a number of things you should avoid when authoring Dags. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
