fshehadeh commented on issue #56446: URL: https://github.com/apache/airflow/issues/56446#issuecomment-3408459563
Thank @potiuk for sharing these additional details. In our setup we run Airflow from container images. In production, we lock down the DAG files between releases, and in lower envs we update the DAG frequently. The one minute reparsing was aggressive for us, so by adding the checksum to it, we are able to skip the unnecessary reparsing, while still being able to detect changes within a minute (because the checksum would have changed). We also ended up adding an upper limit for how long the DAG can go without be reparsed (I think like 10 minutes). We were worried about possible negative side-effects of skipping the reparsing for a long time. Your comment about the callbacks and their relationship with the parsing logic explained why this was necessary. I think ideally the process of parsing should be decoupled from that of the callbacks. I think the approach can help some users, but not necessarily all of them, depending on the number of the DAGs, their frequency, and complexity. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
