greg-seesaw commented on issue #40082:
URL: https://github.com/apache/airflow/issues/40082#issuecomment-2239964163
Funny enough, this error started happening to us after we upgraded MWAA from
2.8.1 to 2.9.2 on Wednesday.
I'm not sure what's happening exactly either, but putting some data points
here:
1. We have a small MWAA environment (`mw1.small`): 2 schedulers, 1-10
workers, and only 6 DAGs, most of which are small.
2. We are using all the MWAA defaults, like `dag_file_processor_timeout =
50` and `standalone_dag_processor = False`. (I'm not sure if we can change the
latter in MWAA.)
3. This tends to happen around the time a DAG is running
4. There have also been intermittent errors in the scheduler log, like: `[
[34m2024-07-17T20:05:36.564+0000 [0m] { [34mscheduler_job_runner.py: [0m1422}
ERROR [0m - Couldn't find DAG SNOWFLAKE_MONITORING_DAG in DAG bag or database!
[0m`. They don't necessarily line up with the DAG Processing errors.
5. We've reduced the amount of top-level Python code as much as we could, to
minimize DAG import time.
@jiananyim Has this error appeared the last 2 days for you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
