potiuk commented on issue #30884: URL: https://github.com/apache/airflow/issues/30884#issuecomment-1523839712
@george-zubrienko - I have some extra questions that might help to find out the root cause. 1) Does it happen all the time for all dags parsed or some of the time for some of the dags? Can you please descibe the pattern you see? 2) Could you please tell more (and possibly share some anonymised examples of) top-level code of the dags of yours that experience this behaviour and whether you use some of those things: * excessive or unusual imports? * reaching out to external sources (HTTP/Similar) while parsing top-level code? * accessing any kind of database while parsing top-level code? * acxessing secrets/variables/connections while parsing top-level code? Generally speaking I would like to know your top-level parsing violates any of the best practices described in https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#top-level-python-code ? 3) Do you see any excessive memory use and swapping while parsing happens for the DAG file processor? 4) Do you use callbacks in your DAGs that experience the problem? Having those answers might help in investigating the root cuase - the problem you describe is not reproducible easily - so we need to know what's so special about your DAGS or environment that triggers that behaviour. With the related #305903 it seemsed that the reason might be MySQL but knowing that you have Postgres suggests that either the problem is different or that it is something completely different than we suspected before. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
