GitHub user IH-Miller edited a discussion: New Dags not loading
`Using
-Airflow version 2.8.4
-Python version 3.11.10
Order of events:
1.In Nov 2024 we upgraded Airflow python from 3.8 -> 3.11
A.This forced apache-airflow-providers-snowflake to then use pandas
2.1.4 instead of 2.0.3 because of the requirement of ">=2.1.2"
B.apache-airflow 2.8.4 says it can handle <2.2
C.While testing existing DAGS seems to work and adding new DAGS worked
2.In April 2025 we did a security vunerability upgrade of the Snowflake
connector
A.Upgrade snowflake-connector-python from 3.7.1 to 3.14.0
B.Testing DAGS worked and we had no errors. However we did not test
adding new DAGS at this time.
3.2025-06-13: We tried to adding multiple new DAGS to the airflow server and it
wasn't showing in the UI with no errors displayed either
A."airflow DAGS list" shows the new dag with the paused field showing
"None" instead of True/False
B."airflow DAGS list-import-errors" shows no errors for the new DAG
C.We tried running "airflow DAGS unpause <dag_name>" and it gave an
error saying the DAG did not exist on the table dag.
D.We tried running existing DAGS an they all ran correctly had no error
and on "airflow DAGS list" have a Paused value of true or false
E.We tried creating empty DAGS and "airflow DAGS list" shows the new
DAG with the paused field showing "None" instead of True/False
F.We downloaded dagbag(Re-serialize any DAGS missing from the
database.) python file from the internet
i.We ran "airflow DAGS list" again it shows less tasks those
not using the snowflake-connector-python library and all the DAGs using
snowflake-connector-python
showed errors even those that were old and still worked.
ii.Then show errors in the "airflow DAGS list-import-errors"
output
Output:
a.File "hashtable.pyx", line 1, in init
pandas._libs.hashtable
b.File "missing.pyx", line 42, in init
pandas._libs.missing
c.AttributeError: partially initialized module 'pandas'
has no attribute '_pandas_datetime_CAPI' (most likely due to a circular import)
iii.We tried using pandas==2.1.2 and 2.1.4 this didn't resolve
the issue
iv.We tried reverting snowflake-connector-python and pandas
back to the versions of libraries we had in November 2024 and we still had the
issue.
[requirements.txt](https://github.com/user-attachments/files/20776472/requirements.txt)`
GitHub link: https://github.com/apache/airflow/discussions/51841
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
