arjunanan6 opened a new issue, #32674:
URL: https://github.com/apache/airflow/issues/32674
### Apache Airflow version
2.6.3
### What happened
Since version 2.6.0, I have noticed that the run-airflow-migrations job
takes extremely long - in some cases 9 minutes when we rebuild our image and
deploy. In the worst of cases, the pod is restarted due to having hit our
timeout of 10 min.
It has taken about 2-3 minutes in the past, and our DAG bag has not grown by
much (perhaps approx 30 DAGs since the 2-3 min deployment).
I cannot see anything out of the usual from the logs that would be causing
this too. This is the entire output from the most recent run that took about 8
minutes to complete. We are aware of the errors listed in here, but this has
been the same even previously, and has not caused the migrations to take so
long.
```DB:
postgresql://psqladmin%40airflow-ps-db-prod:***@airflow-prod-pgbouncer.airflow-prod-ns:6543/airflow-prod-metadata?sslmode=disable
Performing upgrade with database
postgresql://psqladmin%40airflow-ps-db-prod:***@airflow-prod-pgbouncer.airflow-prod-ns:6543/airflow-prod-metadata?sslmode=disable
[2023-07-18T14:25:45.590+0000] {migration.py:213} INFO - Context impl
PostgresqlImpl.
[2023-07-18T14:25:45.591+0000] {migration.py:216} INFO - Will assume
transactional DDL.
[2023-07-18T14:25:45.938+0000] {db.py:1590} INFO - Creating tables
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
ERROR [airflow.models.dagbag.DagBag] Exception bagging dag:
SAP_BANK_JPM_CAMT_D
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagbag.py",
line 500, in _bag_dag
raise AirflowDagDuplicatedIdException(
airflow.exceptions.AirflowDagDuplicatedIdException: Ignoring DAG
SAP_BANK_JPM_CAMT_D from
/opt/airflow/dags/airflow-dags-sap/SAP_BANK/sap_bank_jpm_camt_d.py - also found
in /opt/airflow/dags/airflow-dags-sap/SAP_BANK/sap_bank_jpm_camt_d_SIMU.py
ERROR [airflow.models.dagbag.DagBag] Failed to bag_dag:
/opt/airflow/dags/airflow-dags-sap/SAP_BANK/sap_bank_jpm_camt_d.py
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagbag.py",
line 445, in _process_modules
self.bag_dag(dag=dag, root_dag=dag)
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagbag.py",
line 462, in bag_dag
self._bag_dag(dag=dag, root_dag=root_dag, recursive=True)
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagbag.py",
line 500, in _bag_dag
raise AirflowDagDuplicatedIdException(
airflow.exceptions.AirflowDagDuplicatedIdException: Ignoring DAG
SAP_BANK_JPM_CAMT_D from
/opt/airflow/dags/airflow-dags-sap/SAP_BANK/sap_bank_jpm_camt_d.py - also found
in /opt/airflow/dags/airflow-dags-sap/SAP_BANK/sap_bank_jpm_camt_d_SIMU.py
ERROR [airflow.models.dagbag.DagBag] Exception bagging dag:
SAP_BANK_NOR_CAMT_D
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagbag.py",
line 500, in _bag_dag
raise AirflowDagDuplicatedIdException(
airflow.exceptions.AirflowDagDuplicatedIdException: Ignoring DAG
SAP_BANK_NOR_CAMT_D from
/opt/airflow/dags/airflow-dags-sap/SAP_BANK/sap_bank_nor_camt_d.py - also found
in /opt/airflow/dags/airflow-dags-sap/SAP_BANK/sap_bank_nor_camt_d_SIMU.py
ERROR [airflow.models.dagbag.DagBag] Failed to bag_dag:
/opt/airflow/dags/airflow-dags-sap/SAP_BANK/sap_bank_nor_camt_d.py
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagbag.py",
line 445, in _process_modules
self.bag_dag(dag=dag, root_dag=dag)
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagbag.py",
line 462, in bag_dag
self._bag_dag(dag=dag, root_dag=root_dag, recursive=True)
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagbag.py",
line 500, in _bag_dag
raise AirflowDagDuplicatedIdException(
airflow.exceptions.AirflowDagDuplicatedIdException: Ignoring DAG
SAP_BANK_NOR_CAMT_D from
/opt/airflow/dags/airflow-dags-sap/SAP_BANK/sap_bank_nor_camt_d.py - also found
in /opt/airflow/dags/airflow-dags-sap/SAP_BANK/sap_bank_nor_camt_d_SIMU.py
ERROR [airflow.models.dagbag.DagBag] Exception bagging dag:
SAP_BANK_DAB_CAMT_D
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagbag.py",
line 500, in _bag_dag
raise AirflowDagDuplicatedIdException(
airflow.exceptions.AirflowDagDuplicatedIdException: Ignoring DAG
SAP_BANK_DAB_CAMT_D from
/opt/airflow/dags/airflow-dags-sap/SAP_BANK/sap_bank_dab_camt_d_SIMU.py - also
found in /opt/airflow/dags/airflow-dags-sap/SAP_BANK/sap_bank_dab_camt_d.py
ERROR [airflow.models.dagbag.DagBag] Failed to bag_dag:
/opt/airflow/dags/airflow-dags-sap/SAP_BANK/sap_bank_dab_camt_d_SIMU.py
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagbag.py",
line 445, in _process_modules
self.bag_dag(dag=dag, root_dag=dag)
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagbag.py",
line 462, in bag_dag
self._bag_dag(dag=dag, root_dag=root_dag, recursive=True)
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagbag.py",
line 500, in _bag_dag
raise AirflowDagDuplicatedIdException(
airflow.exceptions.AirflowDagDuplicatedIdException: Ignoring DAG
SAP_BANK_DAB_CAMT_D from
/opt/airflow/dags/airflow-dags-sap/SAP_BANK/sap_bank_dab_camt_d_SIMU.py - also
found in /opt/airflow/dags/airflow-dags-sap/SAP_BANK/sap_bank_dab_camt_d.py
/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/oracle/hooks/oracle.py:31
RemovedInAirflow3Warning: This module is deprecated. Please use
`airflow.providers.common.sql.hooks.sql`.
ERROR [airflow.models.dagbag.DagBag] Exception bagging dag:
ANALYTICS_B2B_A_faktura_all_power_automatic
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagbag.py",
line 500, in _bag_dag
raise AirflowDagDuplicatedIdException(
airflow.exceptions.AirflowDagDuplicatedIdException: Ignoring DAG
ANALYTICS_B2B_A_faktura_all_power_automatic from
/opt/airflow/dags/airflow-dags-analytics/STANDALONE_DAGS/ANALYTICSP_B2B_A_faktura_all_power_automatic.py
- also found in
/opt/airflow/dags/airflow-dags-analytics/STANDALONE_DAGS/ANALYTICSP_B2B_Customer_data_til_Midas_daily_SE.py
ERROR [airflow.models.dagbag.DagBag] Failed to bag_dag:
/opt/airflow/dags/airflow-dags-analytics/STANDALONE_DAGS/ANALYTICSP_B2B_A_faktura_all_power_automatic.py
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagbag.py",
line 445, in _process_modules
self.bag_dag(dag=dag, root_dag=dag)
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagbag.py",
line 462, in bag_dag
self._bag_dag(dag=dag, root_dag=root_dag, recursive=True)
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagbag.py",
line 500, in _bag_dag
raise AirflowDagDuplicatedIdException(
airflow.exceptions.AirflowDagDuplicatedIdException: Ignoring DAG
ANALYTICS_B2B_A_faktura_all_power_automatic from
/opt/airflow/dags/airflow-dags-analytics/STANDALONE_DAGS/ANALYTICSP_B2B_A_faktura_all_power_automatic.py
- also found in
/opt/airflow/dags/airflow-dags-analytics/STANDALONE_DAGS/ANALYTICSP_B2B_Customer_data_til_Midas_daily_SE.py
ERROR [airflow.models.dagbag.DagBag] Failed to import:
/opt/airflow/dags/airflow-dags-analytics/LINKED_DAGS/DELETE_DAG_ANALYTICSP_B2B_SAP_DATA_PO1.py
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagbag.py",
line 346, in parse
loader.exec_module(new_module)
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in
_call_with_frames_removed
File
"/opt/airflow/dags/airflow-dags-analytics/LINKED_DAGS/DELETE_DAG_ANALYTICSP_B2B_SAP_DATA_PO1.py",
line 63, in <module>
sap_data_dp1 >> [trigger_crm_tables_last_updated,
trigger_b2b_customer_data_til_midas_daily_se, trigger_b2b_forbrug_eyc,
trigger_gasdw_01_graddage]
NameError: name 'trigger_gasdw_01_graddage' is not defined
Upgrades done
```
Any idea what we can do to improve the migrations run time? We have not
changed any parameters in regards to the migration, for example:
migrationsWaitTimeout is still 60s.
### What you think should happen instead
_No response_
### How to reproduce
N/A
### Operating System
Debian GNU/Linux
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
We are using the official helm chart v1.9.0 deployed on AKS.
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
