IIPOCTAK opened a new issue, #58401: URL: https://github.com/apache/airflow/issues/58401
### Apache Airflow version 3.1.3 ### If "Other Airflow 2/3 version" selected, which one? _No response_ ### What happened? We are currently using Airflow 3.1.0. After upgrading to the latest versions: - 3.1.0 -> 3.1.1 - 3.1.0 -> 3.1.2 - 3.1.0 -> 3.1.3 We observed significant CPU throttling in the DagProcessor. As shown in the graphs below, CPU throttling has increased dramatically: <img width="687" height="304" alt="Image" src="https://github.com/user-attachments/assets/f7d85d1b-828a-4520-8cbc-9463a275ad63"; /> <img width="769" height="298" alt="Image" src="https://github.com/user-attachments/assets/c19e264c-6837-478a-9354-453ce1bdf477"; /> CPU usage has also increased, but not nearly as much as throttling: <img width="744" height="300" alt="Image" src="https://github.com/user-attachments/assets/d447d13a-cf2c-499d-b1e9-cdb3d27bb451"; /> <img width="1100" height="297" alt="Image" src="https://github.com/user-attachments/assets/05662bb0-b60f-47cf-ac20-5518b939de89"; /> Our Airflow resources run in a Kubernetes cluster. We did not change anything except the package versions. requrements.txt: ``` -apache-airflow[fab,ldap]==3.1.0 -apache-airflow-providers-fab==2.4.4 +apache-airflow[fab,ldap]==3.1.3 +apache-airflow-providers-fab==3.0.1 ``` values.yaml: ``` -airflowVersion: 3.1.0 +airflowVersion: 3.1.3 -defaultAirflowTag: 3.1.0-python3.10 +defaultAirflowTag: 3.1.3-python3.10 ``` I were trying to change this parameters after upgrade: ``` AIRFLOW__DAG_PROCESSOR__FILE_PARSING_SORT_MODE: "random_seeded_by_host" AIRFLOW__DAG_PROCESSOR__PARSING_PROCESSES: "1" AIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL: "300" ``` And increase CPU limits/requests (x2). Nothing helped. ### What you think should happen instead? _No response_ ### How to reproduce Upgrade airflow version, for example, from 3.1.0 -> 3.1.3 version and check CPU usage and throttling. ### Operating System Debian GNU/Linux 12 (bookworm) ### Versions of Apache Airflow Providers \# pip3 list | grep providers ``` apache-airflow-providers-amazon 9.12.0 apache-airflow-providers-celery 3.12.3 apache-airflow-providers-cncf-kubernetes 10.8.1 apache-airflow-providers-common-compat 1.7.4 apache-airflow-providers-common-io 1.6.3 apache-airflow-providers-common-messaging 2.0.0 apache-airflow-providers-common-sql 1.28.1 apache-airflow-providers-docker 4.4.3 apache-airflow-providers-elasticsearch 6.3.3 apache-airflow-providers-fab 2.4.4 apache-airflow-providers-ftp 3.13.2 apache-airflow-providers-git 0.0.8 apache-airflow-providers-google 18.0.0 apache-airflow-providers-grpc 3.8.2 apache-airflow-providers-hashicorp 4.3.2 apache-airflow-providers-http 5.3.4 apache-airflow-providers-microsoft-azure 12.7.1 apache-airflow-providers-mysql 6.3.4 apache-airflow-providers-odbc 4.10.2 apache-airflow-providers-openlineage 2.7.1 apache-airflow-providers-postgres 6.3.0 apache-airflow-providers-redis 4.3.1 apache-airflow-providers-sendgrid 4.1.3 apache-airflow-providers-sftp 5.4.0 apache-airflow-providers-slack 9.3.0 apache-airflow-providers-smtp 2.2.1 apache-airflow-providers-snowflake 6.5.4 apache-airflow-providers-ssh 4.1.4 apache-airflow-providers-standard 1.8.0 ``` ### Deployment Official Apache Airflow Helm Chart ### Deployment details helm ### Anything else? Constantly after upgrade ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
