sebinxavi opened a new issue, #40991: URL: https://github.com/apache/airflow/issues/40991
### Apache Airflow version Other Airflow 2 version (please specify below) ### If "Other Airflow 2 version" selected, which one? 2.8.4 ### What happened? Hello, We have noticed 55% CPU usage in the newly deployed RDS Aurora PostgreSQL . We have connected two airflow environments into this RDS instance. There are no airflow dags running , but still it is taking 55% usage. Is this normal? <img width="1035" alt="image" src="https://github.com/user-attachments/assets/716afffe-674f-4031-a1cc-c64605750f02";> The result of performance insight is provided below, <img width="1558" alt="Screenshot 2024-07-24 at 12 55 50" src="https://github.com/user-attachments/assets/ec39e18f-4b79-446c-a5da-b3d049a17b50";> <img width="396" alt="Screenshot 2024-07-24 at 12 56 06" src="https://github.com/user-attachments/assets/32880341-9607-4200-8d55-0acfd2015e2d";> ### What you think should happen instead? Since no DAGs are running and no DAGs are scheduled, the RDS CPU usage should be very low? ### How to reproduce Deploy airflow 2.8.4 with airflow official helm chart. Connect the airflow to external database RDS Aurora PostgreSQL. ### Operating System EKS - 1.28 ### Versions of Apache Airflow Providers ``` $ pip freeze | grep apache-airflow-providers apache-airflow-providers-amazon==8.19.0 apache-airflow-providers-celery==3.6.1 apache-airflow-providers-cncf-kubernetes==8.0.1 apache-airflow-providers-common-io==1.3.0 apache-airflow-providers-common-sql==1.11.1 apache-airflow-providers-docker==3.9.2 apache-airflow-providers-elasticsearch==5.3.3 apache-airflow-providers-ftp==3.7.0 apache-airflow-providers-google==10.16.0 apache-airflow-providers-grpc==3.4.1 apache-airflow-providers-hashicorp==3.6.4 apache-airflow-providers-http==4.10.0 apache-airflow-providers-imap==3.5.0 apache-airflow-providers-microsoft-azure==9.0.1 apache-airflow-providers-microsoft-mssql==3.4.0 apache-airflow-providers-mysql==5.5.4 apache-airflow-providers-odbc==4.4.1 apache-airflow-providers-openlineage==1.6.0 apache-airflow-providers-postgres==5.10.2 apache-airflow-providers-redis==3.6.0 apache-airflow-providers-sendgrid==3.4.0 apache-airflow-providers-sftp==4.9.0 apache-airflow-providers-slack==8.6.1 apache-airflow-providers-smtp==1.6.1 apache-airflow-providers-snowflake==5.3.1 apache-airflow-providers-sqlite==3.7.1 apache-airflow-providers-ssh==3.10.1 ``` ### Deployment Official Apache Airflow Helm Chart ### Deployment details Airflow is deployed with helm chart https://github.com/airflow-helm/charts/blob/airflow-8.9.0/charts/airflow/values.yaml ### Anything else? This occurs every time. We are seeing below logs in airflow-scheduler pod. There are no DAGs running and no DAGs scheduled to run. ``` {scheduler_job_runner.py:1803} INFO - Orphaning unreferenced dataset {scheduler_job_runner.py:1803} INFO - Orphaning unreferenced dataset {scheduler_job_runner.py:1803} INFO - Orphaning unreferenced dataset [2024-07-24T09:05:27.309+0000] {kubernetes_executor.py:239} INFO - Found 0 queued task instances ``` ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
