Jeoffreybauvin opened a new issue, #74394: URL: https://github.com/apache/airflow/issues/74394
### Under which category would you file this issue? Airflow Core ### Apache Airflow version 3.3.1 ### What happened and how to reproduce it? I'm trying to debug a high memory usage from the api-server. Here is what seems to be the issue : ``` curl -H "Authorization: Bearer ...." -k -s "https://localhost:8080/ui/structure/structure_data?dag_id=my_dag"; -v -o /dev/null -s -w '%{time_total}\n' 15.538918 ``` This call takes always 15 to 17 secondes to run. And in the meantime, ram usage for the api-server rise from 700 Mo to 5 Go : ``` CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 0ae97870d109 airflow-webserver 101.71% 5.126GiB / 47GiB 10.91% 5.19GB / 28.1MB 89.1MB / 16.4kB 20 ``` My dag is pretty big, but not "too big" : This json is the result from the call to structure_data : ``` # cat result.json | jq '.nodes | length' 282 # cat result.json | jq '.edges | length' 281 ``` I tried to use memray, but, sadly, it's not seing the same thing :( . The memory consumption from memray seems flat, not sure if it's profiling the right gunicorn worker... Not sure what to "test" now, I'm looking for help to debug this, and validate if it's a DAG issue ou an Airflow issue :(. ### What you think should happen instead? The call to "/ui/structure/structure_data?dag_id=my_dag" should not take 17 secondes to run and should not consume 4 Go memory. ### Operating System Ubuntu 24.04.5 LTS ### Deployment Docker-Compose ### Apache Airflow Provider(s) _No response_ ### Versions of Apache Airflow Providers ``` apache-airflow-providers-amazon==9.35.1 apache-airflow-providers-apache-cassandra==3.9.5 apache-airflow-providers-apache-hdfs==4.13.0 apache-airflow-providers-apache-kafka==1.16.0 apache-airflow-providers-apache-spark==6.3.2 apache-airflow-providers-celery==3.23.1 apache-airflow-providers-cncf-kubernetes==10.21.1 apache-airflow-providers-common-compat==1.18.0 apache-airflow-providers-common-io==1.8.0 apache-airflow-providers-common-messaging==2.0.4 apache-airflow-providers-common-sql==2.1.1 apache-airflow-providers-docker==4.5.9 apache-airflow-providers-elasticsearch==6.9.0 apache-airflow-providers-fab==3.8.1 apache-airflow-providers-ftp==3.15.2 apache-airflow-providers-git==0.4.3 apache-airflow-providers-google==22.3.0 apache-airflow-providers-grpc==3.9.5 apache-airflow-providers-hashicorp==4.8.1 apache-airflow-providers-http==6.0.5 apache-airflow-providers-microsoft-azure==15.0.1 apache-airflow-providers-microsoft-mssql==4.7.1 apache-airflow-providers-mysql==6.6.2 apache-airflow-providers-odbc==4.12.3 apache-airflow-providers-openlineage==2.20.1 apache-airflow-providers-opensearch==1.12.0 apache-airflow-providers-oracle==4.6.3 apache-airflow-providers-postgres==7.0.2 apache-airflow-providers-redis==4.5.0 apache-airflow-providers-sendgrid==4.2.4 apache-airflow-providers-sftp==6.0.1 apache-airflow-providers-slack==9.10.2 apache-airflow-providers-smtp==3.0.3 apache-airflow-providers-snowflake==6.16.1 apache-airflow-providers-ssh==6.0.1 apache-airflow-providers-standard==1.18.0 ``` ### Official Helm Chart version Not Applicable ### Kubernetes Version n/a ### Helm Chart configuration n/a ### Docker Image customizations n/a ### Anything else? ``` AIRFLOW__EXECUTION_API__JWT_EXPIRATION_TIME: '86400' AIRFLOW__WORKERS__EXECUTION_API_RETRY_WAIT_MIN: '10' AIRFLOW__WORKERS__EXECUTION_API_TIMEOUT: '10' AIRFLOW__EXECUTION_API__JWT_AUDIENCE: airflow-task-api AIRFLOW__API_AUTH__JWT_ALGORITHM: HS512 AIRFLOW__API__SERVER_TYPE: gunicorn AIRFLOW__API__WORKER_REFRESH_INTERVAL: 1800 AIRFLOW__API__WORKERS: 8 ``` ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
