GitHub user jun-roh edited a discussion: Airflow 3.1.1 UI extremely slow when
loading DAGs (running on Docker Compose)
I’m running Apache Airflow 3.1.1 using Docker Compose, and I’ve noticed that
the Web UI is extremely slow when loading data — especially the main DAG list
and DAG details pages.
We currently have about 90 DAGs, and everything else (scheduler, workers, etc.)
seems to be functioning normally. However, the UI becomes noticeably sluggish,
sometimes taking 20–30 seconds to render pages.
Here’s a simplified version of my setup:
`
version: "3.8"
networks:
airflow-net:
driver: bridge
volumes:
airflow-git:
driver: local
x-airflow-common: &airflow-common
image: ${AIRFLOW_IMAGE_NAME}
user: "${AIRFLOW_UID:-50000}:0"
restart: unless-stopped
environment:
# ---- Core ----
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__CORE__STORE_SERIALIZED_DAGS: True
AIRFLOW__CORE__LOAD_EXAMPLES: False
# ---- Database ----
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN:
"mysql+pymysql://${AIRFLOW_DB_USER}:${AIRFLOW_DB_PASSWORD_URL}@${AIRFLOW_DB_HOST}/${AIRFLOW_DB_NAME}"
AIRFLOW__CELERY__RESULT_BACKEND:
"db+mysql+pymysql://${AIRFLOW_DB_USER}:${AIRFLOW_DB_PASSWORD_URL}@${AIRFLOW_DB_HOST}/${AIRFLOW_DB_NAME}"
# ---- Pooling ----
AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_SIZE: 30
AIRFLOW__DATABASE__SQL_ALCHEMY_MAX_OVERFLOW: 60
AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_TIMEOUT: 60
AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_RECYCLE: 1800
AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_PRE_PING: True
# ---- Scheduler ----
AIRFLOW__SCHEDULER__PARSING_PROCESSES: 1
AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: 120
AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL: 300
AIRFLOW__SCHEDULER__MAX_TIS_PER_QUERY: 64
# ---- Logging ----
AIRFLOW__LOGGING__REMOTE_LOGGING: True
AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: s3://example-bucket
AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: aws_default
# ---- Celery ----
AIRFLOW__CELERY__BROKER_URL: redis://redis.example.net:6379/2
AIRFLOW__CELERY__TASK_TRACK_STARTED: True
AIRFLOW__CELERY__PREFETCH_MULTIPLIER: 1
TZ: Asia/Seoul
volumes:
- airflow-git:/opt/airflow/git
- ./airflow_local_settings.py:/opt/airflow/airflow_local_settings.py:ro,Z
networks: [airflow-net]
services:
airflow-apiserver:
<<: *airflow-common
command: >
bash -c 'exec airflow api-server --host 0.0.0.0 --port 8080 --workers 1
--apps all'
ports: ["8080:8080"]
depends_on:
airflow-gitsync: { condition: service_healthy }
airflow-init: { condition: service_completed_successfully }
`
Environment summary
• Airflow version: 3.1.1
• Executor: CeleryExecutor
• DB: MySQL on AWS RDS (db.r7g.large)
• Redis: hosted Redis cluster (for Celery broker)
• DAG count: ~90
• UI: AWS loadbalancer using HTTPS
• Serialized DAGs: enabled (STORE_SERIALIZED_DAGS=True)
The Airflow UI is slow:
• DAG list takes 10–20 seconds to load
• Clicking into a DAG detail or “Graph View” is also very sluggish
• Scheduler and workers seem healthy, tasks execute fine
• Database CPU/memory usage is normal
GitHub link: https://github.com/apache/airflow/discussions/57933
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
