Vinibisineli opened a new issue, #76633:
URL: https://github.com/apache/airflow/issues/76633
### Apache Airflow version
3.4.0
### If "Other Airflow 2 version" selected, which one?
N/A
### What happened?
Manually triggering a DagRun via the REST API (`POST
/api/v2/dags/{dag_id}/dagRuns`) leaves it stuck in `queued` state forever —
task instances never leave `None` state, and the DagRun's
`last_scheduling_decision` is never set (remains `null`).
This happens on a **completely fresh installation** (new metadata DB,
`docker compose up` for the first time), so there's no pre-existing state, no
stale `dag_version`, no prior failed runs — just a clean start, DAGs
auto-paused at creation (`AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION=true`),
and the very first manually-triggered run on one of them.
The `SchedulerJob` itself is healthy the entire time — it's alive in the
`job` table with a current `latest_heartbeat`, and its process is consuming CPU
— but its logs show **no scheduling activity whatsoever** beyond the two
startup lines:
```
[info] Starting the scheduler
[airflow.jobs.scheduler_job_runner.SchedulerJobRunner]
[info] Adopting or resetting orphaned tasks for active dag runs
[airflow.jobs.scheduler_job_runner.SchedulerJobRunner]
```
The second line repeats every ~5 minutes (periodic housekeeping), but there
is never any log line about processing the queued DagRun, finding executable
task instances, or making a scheduling decision — not even an error or warning.
It's completely silent.
Confirmed the DagRun really is untouched at the DB level:
```sql
SELECT dag_id, run_id, state, logical_date, last_scheduling_decision FROM
dag_run;
-- state=queued, last_scheduling_decision is NULL/empty, indefinitely
```
### What you think should happen instead?
The scheduler should pick up the queued DagRun (whose
`logical_date`/`data_interval` is in the past, not the future) within its
normal scheduling loop interval (`scheduler_idle_sleep_time=1`), create/queue
its task instances, and dispatch them to the Celery broker — same as it does
for scheduled runs.
### How to reproduce
1. Fresh `docker compose` install, Airflow 3.4.0, `CeleryExecutor`, Postgres
16 + Redis 7.2 backend, no pre-existing metadata DB (first `airflow db migrate`
on an empty database).
2. Any DAG with a plain cron `schedule` (e.g. `"45 4 * * *"`),
`max_active_runs=1`, decorated with `@dag`/`@task` (TaskFlow API). DAGs come up
paused by default (`dags_are_paused_at_creation=True`).
3. Wait for scheduler/dag-processor/apiserver/worker/triggerer to report
healthy and for the DAG to show up with `import_errors: []`.
4. Trigger a run manually via the API with an explicit **past**
`logical_date`:
```
POST /api/v2/dags/<dag_id>/dagRuns
{"logical_date": "<a few hours in the past, UTC>"}
```
(Confirmed this isn't a future-logical_date mistake — tried both `null`
and explicit past timestamps, same result.)
5. Poll `GET /api/v2/dags/<dag_id>/dagRuns/<run_id>/taskInstances`
repeatedly — states remain `None` indefinitely (tested up to several minutes).
6. Check scheduler container logs — no scheduling activity logged at all
besides the two startup lines above.
Reproduced **identically in two separate environments**:
- macOS + Docker Desktop (Apple Silicon/ARM64)
- Linux x86_64 (production-equivalent server, same `docker-compose.yaml`
topology, isolated project/network/volumes from an existing healthy 3.3.2
deployment running alongside it on the same host)
### Operating System
Reproduced on both: macOS 14 (Docker Desktop, ARM64) and Linux x86_64
(Oracle Linux-based, Debian 13 container base)
### Versions of Apache Airflow Providers
Base image: `apache/airflow:3.4.0-python3.12`. Additional providers
installed via the official `3.4.0`/`constraints-3.12.txt` constraints file:
`apache-airflow-providers-mysql`, `apache-airflow-providers-google`,
`apache-airflow-providers-sftp`.
### Deployment
Docker-Compose
### Deployment details
Standard community `docker-compose.yaml` template (CeleryExecutor + Redis +
Postgres), lightly customized: separate `airflow-dag-processor` service (not
bundled into scheduler), custom Dockerfile layering a few `apt-get`/`pip
install` packages on top of the base image. 6 cores / 30GB RAM host for the
Linux reproduction.
### Anything else?
- The DAG that reproduces this isn't unusual — plain TaskFlow API,
`@dag(schedule="45 4 * * *", max_active_runs=1)`, a handful of `@task`s fanning
out from a `start` `EmptyOperator` and converging into `end`.
- No custom executor, no `AssetOrTimeSchedule`, no `TriggerDagRunOperator`
involved — just a direct manual trigger via the REST API on a cron-scheduled
DAG.
- Didn't happen (as far as we've observed in production) on the same project
under Airflow **3.3.2** with the exact same DAG code and `docker-compose.yaml`
topology — only started when testing the upgrade to 3.4.0.
- Related-looking (but not identical) prior issues: #70056 (pinned
`dag_version` resolution — fixed in 3.3.2, so not the same root cause here) and
#56045 (scheduler silently stuck, Celery, Docker Compose — closed "not planned"
for lack of diagnostic data).
### Are you willing to submit a PR?
- [ ] Yes, I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
