irfansiddiqui opened a new issue, #70958:
URL: https://github.com/apache/airflow/issues/70958
### Apache Airflow version
3.0.6
### What happened?
Tasks that are **directly downstream of a mapped `@task_group` whose
expansion never occurred** (because the task producing the expansion input
finished as `failed`/`upstream_failed`) can **never launch**. The API server
raises an unhandled exception (HTTP 500, `ERROR: Exception in ASGI
application`) on the task runner's startup call:
```
PATCH /execution/task-instances/{ti_id}/run
```
The worker-side SDK client (`airflow.sdk.api.client`) logs `Server error
{'message': 'Internal server error', ...}`, retries via tenacity, and
ultimately raises `ReadTimeout` / `ServerResponseError` inside Celery's
`execute_workload`. The executor then reports the task as failed. The task
instance ends up `failed` with:
- `hostname` empty
- `start_date`/`duration` null (`run_start_date=None` in the scheduler's
"TaskInstance Finished" line)
- no task log stream ever created
Because the failure is reported while the TI is still `queued`, retry
behavior is fragile. We additionally observed that when the DagRun's pinned
serialized DAG version had been replaced by a redeploy, the scheduler's
executor-event handling logged:
```
{scheduler_job_runner.py:952} ERROR - DAG '<dag_id>' for task instance
<TaskInstance: ... [queued]> not found in serialized_dag table
{taskinstance.py:2011} INFO - Marking task as FAILED. ...
```
and force-failed the TI **bypassing its retry policy entirely** (`retries=2`
was configured; the TI went straight to `failed` at `try_number=1`).
Behavioral fingerprint that isolated the cause (observed across ~6 runs on
the same environment, minutes apart):
- every task **directly downstream of the unexpanded mapped group** fails to
launch this way (in our DAG: a `ONE_FAILED` failure-callback leaf and two
`ALL_DONE` cleanup tasks);
- every task **not** directly downstream of the mapped group launches
normally in the same runs (including a task one extra hop downstream of the
affected cleanup tasks);
- the *same* downstream tasks launch normally in runs where the expansion
**did** occur;
- rewiring the failure-callback task so it no longer had the mapped group as
a direct upstream made it launch and execute immediately, same environment,
same failure scenario.
A side effect worth noting: the repeated doomed launch attempts also
produced `celery.concurrency.asynpool` `Timed out waiting for UP message from
<ForkProcess...>` noise, which initially misdirected us toward worker health.
### What you think should happen instead?
The `/run` endpoint should be able to build the startup context for a TI
whose direct upstream is a mapped task group with zero expanded instances (e.g.
resolve upstream map indexes to an empty set), so the task can launch and let
its trigger rule/callable decide what to do. Failing that, the scheduler should
at least respect the TI's retry policy when handling executor-reported launch
failures, and surface the server-side exception in a way that points at the
affected task rather than a generic ASGI 500.
### How to reproduce
Sketch (matches our production DAG's shape):
```python
from airflow.decorators import dag, task, task_group
from airflow.utils.trigger_rule import TriggerRule
@dag(schedule=None)
def repro():
@task
def make_items() -> list:
raise ValueError("expansion input fails") # expansion never happens
@task_group
def per_item(item: dict):
@task
def work(item: dict): ...
work(item)
items = make_items()
mapped = per_item.expand(item=items)
# Any task scheduled despite the failure and directly downstream of
# the mapped group exhibits the launch failure:
@task(trigger_rule=TriggerRule.ONE_FAILED, retries=2)
def on_failure(**context):
...
flag = on_failure()
mapped >> flag # <-- direct mapped-group upstream: never
launches
items >> flag
repro()
```
Trigger the DAG; `make_items` fails; `on_failure` is scheduled (ONE_FAILED
satisfied via upstream_failed) and queued, then fails with no hostname/logs;
API-server log shows the ASGI exception on `PATCH
/execution/task-instances/{id}/run`. Remove the `mapped >> flag` edge and the
task launches and executes normally in the same scenario.
### Operating System
Amazon Linux (AWS MWAA)
### Versions of Apache Airflow Providers
As bundled with MWAA Airflow 3.0.6 (amazon provider; CeleryExecutor,
Postgres metadata DB).
### Deployment
Amazon (AWS) MWAA
### Deployment details
MWAA environment class mw1.small, 2 workers, 2 schedulers. Observed
2026-08-02/03; consistent across many runs and two differently-wired DAGs
(identical structure).
### Anything else?
Occurs every time the expansion input fails; never when expansion succeeds.
Happy to provide fuller (redacted) scheduler/worker/webserver log excerpts.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
