AP24110011695 opened a new issue, #77346:
URL: https://github.com/apache/airflow/issues/77346
### Apache Airflow version
3.1.x (main @ bee092f4, 2026-11-19)
### If "Airflow 2" (experimental)
_No response_
### What happened
With a custom XCom backend configured (`[core] xcom_backend`), XComs written
by a trigger through the task-end event API bypass the backend, while XComs
written by a running task go through it. The two write paths produce
differently-shaped rows for the same key namespace:
- **Task-runtime writes** go through the supervisor, which runs the backend
in a helper process and stores an externalized reference in the XCom row
(`XComBackendCall.store(...)` → the row holds the backend's returned reference).
- **Trigger-written writes** (`Trigger._push_xcoms_if_necessary` →
`XComModel.set_for_attempt(serialize=False)`, in
`airflow-core/src/airflow/models/trigger.py`) store the raw value in the row.
This was pinned deliberately by #76596, and the triggerer has no task context
or backend access.
`TaskSuccessEvent(xcoms=...)` / `TaskFailureEvent(xcoms=...)`
(`airflow-core/src/airflow/triggers/base.py`) are documented public trigger
API, so this path is reachable by user code.
The consequence shows on reads. When a downstream task pulls such a key, the
supervisor's get path resolves row values through the backend's load hook
whenever a custom backend is configured (see #77136 for the current gate). For
a trigger-written row the "row value" is the raw value itself — never
externalized — so a backend that assumes reference-shaped input (the common
case: fetching from external storage, parsing a pointer) either fails with
`XCOM_BACKEND_ERROR` or silently mis-resolves. Backends that fall back to the
serde path on foreign values happen to work, which is why this has not surfaced
widely yet.
### What you think should happen instead
The two write paths should produce consistent rows for the same key, or
reads should distinguish provenance. Options we can see:
1. Route trigger-written xcoms through a backend-aware path (the triggerer
has no backend instance or task-instance coordinate today, so this needs
plumbing — and the helper process model assumes a live task).
2. Mark rows with their provenance (backend-externalized vs inline) so reads
can skip the load hook for inline rows.
3. Document that custom backends must tolerate raw JSON values in `load`
(the base-class fallback does; strict backends do not).
We are not sure which of these the project prefers — 1 is the semantically
cleanest but the heaviest, 2 is targeted, 3 acknowledges the current behavior.
### How to reproduce
1. Configure a custom XCom backend whose `load`/`load_value` assumes
externalized references (e.g. raises on non-reference values — any backend that
fetches from remote storage).
2. Run a Dag whose task defers; its trigger yields
`TaskSuccessEvent(xcoms={"k": "v"})`.
3. A downstream task pulls `k` through the Task SDK.
4. The supervisor routes the row through the backend's load hook with the
raw value `v`, and the backend fails (or mis-resolves) — while the same pull of
a task-written key of the same shape resolves correctly.
### Operating system
macOS / Linux (main branch code path; observed while reviewing #77136)
### Versions of Apache Airflow Providers
n/a — core task-sdk / triggerer paths
### Anything else?
Related but distinct: #76212 (transaction timing on the same write path),
#76596 (the double-encoding fixed there, which also pinned raw storage), #77136
(the read-gate key exclusions this issue complements). No existing issue tracks
the backend bypass itself.
AI disclosure: this report was compiled with AI assistance from the code
paths cited above; the flow is traceable from the referenced sources.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]