potiuk opened a new pull request, #70791:
URL: https://github.com/apache/airflow/pull/70791
A `pandas.DataFrame` does not survive an XCom round trip under pandas 3 the
way it
does under pandas 2, so this version of Airflow constrains pandas to `< 3`
and
refuses DataFrame XComs at runtime when a newer pandas is installed anyway.
### What breaks under pandas 3
**The serializer is never reached.** pandas 3 exposes its public classes
from the
`pandas` namespace, so a DataFrame qualifies as `pandas.DataFrame` rather
than
`pandas.core.frame.DataFrame`. The serde registry is keyed on the latter, so
it does
not dispatch to the pandas serializer at all and the caller gets serde's
generic
```
TypeError: cannot serialize object of type <class 'pandas.DataFrame'>
```
with nothing in it pointing at pandas or at what to do about it.
**The dtypes differ.** Even once dispatch is fixed, the same payload reads
back
differently depending on the reader's pandas version — an `object` column
becomes
`str`, and missing values become `nan` rather than `None`. The *reader's*
pandas
version decides what a task receives, not the writer's, so a mixed-version
deployment silently hands tasks data that differs from what was pushed.
### Approach
`pandas<3` is declared where Airflow declares pandas — the `pandas` extra on
`common-sql` (what `apache-airflow[pandas]` resolves to) and the task-sdk
development group.
**That constraint cannot be relied on by itself.** pandas is an optional
dependency: a deployment can install pandas 3 directly, or pull it in through
another package, and nothing in Airflow's metadata prevents it. So the
constraint
is backed by a runtime check that fails with a message naming pandas, the
version
found, and what to do.
**The check needs the pandas 3 qualname registered to be reachable at all.**
`pandas.DataFrame` is added to the registry *only* so that the request
reaches this
module and can be refused deliberately — not to support it. Without that
entry the
guard is dead code, because serde never dispatches.
**Reads are refused as well as writes.** A payload written under pandas 2
comes
back with pandas 3 dtypes, so accepting it would give the task different
data than
was pushed, with no signal at all. An error on both sides is the safer
outcome, and
the write-side failure is the more recoverable of the two.
### Behaviour change
With pandas >= 3 installed, pushing or pulling a `DataFrame` through XCom
raises:
```
RuntimeError: DataFrame XComs are not supported with pandas 3.0.5: this
version of
Airflow supports pandas < 3 for XCom serialization. Pin pandas < 3 in the
environment that runs your tasks, or avoid passing DataFrames through XCom.
...
```
Previously the same situation produced an opaque serde `TypeError` on write,
and —
once dispatch was corrected — silently different dtypes on read. Nothing
changes for
pandas 2 users.
**This defers pandas 3 support, it does not drop it.** Moving from pandas 2
to
pandas 3 is a deliberate migration for users to make, given the dtype
changes.
Airflow will enable it in a future release; see the follow-up PR linked in
the
comments.
### Test plan
- [x] `task-sdk/tests/task_sdk/serde/test_serializers.py` — 11 pandas cases
pass
- [x] `test_pandas_3_dataframe_xcom_is_refused` — `3.0.0`, `3.0.5` and
`4.1.2` all
refused on **both** `serialize` and `deserialize`
- [x] `test_pandas_3_dataframe_name_is_registered` — guards the reachability
precondition, so the guard cannot silently become dead code if the
registry
entry is dropped
- [x] `test_pandas_2_dataframe_xcom_still_round_trips` — `2.1.2` and `2.3.3`
unaffected
- [x] Existing pandas serde tests unchanged and passing
- [x] `uv.lock` regenerated for the constraint
- [x] `ruff check` / `ruff format` clean
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Opus 5 (1M context)
Generated-by: Claude Opus 5 (1M context) following the guidelines at
https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
