potiuk commented on PR #70791: URL: https://github.com/apache/airflow/pull/70791#issuecomment-5134873159
Cross-reference: **[#70558](https://github.com/apache/airflow/pull/70558) is where pandas 3 support starts**, and it is milestoned for 3.4.0. This PR is the deliberate limit we apply until then. The two are complements rather than alternatives: - **#70558** registers both DataFrame qualnames (`pandas.core.frame.DataFrame` and `pandas.DataFrame`) so that a DataFrame written by either pandas version can be read by either, and documents that the reader's pandas version decides the resulting dtypes. - **this PR** registers the pandas 3 qualname for the opposite purpose — so the request reaches the serializer and can be *refused* with an actionable message instead of dying on serde's generic `cannot serialize object of type`. The reason for limiting rather than shipping support now is that **moving from pandas 2 to pandas 3 should be a deliberate decision by the user, not something an Airflow upgrade does to them**. Under pandas 3 the same XCom payload reads back with different dtypes — an `object` column becomes `str`, missing values become `nan` rather than `None` — so data already written under pandas 2 is read differently afterwards. That is a migration to plan, not a side effect to absorb. So this is a deferral, not a drop. Users who want pandas 3 today can still choose it; they just cannot pass DataFrames through XCom while doing so, and they now get told that explicitly rather than discovering it through an unrelated-looking `TypeError`. Support will be enabled in a future release once #70558 (or its successor) lands. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
