amoghrajesh opened a new pull request, #71103:
URL: https://github.com/apache/airflow/pull/71103
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull
request.
Write a good git commit message following this guide:
https://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the
documentation.
For user-facing UI changes, please attach before/after screenshots (or a
short
screen recording) so reviewers can assess the visual impact.
Feel free to ping (in general) for the review if you do not see reaction for
a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we
sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change below checkbox to `[X]` followed by the name of the tool, uncomment
the "Generated-by".
-->
- [ ] Yes (please specify the tool below)
<!--
Generated-by: [Tool Name] following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
Supersedes https://github.com/apache/airflow/pull/70558
Following discussion from Ash on
https://lists.apache.org/thread/tbv4q04b3xx1hnmtq1b34tkzf43d5yqz
pandas 3 exposes its public classes from the `pandas` namespace, so a
DataFrame is qualified as
`pandas.DataFrame` rather than `pandas.core.frame.DataFrame`. The serde
registry is keyed on that
name and only knew the old one, so under pandas 3 **no DataFrame could be
pushed through XCom at
all**:
```
TypeError: cannot serialize object of type <class 'pandas.DataFrame'>
```
Both names are now registered, so a DataFrame written by either pandas
version can be read by
either. Verified across all four combinations (writer x reader), with the
payload written by one
interpreter and read by the other:
| Reader | pandas-2 payload | pandas-3 payload |
|---|---|---|
| pandas 2.3.3 | `object`, missing = `None` | `object`, missing = `None` |
| pandas 3.0.5 | `str`, missing = `nan` | `str`, missing = `nan` |
The reader's pandas version decides the dtypes, not the writer's -- that,
and the fact that a
component without this change cannot read a pandas-3-written DataFrame XCom,
is what the
significant newsfragment documents.
Two provider tests also asserted on pandas 2 behaviour (`object` dtype,
missing values stringified
to `"nan"` / `"None"`) and are now version-aware. The production paths were
already correct.
Reopened from #70558 -- carries the same three commits unchanged, plus one
added test closing a
review gap: nothing previously forced deserialization through the registry
entry for the *other*
pandas major's qualname, since `serialize()` only ever produces the qualname
of whatever pandas is
actually installed. Depends on the separate revert of #70791 landing first.
related: #70558, #70791
### Testing
#### Testing with pandas v2
DAG:
```
from __future__ import annotations
import pendulum
from airflow.sdk import DAG, task
@task
def push_df():
import pandas as pd
print(f"pandas version (push): {pd.__version__}")
return pd.DataFrame({"strings": ["a", "b", None], "ints": [1, 2, None]})
@task
def pull_df(df):
import pandas as pd
print(f"pandas version (pull): {pd.__version__}")
print(f"dtypes:\n{df.dtypes}")
print(f"strings column: {df['strings'].tolist()}")
print(f"ints column: {df['ints'].tolist()}")
with DAG(
dag_id="pandas3_xcom_check",
start_date=pendulum.datetime(2024, 1, 1, tz="UTC"),
schedule=None,
catchup=False,
):
pull_df(push_df())
```
<img width="2495" height="983" alt="image"
src="https://github.com/user-attachments/assets/7d15f05c-00fa-4b0f-9b9e-a6803001d4e6";
/>
<img width="2495" height="983" alt="image"
src="https://github.com/user-attachments/assets/2bcd95f0-8856-48c5-8969-f2ba49d1c2b4";
/>
#### Testing with pandas v3
<img width="2495" height="983" alt="image"
src="https://github.com/user-attachments/assets/e50f39fc-4765-47c7-9cc2-d918f109a0e0";
/>
<img width="2495" height="983" alt="image"
src="https://github.com/user-attachments/assets/2c7aef2e-aff9-4a20-9b7d-7ee65110eb2b";
/>
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
