dhanusharer commented on issue #40709:
URL: https://github.com/apache/beam/issues/40709#issuecomment-6504195794
### Pandas 3.0+ Scoping Analysis & Decision Matrix
Following up on the upgrade scoping for Pandas 3.0+, here is the inventory
of breaking changes, Beam touchpoints, behavioral decisions, and rollout
strategy:
#### 1. Inventory of Pandas 3.0/3.1 Changes & Beam Touchpoints
- **Python Versioning:** Pandas 3.0 drops Python 3.10 (requires 3.11+). When
the cap is raised, PEP 508 environment markers (`python_version < "3.11"` vs
`>= "3.11"`) will partition the dependency in `setup.py`.
- **String Dtype (`future.infer_string=True`):** Default string columns
infer as `pd.StringDtype` (PyArrow or python-backed) with `<NA>` instead of
`object` with `np.nan`.
- *Touchpoints:* `apache_beam/typehints/pandas_type_compatibility.py` and
`apache_beam/dataframe/schemas.py`.
- *Decision:* Use `pd.api.types.is_string_dtype(dtype)` to map all string
dtypes to `Optional[str]`, while keeping `dtype.kind == 'O'` mapped to `Any`.
- **Copy-on-Write (CoW):** Permanently enforced; `copy=True` deprecated or
ignored.
- *Touchpoints:* `apache_beam/dataframe/frame_base.py`
(`@populate_defaults`).
- *Decision:* Set `copy=None` under Pandas 3.0+ to prevent deprecation
warnings. Beam DataFrames already operate with immutable stage boundaries.
- **Removed Methods (`applymap`, `first`, `last`, `GroupBy.fillna`):**
- *Touchpoints:* `DeferredDataFrame`, `DeferredSeries`,
`_DeferredGroupByCols`.
- *Decision:* Guard definitions with `hasattr(base_type, '...')` to
prevent import-time reflection crashes. In Pandas 3.0, users migrate to
`.map()` and `.ffill()` / `.bfill()`.
- **Removed Arguments (`method` in `fillna`, `axis` in `groupby`):**
- *Touchpoints:* `frames.py` decorators (`@args_to_kwargs`,
`@populate_defaults`).
- *Decision:* Add `removed_args=['method']` and avoid passing
`method=None` to `df.fillna()`. Raise clean `NotImplementedError` for
`groupby(axis=1)` under Pandas 3.0+.
- **Default Containers & Dependencies:**
- *Touchpoints:* Automated weekly dependency bumps update release
containers (`apache/beam_python3.11_sdk`, etc.). Keeping the cap below 3.0 in
preparation PRs prevents prematurely shifting worker container runtimes.
#### 2. Rollout Plan
1. **Phase 1 (PR #40768):** Land all backward-compatible reflection
hardening, `hasattr()` guards, and dtype conversions while keeping the cap
below 3.0 (`pandas>=2.2.2,<2.4`). Validate with a new `py311-pandas-23future`
tox environment running with `PANDAS_MODE__COPY_ON_WRITE=1` and
`PANDAS_FUTURE__INFER_STRING=1`.
2. **Phase 2:** Run experimental CI suites against Pandas 3.0 pre-releases
to verify edge-case doctest formatting and signature parity.
3. **Phase 3:** Raise the cap (`pandas<3.2`) for Python 3.11+ in a dedicated
PR with an entry in `CHANGES.md`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
