tritsystem opened a new issue, #51162:
URL: https://github.com/apache/arrow/issues/51162
Found via a property-based audit (copy/clone independence,
cache-invalidation consistency) run across several widely-used Python
libraries; repro was written and independently re-run twice (identical output
both times) before filing.
## Description
`pyarrow.Table.from_pandas()` on a non-null numeric pandas column takes a
zero-copy fast path that wraps the pandas block's raw numpy buffer directly,
bypassing whatever pandas-level accessor pandas' Copy-on-Write reference
tracker (`BlockValuesRefs`) expects in order to register an external referrer.
As a result, pandas believes the block has no external referrer, and a later
in-place mutation via `df.iloc[...] = ` or `df.loc[...] = ` mutates the block
directly -- silently corrupting the already-produced Arrow `Table`.
Plain pandas-to-pandas CoW sharing (e.g. `df_b = df_a[:]`) is correctly
protected; this gap is specific to pyarrow as a third-party zero-copy consumer
that pandas' CoW tracking doesn't know about.
## Minimal repro
```python
import pandas as pd
import pyarrow as pa
df = pd.DataFrame({"a": [1, 2, 3]})
table = pa.Table.from_pandas(df, preserve_index=False)
df.iloc[0, 0] = 999
print(table.column("a").to_pylist()) # actual: [999, 2, 3] -- expected:
[1, 2, 3]
```
Same result with `df.loc[0, "a"] = 999`. Note a full-column replace
(`df["a"] = [...]`) does NOT leak -- only single-cell in-place mutation reaches
the shared buffer.
## Root cause
`pyarrow/pandas_compat.py`, `dataframe_to_arrays()` -> `convert_column()`:
for a non-null numeric column this calls `pa.array(col, type=type_,
from_pandas=True, safe=safe)`, which takes pyarrow's documented zero-copy fast
path over the pandas block's raw numpy buffer. Confirmed unchanged on
`apache/arrow` `main` (same call site, `_can_definitely_zero_copy` unchanged).
Pandas' own Copy-on-Write user guide documents read-only-array protection
for `to_numpy()`/`.values` outputs, but says nothing about third-party
zero-copy consumers like pyarrow -- this isn't a documented, accepted
limitation on either side, just an interaction gap between the two libraries'
assumptions.
## Expected behavior
`Table.from_pandas()` should produce a Table whose contents are independent
of later mutation to the source DataFrame (or should document that it doesn't,
the way `to_numpy(zero_copy_only=True)` documents its own zero-copy sharing).
## Actual behavior
Mutating a single cell of the source DataFrame after conversion silently
mutates the already-produced Table's data.
## Versions
pyarrow 25.0.1, pandas 3.0.5 (Copy-on-Write unconditional/default since
3.0), Python 3.12.10, Windows 11. Confirmed present in
`pyarrow/pandas_compat.py` on `main` as of 2026-09-04.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
