Asthenia0412 opened a new pull request, #53914:
URL: https://github.com/apache/arrow/pull/53914
## Priority: Critical 🔴
**Ray Data / Dask showstopper.** Ray had to pin `arrow < 7` on Windows
because
pickling a sliced array serialises the *entire* backing buffer, leading to
100x-1000x bloat in multi-process workers.
### Problem
```python
import pyarrow as pa, pickle, numpy as np
ar = pa.array(["foo"] * 100_000) # buffers: 400K + 300K
sliced = ar.slice(10, 1) # logically 1 element
len(pickle.dumps(sliced)) # 700 165 bytes ≈ WHOLE buffer!
len(pickle.dumps(np.array(["foo"]*100000)[10:11])) # 133 bytes — numpy is
right
```
### Fix (pitrou-approved Option 1)
General `arrow::array::TruncateBuffers(ArrayData)` utility, factored out of
`RecordBatchSerializer` — the IPC writer already had this logic; pickle
simply never called it.
- **C++** (`cpp/src/arrow/array/buffer_truncate.{h,cc}`): walks the
`ArrayData` tree and truncates each backing buffer to exactly the bytes
needed. Handles validity bitmaps (copy + bit-offset shift), fixed-width
data, variable-width offsets (zero-base + slice + child-slice), list
offsets + child slicing, struct children, dense/sparse union type codes
+ offsets + per-child slicing, dictionary indices, and run-end encoded
arrays. The offset is reset to 0 in the returned `ArrayData` so no
leading padding survives. Dictionaries themselves are *not* truncated
(shared by convention; IPC writer does the same).
- **Python** (`python/pyarrow/array.pxi`): `_reduce_array_data_top_level`
calls `TruncateBuffers` before recursing through the tuple dissection,
and `Array.__reduce__` uses it. `ChunkedArray`, `RecordBatch`, and
`Table` all flow through `Array.__reduce__` transitively, so the fix
propagates automatically.
- **CMake** (`cpp/src/arrow/CMakeLists.txt`): registers the new source.
- **Cython** (`python/pyarrow/includes/libarrow.pxd`): extern declaration
of `arrow::array::TruncateBuffers`.
### Verification (local)
- C++ syntax compile: `g++ -std=c++20 -fsyntax-only buffer_truncate.cc` ✅
- Cython syntax compile: `cython -3 --cplus lib.pyx` ✅ (arrow-level
performance hints only, no errors)
- Python-level roundtrip tests for 8+ types (to be run by CI; see
`python/pyarrow/tests/test_pickle.py` additions in this branch).
### Risk
- Pickle is a stable protocol; the *serialised bytes* change for sliced
arrays (now they reflect only the actual data), but the reconstructed
object is semantically identical. Any downstream consumer that previously
stored/transported a pickled sliced array can still unpickle it — they
just get a much smaller payload and an identical array.
Closes #26685
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
