zhengruifeng commented on code in PR #57624:
URL: https://github.com/apache/spark/pull/57624#discussion_r3708585101
##########
python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py:
##########
@@ -99,6 +105,77 @@ def _to_pandas_cell(self, arr, **to_pandas_kwargs) -> str:
except Exception as e:
return f"ERR@{type(e).__name__}"
+ @staticmethod
+ def _arrow_buffers(arrow_obj):
+ """
+ Every non-null buffer backing ``arrow_obj``.
+
+ A ChunkedArray has no buffers of its own -- its data lives in its
chunks --
+ so it is expanded first. ``None`` entries (e.g. an absent validity
bitmap)
+ are skipped.
+ """
+ if isinstance(arrow_obj, pa.ChunkedArray):
+ chunks = [arrow_obj.chunk(i) for i in range(arrow_obj.num_chunks)]
+ else:
+ chunks = [arrow_obj]
+
+ buffers = []
+ for chunk in chunks:
+ for buffer in chunk.buffers():
+ if buffer is not None:
+ buffers.append(buffer)
+ return buffers
+
+ @classmethod
+ def _verify_zero_copy(cls, arr, **to_pandas_kwargs) -> str:
+ """
+ Independently verify whether ``to_pandas`` reused ``arr``'s buffers,
+ instead of trusting PyArrow's own ``zero_copy_only`` verdict.
+
+ The check inspects whatever storage pandas returned rather than
assuming a
+ numpy-backed Series, so it stays correct as pandas moves more dtypes to
+ Arrow-backed storage.
+
+ Returns ``"zero-copy"``, ``"copied"``, or ``"ERR@<ExceptionClass>"``.
+ """
+ try:
+ series = arr.to_pandas(**to_pandas_kwargs)
+ except Exception as e:
+ return f"ERR@{type(e).__name__}"
+
+ backing_array = series.array
+
+ # Arrow-backed result: compare buffer addresses, reading the stored
data
+ # back through the public __arrow_array__ protocol. to_numpy() would
+ # materialize a copy here and wrongly report no sharing. Keying on the
+ # protocol rather than on ArrowDtype also covers dtypes that are
+ # Arrow-backed without being ArrowDtype, such as pandas 3's string.
+ if hasattr(backing_array, "__arrow_array__"):
+ stored = pa.array(backing_array)
+ source_addresses = {buffer.address for buffer in
cls._arrow_buffers(arr)}
+ for buffer in cls._arrow_buffers(stored):
+ if buffer.address in source_addresses:
+ return "zero-copy"
Review Comment:
_arrow_buffers includes validity, offset, and value buffers, and this loop
returns `zero-copy` when *any* source buffer is present in pandas storage. A
conversion could retain an offset or validity buffer while materializing the
values, masking the data-copy regression this test is meant to detect. Could we
compare the value buffer specifically (as described in the design doc), require
all relevant buffers to match, or report a distinct partial-copy result?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
