Spenserrrr commented on code in PR #57624:
URL: https://github.com/apache/spark/pull/57624#discussion_r3709002782
##########
python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py:
##########
@@ -99,6 +105,77 @@ def _to_pandas_cell(self, arr, **to_pandas_kwargs) -> str:
except Exception as e:
return f"ERR@{type(e).__name__}"
+ @staticmethod
+ def _arrow_buffers(arrow_obj):
+ """
+ Every non-null buffer backing ``arrow_obj``.
+
+ A ChunkedArray has no buffers of its own -- its data lives in its
chunks --
+ so it is expanded first. ``None`` entries (e.g. an absent validity
bitmap)
+ are skipped.
+ """
+ if isinstance(arrow_obj, pa.ChunkedArray):
+ chunks = [arrow_obj.chunk(i) for i in range(arrow_obj.num_chunks)]
+ else:
+ chunks = [arrow_obj]
+
+ buffers = []
+ for chunk in chunks:
+ for buffer in chunk.buffers():
+ if buffer is not None:
+ buffers.append(buffer)
+ return buffers
+
+ @classmethod
+ def _verify_zero_copy(cls, arr, **to_pandas_kwargs) -> str:
+ """
+ Independently verify whether ``to_pandas`` reused ``arr``'s buffers,
+ instead of trusting PyArrow's own ``zero_copy_only`` verdict.
+
+ The check inspects whatever storage pandas returned rather than
assuming a
+ numpy-backed Series, so it stays correct as pandas moves more dtypes to
+ Arrow-backed storage.
+
+ Returns ``"zero-copy"``, ``"copied"``, or ``"ERR@<ExceptionClass>"``.
+ """
+ try:
+ series = arr.to_pandas(**to_pandas_kwargs)
+ except Exception as e:
+ return f"ERR@{type(e).__name__}"
+
+ backing_array = series.array
+
+ # Arrow-backed result: compare buffer addresses, reading the stored
data
+ # back through the public __arrow_array__ protocol. to_numpy() would
+ # materialize a copy here and wrongly report no sharing. Keying on the
+ # protocol rather than on ArrowDtype also covers dtypes that are
+ # Arrow-backed without being ArrowDtype, such as pandas 3's string.
+ if hasattr(backing_array, "__arrow_array__"):
+ stored = pa.array(backing_array)
+ source_addresses = {buffer.address for buffer in
cls._arrow_buffers(arr)}
+ for buffer in cls._arrow_buffers(stored):
+ if buffer.address in source_addresses:
+ return "zero-copy"
Review Comment:
Thanks for the advice @zhengruifeng! Actually, this problem already appears.
On pandas 2 it's harmless. A numpy result is one dense block, so any/all are
the same question. But on pandas 3 + pyarrow >= 24, strings become
ArrowStringArray with 3 buffers of their own. string:standard borrows the
values while pandas rebuilds the 32-bit offsets as 64-bit because it normalizes
string to large_string. Thus, only 1 of the 2 buffers is shared. The old check
can't tell this.
Thus, I count how many of the result's buffers were borrowed, then report
zero-copy (all), partial-copy (some), copied (none). The edit is in the new
commit. The golden file is identical, and in the override section, two rows are
changed to partial-copy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
