This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 6196e648 [SPARK-60745][CONNECT][FOLLOWUP] Preserve native conversion
for customized pandas strings
6196e648 is described below
commit 6196e648f54fd15fd346700af0e4422e85f079c6
Author: DB Tsai <[email protected]>
AuthorDate: Sun Nov 8 20:31:51 2026 +0900
[SPARK-60745][CONNECT][FOLLOWUP] Preserve native conversion for customized
pandas strings
### What changes were proposed in this pull request?
[SPARK-60745](https://issues.apache.org/jira/browse/SPARK-60745): preserve
native row-conversion behavior for customized Arrow-backed pandas strings in
the columnar input path already on master.
Specialized, generic explicit-schema and inferred/names-only routes share
the relevant hook and alias eligibility checks. Covered custom array/label
conversion hooks and rebound schema-converter aliases decline to native rows,
so another admission route cannot bypass those rejections. The specialized
string helper separately declines recognized storage-normalization and
validation failures, including a column storage length that differs from the
frame. Ordinary supported inputs retain [...]
Inspect current instance hooks and selected canonical class attributes,
including inherited methods and pandas' NumPy-semantics mixin/partial dispatch.
Read raw class attributes without binding customized descriptors, and recheck
between columns because an accessor can change a hook. Only the pandas helper,
regression tests and architecture documentation change.
### Why are the changes needed?
Direct Arrow-storage extraction can bypass a customized conversion method,
change values, suppress an exception or run label callbacks earlier than native
row conversion. The covered fixtures must retain native values, errors and
ordering through every applicable admission route.
### Does this PR introduce _any_ user-facing change?
Yes. Inputs with the covered customizations fall back to native rows;
ordinary eligible inputs retain the existing fast path. Authentication adds
per-column work, and conservative fallback can cost more than columnar
conversion.
The guarantee is bounded to the selected canonical Arrow-string classes,
instance/class hooks and explicitly checked aliases. **Class data read by those
methods, including `_storage` and `_na_value`, and other module globals are
outside the checks.** Other array classes retain the existing generic-route
module-origin policy. Arbitrary private storage replacement or inconsistent
frame internals are not covered by an all-route consistency guarantee. This
does not establish an invariant [...]
Existing importer costs remain: configuration RPCs, validation/casting,
alignment, IPC serialization and threshold-triggered artifact uploads.
Narrowing `large_string` to Arrow `string` can copy the offset buffer; this
path is not universally zero-copy. Simulated cast failures do not establish
support for actual storage above 2 GiB or malformed offsets. The inherited
package/runtime pandas-minimum mismatch and support for older/broken
dependencies are not repaired here.
### How was this patch tested?
Candidate `33559dc2af3df57b4db56b1cb90f16d1f5bbd8c5` against baseline
`e8b1d4284e11d119c4fda9563a913ece22f43e7c`:
- Independent full-patch source review found no actionable findings. **216
focused native cases passed**. Full native suite: **2,378 passed, eight
skipped, one failed**; the sole failure matches the baseline's
unsupported-NumPy-scalar case. The raw full result remains failed.
- Version matrix: **216 passed in each of four
environments**—pandas/PyArrow 2.2.3/18, 2.3.3/26, 3.0.6/26 and 3.1.0/26. Actual
imports, source and binary hashes were checked. Controls produced **132
candidate passes and 132 expected assertion failures**: 120 on baseline and 12
with intentionally memoized guards. Wrong-origin failures were not accepted.
Simulated cast failures remain distinct from real capacity/offset coverage.
- Rust core/API/deadline checks and default/Wasm compilation are carried
from the fresh baseline through equality of all **190 native build inputs**.
The native Python tests imported the recorded baseline extension; no fresh
Arrow Rust build is claimed. Baseline workspace Clippy remains a disclosed
nonzero failure. A 12-class name audit found zero missing reference names and
five extra native names; it does not certify signatures, behavior or full API
parity.
- The completed fresh official transport gate has **91 selected files,
2,601 passed and 655 runtime skips**, with zero unexpected selected-suite
failures/errors. Eleven all-skipped modules account for 267 skips. A separately
run, checked-in excluded catalog case fails because the distribution lacks
`InMemoryCatalog`. A final 48-input transport/build/harness equality audit
supports carrying this result to Arrow. **Arrow's binary did not execute this
official suite**; it used the Decima [...]
- The initial final-revision run completed all six measurement workers and
contracts, then its large-upload live check exhausted the **1 GiB JVM heap**;
cleanup also raised `PermissionError`. The incomplete raw summary and
failed-execution receipt are preserved; that run is not accepted as passed. A
complete repeat uses an independently verified **3 GiB JVM heap**, with
unchanged sources, worker, driver and plan.
- The complete **3 GiB rerun passed**: all six timing workers,
output-contract checks and **eight native live tests per candidate** (including
the >128 MiB upload), with no live skips or failures. All six
inferred/names-only Arrow-string contract cases passed on the baseline,
candidate and official PySpark, with full values and schemas checked. All final
source, import, binary and reference-input guards passed; the owned JVM was
reaped.
- Published-head CI remains pending publication and exact-head verification.
Measurements use macOS, Spark Connect 4.3.0, pandas 2.3.3, PyArrow 26.0.0
and one shared owned local server. Decimal workloads have 200,000 rows and two
fields; string workloads have three fields. Each worker is a separate process.
The following values are client `createDataFrame` CPU / wall milliseconds.
Complete output and schema validation ran outside the timer.
**Warm medians of five calls:**
| Workload | Frozen master CPU / wall | Candidate CPU / wall | Pure Python
CPU / wall |
| --- | ---: | ---: | ---: |
| arrow-strings-1000 | 0.844 / 1.355 | 0.888 / 1.377 | 0.768 / 0.784 |
| arrow-strings-200000 | 5.823 / 6.501 | 5.915 / 6.504 | 4.586 / 5.180 |
**First measured call per workload:** imports, session setup and input
preparation precede these samples; later workloads follow earlier ones in their
worker.
| Workload | Frozen master CPU / wall | Candidate CPU / wall | Pure Python
CPU / wall |
| --- | ---: | ---: | ---: |
| arrow-strings-1000 | 9.802 / 10.918 | 11.856 / 12.689 | 1.257 / 1.820 |
| arrow-strings-200000 | 10.882 / 37.037 | 10.735 / 42.028 | 13.557 /
39.020 |
Observed warm CPU cost increased **5.2% at 1,000 rows** and **1.6% at
200,000 rows** versus master; warm wall time changed by +1.6% and approximately
0.0%. The candidate remains slower than Python on these two workloads; the
already-landed fast path is part of the baseline. Whole-worker cumulative peak
RSS was 163.1 MiB for master, 165.4 MiB for the candidate and 174.9 MiB for
Python.
This is one fixed-order paired run without confidence intervals. RSS
includes input preparation, expected tables, collection, validation and
allocator retention. Guard-only cost, fallback/wide/nested workload
performance, RPC/upload counts, conversion-only memory, process cold starts and
whole-query performance are outside these measurements.
### Was this patch authored or co-authored using generative AI tooling?
OpenAI Codex.
This pull request and its description were written by OpenAI Codex.
Closes #366 from dbtsai/codex/followup-arrow-string-hooks-current.
Lead-authored-by: DB Tsai <[email protected]>
Co-authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
dev/design/architecture.md | 38 ++
docs/architecture.md | 6 +-
python/pyspark/sql/_pandas_input.py | 348 +++++++++---
python/tests/test_arrow_handoff.py | 1021 ++++++++++++++++++++++++++++++++++-
4 files changed, 1317 insertions(+), 96 deletions(-)
diff --git a/dev/design/architecture.md b/dev/design/architecture.md
index cd625c30..f8931f8e 100644
--- a/dev/design/architecture.md
+++ b/dev/design/architecture.md
@@ -101,6 +101,44 @@ against its live server and built native extension.
Official parity tests
exercise the Rust transport with official PySpark; the separate native smoke
step covers the drop-in `toArrow()` path.
+The pandas Arrow-string input guard recognizes pandas 2.2 through 3.x storage
+layouts. The Python Connect and pandas parity workflows pin `pandas>=2.2,<3.0`;
+coverage and pipelines parity install unpinned pandas. CI has no explicit
+pandas-version matrix. The inherited `pandas>=1.0.0` package declaration is
+below the runtime utilities' `>=2.2` requirement. Aligning the package metadata
+is a separate packaging follow-up; neither that declaration nor the
Arrow-string
+checks establish support for older pandas versions.
+
+Arrow-string input eligibility checks both array instances and canonical
+classes' row/Arrow conversion methods and label casting, including inherited
+methods, on the specialized, generic explicit-schema and inferred-schema
routes.
+Other pandas array classes on the generic routes retain the existing
+module-origin policy, which does not authenticate mutations of their canonical
+classes' conversion hooks. The generic routes identify Arrow-string classes
+through their loaded defining module in `sys.modules`. Removing or replacing
+`pandas.core.arrays.string_arrow` there while retaining instances from the old
+module is outside these checks.
+
+Authentication occurs when the concrete array is inspected. With NumPy-backed
+column labels, a specialized guard may first read `frame.dtypes`, matching the
+native row path's dtype read before cell normalization. A customized dtype
+property can therefore raise before the later array-class check, with the same
+local error and no configuration or upload request. The bounded Arrow-string
+class checks for data and labels inspect raw MRO dictionaries for conversion
and
+label-iteration methods, dtype/missing-value properties and the null-subarray
+constructor (including its immediate super-constructor), optional factory and
+allocator used by supported pandas layouts. The label checks authenticate the
+class before inspecting its other methods, which also use the raw lookup. This
+avoids binding customized descriptors during those checks. The pandas 2.2
+NumPy-semantics accessor also dispatches through a mixin before ordinary MRO
+lookup; its mixin and partial-function aliases are checked, and its namespace
is
+limited to string-method names, initializer/storage declarations and class
+metadata. Its mixin initializer remains allowed because construction uses the
+class slots authenticated above. Class data read by the authenticated methods,
+such as `_storage` and `_na_value`, and module globals other than the
explicitly
+checked aliases are outside these checks. These checks do not establish a
+general invariant for arbitrary mutations of pandas internals.
+
## UDF strategy (full compatibility)
Python/pandas UDFs are **serialized, not executed** on the client:
diff --git a/docs/architecture.md b/docs/architecture.md
index cb8663de..0365402d 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -163,9 +163,9 @@ Pandas object columns containing exact `datetime.date`
values or `None` use the
Arrow importer at 10,000 rows or more with an explicit `StructType` of
`DateType`
fields. Unsupported values and customized inputs retain the row conversion
path.
Pandas frames of at least 10,000 rows whose columns are all kept in pandas'
Arrow
-string storage (`string[pyarrow]`, and the `str` dtype of pandas 3), with an
-explicit `StructType` of `StringType` fields, pass that storage to this
importer
-as Arrow string columns, without Python rows.
+string storage (`string[pyarrow]` in pandas 2.2 and later, and the `str` dtype
of
+pandas 3), with an explicit `StructType` of `StringType` fields, pass that
storage
+to this importer as Arrow string columns, without Python rows.
Inputs at or above `spark.sql.session.localRelationCacheThreshold` are
serialized
into independent IPC streams and uploaded as session-scoped cache artifacts.
The plan carries `ChunkedCachedLocalRelation` hashes instead of embedding the
diff --git a/python/pyspark/sql/_pandas_input.py
b/python/pyspark/sql/_pandas_input.py
index fa92a3f9..7329ea4c 100644
--- a/python/pyspark/sql/_pandas_input.py
+++ b/python/pyspark/sql/_pandas_input.py
@@ -19,10 +19,135 @@
"""Conservative columnar pandas input; unsupported conversions use the row
path."""
+_ARROW_STRING_HOOKS = ("__arrow_array__", "__array__", "__getitem__", "isna",
+ "to_numpy", "__len__")
+_ARROW_STRING_CONSTRUCTION_HOOKS = ("__init__", "_from_pyarrow_array",
"__new__")
+_ARROW_STRING_MIXIN_ALLOWED_NAMES = (
+ "__module__", "__qualname__", "__doc__", "__dict__", "__weakref__",
+ "__firstlineno__", "__static_attributes__", "_pa_array", "__init__",
+)
+
+
+def _defined_in(method, name, *modules):
+ """Whether `method` is a plain function named `name` whose module name and
+ globals belong to one of `modules`.
+ """
+ from types import FunctionType
+
+ return (type(method) is FunctionType and method.__name__ == name
+ and any(method.__module__ == module.__name__
+ and method.__globals__ is vars(module) for module in
modules))
+
+
+def _raw_class_attribute(cls, name, start=0):
+ """Return the defining class and raw attribute, or (None, None) if
absent."""
+ if cls is not None:
+ for base in cls.__mro__[start:]:
+ namespace = vars(base)
+ if name in namespace:
+ return base, namespace[name]
+ return None, None
+
+
+def _has_canonical_arrow_string_super_method(cls, start, name, owner, module):
+ """Authenticate the immediate method reached through ArrowStringArray's
super()."""
+ actual_owner, method = _raw_class_attribute(cls, name, start)
+ return actual_owner is owner and _defined_in(method, name, module)
+
+
+def _arrow_string_class_is_canonical(cls):
+ """Check the identity and supported hooks of a concrete pandas
Arrow-string class.
+
+ Authenticate conversion, construction and allocation, dtype/missing-value
+ properties, and NumPy-semantics accessor dispatch using raw MRO lookups.
+ """
+ import pandas as pd
+ from pandas.core.arrays import base as array_base_module, string_ as
string_module, string_arrow
+ from pandas.core.arrays.arrow import array as arrow_module
+
+ numpy_semantics = getattr(string_arrow, "ArrowStringArrayNumpySemantics",
None)
+ # Reject subclasses by identity before inspecting them. In particular, do
not
+ # hash an untrusted class whose metaclass could customize __hash__.
+ if (not any(cls is candidate for candidate in
+ (string_arrow.ArrowStringArray, numpy_semantics))
+ or pd.arrays.ArrowStringArray is not string_arrow.ArrowStringArray
+ or pd.StringDtype is not string_module.StringDtype):
+ return False
+ try:
+ super_start = cls.__mro__.index(string_arrow.ArrowStringArray) + 1
+ except ValueError:
+ # A rebound optional NumPy-semantics class alias may lack that
ancestry.
+ return False
+
+ # pandas 2.2's NumPy-semantics variant has its own string-method accessor.
+ access = _raw_class_attribute(cls, "__getattribute__")[1]
+ if access is not object.__getattribute__:
+ if cls is not numpy_semantics or not _defined_in(
+ access, "__getattribute__", string_arrow):
+ return False
+ from functools import partial
+ from pandas.core.arrays import _arrow_string_mixins as string_mixins
+
+ mixin = string_mixins.ArrowStringArrayMixin
+ # This accessor dispatches through the mixin before the ordinary MRO.
+ # Check the mixin/partial aliases and allow only string-method names,
class
+ # metadata, and initializer/storage declarations in the mixin
namespace.
+ # Construction itself uses class slots, checked separately below;
+ # the mixin's canonical __init__ is not an instance conversion hook.
+ if (not any(base is mixin for base in cls.__mro__)
+ or vars(string_arrow).get("ArrowStringArrayMixin") is not mixin
+ or vars(string_arrow).get("partial") is not partial
+ or any(name not in _ARROW_STRING_MIXIN_ALLOWED_NAMES
+ and not name.startswith("_str_") for name in
vars(mixin))):
+ return False
+ # Authenticate row/Arrow conversion hooks and __len__, which sizes the
table.
+ # to_numpy indexes the non-null portion through __getitem__ before reading
storage.
+ if (any(not _defined_in(_raw_class_attribute(cls, name)[1], name,
+ string_arrow, arrow_module) for name in
_ARROW_STRING_HOOKS)
+ or not _defined_in(_raw_class_attribute(cls, "astype")[1],
"astype", string_arrow)
+ or not _has_canonical_arrow_string_super_method(
+ cls, super_start, "astype", array_base_module.ExtensionArray,
array_base_module)):
+ return False
+ # Null-subarray conversion may construct another array. Older pandas
versions
+ # have no _from_pyarrow_array helper; a present custom descriptor must
decline.
+ if (not _defined_in(_raw_class_attribute(cls, "__init__")[1], "__init__",
string_arrow)
+ or not _has_canonical_arrow_string_super_method(
+ cls, super_start, "__init__",
arrow_module.ArrowExtensionArray, arrow_module)):
+ return False
+ factory_owner, factory = _raw_class_attribute(cls, "_from_pyarrow_array")
+ if factory_owner is not None and not _defined_in(factory,
"_from_pyarrow_array", string_arrow):
+ return False
+ if _raw_class_attribute(cls, "__new__")[1] is not object.__new__:
+ return False
+ dtype = _raw_class_attribute(cls, "dtype")[1]
+ hasna = _raw_class_attribute(cls, "_hasna")[1]
+ missing_value = _raw_class_attribute(string_module.StringDtype,
"na_value")[1]
+ return (type(dtype) is property
+ and _defined_in(dtype.fget, "dtype", string_arrow, arrow_module)
+ and type(hasna) is property
+ and _defined_in(hasna.fget, "_hasna", arrow_module)
+ and type(missing_value) is property
+ and _defined_in(missing_value.fget, "na_value", string_module))
+
+
+def _has_canonical_arrow_string_array(array):
+ """Authenticate a column at this point without caching mutable hooks.
+
+ Callers may already have read frame.dtypes, as the native row path does.
+ """
+ # Column lookup can run customized pandas accessors between these checks.
+ # An earlier column authenticates neither the next instance nor its class.
+ if not _arrow_string_class_is_canonical(type(array)):
+ return False
+ # Deliberately include descriptors and special names for future explicit
lookups.
+ return not any(name in vars(array) for name in
+ _ARROW_STRING_HOOKS + _ARROW_STRING_CONSTRUCTION_HOOKS
+ + ("astype", "dtype", "_hasna"))
+
+
def _has_canonical_columns(columns):
"""Decline label containers whose row-path iteration can run custom
hooks."""
import sys
- from types import FunctionType
import numpy as np
import pandas as pd
@@ -35,19 +160,14 @@ def _has_canonical_columns(columns):
or "__iter__" in vars(columns)):
return False
- def ordinary(method, name, module):
- return (type(method) is FunctionType and method.__name__ == name
- and method.__module__ == module.__name__
- and method.__globals__ is vars(module))
-
# The native row path iterates columns before normalizing any cells.
# Decline overrides without invoking them, so fallback preserves that
order.
if (Index.__iter__ is not base_module.IndexOpsMixin.__iter__
- or not ordinary(Index.__iter__, "__iter__", base_module)):
+ or not _defined_in(Index.__iter__, "__iter__", base_module)):
return False
values = vars(Index).get("_values")
if (type(values) is not property
- or not ordinary(values.fget, "_values", index_module)):
+ or not _defined_in(values.fget, "_values", index_module)):
return False
if any("_data" in vars(base) for base in Index.__mro__):
return False
@@ -78,13 +198,26 @@ def _has_canonical_columns(columns):
cls = type(backing)
if not any(cls is candidate for candidate in python_strings +
arrow_strings):
return False
- if (cls.__getattribute__ is not object.__getattribute__
+ arrow_backed = any(cls is candidate for candidate in arrow_strings)
+ if arrow_backed and not _arrow_string_class_is_canonical(cls):
+ return False
+
+ def class_attribute(owner, name):
+ # Arrow labels share the raw lookup; Python-string class lookup stays
unchanged.
+ if arrow_backed:
+ return _raw_class_attribute(owner, name)[1]
+ return getattr(owner, name, None)
+
+ # The shared instance-shadow list conservatively includes descriptors and
special
+ # names for both Arrow-backed and Python-backed string labels.
+ if (class_attribute(cls, "__getattribute__") is not object.__getattribute__
or any(name in vars(backing) for name in
- ("__iter__", "__getitem__", "__len__", "_box_func",
- "tolist", "to_numpy", "isna", "fillna", "_hasna"))
- or getattr(cls, "tolist", None) is not getattr(
- vars(string_module).get("BaseStringArray"), "tolist", None)
- or not ordinary(getattr(cls, "tolist", None), "tolist",
string_module)):
+ ("__iter__", "__getitem__", "__len__", "__array__",
"_box_func",
+ "tolist", "to_numpy", "isna", "fillna", "_hasna")
+ + _ARROW_STRING_CONSTRUCTION_HOOKS)
+ or class_attribute(cls, "tolist") is not class_attribute(
+ vars(string_module).get("BaseStringArray"), "tolist")
+ or not _defined_in(class_attribute(cls, "tolist"), "tolist",
string_module)):
return False
if any(cls is candidate for candidate in python_strings):
owner, module = vars(numpy_module).get("NumpyExtensionArray"),
numpy_module
@@ -100,7 +233,7 @@ def _has_canonical_columns(columns):
# ExtensionArray iteration dispatches through len and indexed boxing.
for name, hook_module in (("__iter__", array_base_module),
("__getitem__", mixins_module),
("_box_func", mixins_module)):
- if not ordinary(getattr(cls, name, None), name, hook_module):
+ if not _defined_in(getattr(cls, name, None), name, hook_module):
return False
if (getattr(cls, "__len__", None) is not NDArrayBacked.__len__
or getattr(cls, "ndim", None) is not NDArrayBacked.ndim
@@ -113,26 +246,31 @@ def _has_canonical_columns(columns):
owner, module = vars(arrow_module).get("ArrowExtensionArray"),
arrow_module
if owner is None:
return False
+ # pandas builds an Arrow string Index's uniqueness engine with
astype(object).
+ # Authenticate the recognized object-cast methods and
dtype/missing-value
+ # properties before is_unique. This is a bounded check of pandas
internals.
+ if "astype" in vars(backing):
+ return False
try:
storage = object.__getattribute__(backing, "_pa_array")
except AttributeError:
return False
if (type(storage) is not pa.ChunkedArray
- or not ordinary(getattr(cls, "__iter__", None), "__iter__",
module)):
+ or not _defined_in(class_attribute(cls, "__iter__"),
"__iter__", module)):
return False
- isna, isna_module = getattr(owner, "isna", None), module
- hasna = getattr(cls, "_hasna", None)
+ isna, isna_module = class_attribute(owner, "isna"), module
+ hasna = class_attribute(cls, "_hasna")
if (type(hasna) is not property
- or not ordinary(hasna.fget, "_hasna", module)
- or getattr(cls, "fillna", None) is not getattr(owner,
"fillna", None)
- or not ordinary(getattr(cls, "fillna", None), "fillna",
module)):
+ or not _defined_in(hasna.fget, "_hasna", module)
+ or class_attribute(cls, "fillna") is not
class_attribute(owner, "fillna")
+ or not _defined_in(class_attribute(cls, "fillna"), "fillna",
module)):
return False
# Keep the existing storage/conversion authentication as well as the
# iterator hooks: these objects also supply labels to pandas normalization.
- return (getattr(cls, "to_numpy", None) is getattr(owner, "to_numpy", None)
- and ordinary(getattr(cls, "to_numpy", None), "to_numpy", module)
- and getattr(cls, "isna", None) is isna
- and ordinary(getattr(cls, "isna", None), "isna", isna_module))
+ return (class_attribute(cls, "to_numpy") is class_attribute(owner,
"to_numpy")
+ and _defined_in(class_attribute(cls, "to_numpy"), "to_numpy",
module)
+ and class_attribute(cls, "isna") is isna
+ and _defined_in(class_attribute(cls, "isna"), "isna", isna_module))
def prepare_int64_table(frame, schema):
@@ -577,7 +715,7 @@ def prepare_string_input(frame, schema):
from pandas.core import frame as frame_module, generic as generic_module
from pandas.core.frame import DataFrame
from pandas.core.series import Series
- from types import FunctionType, GetSetDescriptorType,
MethodDescriptorType, WrapperDescriptorType
+ from types import GetSetDescriptorType, MethodDescriptorType,
WrapperDescriptorType
from pyspark._pyspark import StringType, StructField, StructType as
NativeStructType
from pyspark.sql import types as sql_types
@@ -592,8 +730,7 @@ def prepare_string_input(frame, schema):
for name, module in (("astype", generic_module), ("notna", frame_module),
("where", generic_module)):
method = getattr(DataFrame, name)
- if (type(method) is not FunctionType or method.__name__ != name
- or method.__module__ != module.__name__ or method.__globals__
is not vars(module)):
+ if not _defined_in(method, name, module):
return None
if (DataFrame.astype is not generic_module.NDFrame.astype
or DataFrame.where is not generic_module.NDFrame.where):
@@ -671,29 +808,48 @@ def _arrow_string_table(frame, schema, dtypes):
For the row path, pandas converts such a column from that storage, with
its missing
values as None, so the storage holds the same values. Only pandas' own
array classes are
- trusted to keep them there: a subclass could convert differently. The
caller's guards of
- prepare_string_input (frame, schema, names, size) have passed.
+ trusted with their original conversion hooks: customization could convert
differently.
+ The caller's guards of prepare_string_input (frame, schema, names, size)
have passed.
"""
- import pandas as pd
- import pyarrow as pa
- from pandas.core.arrays import string_arrow
+ from pandas.core.arrays import string_ as string_module
- classes = {pd.arrays.ArrowStringArray,
- getattr(string_arrow, "ArrowStringArrayNumpySemantics", None)}
- {None}
- if not all(type(dtype) is pd.StringDtype and dtype.storage in ("pyarrow",
"pyarrow_numpy")
+ if not all(type(dtype) is string_module.StringDtype
+ and dtype.storage in ("pyarrow", "pyarrow_numpy")
for dtype in dtypes):
return None
+ # The importer uses mutable public/captured schema names. Decline a rebound
+ # namespace before conversion or configuration/upload RPCs, as the generic
route does.
+ if not _converter_namespaces_intact():
+ return None
+
+ import pyarrow as pa
+ from pandas.core.series import Series
+
arrays = []
- for (_, column), field in zip(frame.items(), schema.fields):
+ for index, field in enumerate(schema.fields):
+ column = frame.iloc[:, index]
+ if type(column) is not Series:
+ return None
array = column.array
- if type(array) not in classes:
+ if not _has_canonical_arrow_string_array(array):
+ return None
+ values = array.__arrow_array__()
+ # pandas returns a ChunkedArray; normalize a plain Array defensively if
+ # an instance's _pa_array storage was replaced.
+ if issubclass(type(values), pa.Array):
+ values = pa.chunked_array([values])
+ elif type(values) is not pa.ChunkedArray:
+ return None
+ if len(values) != len(frame):
return None
- values = pa.chunked_array(array.__arrow_array__())
if values.type not in (pa.string(), pa.large_string()):
return None
try:
# Malformed offsets or UTF-8 keep the row path and its error.
values.validate(full=True)
+ # Check normal-string capacity before configuration/upload RPCs.
The importer
+ # may widen back to large_string when useLargeVarTypes is enabled.
+ # pandas 2.2+ stores large_string, so this narrowing copies each
column's offsets.
values = values.cast(pa.string())
except (pa.ArrowException, OverflowError):
return None
@@ -868,6 +1024,45 @@ def _native_names_intact(namespace):
)
+def _converter_namespaces_intact():
+ """Whether loaded schema converters/importers still resolve native type
names."""
+ import sys
+
+ from pyspark.sql import types as sql_types
+
+ if sys.modules.get("pyspark.sql.types") is not sql_types or not
_native_names_intact(
+ vars(sql_types)
+ ):
+ return False
+ for name in ("pyspark.sql.pandas.types", "pyspark.sql._arrow_input"):
+ module = sys.modules.get(name)
+ if module is not None and not _native_names_intact(vars(module)):
+ return False
+ # The importer resolves its schema classes by name, and its conversion
functions may
+ # keep the globals of an older type module.
+ importer = sys.modules.get("pyspark.sql._arrow_input")
+ if importer is not None:
+ from pyspark._pyspark import AtomicType, StructType
+
+ if any(vars(importer).get(name) is not cls
+ for name, cls in (("StructType", StructType), ("AtomicType",
AtomicType))):
+ return False
+ for function in list(vars(importer).values()):
+ namespace = getattr(function, "__globals__", None)
+ if type(namespace) is dict and not _native_names_intact(namespace):
+ return False
+ # The importer calls its converters by these names, which must still
be functions.
+ converter = sys.modules.get("pyspark.sql.pandas.types")
+ for name in ("to_arrow_schema", "from_arrow_schema"):
+ if name in vars(importer) and (
+ getattr(vars(importer)[name], "__globals__", None) is None
+ or (converter is not None
+ and vars(importer)[name] is not getattr(converter, name,
None))
+ ):
+ return False
+ return True
+
+
def _canonical_struct(schema):
"""Whether a native struct schema and its fields keep their own classes'
behavior.
@@ -904,39 +1099,57 @@ def _canonical_struct(schema):
def accepts_pandas_table(frame, schema):
"""Whether `prepare_pandas_table` may convert this frame under this schema.
- Customized frames, columns, schema classes and rebound type names keep the
row path,
- which reads them in its own order. The conversion and the importer resolve
the type
- names by lookup, so each namespace must still hold the native classes. The
schema is
- the caller's object, or a DDL string, which is parsed.
+ Customized frames, column labels, schema classes and rebound type names
keep the row
+ path, which reads them in its own order. Conversion-hook authentication
covers Arrow
+ string storage, including inherited hooks. Other pandas arrays retain the
existing
+ module-origin checks; rebinding their canonical classes' hooks is not
covered here.
+ The conversion and importer resolve type names by lookup, so each
namespace must hold
+ the native classes. The schema is the caller's object or a DDL string,
which is parsed.
"""
return accepts_pandas_frame(frame) and (type(schema) is str or
_canonical_struct(schema))
def accepts_pandas_frame(frame):
- """Whether PySpark's Arrow conversion may convert this frame, as
`accepts_pandas_table`
- checks it, but for a schema."""
+ """Check frame eligibility for explicit or inferred Arrow schema
conversion."""
import sys
import pandas as pd
from pandas.core.frame import DataFrame
- from pyspark.sql import types as sql_types
-
if pd.DataFrame is not DataFrame or type(frame) is not DataFrame:
return False
if not _has_canonical_columns(frame.columns):
return False
- # Custom extension arrays and dtypes keep the row path: Arrow would call
their own
- # `__arrow_array__`, and their dtype may impersonate a NumPy type.
+ # Custom extension-array/dtype classes outside pandas and NumPy keep the
row path.
+ # Canonical non-string array classes retain the inherited module-origin
policy.
+ # Arrow string storage already loads its defining module; do not import
optional
+ # Arrow dependencies merely to inspect unrelated pandas columns.
+ # The generic routes assume this sys.modules entry still defines live
arrays;
+ # removing or replacing that module while keeping old instances is not
covered.
+ string_arrow = sys.modules.get("pandas.core.arrays.string_arrow")
+ arrow_strings = () if string_arrow is None else tuple(
+ vars(string_arrow).get(name)
+ for name in ("ArrowStringArray", "ArrowStringArrayNumpySemantics")
+ )
+ # Direct type checks bypass __subclasscheck__ and __mro__ metaclass
callbacks.
+ # Use the real type: isinstance can also consult a custom __class__
property.
+ # Non-class aliases cannot own array storage.
+ arrow_strings = tuple(cls for cls in arrow_strings
+ if type.__subclasscheck__(type, type(cls)))
for i in range(len(frame.columns)):
column = frame.iloc[:, i]
+ array = column.array
+ # A canonical class can still have customized hooks. Generic explicit
and inferred
+ # schema routes must preserve the specialized routes' row fallback.
+ if (any(type.__subclasscheck__(cls, type(array)) for cls in
arrow_strings)
+ and not _has_canonical_arrow_string_array(array)):
+ return False
if not all(type(value).__module__.split(".")[0] in ("pandas", "numpy")
- for value in (column.array, column.dtype)):
+ for value in (array, column.dtype)):
return False
# The native row path normalizes cells through these methods, which a
customized class
# or instance keeps on the row path.
from pandas.core import frame as frame_module, generic as generic_module
- from types import FunctionType
if any(name in vars(frame) for name in ("astype", "notna", "where")):
return False
@@ -946,39 +1159,10 @@ def accepts_pandas_frame(frame):
("where", generic_module, "NDFrame.where"),
):
method = getattr(DataFrame, name)
- if (type(method) is not FunctionType or method.__qualname__ != qualname
- or method.__module__ != module.__name__ or method.__globals__
is not vars(module)):
+ if not _defined_in(method, name, module) or method.__qualname__ !=
qualname:
return False
- if sys.modules.get("pyspark.sql.types") is not sql_types or not
_native_names_intact(
- vars(sql_types)
- ):
+ if not _converter_namespaces_intact():
return False
- for name in ("pyspark.sql.pandas.types", "pyspark.sql._arrow_input"):
- module = sys.modules.get(name)
- if module is not None and not _native_names_intact(vars(module)):
- return False
- # The importer resolves its schema classes by name, and its conversion
functions may
- # keep the globals of an older type module.
- importer = sys.modules.get("pyspark.sql._arrow_input")
- if importer is not None:
- from pyspark._pyspark import AtomicType, StructType
-
- if any(vars(importer).get(name) is not cls
- for name, cls in (("StructType", StructType), ("AtomicType",
AtomicType))):
- return False
- for function in list(vars(importer).values()):
- namespace = getattr(function, "__globals__", None)
- if type(namespace) is dict and not _native_names_intact(namespace):
- return False
- # The importer calls its converters by these names, which must still
be functions.
- converter = sys.modules.get("pyspark.sql.pandas.types")
- for name in ("to_arrow_schema", "from_arrow_schema"):
- if name in vars(importer) and (
- getattr(vars(importer)[name], "__globals__", None) is None
- or (converter is not None
- and vars(importer)[name] is not getattr(converter, name,
None))
- ):
- return False
return True
diff --git a/python/tests/test_arrow_handoff.py
b/python/tests/test_arrow_handoff.py
index cfe35c1f..d82c6d55 100644
--- a/python/tests/test_arrow_handoff.py
+++ b/python/tests/test_arrow_handoff.py
@@ -4193,7 +4193,7 @@ def test_string_columns_preserve_every_ipc_cell(protocol,
monkeypatch, cached, v
"nonnullable_null", "frame_subclass", "string_subclass", "string_dtype",
"arrow_string_dtype", "late_unsupported", "surrogate", "late_surrogate",
"collation",
])
-def test_string_column_guard_preserves_row_fallback(reason):
+def test_string_column_guard_handles_supported_and_unsupported_inputs(reason):
from pyspark.sql._pandas_input import prepare_string_input
class Frame(pd.DataFrame):
@@ -4269,25 +4269,37 @@ def
test_string_column_guard_preserves_row_fallback(reason):
assert result is None
-
# Columns kept in pandas' Arrow string storage reach the importer as Arrow,
without Python rows.
def _arrow_string_column(texts, storage):
- import pyarrow as pa
if storage == "numpy_semantics":
try:
dtype = pd.StringDtype("pyarrow", na_value=float("nan"))
except TypeError: # pandas 2.2
dtype = pd.StringDtype("pyarrow_numpy")
return pd.array(texts, dtype=dtype)
- if storage == "large_string":
+ if storage == "chunked":
return pd.arrays.ArrowStringArray(pa.chunked_array(
[pa.array(texts[:7], pa.large_string()), pa.array(texts[7:],
pa.large_string())]))
+ # The sliced case uses this storage, then slices the constructed frame
below.
return pd.array(texts, dtype="string[pyarrow]")
+def _force_native_pandas_rows(monkeypatch):
+ import importlib
+
+ # Earlier import-hook tests can leave the package attribute pointing at a
+ # different module object from the one imported by the native binding.
+ helper = importlib.import_module("pyspark.sql._pandas_input")
+ for name in ("prepare_int64_table", "timestamp_columns",
"prepare_float64_input",
+ "prepare_string_input", "prepare_date_table"):
+ monkeypatch.setattr(helper, name, lambda *_: None)
+ for name in ("accepts_pandas_table", "accepts_pandas_frame"):
+ monkeypatch.setattr(helper, name, lambda *_: False)
+
+
@pytest.mark.parametrize("cached", [False, True])
[email protected]("storage", ["pyarrow", "numpy_semantics",
"large_string", "sliced"])
[email protected]("storage", ["pyarrow", "numpy_semantics", "chunked",
"sliced"])
def test_arrow_string_columns_preserve_every_ipc_cell(protocol, monkeypatch,
cached, storage):
from pyspark.sql.pandas.types import from_arrow_metadata
@@ -4359,15 +4371,1002 @@ def
test_arrow_string_columns_keep_the_row_path(reason):
assert prepare_string_input(frame, T.StructType(fields)) is None
-def test_arrow_string_columns_match_the_row_path(protocol):
[email protected]("storage", ["pyarrow", "numpy_semantics"])
+def test_arrow_string_columns_match_the_row_path(protocol, monkeypatch,
storage):
+ import json
+
+ from pyspark.sql.pandas.types import from_arrow_metadata
+
texts = ["x" * (i % 5) if i % 4 else None for i in range(12000)]
- schema = T.StructType([T.StructField("v", T.StringType())])
+ frame = pd.DataFrame({"v": _arrow_string_column(texts, storage)})
+ schema = T.StructType([T.StructField("v", T.StringType(), True, {"source":
"row-control"})])
tables = []
- for dtype in ("string[pyarrow]", object):
- with _arrow_input_service(protocol) as (spark, state):
- spark.createDataFrame(pd.DataFrame({"v": pd.array(texts,
dtype=dtype)}), schema).toArrow()
- tables.append(state["input_tables"][0])
+ logical_schemas = []
+ for row_control in (False, True):
+ with monkeypatch.context() as control:
+ if row_control:
+ _force_native_pandas_rows(control)
+ with _arrow_input_service(protocol) as (spark, state):
+ spark.createDataFrame(frame, schema).toArrow()
+ tables.append(state["input_tables"][0])
+ logical_schemas.append(json.loads(state["spark_schema"]))
+ assert bool(state["config_requests"]) == (not row_control)
assert tables[0].equals(tables[1])
+ # Row IPC omits field metadata; Spark overlays the explicit logical schema.
+ # The Arrow importer also embeds metadata in IPC, so compare both
contracts.
+ assert from_arrow_metadata(tables[0].schema.field(0).metadata) ==
{"source": "row-control"}
+ assert tables[1].schema.field(0).metadata is None
+ assert logical_schemas == [json.loads(schema.json())] * 2
+
+
[email protected]("storage", ["pyarrow", "numpy_semantics"])
[email protected]("layout", ["small", "mixed"])
+def test_arrow_string_generic_explicit_schema_stays_columnar(
+ protocol, monkeypatch, storage, layout
+):
+ from pyspark.sql._pandas_input import accepts_pandas_table,
prepare_string_input
+
+ texts = ["", None, "中文", "a\x00b"] * (3 if layout == "small" else 3000)
+ frame = pd.DataFrame({"v": _arrow_string_column(texts, storage)})
+ fields = [T.StructField("renamed", T.StringType())]
+ expected = {"renamed": texts}
+ if layout == "mixed":
+ frame["number"] = np.arange(len(texts), dtype=np.int64)
+ fields.append(T.StructField("number", T.LongType()))
+ expected["number"] = list(range(len(texts)))
+ schema = T.StructType(fields)
+ # These ordinary inputs miss the specialized string route but remain
eligible
+ # for upstream's generic explicit-schema conversion.
+ assert prepare_string_input(frame, schema) is None
+ assert accepts_pandas_table(frame, schema)
+
+ def no_rows(*args, **kwargs):
+ raise AssertionError("generic Arrow strings must not materialize
pandas rows")
+
+ monkeypatch.setattr(pd.DataFrame, "values", property(no_rows))
+ monkeypatch.setattr(pd.Series, "tolist", no_rows)
+ with _arrow_input_service(protocol) as (spark, state):
+ spark.createDataFrame(frame, schema).toArrow()
+ assert state["config_requests"]
+ assert state["input_tables"][0].to_pydict() == expected
+ assert state["input_tables"][0].schema.field(0).type == pa.string()
+
+
[email protected]("scope", ["public", "converter", "captured"])
[email protected]("class_name", ["BooleanType", "StringType",
"StructType"])
+def test_arrow_string_rebound_converter_alias_keeps_native_rows(scope,
class_name):
+ import sys
+
+ script = r'''
+import importlib
+import importlib.util
+import json
+import sys
+import tempfile
+from pathlib import Path
+import pandas as pd
+import pytest
+from pyspark import _pyspark as native
+from pyspark.sql import types as T
+
+_pandas_input = importlib.import_module("pyspark.sql._pandas_input")
+spec = importlib.util.spec_from_file_location("arrow_string_alias_capture",
sys.argv[3])
+capture = importlib.util.module_from_spec(spec)
+sys.modules[spec.name] = capture
+spec.loader.exec_module(capture)
+texts = ["original", None] * 5000
+frame = pd.DataFrame({"v": pd.array(texts, dtype="string[pyarrow]")})
+schema = T.StructType([T.StructField("v", T.StringType(), True, {"source":
"native"})])
+scope, name = sys.argv[1:3]
+replacement = native.BooleanType if name == "StringType" else native.StringType
+if scope == "public":
+ sys.modules.pop("pyspark.sql.pandas.types", None)
+ setattr(T, name, replacement)
+else:
+ converter = importlib.import_module("pyspark.sql.pandas.types")
+ if scope == "captured":
+ importer = importlib.import_module("pyspark.sql._arrow_input")
+ setattr(converter, name, replacement)
+ if scope == "captured":
+ sys.modules.pop("pyspark.sql.pandas.types")
+ assert importlib.import_module("pyspark.sql.pandas.types") is not
converter
+ assert importer.to_arrow_schema.__globals__ is vars(converter)
+assert _pandas_input.prepare_string_input(frame, schema) is None
+assert not _pandas_input.accepts_pandas_table(frame, schema)
+class Factory:
+ def mktemp(self, name):
+ return Path(tempfile.mkdtemp(prefix=name))
+protocol = capture.protocol.__wrapped__(Factory())
+tables = []
+for row_control in (False, True):
+ with pytest.MonkeyPatch.context() as control:
+ if row_control:
+ capture._force_native_pandas_rows(control)
+ with capture._arrow_input_service(protocol) as (spark, state):
+ spark.createDataFrame(frame, schema).toArrow()
+ assert state["config_requests"] == []
+ assert state["uploads"] == 0
+ assert json.loads(state["spark_schema"]) ==
json.loads(schema.json())
+ tables.append(state["input_tables"][0])
+assert tables[0].column(0).to_pylist() == texts
+assert tables[0].equals(tables[1], check_metadata=True)
+'''
+ result = subprocess.run([sys.executable, "-c", script, scope, class_name,
__file__],
+ capture_output=True, text=True, timeout=60)
+ assert result.returncode == 0, result.stdout + result.stderr
+
+
[email protected]("storage", ["pyarrow", "numpy_semantics"])
[email protected]("schema_kind", ["explicit", "inferred", "names"])
[email protected]("instance_hook", [False, True])
+def test_arrow_string_custom_arrow_hook_keeps_native_row_values(
+ protocol, monkeypatch, instance_hook, schema_kind, storage
+):
+ import importlib
+
+ _pandas_input = importlib.import_module("pyspark.sql._pandas_input")
+
+ texts = ["original", None] * 5000
+ # The first column authenticates this class before the second instance's
+ # override is checked; every instance must still be inspected.
+ frame = pd.DataFrame({name: _arrow_string_column(texts, storage)
+ for name in ("first", "v")})
+ schema = T.StructType([T.StructField(name, T.StringType()) for name in
("first", "v")])
+ input_schema = {"explicit": schema, "inferred": None, "names": ["first",
"v"]}[schema_kind]
+ array = frame.iloc[:, 1].array
+ calls = []
+
+ def changed_arrow(*args, **kwargs):
+ calls.append("arrow")
+ return pa.array(["changed"] * len(texts), type=pa.string())
+
+ monkeypatch.setattr(array if instance_hook else type(array),
"__arrow_array__", changed_arrow)
+ assert _pandas_input.prepare_string_input(frame, schema) is None
+ assert not _pandas_input.accepts_pandas_table(frame, schema)
+ assert not _pandas_input.accepts_pandas_frame(frame)
+ assert calls == []
+ tables = []
+ for row_control in (False, True):
+ with monkeypatch.context() as control:
+ if row_control:
+ _force_native_pandas_rows(control)
+ with _arrow_input_service(protocol, config_error=True) as (spark,
state):
+ spark.createDataFrame(frame, input_schema).toArrow()
+ tables.append(state["input_tables"][0])
+ assert state["config_requests"] == []
+ assert state["uploads"] == 0
+ assert calls == []
+ assert tables[0].to_pydict() == {"first": texts, "v": texts}
+ assert tables[0].equals(tables[1], check_metadata=True)
+
+
[email protected]("schema_kind", ["explicit", "inferred", "names"])
[email protected]("customization", [
+ "mixin_hook", "mixin_alias", "partial_alias", "mixin_storage",
"mixin_dtype",
+])
+def test_arrow_string_numpy_semantics_accessor_keeps_native_outcomes(
+ protocol, monkeypatch, customization, schema_kind
+):
+ import importlib
+ import json
+ import traceback
+ from pandas.core.arrays import _arrow_string_mixins as string_mixins,
string_arrow
+ from pyspark.sql.pandas.types import from_arrow_schema
+
+ helper = importlib.import_module("pyspark.sql._pandas_input")
+ texts = ["original", None] * 5000
+ frames = [pd.DataFrame({"v": _arrow_string_column(texts,
"numpy_semantics")})
+ for _ in range(2)]
+ for frame in frames:
+ frame.columns = pd.Index(["v"], dtype=object)
+ schema = T.StructType([T.StructField("v", T.StringType())])
+ input_schema = {"explicit": schema, "inferred": None, "names":
["v"]}[schema_kind]
+ array_class = type(frames[0].iloc[:, 0].array)
+ dynamic_accessor = (helper._raw_class_attribute(array_class,
"__getattribute__")[1]
+ is not object.__getattribute__)
+ # These warm-ups populate Block.dtype so mixin_dtype reaches class
authentication
+ # instead of raising during the earlier native-compatible frame.dtypes
read.
+ for frame in frames:
+ assert helper.prepare_string_input(frame, schema) is not None
+ assert helper.accepts_pandas_table(frame, schema)
+ assert helper.accepts_pandas_frame(frame)
+ calls = []
+
+ def changed_arrow(*args, **kwargs):
+ calls.append("arrow")
+ return pa.array(["changed"] * len(texts), type=pa.string())
+
+ def changed_partial(*args, **kwargs):
+ calls.append("partial")
+ raise AssertionError("eligibility must not invoke the rebound partial
alias")
+
+ def changed_attribute(*args, **kwargs):
+ calls.append(customization)
+ raise AssertionError("the mixin state callback must not be called")
+
+ if customization == "mixin_hook":
+ monkeypatch.setattr(string_mixins.ArrowStringArrayMixin,
"__arrow_array__",
+ changed_arrow, raising=False)
+ elif customization == "mixin_alias":
+ class ChangedMixin:
+ __arrow_array__ = changed_arrow
+
+ monkeypatch.setattr(string_arrow, "ArrowStringArrayMixin",
ChangedMixin, raising=False)
+ elif customization in ("mixin_storage", "mixin_dtype"):
+ name = {"mixin_storage": "_storage", "mixin_dtype":
"_dtype"}[customization]
+ monkeypatch.setattr(string_mixins.ArrowStringArrayMixin, name,
+ changed_attribute, raising=False)
+ else:
+ monkeypatch.setattr(string_arrow, "partial", changed_partial,
raising=False)
+ for frame in frames:
+ # pandas 2.2 dispatches instance lookups through these mutable aliases.
+ # Later layouts use object access and remain unaffected by these
changes.
+ assert (helper.prepare_string_input(frame, schema) is None) ==
dynamic_accessor
+ assert helper.accepts_pandas_table(frame, schema) == (not
dynamic_accessor)
+ assert helper.accepts_pandas_frame(frame) == (not dynamic_accessor)
+ assert calls == []
+ logical_tables = []
+ outcomes = []
+ for row_control, frame in zip((True, False), frames):
+ with monkeypatch.context() as control:
+ if row_control:
+ _force_native_pandas_rows(control)
+ with _arrow_input_service(protocol, config_error=dynamic_accessor)
as (spark, state):
+ # The mock returns only a row count, not the uploaded data or
resolved schema.
+ try:
+ reply = spark.createDataFrame(frame,
input_schema).toArrow()
+ except Exception as error:
+ assert dynamic_accessor and customization in
("mixin_storage", "mixin_dtype")
+ assert not isinstance(error, AssertionError)
+ assert any(
+ trace_frame.f_globals.get("__name__",
"").startswith("pandas.core.arrays.")
+ for trace_frame, _ in
traceback.walk_tb(error.__traceback__)
+ )
+ assert state["input_tables"] == []
+ assert state["config_requests"] == []
+ assert state["uploads"] == 0
+ outcomes.append((type(error), str(error)))
+ continue
+ outcomes.append(None)
+ assert reply.to_pydict() == {"rows": [len(texts)]}
+ assert len(state["input_tables"]) == 1
+ uploaded = state["input_tables"][0]
+ assert uploaded.num_rows == len(texts)
+ assert uploaded.num_columns == 1
+ assert uploaded.column(0).to_pylist() == texts
+ assert uploaded.column(0).is_null().to_pylist() == [False,
True] * 5000
+ assert state["column_names"] == ["v"]
+ if row_control or dynamic_accessor or schema_kind ==
"explicit":
+ expected_wire = pa.schema([pa.field("v", pa.string())])
+ assert json.loads(state["spark_schema"]) ==
json.loads(schema.json())
+ else:
+ # Inherited inferred Arrow conversion sends _n fields
without a
+ # declared schema; Spark infers large_string as StringType
and toDF
+ # applies the actual captured column names above.
+ expected_wire = pa.schema([pa.field("_0",
pa.large_string())])
+ assert state["spark_schema"] is None
+ assert uploaded.schema.equals(expected_wire,
check_metadata=True)
+ if row_control or dynamic_accessor:
+ assert state["config_requests"] == []
+ assert state["uploads"] == 0
+ else:
+ assert state["config_requests"]
+ # Compare logical uploaded data only after checking the exact
wire
+ # schema and rename plan. This is not a real server schema
assertion.
+ renamed = uploaded.rename_columns(state["column_names"])
+ assert from_arrow_schema(renamed.schema) == schema
+ logical_tables.append(renamed.cast(pa.schema([pa.field("v",
pa.string())])))
+ assert calls == []
+ assert outcomes[0] == outcomes[1]
+ if dynamic_accessor and customization in ("mixin_storage", "mixin_dtype"):
+ assert outcomes[0] is not None
+ if logical_tables:
+ assert len(logical_tables) == 2
+ assert logical_tables[0].equals(logical_tables[1], check_metadata=True)
+
+
[email protected]("route", ["specialized", "generic", "inferred"])
+def test_arrow_string_guards_recheck_hooks_between_columns(monkeypatch, route):
+ import importlib
+
+ _pandas_input = importlib.import_module("pyspark.sql._pandas_input")
+ texts = ["original", None] * 5000
+ frame = pd.DataFrame({name: pd.array(texts, dtype="string[pyarrow]")
+ for name in ("first", "second")})
+ schema = T.StructType([T.StructField(name, T.StringType())
+ for name in ("first", "second")])
+ array_class = type(frame.iloc[:, 0].array)
+ original_iloc = pd.DataFrame.iloc
+ lookups = []
+ calls = []
+
+ def customized(*args, **kwargs):
+ calls.append("__getitem__")
+ raise AssertionError("eligibility must not run the customized row
hook")
+
+ class RebindingIndexer:
+ def __init__(self, indexer):
+ self.indexer = indexer
+
+ def __getitem__(self, key):
+ column = self.indexer[key]
+ lookups.append(key[1])
+ if key[1] == 1:
+ monkeypatch.setattr(array_class, "__getitem__", customized)
+ return column
+
+ def iloc(self):
+ indexer = original_iloc.fget(self)
+ return RebindingIndexer(indexer) if self is frame else indexer
+
+ # Both guards allow this accessor today. It returns ordinary Series while
+ # rebinding a hook after the first column, invalidating any per-class
cache.
+ monkeypatch.setattr(pd.DataFrame, "iloc", property(iloc))
+ if route == "specialized":
+ assert _pandas_input.prepare_string_input(frame, schema) is None
+ elif route == "generic":
+ assert not _pandas_input.accepts_pandas_table(frame, schema)
+ else:
+ assert not _pandas_input.accepts_pandas_frame(frame)
+ assert lookups == [0, 1]
+ assert calls == []
+
+
+def test_arrow_string_subclass_guard_does_not_hash_custom_metaclass():
+ from pyspark.sql._pandas_input import _arrow_string_class_is_canonical
+
+ calls = []
+
+ class CustomMeta(type(pd.arrays.ArrowStringArray)):
+ def __hash__(cls):
+ calls.append("class hash")
+ raise AssertionError("declining a subclass must not run its hash
hook")
+
+ class Custom(pd.arrays.ArrowStringArray, metaclass=CustomMeta):
+ pass
+
+ assert not _arrow_string_class_is_canonical(Custom)
+ assert calls == []
+
+
+def test_arrow_string_generic_subclass_guard_does_not_read_custom_mro():
+ from pyspark.sql._pandas_input import accepts_pandas_table
+
+ calls = []
+ armed = False
+
+ class CustomMeta(type(pd.arrays.ArrowStringArray)):
+ @property
+ def __mro__(cls):
+ if armed:
+ calls.append("class mro")
+ raise AssertionError("declining a subclass must not read its
custom MRO")
+ return type.__dict__["__mro__"].__get__(cls, type(cls))
+
+ class Custom(pd.arrays.ArrowStringArray, metaclass=CustomMeta):
+ pass
+
+ # Finish ordinary pandas construction before arming the metaclass callback.
+ frame = pd.DataFrame({"v": Custom(pa.chunked_array([pa.array(["a",
None])]))})
+ assert type(frame.iloc[:, 0].array) is Custom
+ schema = T.StructType([T.StructField("v", T.StringType())])
+ armed = True
+ assert not accepts_pandas_table(frame, schema)
+ assert calls == []
+
+
+def _arrow_string_allocator_process(case, schema_kind="explicit",
storage="pyarrow"):
+ """Keep CPython allocator-slot changes out of the main pytest
interpreter."""
+ import signal
+ import sys
+
+ script = r'''
+import importlib.util
+import sys
+import tempfile
+from pathlib import Path
+import pytest
+
+spec =
importlib.util.spec_from_file_location("arrow_string_allocator_capture",
sys.argv[4])
+capture = importlib.util.module_from_spec(spec)
+sys.modules[spec.name] = capture
+spec.loader.exec_module(capture)
+case, schema_kind, storage = sys.argv[1:4]
+with pytest.MonkeyPatch.context() as control:
+ if case == "row":
+ with tempfile.TemporaryDirectory(prefix="arrow-string-allocator-") as
directory:
+ class Factory:
+ def mktemp(self, name):
+ path = Path(directory) / name
+ path.mkdir()
+ return path
+ protocol = capture.protocol.__wrapped__(Factory())
+ capture._check_arrow_string_custom_row_hook(
+ protocol, control, "__new__", schema_kind, storage)
+ else:
+ assert case == "descriptor"
+ capture._check_arrow_string_class_descriptor(control, "__new__")
+'''
+ owned_group = os.name == "posix"
+ process = subprocess.Popen(
+ [sys.executable, "-c", script, case, schema_kind, storage,
str(Path(__file__).resolve())],
+ stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True,
+ start_new_session=owned_group,
+ )
+
+ def terminate_child():
+ try:
+ if owned_group:
+ # protoc inherits this session, so a timeout also cleans up
that child.
+ os.killpg(process.pid, signal.SIGKILL)
+ else:
+ process.kill()
+ except ProcessLookupError:
+ pass
+
+ try:
+ stdout, stderr = process.communicate(timeout=60)
+ except subprocess.TimeoutExpired:
+ terminate_child()
+ try:
+ stdout, stderr = process.communicate(timeout=5)
+ except subprocess.TimeoutExpired:
+ stdout, stderr = "", ""
+ pytest.fail("Arrow-string allocator child timed out\n" + stdout +
stderr)
+ except BaseException:
+ terminate_child()
+ try:
+ process.communicate(timeout=5)
+ except subprocess.TimeoutExpired:
+ pass
+ raise
+ assert process.returncode == 0, stdout + stderr
+ # Guard against accidentally reintroducing in-process allocator mutation.
+ healthy = pd.array(["original", None], dtype="string[pyarrow]")
+ assert healthy.__arrow_array__().to_pylist() == ["original", None]
+
+
[email protected]("storage", ["pyarrow", "numpy_semantics"])
[email protected]("schema_kind", ["explicit", "inferred", "names"])
[email protected]("method", [
+ "astype", "isna", "to_numpy", "__getitem__", "dtype", "na_value",
"_hasna", "__init__",
+ "_from_pyarrow_array", "__new__",
+])
+def test_arrow_string_custom_row_hooks_keep_native_errors(
+ protocol, monkeypatch, method, schema_kind, storage
+):
+ if method == "__new__":
+ # Other hooks use this shared fixture signature; the child needs its
own protocol.
+ _arrow_string_allocator_process("row", schema_kind, storage)
+ else:
+ _check_arrow_string_custom_row_hook(protocol, monkeypatch, method,
schema_kind, storage)
+
+
+def _check_arrow_string_custom_row_hook(
+ protocol, monkeypatch, method, schema_kind, storage
+):
+ import importlib
+
+ _pandas_input = importlib.import_module("pyspark.sql._pandas_input")
+
+ texts = ["original", None] * 5000
+ frames = []
+ for _ in range(2):
+ frame = pd.DataFrame({"v": _arrow_string_column(texts, storage)})
+ # Object labels isolate the additional construction/_hasna data hooks.
+ # The other Arrow labels exercise earlier checks when they share the
data
+ # class (pyarrow storage); a separate NumPy-semantics class is checked
later.
+ frame.columns = pd.Index(["v"], dtype=object) if method in (
+ "_hasna", "__init__", "_from_pyarrow_array", "__new__",
+ ) else pd.Index(
+ pd.array(["v"], dtype="string[pyarrow]"))
+ frames.append(frame)
+ schema = T.StructType([T.StructField("v", T.StringType())])
+ input_schema = {"explicit": schema, "inferred": None, "names":
["v"]}[schema_kind]
+ array_class = type(frames[0].iloc[:, 0].array)
+ calls = []
+ # Warm all guards before rebinding: class authentication must not be cached
+ # across calls, even when this exact class was previously accepted. This
also
+ # populates pandas' Block.dtype cache, so a customized data dtype reaches
the
+ # class check; the separate cold-dtype test covers its earlier native
error.
+ for frame in frames:
+ assert _pandas_input.prepare_string_input(frame, schema) is not None
+ assert _pandas_input.accepts_pandas_table(frame, schema)
+ assert _pandas_input.accepts_pandas_frame(frame)
+
+ class HookError(Exception):
+ pass
+
+ def customized(*args, **kwargs):
+ if method == "na_value":
+ # This property receives a StringDtype, including during native
label iteration.
+ assert type(args[0]) is pd.StringDtype
+ elif method == "__new__":
+ # pandas may pass only the class; no initialized receiver is
available.
+ assert args[0] is array_class
+ elif method in ("__init__", "_from_pyarrow_array"):
+ # __init__ receives an uninitialized object; inspect the subset
instead.
+ assert type(args[0]) is array_class
+ assert len(args) == 2
+ values = args[1]
+ assert type(values) is pa.ChunkedArray
+ assert values.null_count == 0
+ assert values.to_pylist() == ["original"] * 5000
+ assert set(kwargs) <= {"dtype"}
+ if kwargs:
+ assert type(kwargs["dtype"]) is pd.StringDtype
+ else:
+ assert len(args[0]) == 10000 # Labels have length one; guards
must not cast them.
+ calls.append(method)
+ raise HookError(f"{method} sentinel")
+
+ class FactoryHook:
+ def __get__(self, receiver, owner):
+ assert receiver is not None, "eligibility must not bind the
factory descriptor"
+ assert owner is array_class
+ return customized.__get__(receiver, owner)
+
+ if method == "na_value":
+ monkeypatch.setattr(pd.StringDtype, method, property(customized))
+ elif method == "_from_pyarrow_array":
+ # An absent helper on older pandas is also a conservative decline.
+ monkeypatch.setattr(array_class, method, FactoryHook(), raising=False)
+ elif method == "__new__":
+ monkeypatch.setattr(array_class, method, staticmethod(customized))
+ elif method in ("dtype", "_hasna"):
+ monkeypatch.setattr(array_class, method, property(customized))
+ else:
+ monkeypatch.setattr(array_class, method, customized)
+ for frame in frames:
+ assert _pandas_input.prepare_string_input(frame, schema) is None
+ assert not _pandas_input.accepts_pandas_table(frame, schema)
+ assert not _pandas_input.accepts_pandas_frame(frame)
+ assert calls == []
+
+ outcomes = []
+ tables = []
+ for row_control, frame in zip((True, False), frames):
+ calls.clear()
+ with monkeypatch.context() as control:
+ if row_control:
+ _force_native_pandas_rows(control)
+ with _arrow_input_service(protocol, config_error=True) as (spark,
state):
+ try:
+ result = spark.createDataFrame(frame, input_schema)
+ result.toArrow()
+ except HookError as error:
+ assert type(error) is HookError
+ assert str(error) == f"{method} sentinel"
+ if method in ("__init__", "_from_pyarrow_array",
"__new__"):
+ assert calls
+ else:
+ assert calls == [method]
+ outcome = (type(error), str(error))
+ assert state["input_tables"] == []
+ else:
+ # pandas versions use different subset-construction
helpers.
+ assert method in ("__init__", "_from_pyarrow_array",
"__new__")
+ assert len(state["input_tables"]) == 1
+ table = state["input_tables"][0]
+ assert table.column(0).to_pylist() == texts
+ tables.append(table)
+ outcome = None
+ outcomes.append((outcome, tuple(calls)))
+ assert state["config_requests"] == []
+ assert state["uploads"] == 0
+ assert outcomes[0] == outcomes[1]
+ if tables:
+ assert len(tables) == 2
+ assert tables[0].equals(tables[1], check_metadata=True)
+
+
[email protected]("method", [
+ "__arrow_array__", "__array__", "__getitem__", "isna", "to_numpy",
"__len__", "astype",
+ "__init__", "_from_pyarrow_array", "__new__", "__getattribute__", "dtype",
"_hasna", "na_value",
+])
+def test_arrow_string_class_descriptors_decline_without_binding(monkeypatch,
method):
+ if method == "__new__":
+ _arrow_string_allocator_process("descriptor")
+ else:
+ _check_arrow_string_class_descriptor(monkeypatch, method)
+
+
+def _check_arrow_string_class_descriptor(monkeypatch, method):
+ import importlib
+
+ _pandas_input = importlib.import_module("pyspark.sql._pandas_input")
+ array = pd.array(["original", None], dtype="string[pyarrow]")
+ array_class = type(array)
+ assert _pandas_input._arrow_string_class_is_canonical(array_class)
+ assert _pandas_input._has_canonical_arrow_string_array(array)
+ calls = []
+
+ class HookError(Exception):
+ pass
+
+ class CustomizedDescriptor:
+ def __get__(self, receiver, owner):
+ calls.append((receiver, owner))
+ raise HookError(f"{method} descriptor sentinel")
+
+ owner = pd.StringDtype if method == "na_value" else array_class
+ monkeypatch.setattr(owner, method, CustomizedDescriptor(), raising=False)
+ # These concrete-class checks inspect raw dictionaries. Earlier
frame.dtypes
+ # reads are covered separately because they intentionally keep native
ordering.
+ assert not _pandas_input._arrow_string_class_is_canonical(array_class)
+ assert not _pandas_input._has_canonical_arrow_string_array(array)
+ assert calls == []
+
+
[email protected]("method", [
+ "__getattribute__", "tolist", "__iter__", "fillna", "to_numpy", "isna",
"_hasna",
+ "dtype", "na_value",
+])
+def test_arrow_string_label_descriptors_decline_without_binding(monkeypatch,
method):
+ import importlib
+
+ _pandas_input = importlib.import_module("pyspark.sql._pandas_input")
+ # Concrete object data isolates the Arrow-string class used by the labels.
+ frame = pd.DataFrame({"v": pd.Series(["original", None] * 5000,
dtype=object)})
+ frame.columns = pd.Index(pd.array(["v"], dtype="string[pyarrow]"))
+ array_class = type(frame.columns._values)
+ schema = T.StructType([T.StructField("v", T.StringType())])
+ assert _pandas_input.prepare_string_input(frame, schema) is not None
+ assert _pandas_input.accepts_pandas_table(frame, schema)
+ assert _pandas_input.accepts_pandas_frame(frame)
+ calls = []
+
+ class HookError(Exception):
+ pass
+
+ class CustomizedDescriptor:
+ def __get__(self, receiver, owner):
+ calls.append((receiver, owner))
+ raise HookError(f"label {method} descriptor sentinel")
+
+ owner = pd.StringDtype if method == "na_value" else array_class
+ monkeypatch.setattr(owner, method, CustomizedDescriptor())
+ assert _pandas_input.prepare_string_input(frame, schema) is None
+ assert not _pandas_input.accepts_pandas_table(frame, schema)
+ assert not _pandas_input.accepts_pandas_frame(frame)
+ assert calls == []
+
+
[email protected]("storage_kind", [
+ "plain_array", "nonstring", "invalid_utf8", "bad_storage", "spoofed_array",
+ "short_storage", "long_storage",
+])
+def test_arrow_string_storage_normalization_and_fallback(monkeypatch,
storage_kind):
+ import importlib
+ import struct
+
+ _pandas_input = importlib.import_module("pyspark.sql._pandas_input")
+ texts = ["é", None]
+ frame = pd.DataFrame({"v": pd.array(texts, dtype="string[pyarrow]")})
+ schema = T.StructType([T.StructField("v", T.StringType())])
+ # Capture pandas metadata before intentionally replacing its Arrow storage.
+ dtypes = frame.dtypes
+ array = frame.iloc[:, 0].array
+ if storage_kind == "plain_array":
+ storage = pa.array(texts, type=pa.string())
+ assert isinstance(storage, pa.Array)
+ elif storage_kind == "nonstring":
+ storage = pa.chunked_array([pa.array([1, None], type=pa.int64())])
+ elif storage_kind in ("short_storage", "long_storage"):
+ count = 1 if storage_kind == "short_storage" else 3
+ storage = pa.chunked_array([pa.array(["é"] * count, type=pa.string())])
+ assert len(storage) != len(frame)
+ elif storage_kind == "invalid_utf8":
+ invalid = pa.Array.from_buffers(pa.string(), 2, [
+ None, pa.py_buffer(struct.pack("<iii", 0, 5, 6)),
pa.py_buffer(b"valid\xff"),
+ ])
+ storage = pa.chunked_array([invalid])
+ with pytest.raises(pa.ArrowInvalid):
+ storage.validate(full=True)
+ elif storage_kind == "bad_storage":
+ class UnsupportedStorage:
+ # pandas' column boxing reads length before the storage-type guard.
+ def __len__(self):
+ return len(texts)
+
+ storage = UnsupportedStorage()
+ assert not isinstance(storage, (pa.Array, pa.ChunkedArray))
+ else:
+ class SpoofedArray:
+ @property
+ def __class__(self):
+ return pa.Array
+
+ def __len__(self):
+ return len(texts)
+
+ storage = SpoofedArray()
+ assert isinstance(storage, pa.Array)
+ assert not issubclass(type(storage), pa.Array)
+ monkeypatch.setattr(array, "_pa_array", storage)
+ assert object.__getattribute__(frame.iloc[:, 0].array, "_pa_array") is
storage
+ result = _pandas_input._arrow_string_table(frame, schema, dtypes)
+ if storage_kind == "plain_array":
+ assert type(result) is pa.Table
+ result.validate(full=True)
+ assert result.schema.names == ["v"]
+ assert result.column(0).type == pa.string()
+ assert result.column(0).to_pylist() == texts
+ else:
+ assert result is None
+
+
[email protected]("error_kind", ["arrow", "overflow"])
+def test_arrow_string_simulated_cast_failures_decline(monkeypatch, error_kind):
+ """Exercise exception handling without allocating a greater-than-2-GiB
column."""
+ import importlib
+ import sys
+ from types import ModuleType
+
+ _pandas_input = importlib.import_module("pyspark.sql._pandas_input")
+ frame = pd.DataFrame({"v": pd.array(["original", None],
dtype="string[pyarrow]")})
+ schema = T.StructType([T.StructField("v", T.StringType())])
+ dtypes = frame.dtypes
+ array = frame.iloc[:, 0].array
+ values = array.__arrow_array__()
+ error_class = pa.ArrowInvalid if error_kind == "arrow" else OverflowError
+ calls = []
+
+ class SimulatedCastFailure:
+ type = values.type
+
+ def __len__(self):
+ return len(values)
+
+ def validate(self, *, full):
+ calls.append(("validate", full))
+ values.validate(full=full)
+
+ def cast(self, target):
+ assert target == pa.string()
+ calls.append(("cast", target))
+ raise error_class("simulated Arrow-string cast failure")
+
+ # PyArrow extension types are immutable. This local module proxy
substitutes
+ # the exact recognized storage class only for this simulated cast branch;
+ # no real ChunkedArray subclass or capacity behavior is being tested.
+ proxy = ModuleType("pyarrow")
+ vars(proxy).update(vars(pa))
+ proxy.ChunkedArray = SimulatedCastFailure
+ with monkeypatch.context() as simulated:
+ simulated.setattr(array, "_pa_array", SimulatedCastFailure())
+ simulated.setitem(sys.modules, "pyarrow", proxy)
+ assert _pandas_input._arrow_string_table(frame, schema, dtypes) is None
+ assert calls == [("validate", True), ("cast", pa.string())]
+
+
+def test_arrow_string_object_labels_dtype_error_matches_native_rows(protocol,
monkeypatch):
+ import importlib
+
+ _pandas_input = importlib.import_module("pyspark.sql._pandas_input")
+ frames = []
+ for _ in range(3):
+ frame = pd.DataFrame({"v": pd.array(["original", None] * 5000,
+ dtype="string[pyarrow]")})
+ frame.columns = pd.Index(["v"], dtype=object)
+ frames.append(frame)
+ schema = T.StructType([T.StructField("v", T.StringType())])
+ array_class = type(frames[0].iloc[:, 0].array)
+ calls = []
+
+ class HookError(Exception):
+ pass
+
+ def customized(self):
+ assert type(self) is array_class
+ assert len(self) == 10000
+ calls.append("dtype")
+ raise HookError("dtype sentinel")
+
+ monkeypatch.setattr(array_class, "dtype", property(customized))
+ # The generic route can authenticate the data class before reading its
dtype.
+ assert not _pandas_input.accepts_pandas_table(frames[0], schema)
+ assert calls == []
+ errors = []
+ with pytest.raises(HookError, match="^dtype sentinel$") as error:
+ _pandas_input.prepare_string_input(frames[0], schema)
+ errors.append((type(error.value), str(error.value)))
+ assert calls == ["dtype"]
+ for row_control, frame in zip((False, True), frames[1:]):
+ calls.clear()
+ with monkeypatch.context() as control:
+ if row_control:
+ _force_native_pandas_rows(control)
+ with _arrow_input_service(protocol, config_error=True) as (spark,
state):
+ with pytest.raises(HookError, match="^dtype sentinel$") as
error:
+ spark.createDataFrame(frame, schema)
+ errors.append((type(error.value), str(error.value)))
+ assert state["config_requests"] == []
+ assert state["uploads"] == 0
+ assert state["input_tables"] == []
+ assert calls == ["dtype"]
+ assert errors[0] == errors[1] == errors[2]
+
+
+def test_arrow_string_custom_label_astype_keeps_native_row_values(protocol,
monkeypatch):
+ import importlib
+
+ _pandas_input = importlib.import_module("pyspark.sql._pandas_input")
+
+ texts = ["original", None] * 5000
+ frame = pd.DataFrame({"v": pd.array(texts, dtype="string[pyarrow]")})
+ frame.columns = pd.Index(pd.array(["v"], dtype="string[pyarrow]"))
+ schema = T.StructType([T.StructField("v", T.StringType())])
+ calls = []
+
+ def customized(*args, **kwargs):
+ calls.append("label astype")
+ raise AssertionError("native row conversion does not cast column
labels")
+
+ monkeypatch.setattr(frame.columns._values, "astype", customized)
+ assert _pandas_input.prepare_string_input(frame, schema) is None
+ assert not _pandas_input.accepts_pandas_table(frame, schema)
+ assert calls == []
+ tables = []
+ for row_control in (False, True):
+ with monkeypatch.context() as control:
+ if row_control:
+ _force_native_pandas_rows(control)
+ with _arrow_input_service(protocol, config_error=True) as (spark,
state):
+ spark.createDataFrame(frame, schema).toArrow()
+ tables.append(state["input_tables"][0])
+ assert state["config_requests"] == []
+ assert state["uploads"] == 0
+ assert calls == []
+ assert tables[0].column(0).to_pylist() == texts
+ assert tables[0].equals(tables[1], check_metadata=True)
+
+
+def test_arrow_string_null_labels_decline_without_getitem_callback(protocol,
monkeypatch):
+ import importlib
+
+ from pandas.core.arrays.arrow.array import ArrowExtensionArray
+
+ _pandas_input = importlib.import_module("pyspark.sql._pandas_input")
+ # Index uniqueness can cast null labels and invoke indexed masking. Native
+ # normalization may also slice labels while aligning DataFrame.where;
guards
+ # must decline without an early callback and preserve that native outcome.
+ expected = {"a": ["first"] * 10000, "b": ["second"] * 10000}
+ frames = []
+ for _ in range(2):
+ frame = pd.DataFrame({name: pd.array(values, dtype="string[pyarrow]")
+ for name, values in expected.items()})
+ frame.columns = pd.Index(pd.array(["a", None],
dtype="string[pyarrow]"))
+ frames.append(frame)
+ schema = T.StructType([T.StructField(name, T.StringType()) for name in
expected])
+ calls = []
+
+ class HookError(Exception):
+ pass
+
+ def customized(*args, **kwargs):
+ calls.append("label getitem")
+ raise HookError("label getitem sentinel")
+
+ monkeypatch.setattr(ArrowExtensionArray, "__getitem__", customized)
+ for frame in frames:
+ assert _pandas_input.prepare_string_input(frame, schema) is None
+ assert not _pandas_input.accepts_pandas_table(frame, schema)
+ assert calls == []
+
+ outcomes = []
+ tables = []
+ # Observe native behavior first, using a fresh equal frame for each route.
+ for row_control, frame in zip((True, False), frames):
+ calls.clear()
+ with monkeypatch.context() as control:
+ if row_control:
+ _force_native_pandas_rows(control)
+ with _arrow_input_service(protocol, config_error=True) as (spark,
state):
+ try:
+ spark.createDataFrame(frame, schema).toArrow()
+ except HookError as error:
+ assert type(error) is HookError
+ assert str(error) == "label getitem sentinel"
+ assert calls
+ outcome = (type(error), str(error))
+ assert state["input_tables"] == []
+ else:
+ assert len(state["input_tables"]) == 1
+ table = state["input_tables"][0]
+ assert table.to_pydict() == expected
+ tables.append(table)
+ outcome = None
+ outcomes.append((outcome, tuple(calls)))
+ assert state["config_requests"] == []
+ assert state["uploads"] == 0
+ assert outcomes[0] == outcomes[1]
+ if tables:
+ assert len(tables) == 2
+ assert tables[0].equals(tables[1], check_metadata=True)
+
+
[email protected]("label_storage", ["object", "pyarrow"])
[email protected]("method", ["astype", "__array__", "__getitem__",
"__init__"])
[email protected]("storage", ["pyarrow", "numpy_semantics"])
+def test_arrow_string_super_row_hooks_keep_native_errors(
+ protocol, monkeypatch, label_storage, method, storage
+):
+ import importlib
+
+ from pandas.core.arrays.arrow.array import ArrowExtensionArray
+
+ _pandas_input = importlib.import_module("pyspark.sql._pandas_input")
+ frames = []
+ for _ in range(2):
+ frame = pd.DataFrame({"v": _arrow_string_column(["original", None] *
5000, storage)})
+ frame.columns = pd.Index(["v"], dtype=object) if label_storage ==
"object" else pd.Index(
+ pd.array(["v"], dtype="string[pyarrow]"))
+ frames.append(frame)
+ schema = T.StructType([T.StructField("v", T.StringType())])
+ calls = []
+ for frame in frames:
+ assert _pandas_input.prepare_string_input(frame, schema) is not None
+ assert _pandas_input.accepts_pandas_table(frame, schema)
+ assert _pandas_input.accepts_pandas_frame(frame)
+
+ class HookError(Exception):
+ pass
+
+ def customized(self, *args, **kwargs):
+ if method == "__init__":
+ # The receiver has no storage yet. Null-subarray construction
passes
+ # the 5,000 non-null data values; labels have only one value.
+ values = args[0]
+ assert values.to_pylist() == ["original"] * 5000
+ else:
+ assert len(self) == 10000 # Preflight must not cast the
length-one labels.
+ calls.append(method)
+ raise HookError(f"super {method} sentinel")
+
+ monkeypatch.setattr(ArrowExtensionArray, method, customized)
+ for frame in frames:
+ assert _pandas_input.prepare_string_input(frame, schema) is None
+ assert not _pandas_input.accepts_pandas_table(frame, schema)
+ assert not _pandas_input.accepts_pandas_frame(frame)
+ assert calls == []
+ outcomes = []
+ tables = []
+ # The native control determines whether this pandas layout calls __init__.
+ # pandas 3.1's null-subarray factory instead allocates through __new__.
+ for row_control, frame in zip((True, False), frames):
+ calls.clear()
+ with monkeypatch.context() as control:
+ if row_control:
+ _force_native_pandas_rows(control)
+ with _arrow_input_service(protocol, config_error=True) as (spark,
state):
+ try:
+ spark.createDataFrame(frame, schema).toArrow()
+ except HookError as error:
+ assert type(error) is HookError
+ assert str(error) == f"super {method} sentinel"
+ assert calls == [method]
+ assert state["input_tables"] == []
+ outcome = (type(error), str(error))
+ else:
+ assert method == "__init__"
+ assert calls == []
+ assert len(state["input_tables"]) == 1
+ table = state["input_tables"][0]
+ table.validate(full=True)
+ assert table.to_pydict() == {"v": ["original", None] *
5000}
+ assert table.schema.names == ["v"]
+ assert table.schema.field(0).type == pa.string()
+ assert table.schema.field(0).nullable
+ tables.append(table)
+ outcome = None
+ outcomes.append((outcome, tuple(calls)))
+ assert state["config_requests"] == []
+ assert state["uploads"] == 0
+ assert outcomes[0] == outcomes[1]
+ if tables:
+ assert len(tables) == 2
+ assert tables[0].equals(tables[1], check_metadata=True)
@pytest.mark.parametrize("reason", [
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]