This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 6196e648 [SPARK-60745][CONNECT][FOLLOWUP] Preserve native conversion 
for customized pandas strings
6196e648 is described below

commit 6196e648f54fd15fd346700af0e4422e85f079c6
Author: DB Tsai <[email protected]>
AuthorDate: Sun Nov 8 20:31:51 2026 +0900

    [SPARK-60745][CONNECT][FOLLOWUP] Preserve native conversion for customized 
pandas strings
    
    ### What changes were proposed in this pull request?
    
    [SPARK-60745](https://issues.apache.org/jira/browse/SPARK-60745): preserve 
native row-conversion behavior for customized Arrow-backed pandas strings in 
the columnar input path already on master.
    
    Specialized, generic explicit-schema and inferred/names-only routes share 
the relevant hook and alias eligibility checks. Covered custom array/label 
conversion hooks and rebound schema-converter aliases decline to native rows, 
so another admission route cannot bypass those rejections. The specialized 
string helper separately declines recognized storage-normalization and 
validation failures, including a column storage length that differs from the 
frame. Ordinary supported inputs retain [...]
    
    Inspect current instance hooks and selected canonical class attributes, 
including inherited methods and pandas' NumPy-semantics mixin/partial dispatch. 
Read raw class attributes without binding customized descriptors, and recheck 
between columns because an accessor can change a hook. Only the pandas helper, 
regression tests and architecture documentation change.
    
    ### Why are the changes needed?
    
    Direct Arrow-storage extraction can bypass a customized conversion method, 
change values, suppress an exception or run label callbacks earlier than native 
row conversion. The covered fixtures must retain native values, errors and 
ordering through every applicable admission route.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Inputs with the covered customizations fall back to native rows; 
ordinary eligible inputs retain the existing fast path. Authentication adds 
per-column work, and conservative fallback can cost more than columnar 
conversion.
    
    The guarantee is bounded to the selected canonical Arrow-string classes, 
instance/class hooks and explicitly checked aliases. **Class data read by those 
methods, including `_storage` and `_na_value`, and other module globals are 
outside the checks.** Other array classes retain the existing generic-route 
module-origin policy. Arbitrary private storage replacement or inconsistent 
frame internals are not covered by an all-route consistency guarantee. This 
does not establish an invariant  [...]
    
    Existing importer costs remain: configuration RPCs, validation/casting, 
alignment, IPC serialization and threshold-triggered artifact uploads. 
Narrowing `large_string` to Arrow `string` can copy the offset buffer; this 
path is not universally zero-copy. Simulated cast failures do not establish 
support for actual storage above 2 GiB or malformed offsets. The inherited 
package/runtime pandas-minimum mismatch and support for older/broken 
dependencies are not repaired here.
    
    ### How was this patch tested?
    
    Candidate `33559dc2af3df57b4db56b1cb90f16d1f5bbd8c5` against baseline 
`e8b1d4284e11d119c4fda9563a913ece22f43e7c`:
    
    - Independent full-patch source review found no actionable findings. **216 
focused native cases passed**. Full native suite: **2,378 passed, eight 
skipped, one failed**; the sole failure matches the baseline's 
unsupported-NumPy-scalar case. The raw full result remains failed.
    - Version matrix: **216 passed in each of four 
environments**—pandas/PyArrow 2.2.3/18, 2.3.3/26, 3.0.6/26 and 3.1.0/26. Actual 
imports, source and binary hashes were checked. Controls produced **132 
candidate passes and 132 expected assertion failures**: 120 on baseline and 12 
with intentionally memoized guards. Wrong-origin failures were not accepted. 
Simulated cast failures remain distinct from real capacity/offset coverage.
    - Rust core/API/deadline checks and default/Wasm compilation are carried 
from the fresh baseline through equality of all **190 native build inputs**. 
The native Python tests imported the recorded baseline extension; no fresh 
Arrow Rust build is claimed. Baseline workspace Clippy remains a disclosed 
nonzero failure. A 12-class name audit found zero missing reference names and 
five extra native names; it does not certify signatures, behavior or full API 
parity.
    - The completed fresh official transport gate has **91 selected files, 
2,601 passed and 655 runtime skips**, with zero unexpected selected-suite 
failures/errors. Eleven all-skipped modules account for 267 skips. A separately 
run, checked-in excluded catalog case fails because the distribution lacks 
`InMemoryCatalog`. A final 48-input transport/build/harness equality audit 
supports carrying this result to Arrow. **Arrow's binary did not execute this 
official suite**; it used the Decima [...]
    - The initial final-revision run completed all six measurement workers and 
contracts, then its large-upload live check exhausted the **1 GiB JVM heap**; 
cleanup also raised `PermissionError`. The incomplete raw summary and 
failed-execution receipt are preserved; that run is not accepted as passed. A 
complete repeat uses an independently verified **3 GiB JVM heap**, with 
unchanged sources, worker, driver and plan.
    - The complete **3 GiB rerun passed**: all six timing workers, 
output-contract checks and **eight native live tests per candidate** (including 
the >128 MiB upload), with no live skips or failures. All six 
inferred/names-only Arrow-string contract cases passed on the baseline, 
candidate and official PySpark, with full values and schemas checked. All final 
source, import, binary and reference-input guards passed; the owned JVM was 
reaped.
    - Published-head CI remains pending publication and exact-head verification.
    
    Measurements use macOS, Spark Connect 4.3.0, pandas 2.3.3, PyArrow 26.0.0 
and one shared owned local server. Decimal workloads have 200,000 rows and two 
fields; string workloads have three fields. Each worker is a separate process. 
The following values are client `createDataFrame` CPU / wall milliseconds. 
Complete output and schema validation ran outside the timer.
    
    **Warm medians of five calls:**
    
    | Workload | Frozen master CPU / wall | Candidate CPU / wall | Pure Python 
CPU / wall |
    | --- | ---: | ---: | ---: |
    | arrow-strings-1000 | 0.844 / 1.355 | 0.888 / 1.377 | 0.768 / 0.784 |
    | arrow-strings-200000 | 5.823 / 6.501 | 5.915 / 6.504 | 4.586 / 5.180 |
    
    **First measured call per workload:** imports, session setup and input 
preparation precede these samples; later workloads follow earlier ones in their 
worker.
    
    | Workload | Frozen master CPU / wall | Candidate CPU / wall | Pure Python 
CPU / wall |
    | --- | ---: | ---: | ---: |
    | arrow-strings-1000 | 9.802 / 10.918 | 11.856 / 12.689 | 1.257 / 1.820 |
    | arrow-strings-200000 | 10.882 / 37.037 | 10.735 / 42.028 | 13.557 / 
39.020 |
    
    Observed warm CPU cost increased **5.2% at 1,000 rows** and **1.6% at 
200,000 rows** versus master; warm wall time changed by +1.6% and approximately 
0.0%. The candidate remains slower than Python on these two workloads; the 
already-landed fast path is part of the baseline. Whole-worker cumulative peak 
RSS was 163.1 MiB for master, 165.4 MiB for the candidate and 174.9 MiB for 
Python.
    
    This is one fixed-order paired run without confidence intervals. RSS 
includes input preparation, expected tables, collection, validation and 
allocator retention. Guard-only cost, fallback/wide/nested workload 
performance, RPC/upload counts, conversion-only memory, process cold starts and 
whole-query performance are outside these measurements.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    OpenAI Codex.
    
    This pull request and its description were written by OpenAI Codex.
    
    Closes #366 from dbtsai/codex/followup-arrow-string-hooks-current.
    
    Lead-authored-by: DB Tsai <[email protected]>
    Co-authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 dev/design/architecture.md          |   38 ++
 docs/architecture.md                |    6 +-
 python/pyspark/sql/_pandas_input.py |  348 +++++++++---
 python/tests/test_arrow_handoff.py  | 1021 ++++++++++++++++++++++++++++++++++-
 4 files changed, 1317 insertions(+), 96 deletions(-)

diff --git a/dev/design/architecture.md b/dev/design/architecture.md
index cd625c30..f8931f8e 100644
--- a/dev/design/architecture.md
+++ b/dev/design/architecture.md
@@ -101,6 +101,44 @@ against its live server and built native extension. 
Official parity tests
 exercise the Rust transport with official PySpark; the separate native smoke
 step covers the drop-in `toArrow()` path.
 
+The pandas Arrow-string input guard recognizes pandas 2.2 through 3.x storage
+layouts. The Python Connect and pandas parity workflows pin `pandas>=2.2,<3.0`;
+coverage and pipelines parity install unpinned pandas. CI has no explicit
+pandas-version matrix. The inherited `pandas>=1.0.0` package declaration is
+below the runtime utilities' `>=2.2` requirement. Aligning the package metadata
+is a separate packaging follow-up; neither that declaration nor the 
Arrow-string
+checks establish support for older pandas versions.
+
+Arrow-string input eligibility checks both array instances and canonical
+classes' row/Arrow conversion methods and label casting, including inherited
+methods, on the specialized, generic explicit-schema and inferred-schema 
routes.
+Other pandas array classes on the generic routes retain the existing
+module-origin policy, which does not authenticate mutations of their canonical
+classes' conversion hooks. The generic routes identify Arrow-string classes
+through their loaded defining module in `sys.modules`. Removing or replacing
+`pandas.core.arrays.string_arrow` there while retaining instances from the old
+module is outside these checks.
+
+Authentication occurs when the concrete array is inspected. With NumPy-backed
+column labels, a specialized guard may first read `frame.dtypes`, matching the
+native row path's dtype read before cell normalization. A customized dtype
+property can therefore raise before the later array-class check, with the same
+local error and no configuration or upload request. The bounded Arrow-string
+class checks for data and labels inspect raw MRO dictionaries for conversion 
and
+label-iteration methods, dtype/missing-value properties and the null-subarray
+constructor (including its immediate super-constructor), optional factory and
+allocator used by supported pandas layouts. The label checks authenticate the
+class before inspecting its other methods, which also use the raw lookup. This
+avoids binding customized descriptors during those checks. The pandas 2.2
+NumPy-semantics accessor also dispatches through a mixin before ordinary MRO
+lookup; its mixin and partial-function aliases are checked, and its namespace 
is
+limited to string-method names, initializer/storage declarations and class
+metadata. Its mixin initializer remains allowed because construction uses the
+class slots authenticated above. Class data read by the authenticated methods,
+such as `_storage` and `_na_value`, and module globals other than the 
explicitly
+checked aliases are outside these checks. These checks do not establish a
+general invariant for arbitrary mutations of pandas internals.
+
 ## UDF strategy (full compatibility)
 
 Python/pandas UDFs are **serialized, not executed** on the client:
diff --git a/docs/architecture.md b/docs/architecture.md
index cb8663de..0365402d 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -163,9 +163,9 @@ Pandas object columns containing exact `datetime.date` 
values or `None` use the
 Arrow importer at 10,000 rows or more with an explicit `StructType` of 
`DateType`
 fields. Unsupported values and customized inputs retain the row conversion 
path.
 Pandas frames of at least 10,000 rows whose columns are all kept in pandas' 
Arrow
-string storage (`string[pyarrow]`, and the `str` dtype of pandas 3), with an
-explicit `StructType` of `StringType` fields, pass that storage to this 
importer
-as Arrow string columns, without Python rows.
+string storage (`string[pyarrow]` in pandas 2.2 and later, and the `str` dtype 
of
+pandas 3), with an explicit `StructType` of `StringType` fields, pass that 
storage
+to this importer as Arrow string columns, without Python rows.
 Inputs at or above `spark.sql.session.localRelationCacheThreshold` are 
serialized
 into independent IPC streams and uploaded as session-scoped cache artifacts.
 The plan carries `ChunkedCachedLocalRelation` hashes instead of embedding the
diff --git a/python/pyspark/sql/_pandas_input.py 
b/python/pyspark/sql/_pandas_input.py
index fa92a3f9..7329ea4c 100644
--- a/python/pyspark/sql/_pandas_input.py
+++ b/python/pyspark/sql/_pandas_input.py
@@ -19,10 +19,135 @@
 """Conservative columnar pandas input; unsupported conversions use the row 
path."""
 
 
+_ARROW_STRING_HOOKS = ("__arrow_array__", "__array__", "__getitem__", "isna",
+                       "to_numpy", "__len__")
+_ARROW_STRING_CONSTRUCTION_HOOKS = ("__init__", "_from_pyarrow_array", 
"__new__")
+_ARROW_STRING_MIXIN_ALLOWED_NAMES = (
+    "__module__", "__qualname__", "__doc__", "__dict__", "__weakref__",
+    "__firstlineno__", "__static_attributes__", "_pa_array", "__init__",
+)
+
+
+def _defined_in(method, name, *modules):
+    """Whether `method` is a plain function named `name` whose module name and
+    globals belong to one of `modules`.
+    """
+    from types import FunctionType
+
+    return (type(method) is FunctionType and method.__name__ == name
+            and any(method.__module__ == module.__name__
+                    and method.__globals__ is vars(module) for module in 
modules))
+
+
+def _raw_class_attribute(cls, name, start=0):
+    """Return the defining class and raw attribute, or (None, None) if 
absent."""
+    if cls is not None:
+        for base in cls.__mro__[start:]:
+            namespace = vars(base)
+            if name in namespace:
+                return base, namespace[name]
+    return None, None
+
+
+def _has_canonical_arrow_string_super_method(cls, start, name, owner, module):
+    """Authenticate the immediate method reached through ArrowStringArray's 
super()."""
+    actual_owner, method = _raw_class_attribute(cls, name, start)
+    return actual_owner is owner and _defined_in(method, name, module)
+
+
+def _arrow_string_class_is_canonical(cls):
+    """Check the identity and supported hooks of a concrete pandas 
Arrow-string class.
+
+    Authenticate conversion, construction and allocation, dtype/missing-value
+    properties, and NumPy-semantics accessor dispatch using raw MRO lookups.
+    """
+    import pandas as pd
+    from pandas.core.arrays import base as array_base_module, string_ as 
string_module, string_arrow
+    from pandas.core.arrays.arrow import array as arrow_module
+
+    numpy_semantics = getattr(string_arrow, "ArrowStringArrayNumpySemantics", 
None)
+    # Reject subclasses by identity before inspecting them. In particular, do 
not
+    # hash an untrusted class whose metaclass could customize __hash__.
+    if (not any(cls is candidate for candidate in
+                (string_arrow.ArrowStringArray, numpy_semantics))
+            or pd.arrays.ArrowStringArray is not string_arrow.ArrowStringArray
+            or pd.StringDtype is not string_module.StringDtype):
+        return False
+    try:
+        super_start = cls.__mro__.index(string_arrow.ArrowStringArray) + 1
+    except ValueError:
+        # A rebound optional NumPy-semantics class alias may lack that 
ancestry.
+        return False
+
+    # pandas 2.2's NumPy-semantics variant has its own string-method accessor.
+    access = _raw_class_attribute(cls, "__getattribute__")[1]
+    if access is not object.__getattribute__:
+        if cls is not numpy_semantics or not _defined_in(
+                access, "__getattribute__", string_arrow):
+            return False
+        from functools import partial
+        from pandas.core.arrays import _arrow_string_mixins as string_mixins
+
+        mixin = string_mixins.ArrowStringArrayMixin
+        # This accessor dispatches through the mixin before the ordinary MRO.
+        # Check the mixin/partial aliases and allow only string-method names, 
class
+        # metadata, and initializer/storage declarations in the mixin 
namespace.
+        # Construction itself uses class slots, checked separately below;
+        # the mixin's canonical __init__ is not an instance conversion hook.
+        if (not any(base is mixin for base in cls.__mro__)
+                or vars(string_arrow).get("ArrowStringArrayMixin") is not mixin
+                or vars(string_arrow).get("partial") is not partial
+                or any(name not in _ARROW_STRING_MIXIN_ALLOWED_NAMES
+                       and not name.startswith("_str_") for name in 
vars(mixin))):
+            return False
+    # Authenticate row/Arrow conversion hooks and __len__, which sizes the 
table.
+    # to_numpy indexes the non-null portion through __getitem__ before reading 
storage.
+    if (any(not _defined_in(_raw_class_attribute(cls, name)[1], name,
+                           string_arrow, arrow_module) for name in 
_ARROW_STRING_HOOKS)
+            or not _defined_in(_raw_class_attribute(cls, "astype")[1], 
"astype", string_arrow)
+            or not _has_canonical_arrow_string_super_method(
+                cls, super_start, "astype", array_base_module.ExtensionArray, 
array_base_module)):
+        return False
+    # Null-subarray conversion may construct another array. Older pandas 
versions
+    # have no _from_pyarrow_array helper; a present custom descriptor must 
decline.
+    if (not _defined_in(_raw_class_attribute(cls, "__init__")[1], "__init__", 
string_arrow)
+            or not _has_canonical_arrow_string_super_method(
+                cls, super_start, "__init__", 
arrow_module.ArrowExtensionArray, arrow_module)):
+        return False
+    factory_owner, factory = _raw_class_attribute(cls, "_from_pyarrow_array")
+    if factory_owner is not None and not _defined_in(factory, 
"_from_pyarrow_array", string_arrow):
+        return False
+    if _raw_class_attribute(cls, "__new__")[1] is not object.__new__:
+        return False
+    dtype = _raw_class_attribute(cls, "dtype")[1]
+    hasna = _raw_class_attribute(cls, "_hasna")[1]
+    missing_value = _raw_class_attribute(string_module.StringDtype, 
"na_value")[1]
+    return (type(dtype) is property
+            and _defined_in(dtype.fget, "dtype", string_arrow, arrow_module)
+            and type(hasna) is property
+            and _defined_in(hasna.fget, "_hasna", arrow_module)
+            and type(missing_value) is property
+            and _defined_in(missing_value.fget, "na_value", string_module))
+
+
+def _has_canonical_arrow_string_array(array):
+    """Authenticate a column at this point without caching mutable hooks.
+
+    Callers may already have read frame.dtypes, as the native row path does.
+    """
+    # Column lookup can run customized pandas accessors between these checks.
+    # An earlier column authenticates neither the next instance nor its class.
+    if not _arrow_string_class_is_canonical(type(array)):
+        return False
+    # Deliberately include descriptors and special names for future explicit 
lookups.
+    return not any(name in vars(array) for name in
+                   _ARROW_STRING_HOOKS + _ARROW_STRING_CONSTRUCTION_HOOKS
+                   + ("astype", "dtype", "_hasna"))
+
+
 def _has_canonical_columns(columns):
     """Decline label containers whose row-path iteration can run custom 
hooks."""
     import sys
-    from types import FunctionType
 
     import numpy as np
     import pandas as pd
@@ -35,19 +160,14 @@ def _has_canonical_columns(columns):
             or "__iter__" in vars(columns)):
         return False
 
-    def ordinary(method, name, module):
-        return (type(method) is FunctionType and method.__name__ == name
-                and method.__module__ == module.__name__
-                and method.__globals__ is vars(module))
-
     # The native row path iterates columns before normalizing any cells.
     # Decline overrides without invoking them, so fallback preserves that 
order.
     if (Index.__iter__ is not base_module.IndexOpsMixin.__iter__
-            or not ordinary(Index.__iter__, "__iter__", base_module)):
+            or not _defined_in(Index.__iter__, "__iter__", base_module)):
         return False
     values = vars(Index).get("_values")
     if (type(values) is not property
-            or not ordinary(values.fget, "_values", index_module)):
+            or not _defined_in(values.fget, "_values", index_module)):
         return False
     if any("_data" in vars(base) for base in Index.__mro__):
         return False
@@ -78,13 +198,26 @@ def _has_canonical_columns(columns):
     cls = type(backing)
     if not any(cls is candidate for candidate in python_strings + 
arrow_strings):
         return False
-    if (cls.__getattribute__ is not object.__getattribute__
+    arrow_backed = any(cls is candidate for candidate in arrow_strings)
+    if arrow_backed and not _arrow_string_class_is_canonical(cls):
+        return False
+
+    def class_attribute(owner, name):
+        # Arrow labels share the raw lookup; Python-string class lookup stays 
unchanged.
+        if arrow_backed:
+            return _raw_class_attribute(owner, name)[1]
+        return getattr(owner, name, None)
+
+    # The shared instance-shadow list conservatively includes descriptors and 
special
+    # names for both Arrow-backed and Python-backed string labels.
+    if (class_attribute(cls, "__getattribute__") is not object.__getattribute__
             or any(name in vars(backing) for name in
-                   ("__iter__", "__getitem__", "__len__", "_box_func",
-                    "tolist", "to_numpy", "isna", "fillna", "_hasna"))
-            or getattr(cls, "tolist", None) is not getattr(
-                vars(string_module).get("BaseStringArray"), "tolist", None)
-            or not ordinary(getattr(cls, "tolist", None), "tolist", 
string_module)):
+                   ("__iter__", "__getitem__", "__len__", "__array__", 
"_box_func",
+                    "tolist", "to_numpy", "isna", "fillna", "_hasna")
+                   + _ARROW_STRING_CONSTRUCTION_HOOKS)
+            or class_attribute(cls, "tolist") is not class_attribute(
+                vars(string_module).get("BaseStringArray"), "tolist")
+            or not _defined_in(class_attribute(cls, "tolist"), "tolist", 
string_module)):
         return False
     if any(cls is candidate for candidate in python_strings):
         owner, module = vars(numpy_module).get("NumpyExtensionArray"), 
numpy_module
@@ -100,7 +233,7 @@ def _has_canonical_columns(columns):
         # ExtensionArray iteration dispatches through len and indexed boxing.
         for name, hook_module in (("__iter__", array_base_module),
                                   ("__getitem__", mixins_module), 
("_box_func", mixins_module)):
-            if not ordinary(getattr(cls, name, None), name, hook_module):
+            if not _defined_in(getattr(cls, name, None), name, hook_module):
                 return False
         if (getattr(cls, "__len__", None) is not NDArrayBacked.__len__
                 or getattr(cls, "ndim", None) is not NDArrayBacked.ndim
@@ -113,26 +246,31 @@ def _has_canonical_columns(columns):
         owner, module = vars(arrow_module).get("ArrowExtensionArray"), 
arrow_module
         if owner is None:
             return False
+        # pandas builds an Arrow string Index's uniqueness engine with 
astype(object).
+        # Authenticate the recognized object-cast methods and 
dtype/missing-value
+        # properties before is_unique. This is a bounded check of pandas 
internals.
+        if "astype" in vars(backing):
+            return False
         try:
             storage = object.__getattribute__(backing, "_pa_array")
         except AttributeError:
             return False
         if (type(storage) is not pa.ChunkedArray
-                or not ordinary(getattr(cls, "__iter__", None), "__iter__", 
module)):
+                or not _defined_in(class_attribute(cls, "__iter__"), 
"__iter__", module)):
             return False
-        isna, isna_module = getattr(owner, "isna", None), module
-        hasna = getattr(cls, "_hasna", None)
+        isna, isna_module = class_attribute(owner, "isna"), module
+        hasna = class_attribute(cls, "_hasna")
         if (type(hasna) is not property
-                or not ordinary(hasna.fget, "_hasna", module)
-                or getattr(cls, "fillna", None) is not getattr(owner, 
"fillna", None)
-                or not ordinary(getattr(cls, "fillna", None), "fillna", 
module)):
+                or not _defined_in(hasna.fget, "_hasna", module)
+                or class_attribute(cls, "fillna") is not 
class_attribute(owner, "fillna")
+                or not _defined_in(class_attribute(cls, "fillna"), "fillna", 
module)):
             return False
     # Keep the existing storage/conversion authentication as well as the
     # iterator hooks: these objects also supply labels to pandas normalization.
-    return (getattr(cls, "to_numpy", None) is getattr(owner, "to_numpy", None)
-            and ordinary(getattr(cls, "to_numpy", None), "to_numpy", module)
-            and getattr(cls, "isna", None) is isna
-            and ordinary(getattr(cls, "isna", None), "isna", isna_module))
+    return (class_attribute(cls, "to_numpy") is class_attribute(owner, 
"to_numpy")
+            and _defined_in(class_attribute(cls, "to_numpy"), "to_numpy", 
module)
+            and class_attribute(cls, "isna") is isna
+            and _defined_in(class_attribute(cls, "isna"), "isna", isna_module))
 
 
 def prepare_int64_table(frame, schema):
@@ -577,7 +715,7 @@ def prepare_string_input(frame, schema):
     from pandas.core import frame as frame_module, generic as generic_module
     from pandas.core.frame import DataFrame
     from pandas.core.series import Series
-    from types import FunctionType, GetSetDescriptorType, 
MethodDescriptorType, WrapperDescriptorType
+    from types import GetSetDescriptorType, MethodDescriptorType, 
WrapperDescriptorType
 
     from pyspark._pyspark import StringType, StructField, StructType as 
NativeStructType
     from pyspark.sql import types as sql_types
@@ -592,8 +730,7 @@ def prepare_string_input(frame, schema):
     for name, module in (("astype", generic_module), ("notna", frame_module),
                          ("where", generic_module)):
         method = getattr(DataFrame, name)
-        if (type(method) is not FunctionType or method.__name__ != name
-                or method.__module__ != module.__name__ or method.__globals__ 
is not vars(module)):
+        if not _defined_in(method, name, module):
             return None
     if (DataFrame.astype is not generic_module.NDFrame.astype
             or DataFrame.where is not generic_module.NDFrame.where):
@@ -671,29 +808,48 @@ def _arrow_string_table(frame, schema, dtypes):
 
     For the row path, pandas converts such a column from that storage, with 
its missing
     values as None, so the storage holds the same values. Only pandas' own 
array classes are
-    trusted to keep them there: a subclass could convert differently. The 
caller's guards of
-    prepare_string_input (frame, schema, names, size) have passed.
+    trusted with their original conversion hooks: customization could convert 
differently.
+    The caller's guards of prepare_string_input (frame, schema, names, size) 
have passed.
     """
-    import pandas as pd
-    import pyarrow as pa
-    from pandas.core.arrays import string_arrow
+    from pandas.core.arrays import string_ as string_module
 
-    classes = {pd.arrays.ArrowStringArray,
-               getattr(string_arrow, "ArrowStringArrayNumpySemantics", None)} 
- {None}
-    if not all(type(dtype) is pd.StringDtype and dtype.storage in ("pyarrow", 
"pyarrow_numpy")
+    if not all(type(dtype) is string_module.StringDtype
+               and dtype.storage in ("pyarrow", "pyarrow_numpy")
                for dtype in dtypes):
         return None
+    # The importer uses mutable public/captured schema names. Decline a rebound
+    # namespace before conversion or configuration/upload RPCs, as the generic 
route does.
+    if not _converter_namespaces_intact():
+        return None
+
+    import pyarrow as pa
+    from pandas.core.series import Series
+
     arrays = []
-    for (_, column), field in zip(frame.items(), schema.fields):
+    for index, field in enumerate(schema.fields):
+        column = frame.iloc[:, index]
+        if type(column) is not Series:
+            return None
         array = column.array
-        if type(array) not in classes:
+        if not _has_canonical_arrow_string_array(array):
+            return None
+        values = array.__arrow_array__()
+        # pandas returns a ChunkedArray; normalize a plain Array defensively if
+        # an instance's _pa_array storage was replaced.
+        if issubclass(type(values), pa.Array):
+            values = pa.chunked_array([values])
+        elif type(values) is not pa.ChunkedArray:
+            return None
+        if len(values) != len(frame):
             return None
-        values = pa.chunked_array(array.__arrow_array__())
         if values.type not in (pa.string(), pa.large_string()):
             return None
         try:
             # Malformed offsets or UTF-8 keep the row path and its error.
             values.validate(full=True)
+            # Check normal-string capacity before configuration/upload RPCs. 
The importer
+            # may widen back to large_string when useLargeVarTypes is enabled.
+            # pandas 2.2+ stores large_string, so this narrowing copies each 
column's offsets.
             values = values.cast(pa.string())
         except (pa.ArrowException, OverflowError):
             return None
@@ -868,6 +1024,45 @@ def _native_names_intact(namespace):
     )
 
 
+def _converter_namespaces_intact():
+    """Whether loaded schema converters/importers still resolve native type 
names."""
+    import sys
+
+    from pyspark.sql import types as sql_types
+
+    if sys.modules.get("pyspark.sql.types") is not sql_types or not 
_native_names_intact(
+        vars(sql_types)
+    ):
+        return False
+    for name in ("pyspark.sql.pandas.types", "pyspark.sql._arrow_input"):
+        module = sys.modules.get(name)
+        if module is not None and not _native_names_intact(vars(module)):
+            return False
+    # The importer resolves its schema classes by name, and its conversion 
functions may
+    # keep the globals of an older type module.
+    importer = sys.modules.get("pyspark.sql._arrow_input")
+    if importer is not None:
+        from pyspark._pyspark import AtomicType, StructType
+
+        if any(vars(importer).get(name) is not cls
+               for name, cls in (("StructType", StructType), ("AtomicType", 
AtomicType))):
+            return False
+        for function in list(vars(importer).values()):
+            namespace = getattr(function, "__globals__", None)
+            if type(namespace) is dict and not _native_names_intact(namespace):
+                return False
+        # The importer calls its converters by these names, which must still 
be functions.
+        converter = sys.modules.get("pyspark.sql.pandas.types")
+        for name in ("to_arrow_schema", "from_arrow_schema"):
+            if name in vars(importer) and (
+                getattr(vars(importer)[name], "__globals__", None) is None
+                or (converter is not None
+                    and vars(importer)[name] is not getattr(converter, name, 
None))
+            ):
+                return False
+    return True
+
+
 def _canonical_struct(schema):
     """Whether a native struct schema and its fields keep their own classes' 
behavior.
 
@@ -904,39 +1099,57 @@ def _canonical_struct(schema):
 def accepts_pandas_table(frame, schema):
     """Whether `prepare_pandas_table` may convert this frame under this schema.
 
-    Customized frames, columns, schema classes and rebound type names keep the 
row path,
-    which reads them in its own order. The conversion and the importer resolve 
the type
-    names by lookup, so each namespace must still hold the native classes. The 
schema is
-    the caller's object, or a DDL string, which is parsed.
+    Customized frames, column labels, schema classes and rebound type names 
keep the row
+    path, which reads them in its own order. Conversion-hook authentication 
covers Arrow
+    string storage, including inherited hooks. Other pandas arrays retain the 
existing
+    module-origin checks; rebinding their canonical classes' hooks is not 
covered here.
+    The conversion and importer resolve type names by lookup, so each 
namespace must hold
+    the native classes. The schema is the caller's object or a DDL string, 
which is parsed.
     """
     return accepts_pandas_frame(frame) and (type(schema) is str or 
_canonical_struct(schema))
 
 
 def accepts_pandas_frame(frame):
-    """Whether PySpark's Arrow conversion may convert this frame, as 
`accepts_pandas_table`
-    checks it, but for a schema."""
+    """Check frame eligibility for explicit or inferred Arrow schema 
conversion."""
     import sys
 
     import pandas as pd
     from pandas.core.frame import DataFrame
 
-    from pyspark.sql import types as sql_types
-
     if pd.DataFrame is not DataFrame or type(frame) is not DataFrame:
         return False
     if not _has_canonical_columns(frame.columns):
         return False
-    # Custom extension arrays and dtypes keep the row path: Arrow would call 
their own
-    # `__arrow_array__`, and their dtype may impersonate a NumPy type.
+    # Custom extension-array/dtype classes outside pandas and NumPy keep the 
row path.
+    # Canonical non-string array classes retain the inherited module-origin 
policy.
+    # Arrow string storage already loads its defining module; do not import 
optional
+    # Arrow dependencies merely to inspect unrelated pandas columns.
+    # The generic routes assume this sys.modules entry still defines live 
arrays;
+    # removing or replacing that module while keeping old instances is not 
covered.
+    string_arrow = sys.modules.get("pandas.core.arrays.string_arrow")
+    arrow_strings = () if string_arrow is None else tuple(
+        vars(string_arrow).get(name)
+        for name in ("ArrowStringArray", "ArrowStringArrayNumpySemantics")
+    )
+    # Direct type checks bypass __subclasscheck__ and __mro__ metaclass 
callbacks.
+    # Use the real type: isinstance can also consult a custom __class__ 
property.
+    # Non-class aliases cannot own array storage.
+    arrow_strings = tuple(cls for cls in arrow_strings
+                          if type.__subclasscheck__(type, type(cls)))
     for i in range(len(frame.columns)):
         column = frame.iloc[:, i]
+        array = column.array
+        # A canonical class can still have customized hooks. Generic explicit 
and inferred
+        # schema routes must preserve the specialized routes' row fallback.
+        if (any(type.__subclasscheck__(cls, type(array)) for cls in 
arrow_strings)
+                and not _has_canonical_arrow_string_array(array)):
+            return False
         if not all(type(value).__module__.split(".")[0] in ("pandas", "numpy")
-                   for value in (column.array, column.dtype)):
+                   for value in (array, column.dtype)):
             return False
     # The native row path normalizes cells through these methods, which a 
customized class
     # or instance keeps on the row path.
     from pandas.core import frame as frame_module, generic as generic_module
-    from types import FunctionType
 
     if any(name in vars(frame) for name in ("astype", "notna", "where")):
         return False
@@ -946,39 +1159,10 @@ def accepts_pandas_frame(frame):
         ("where", generic_module, "NDFrame.where"),
     ):
         method = getattr(DataFrame, name)
-        if (type(method) is not FunctionType or method.__qualname__ != qualname
-                or method.__module__ != module.__name__ or method.__globals__ 
is not vars(module)):
+        if not _defined_in(method, name, module) or method.__qualname__ != 
qualname:
             return False
-    if sys.modules.get("pyspark.sql.types") is not sql_types or not 
_native_names_intact(
-        vars(sql_types)
-    ):
+    if not _converter_namespaces_intact():
         return False
-    for name in ("pyspark.sql.pandas.types", "pyspark.sql._arrow_input"):
-        module = sys.modules.get(name)
-        if module is not None and not _native_names_intact(vars(module)):
-            return False
-    # The importer resolves its schema classes by name, and its conversion 
functions may
-    # keep the globals of an older type module.
-    importer = sys.modules.get("pyspark.sql._arrow_input")
-    if importer is not None:
-        from pyspark._pyspark import AtomicType, StructType
-
-        if any(vars(importer).get(name) is not cls
-               for name, cls in (("StructType", StructType), ("AtomicType", 
AtomicType))):
-            return False
-        for function in list(vars(importer).values()):
-            namespace = getattr(function, "__globals__", None)
-            if type(namespace) is dict and not _native_names_intact(namespace):
-                return False
-        # The importer calls its converters by these names, which must still 
be functions.
-        converter = sys.modules.get("pyspark.sql.pandas.types")
-        for name in ("to_arrow_schema", "from_arrow_schema"):
-            if name in vars(importer) and (
-                getattr(vars(importer)[name], "__globals__", None) is None
-                or (converter is not None
-                    and vars(importer)[name] is not getattr(converter, name, 
None))
-            ):
-                return False
     return True
 
 
diff --git a/python/tests/test_arrow_handoff.py 
b/python/tests/test_arrow_handoff.py
index cfe35c1f..d82c6d55 100644
--- a/python/tests/test_arrow_handoff.py
+++ b/python/tests/test_arrow_handoff.py
@@ -4193,7 +4193,7 @@ def test_string_columns_preserve_every_ipc_cell(protocol, 
monkeypatch, cached, v
     "nonnullable_null", "frame_subclass", "string_subclass", "string_dtype",
     "arrow_string_dtype", "late_unsupported", "surrogate", "late_surrogate", 
"collation",
 ])
-def test_string_column_guard_preserves_row_fallback(reason):
+def test_string_column_guard_handles_supported_and_unsupported_inputs(reason):
     from pyspark.sql._pandas_input import prepare_string_input
 
     class Frame(pd.DataFrame):
@@ -4269,25 +4269,37 @@ def 
test_string_column_guard_preserves_row_fallback(reason):
         assert result is None
 
 
-
 # Columns kept in pandas' Arrow string storage reach the importer as Arrow, 
without Python rows.
 
 def _arrow_string_column(texts, storage):
-    import pyarrow as pa
     if storage == "numpy_semantics":
         try:
             dtype = pd.StringDtype("pyarrow", na_value=float("nan"))
         except TypeError:  # pandas 2.2
             dtype = pd.StringDtype("pyarrow_numpy")
         return pd.array(texts, dtype=dtype)
-    if storage == "large_string":
+    if storage == "chunked":
         return pd.arrays.ArrowStringArray(pa.chunked_array(
             [pa.array(texts[:7], pa.large_string()), pa.array(texts[7:], 
pa.large_string())]))
+    # The sliced case uses this storage, then slices the constructed frame 
below.
     return pd.array(texts, dtype="string[pyarrow]")
 
 
+def _force_native_pandas_rows(monkeypatch):
+    import importlib
+
+    # Earlier import-hook tests can leave the package attribute pointing at a
+    # different module object from the one imported by the native binding.
+    helper = importlib.import_module("pyspark.sql._pandas_input")
+    for name in ("prepare_int64_table", "timestamp_columns", 
"prepare_float64_input",
+                 "prepare_string_input", "prepare_date_table"):
+        monkeypatch.setattr(helper, name, lambda *_: None)
+    for name in ("accepts_pandas_table", "accepts_pandas_frame"):
+        monkeypatch.setattr(helper, name, lambda *_: False)
+
+
 @pytest.mark.parametrize("cached", [False, True])
[email protected]("storage", ["pyarrow", "numpy_semantics", 
"large_string", "sliced"])
[email protected]("storage", ["pyarrow", "numpy_semantics", "chunked", 
"sliced"])
 def test_arrow_string_columns_preserve_every_ipc_cell(protocol, monkeypatch, 
cached, storage):
     from pyspark.sql.pandas.types import from_arrow_metadata
 
@@ -4359,15 +4371,1002 @@ def 
test_arrow_string_columns_keep_the_row_path(reason):
     assert prepare_string_input(frame, T.StructType(fields)) is None
 
 
-def test_arrow_string_columns_match_the_row_path(protocol):
[email protected]("storage", ["pyarrow", "numpy_semantics"])
+def test_arrow_string_columns_match_the_row_path(protocol, monkeypatch, 
storage):
+    import json
+
+    from pyspark.sql.pandas.types import from_arrow_metadata
+
     texts = ["x" * (i % 5) if i % 4 else None for i in range(12000)]
-    schema = T.StructType([T.StructField("v", T.StringType())])
+    frame = pd.DataFrame({"v": _arrow_string_column(texts, storage)})
+    schema = T.StructType([T.StructField("v", T.StringType(), True, {"source": 
"row-control"})])
     tables = []
-    for dtype in ("string[pyarrow]", object):
-        with _arrow_input_service(protocol) as (spark, state):
-            spark.createDataFrame(pd.DataFrame({"v": pd.array(texts, 
dtype=dtype)}), schema).toArrow()
-            tables.append(state["input_tables"][0])
+    logical_schemas = []
+    for row_control in (False, True):
+        with monkeypatch.context() as control:
+            if row_control:
+                _force_native_pandas_rows(control)
+            with _arrow_input_service(protocol) as (spark, state):
+                spark.createDataFrame(frame, schema).toArrow()
+                tables.append(state["input_tables"][0])
+                logical_schemas.append(json.loads(state["spark_schema"]))
+                assert bool(state["config_requests"]) == (not row_control)
     assert tables[0].equals(tables[1])
+    # Row IPC omits field metadata; Spark overlays the explicit logical schema.
+    # The Arrow importer also embeds metadata in IPC, so compare both 
contracts.
+    assert from_arrow_metadata(tables[0].schema.field(0).metadata) == 
{"source": "row-control"}
+    assert tables[1].schema.field(0).metadata is None
+    assert logical_schemas == [json.loads(schema.json())] * 2
+
+
[email protected]("storage", ["pyarrow", "numpy_semantics"])
[email protected]("layout", ["small", "mixed"])
+def test_arrow_string_generic_explicit_schema_stays_columnar(
+    protocol, monkeypatch, storage, layout
+):
+    from pyspark.sql._pandas_input import accepts_pandas_table, 
prepare_string_input
+
+    texts = ["", None, "中文", "a\x00b"] * (3 if layout == "small" else 3000)
+    frame = pd.DataFrame({"v": _arrow_string_column(texts, storage)})
+    fields = [T.StructField("renamed", T.StringType())]
+    expected = {"renamed": texts}
+    if layout == "mixed":
+        frame["number"] = np.arange(len(texts), dtype=np.int64)
+        fields.append(T.StructField("number", T.LongType()))
+        expected["number"] = list(range(len(texts)))
+    schema = T.StructType(fields)
+    # These ordinary inputs miss the specialized string route but remain 
eligible
+    # for upstream's generic explicit-schema conversion.
+    assert prepare_string_input(frame, schema) is None
+    assert accepts_pandas_table(frame, schema)
+
+    def no_rows(*args, **kwargs):
+        raise AssertionError("generic Arrow strings must not materialize 
pandas rows")
+
+    monkeypatch.setattr(pd.DataFrame, "values", property(no_rows))
+    monkeypatch.setattr(pd.Series, "tolist", no_rows)
+    with _arrow_input_service(protocol) as (spark, state):
+        spark.createDataFrame(frame, schema).toArrow()
+        assert state["config_requests"]
+        assert state["input_tables"][0].to_pydict() == expected
+        assert state["input_tables"][0].schema.field(0).type == pa.string()
+
+
[email protected]("scope", ["public", "converter", "captured"])
[email protected]("class_name", ["BooleanType", "StringType", 
"StructType"])
+def test_arrow_string_rebound_converter_alias_keeps_native_rows(scope, 
class_name):
+    import sys
+
+    script = r'''
+import importlib
+import importlib.util
+import json
+import sys
+import tempfile
+from pathlib import Path
+import pandas as pd
+import pytest
+from pyspark import _pyspark as native
+from pyspark.sql import types as T
+
+_pandas_input = importlib.import_module("pyspark.sql._pandas_input")
+spec = importlib.util.spec_from_file_location("arrow_string_alias_capture", 
sys.argv[3])
+capture = importlib.util.module_from_spec(spec)
+sys.modules[spec.name] = capture
+spec.loader.exec_module(capture)
+texts = ["original", None] * 5000
+frame = pd.DataFrame({"v": pd.array(texts, dtype="string[pyarrow]")})
+schema = T.StructType([T.StructField("v", T.StringType(), True, {"source": 
"native"})])
+scope, name = sys.argv[1:3]
+replacement = native.BooleanType if name == "StringType" else native.StringType
+if scope == "public":
+    sys.modules.pop("pyspark.sql.pandas.types", None)
+    setattr(T, name, replacement)
+else:
+    converter = importlib.import_module("pyspark.sql.pandas.types")
+    if scope == "captured":
+        importer = importlib.import_module("pyspark.sql._arrow_input")
+    setattr(converter, name, replacement)
+    if scope == "captured":
+        sys.modules.pop("pyspark.sql.pandas.types")
+        assert importlib.import_module("pyspark.sql.pandas.types") is not 
converter
+        assert importer.to_arrow_schema.__globals__ is vars(converter)
+assert _pandas_input.prepare_string_input(frame, schema) is None
+assert not _pandas_input.accepts_pandas_table(frame, schema)
+class Factory:
+    def mktemp(self, name):
+        return Path(tempfile.mkdtemp(prefix=name))
+protocol = capture.protocol.__wrapped__(Factory())
+tables = []
+for row_control in (False, True):
+    with pytest.MonkeyPatch.context() as control:
+        if row_control:
+            capture._force_native_pandas_rows(control)
+        with capture._arrow_input_service(protocol) as (spark, state):
+            spark.createDataFrame(frame, schema).toArrow()
+            assert state["config_requests"] == []
+            assert state["uploads"] == 0
+            assert json.loads(state["spark_schema"]) == 
json.loads(schema.json())
+            tables.append(state["input_tables"][0])
+assert tables[0].column(0).to_pylist() == texts
+assert tables[0].equals(tables[1], check_metadata=True)
+'''
+    result = subprocess.run([sys.executable, "-c", script, scope, class_name, 
__file__],
+                            capture_output=True, text=True, timeout=60)
+    assert result.returncode == 0, result.stdout + result.stderr
+
+
[email protected]("storage", ["pyarrow", "numpy_semantics"])
[email protected]("schema_kind", ["explicit", "inferred", "names"])
[email protected]("instance_hook", [False, True])
+def test_arrow_string_custom_arrow_hook_keeps_native_row_values(
+    protocol, monkeypatch, instance_hook, schema_kind, storage
+):
+    import importlib
+
+    _pandas_input = importlib.import_module("pyspark.sql._pandas_input")
+
+    texts = ["original", None] * 5000
+    # The first column authenticates this class before the second instance's
+    # override is checked; every instance must still be inspected.
+    frame = pd.DataFrame({name: _arrow_string_column(texts, storage)
+                          for name in ("first", "v")})
+    schema = T.StructType([T.StructField(name, T.StringType()) for name in 
("first", "v")])
+    input_schema = {"explicit": schema, "inferred": None, "names": ["first", 
"v"]}[schema_kind]
+    array = frame.iloc[:, 1].array
+    calls = []
+
+    def changed_arrow(*args, **kwargs):
+        calls.append("arrow")
+        return pa.array(["changed"] * len(texts), type=pa.string())
+
+    monkeypatch.setattr(array if instance_hook else type(array), 
"__arrow_array__", changed_arrow)
+    assert _pandas_input.prepare_string_input(frame, schema) is None
+    assert not _pandas_input.accepts_pandas_table(frame, schema)
+    assert not _pandas_input.accepts_pandas_frame(frame)
+    assert calls == []
+    tables = []
+    for row_control in (False, True):
+        with monkeypatch.context() as control:
+            if row_control:
+                _force_native_pandas_rows(control)
+            with _arrow_input_service(protocol, config_error=True) as (spark, 
state):
+                spark.createDataFrame(frame, input_schema).toArrow()
+                tables.append(state["input_tables"][0])
+                assert state["config_requests"] == []
+                assert state["uploads"] == 0
+    assert calls == []
+    assert tables[0].to_pydict() == {"first": texts, "v": texts}
+    assert tables[0].equals(tables[1], check_metadata=True)
+
+
[email protected]("schema_kind", ["explicit", "inferred", "names"])
[email protected]("customization", [
+    "mixin_hook", "mixin_alias", "partial_alias", "mixin_storage", 
"mixin_dtype",
+])
+def test_arrow_string_numpy_semantics_accessor_keeps_native_outcomes(
+    protocol, monkeypatch, customization, schema_kind
+):
+    import importlib
+    import json
+    import traceback
+    from pandas.core.arrays import _arrow_string_mixins as string_mixins, 
string_arrow
+    from pyspark.sql.pandas.types import from_arrow_schema
+
+    helper = importlib.import_module("pyspark.sql._pandas_input")
+    texts = ["original", None] * 5000
+    frames = [pd.DataFrame({"v": _arrow_string_column(texts, 
"numpy_semantics")})
+              for _ in range(2)]
+    for frame in frames:
+        frame.columns = pd.Index(["v"], dtype=object)
+    schema = T.StructType([T.StructField("v", T.StringType())])
+    input_schema = {"explicit": schema, "inferred": None, "names": 
["v"]}[schema_kind]
+    array_class = type(frames[0].iloc[:, 0].array)
+    dynamic_accessor = (helper._raw_class_attribute(array_class, 
"__getattribute__")[1]
+                        is not object.__getattribute__)
+    # These warm-ups populate Block.dtype so mixin_dtype reaches class 
authentication
+    # instead of raising during the earlier native-compatible frame.dtypes 
read.
+    for frame in frames:
+        assert helper.prepare_string_input(frame, schema) is not None
+        assert helper.accepts_pandas_table(frame, schema)
+        assert helper.accepts_pandas_frame(frame)
+    calls = []
+
+    def changed_arrow(*args, **kwargs):
+        calls.append("arrow")
+        return pa.array(["changed"] * len(texts), type=pa.string())
+
+    def changed_partial(*args, **kwargs):
+        calls.append("partial")
+        raise AssertionError("eligibility must not invoke the rebound partial 
alias")
+
+    def changed_attribute(*args, **kwargs):
+        calls.append(customization)
+        raise AssertionError("the mixin state callback must not be called")
+
+    if customization == "mixin_hook":
+        monkeypatch.setattr(string_mixins.ArrowStringArrayMixin, 
"__arrow_array__",
+                            changed_arrow, raising=False)
+    elif customization == "mixin_alias":
+        class ChangedMixin:
+            __arrow_array__ = changed_arrow
+
+        monkeypatch.setattr(string_arrow, "ArrowStringArrayMixin", 
ChangedMixin, raising=False)
+    elif customization in ("mixin_storage", "mixin_dtype"):
+        name = {"mixin_storage": "_storage", "mixin_dtype": 
"_dtype"}[customization]
+        monkeypatch.setattr(string_mixins.ArrowStringArrayMixin, name,
+                            changed_attribute, raising=False)
+    else:
+        monkeypatch.setattr(string_arrow, "partial", changed_partial, 
raising=False)
+    for frame in frames:
+        # pandas 2.2 dispatches instance lookups through these mutable aliases.
+        # Later layouts use object access and remain unaffected by these 
changes.
+        assert (helper.prepare_string_input(frame, schema) is None) == 
dynamic_accessor
+        assert helper.accepts_pandas_table(frame, schema) == (not 
dynamic_accessor)
+        assert helper.accepts_pandas_frame(frame) == (not dynamic_accessor)
+        assert calls == []
+    logical_tables = []
+    outcomes = []
+    for row_control, frame in zip((True, False), frames):
+        with monkeypatch.context() as control:
+            if row_control:
+                _force_native_pandas_rows(control)
+            with _arrow_input_service(protocol, config_error=dynamic_accessor) 
as (spark, state):
+                # The mock returns only a row count, not the uploaded data or 
resolved schema.
+                try:
+                    reply = spark.createDataFrame(frame, 
input_schema).toArrow()
+                except Exception as error:
+                    assert dynamic_accessor and customization in 
("mixin_storage", "mixin_dtype")
+                    assert not isinstance(error, AssertionError)
+                    assert any(
+                        trace_frame.f_globals.get("__name__", 
"").startswith("pandas.core.arrays.")
+                        for trace_frame, _ in 
traceback.walk_tb(error.__traceback__)
+                    )
+                    assert state["input_tables"] == []
+                    assert state["config_requests"] == []
+                    assert state["uploads"] == 0
+                    outcomes.append((type(error), str(error)))
+                    continue
+                outcomes.append(None)
+                assert reply.to_pydict() == {"rows": [len(texts)]}
+                assert len(state["input_tables"]) == 1
+                uploaded = state["input_tables"][0]
+                assert uploaded.num_rows == len(texts)
+                assert uploaded.num_columns == 1
+                assert uploaded.column(0).to_pylist() == texts
+                assert uploaded.column(0).is_null().to_pylist() == [False, 
True] * 5000
+                assert state["column_names"] == ["v"]
+                if row_control or dynamic_accessor or schema_kind == 
"explicit":
+                    expected_wire = pa.schema([pa.field("v", pa.string())])
+                    assert json.loads(state["spark_schema"]) == 
json.loads(schema.json())
+                else:
+                    # Inherited inferred Arrow conversion sends _n fields 
without a
+                    # declared schema; Spark infers large_string as StringType 
and toDF
+                    # applies the actual captured column names above.
+                    expected_wire = pa.schema([pa.field("_0", 
pa.large_string())])
+                    assert state["spark_schema"] is None
+                assert uploaded.schema.equals(expected_wire, 
check_metadata=True)
+                if row_control or dynamic_accessor:
+                    assert state["config_requests"] == []
+                    assert state["uploads"] == 0
+                else:
+                    assert state["config_requests"]
+                # Compare logical uploaded data only after checking the exact 
wire
+                # schema and rename plan. This is not a real server schema 
assertion.
+                renamed = uploaded.rename_columns(state["column_names"])
+                assert from_arrow_schema(renamed.schema) == schema
+                logical_tables.append(renamed.cast(pa.schema([pa.field("v", 
pa.string())])))
+    assert calls == []
+    assert outcomes[0] == outcomes[1]
+    if dynamic_accessor and customization in ("mixin_storage", "mixin_dtype"):
+        assert outcomes[0] is not None
+    if logical_tables:
+        assert len(logical_tables) == 2
+        assert logical_tables[0].equals(logical_tables[1], check_metadata=True)
+
+
[email protected]("route", ["specialized", "generic", "inferred"])
+def test_arrow_string_guards_recheck_hooks_between_columns(monkeypatch, route):
+    import importlib
+
+    _pandas_input = importlib.import_module("pyspark.sql._pandas_input")
+    texts = ["original", None] * 5000
+    frame = pd.DataFrame({name: pd.array(texts, dtype="string[pyarrow]")
+                          for name in ("first", "second")})
+    schema = T.StructType([T.StructField(name, T.StringType())
+                           for name in ("first", "second")])
+    array_class = type(frame.iloc[:, 0].array)
+    original_iloc = pd.DataFrame.iloc
+    lookups = []
+    calls = []
+
+    def customized(*args, **kwargs):
+        calls.append("__getitem__")
+        raise AssertionError("eligibility must not run the customized row 
hook")
+
+    class RebindingIndexer:
+        def __init__(self, indexer):
+            self.indexer = indexer
+
+        def __getitem__(self, key):
+            column = self.indexer[key]
+            lookups.append(key[1])
+            if key[1] == 1:
+                monkeypatch.setattr(array_class, "__getitem__", customized)
+            return column
+
+    def iloc(self):
+        indexer = original_iloc.fget(self)
+        return RebindingIndexer(indexer) if self is frame else indexer
+
+    # Both guards allow this accessor today. It returns ordinary Series while
+    # rebinding a hook after the first column, invalidating any per-class 
cache.
+    monkeypatch.setattr(pd.DataFrame, "iloc", property(iloc))
+    if route == "specialized":
+        assert _pandas_input.prepare_string_input(frame, schema) is None
+    elif route == "generic":
+        assert not _pandas_input.accepts_pandas_table(frame, schema)
+    else:
+        assert not _pandas_input.accepts_pandas_frame(frame)
+    assert lookups == [0, 1]
+    assert calls == []
+
+
+def test_arrow_string_subclass_guard_does_not_hash_custom_metaclass():
+    from pyspark.sql._pandas_input import _arrow_string_class_is_canonical
+
+    calls = []
+
+    class CustomMeta(type(pd.arrays.ArrowStringArray)):
+        def __hash__(cls):
+            calls.append("class hash")
+            raise AssertionError("declining a subclass must not run its hash 
hook")
+
+    class Custom(pd.arrays.ArrowStringArray, metaclass=CustomMeta):
+        pass
+
+    assert not _arrow_string_class_is_canonical(Custom)
+    assert calls == []
+
+
+def test_arrow_string_generic_subclass_guard_does_not_read_custom_mro():
+    from pyspark.sql._pandas_input import accepts_pandas_table
+
+    calls = []
+    armed = False
+
+    class CustomMeta(type(pd.arrays.ArrowStringArray)):
+        @property
+        def __mro__(cls):
+            if armed:
+                calls.append("class mro")
+                raise AssertionError("declining a subclass must not read its 
custom MRO")
+            return type.__dict__["__mro__"].__get__(cls, type(cls))
+
+    class Custom(pd.arrays.ArrowStringArray, metaclass=CustomMeta):
+        pass
+
+    # Finish ordinary pandas construction before arming the metaclass callback.
+    frame = pd.DataFrame({"v": Custom(pa.chunked_array([pa.array(["a", 
None])]))})
+    assert type(frame.iloc[:, 0].array) is Custom
+    schema = T.StructType([T.StructField("v", T.StringType())])
+    armed = True
+    assert not accepts_pandas_table(frame, schema)
+    assert calls == []
+
+
+def _arrow_string_allocator_process(case, schema_kind="explicit", 
storage="pyarrow"):
+    """Keep CPython allocator-slot changes out of the main pytest 
interpreter."""
+    import signal
+    import sys
+
+    script = r'''
+import importlib.util
+import sys
+import tempfile
+from pathlib import Path
+import pytest
+
+spec = 
importlib.util.spec_from_file_location("arrow_string_allocator_capture", 
sys.argv[4])
+capture = importlib.util.module_from_spec(spec)
+sys.modules[spec.name] = capture
+spec.loader.exec_module(capture)
+case, schema_kind, storage = sys.argv[1:4]
+with pytest.MonkeyPatch.context() as control:
+    if case == "row":
+        with tempfile.TemporaryDirectory(prefix="arrow-string-allocator-") as 
directory:
+            class Factory:
+                def mktemp(self, name):
+                    path = Path(directory) / name
+                    path.mkdir()
+                    return path
+            protocol = capture.protocol.__wrapped__(Factory())
+            capture._check_arrow_string_custom_row_hook(
+                protocol, control, "__new__", schema_kind, storage)
+    else:
+        assert case == "descriptor"
+        capture._check_arrow_string_class_descriptor(control, "__new__")
+'''
+    owned_group = os.name == "posix"
+    process = subprocess.Popen(
+        [sys.executable, "-c", script, case, schema_kind, storage, 
str(Path(__file__).resolve())],
+        stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True,
+        start_new_session=owned_group,
+    )
+
+    def terminate_child():
+        try:
+            if owned_group:
+                # protoc inherits this session, so a timeout also cleans up 
that child.
+                os.killpg(process.pid, signal.SIGKILL)
+            else:
+                process.kill()
+        except ProcessLookupError:
+            pass
+
+    try:
+        stdout, stderr = process.communicate(timeout=60)
+    except subprocess.TimeoutExpired:
+        terminate_child()
+        try:
+            stdout, stderr = process.communicate(timeout=5)
+        except subprocess.TimeoutExpired:
+            stdout, stderr = "", ""
+        pytest.fail("Arrow-string allocator child timed out\n" + stdout + 
stderr)
+    except BaseException:
+        terminate_child()
+        try:
+            process.communicate(timeout=5)
+        except subprocess.TimeoutExpired:
+            pass
+        raise
+    assert process.returncode == 0, stdout + stderr
+    # Guard against accidentally reintroducing in-process allocator mutation.
+    healthy = pd.array(["original", None], dtype="string[pyarrow]")
+    assert healthy.__arrow_array__().to_pylist() == ["original", None]
+
+
[email protected]("storage", ["pyarrow", "numpy_semantics"])
[email protected]("schema_kind", ["explicit", "inferred", "names"])
[email protected]("method", [
+    "astype", "isna", "to_numpy", "__getitem__", "dtype", "na_value", 
"_hasna", "__init__",
+    "_from_pyarrow_array", "__new__",
+])
+def test_arrow_string_custom_row_hooks_keep_native_errors(
+    protocol, monkeypatch, method, schema_kind, storage
+):
+    if method == "__new__":
+        # Other hooks use this shared fixture signature; the child needs its 
own protocol.
+        _arrow_string_allocator_process("row", schema_kind, storage)
+    else:
+        _check_arrow_string_custom_row_hook(protocol, monkeypatch, method, 
schema_kind, storage)
+
+
+def _check_arrow_string_custom_row_hook(
+    protocol, monkeypatch, method, schema_kind, storage
+):
+    import importlib
+
+    _pandas_input = importlib.import_module("pyspark.sql._pandas_input")
+
+    texts = ["original", None] * 5000
+    frames = []
+    for _ in range(2):
+        frame = pd.DataFrame({"v": _arrow_string_column(texts, storage)})
+        # Object labels isolate the additional construction/_hasna data hooks.
+        # The other Arrow labels exercise earlier checks when they share the 
data
+        # class (pyarrow storage); a separate NumPy-semantics class is checked 
later.
+        frame.columns = pd.Index(["v"], dtype=object) if method in (
+            "_hasna", "__init__", "_from_pyarrow_array", "__new__",
+        ) else pd.Index(
+            pd.array(["v"], dtype="string[pyarrow]"))
+        frames.append(frame)
+    schema = T.StructType([T.StructField("v", T.StringType())])
+    input_schema = {"explicit": schema, "inferred": None, "names": 
["v"]}[schema_kind]
+    array_class = type(frames[0].iloc[:, 0].array)
+    calls = []
+    # Warm all guards before rebinding: class authentication must not be cached
+    # across calls, even when this exact class was previously accepted. This 
also
+    # populates pandas' Block.dtype cache, so a customized data dtype reaches 
the
+    # class check; the separate cold-dtype test covers its earlier native 
error.
+    for frame in frames:
+        assert _pandas_input.prepare_string_input(frame, schema) is not None
+        assert _pandas_input.accepts_pandas_table(frame, schema)
+        assert _pandas_input.accepts_pandas_frame(frame)
+
+    class HookError(Exception):
+        pass
+
+    def customized(*args, **kwargs):
+        if method == "na_value":
+            # This property receives a StringDtype, including during native 
label iteration.
+            assert type(args[0]) is pd.StringDtype
+        elif method == "__new__":
+            # pandas may pass only the class; no initialized receiver is 
available.
+            assert args[0] is array_class
+        elif method in ("__init__", "_from_pyarrow_array"):
+            # __init__ receives an uninitialized object; inspect the subset 
instead.
+            assert type(args[0]) is array_class
+            assert len(args) == 2
+            values = args[1]
+            assert type(values) is pa.ChunkedArray
+            assert values.null_count == 0
+            assert values.to_pylist() == ["original"] * 5000
+            assert set(kwargs) <= {"dtype"}
+            if kwargs:
+                assert type(kwargs["dtype"]) is pd.StringDtype
+        else:
+            assert len(args[0]) == 10000  # Labels have length one; guards 
must not cast them.
+        calls.append(method)
+        raise HookError(f"{method} sentinel")
+
+    class FactoryHook:
+        def __get__(self, receiver, owner):
+            assert receiver is not None, "eligibility must not bind the 
factory descriptor"
+            assert owner is array_class
+            return customized.__get__(receiver, owner)
+
+    if method == "na_value":
+        monkeypatch.setattr(pd.StringDtype, method, property(customized))
+    elif method == "_from_pyarrow_array":
+        # An absent helper on older pandas is also a conservative decline.
+        monkeypatch.setattr(array_class, method, FactoryHook(), raising=False)
+    elif method == "__new__":
+        monkeypatch.setattr(array_class, method, staticmethod(customized))
+    elif method in ("dtype", "_hasna"):
+        monkeypatch.setattr(array_class, method, property(customized))
+    else:
+        monkeypatch.setattr(array_class, method, customized)
+    for frame in frames:
+        assert _pandas_input.prepare_string_input(frame, schema) is None
+        assert not _pandas_input.accepts_pandas_table(frame, schema)
+        assert not _pandas_input.accepts_pandas_frame(frame)
+        assert calls == []
+
+    outcomes = []
+    tables = []
+    for row_control, frame in zip((True, False), frames):
+        calls.clear()
+        with monkeypatch.context() as control:
+            if row_control:
+                _force_native_pandas_rows(control)
+            with _arrow_input_service(protocol, config_error=True) as (spark, 
state):
+                try:
+                    result = spark.createDataFrame(frame, input_schema)
+                    result.toArrow()
+                except HookError as error:
+                    assert type(error) is HookError
+                    assert str(error) == f"{method} sentinel"
+                    if method in ("__init__", "_from_pyarrow_array", 
"__new__"):
+                        assert calls
+                    else:
+                        assert calls == [method]
+                    outcome = (type(error), str(error))
+                    assert state["input_tables"] == []
+                else:
+                    # pandas versions use different subset-construction 
helpers.
+                    assert method in ("__init__", "_from_pyarrow_array", 
"__new__")
+                    assert len(state["input_tables"]) == 1
+                    table = state["input_tables"][0]
+                    assert table.column(0).to_pylist() == texts
+                    tables.append(table)
+                    outcome = None
+                outcomes.append((outcome, tuple(calls)))
+                assert state["config_requests"] == []
+                assert state["uploads"] == 0
+    assert outcomes[0] == outcomes[1]
+    if tables:
+        assert len(tables) == 2
+        assert tables[0].equals(tables[1], check_metadata=True)
+
+
[email protected]("method", [
+    "__arrow_array__", "__array__", "__getitem__", "isna", "to_numpy", 
"__len__", "astype",
+    "__init__", "_from_pyarrow_array", "__new__", "__getattribute__", "dtype", 
"_hasna", "na_value",
+])
+def test_arrow_string_class_descriptors_decline_without_binding(monkeypatch, 
method):
+    if method == "__new__":
+        _arrow_string_allocator_process("descriptor")
+    else:
+        _check_arrow_string_class_descriptor(monkeypatch, method)
+
+
+def _check_arrow_string_class_descriptor(monkeypatch, method):
+    import importlib
+
+    _pandas_input = importlib.import_module("pyspark.sql._pandas_input")
+    array = pd.array(["original", None], dtype="string[pyarrow]")
+    array_class = type(array)
+    assert _pandas_input._arrow_string_class_is_canonical(array_class)
+    assert _pandas_input._has_canonical_arrow_string_array(array)
+    calls = []
+
+    class HookError(Exception):
+        pass
+
+    class CustomizedDescriptor:
+        def __get__(self, receiver, owner):
+            calls.append((receiver, owner))
+            raise HookError(f"{method} descriptor sentinel")
+
+    owner = pd.StringDtype if method == "na_value" else array_class
+    monkeypatch.setattr(owner, method, CustomizedDescriptor(), raising=False)
+    # These concrete-class checks inspect raw dictionaries. Earlier 
frame.dtypes
+    # reads are covered separately because they intentionally keep native 
ordering.
+    assert not _pandas_input._arrow_string_class_is_canonical(array_class)
+    assert not _pandas_input._has_canonical_arrow_string_array(array)
+    assert calls == []
+
+
[email protected]("method", [
+    "__getattribute__", "tolist", "__iter__", "fillna", "to_numpy", "isna", 
"_hasna",
+    "dtype", "na_value",
+])
+def test_arrow_string_label_descriptors_decline_without_binding(monkeypatch, 
method):
+    import importlib
+
+    _pandas_input = importlib.import_module("pyspark.sql._pandas_input")
+    # Concrete object data isolates the Arrow-string class used by the labels.
+    frame = pd.DataFrame({"v": pd.Series(["original", None] * 5000, 
dtype=object)})
+    frame.columns = pd.Index(pd.array(["v"], dtype="string[pyarrow]"))
+    array_class = type(frame.columns._values)
+    schema = T.StructType([T.StructField("v", T.StringType())])
+    assert _pandas_input.prepare_string_input(frame, schema) is not None
+    assert _pandas_input.accepts_pandas_table(frame, schema)
+    assert _pandas_input.accepts_pandas_frame(frame)
+    calls = []
+
+    class HookError(Exception):
+        pass
+
+    class CustomizedDescriptor:
+        def __get__(self, receiver, owner):
+            calls.append((receiver, owner))
+            raise HookError(f"label {method} descriptor sentinel")
+
+    owner = pd.StringDtype if method == "na_value" else array_class
+    monkeypatch.setattr(owner, method, CustomizedDescriptor())
+    assert _pandas_input.prepare_string_input(frame, schema) is None
+    assert not _pandas_input.accepts_pandas_table(frame, schema)
+    assert not _pandas_input.accepts_pandas_frame(frame)
+    assert calls == []
+
+
[email protected]("storage_kind", [
+    "plain_array", "nonstring", "invalid_utf8", "bad_storage", "spoofed_array",
+    "short_storage", "long_storage",
+])
+def test_arrow_string_storage_normalization_and_fallback(monkeypatch, 
storage_kind):
+    import importlib
+    import struct
+
+    _pandas_input = importlib.import_module("pyspark.sql._pandas_input")
+    texts = ["é", None]
+    frame = pd.DataFrame({"v": pd.array(texts, dtype="string[pyarrow]")})
+    schema = T.StructType([T.StructField("v", T.StringType())])
+    # Capture pandas metadata before intentionally replacing its Arrow storage.
+    dtypes = frame.dtypes
+    array = frame.iloc[:, 0].array
+    if storage_kind == "plain_array":
+        storage = pa.array(texts, type=pa.string())
+        assert isinstance(storage, pa.Array)
+    elif storage_kind == "nonstring":
+        storage = pa.chunked_array([pa.array([1, None], type=pa.int64())])
+    elif storage_kind in ("short_storage", "long_storage"):
+        count = 1 if storage_kind == "short_storage" else 3
+        storage = pa.chunked_array([pa.array(["é"] * count, type=pa.string())])
+        assert len(storage) != len(frame)
+    elif storage_kind == "invalid_utf8":
+        invalid = pa.Array.from_buffers(pa.string(), 2, [
+            None, pa.py_buffer(struct.pack("<iii", 0, 5, 6)), 
pa.py_buffer(b"valid\xff"),
+        ])
+        storage = pa.chunked_array([invalid])
+        with pytest.raises(pa.ArrowInvalid):
+            storage.validate(full=True)
+    elif storage_kind == "bad_storage":
+        class UnsupportedStorage:
+            # pandas' column boxing reads length before the storage-type guard.
+            def __len__(self):
+                return len(texts)
+
+        storage = UnsupportedStorage()
+        assert not isinstance(storage, (pa.Array, pa.ChunkedArray))
+    else:
+        class SpoofedArray:
+            @property
+            def __class__(self):
+                return pa.Array
+
+            def __len__(self):
+                return len(texts)
+
+        storage = SpoofedArray()
+        assert isinstance(storage, pa.Array)
+        assert not issubclass(type(storage), pa.Array)
+    monkeypatch.setattr(array, "_pa_array", storage)
+    assert object.__getattribute__(frame.iloc[:, 0].array, "_pa_array") is 
storage
+    result = _pandas_input._arrow_string_table(frame, schema, dtypes)
+    if storage_kind == "plain_array":
+        assert type(result) is pa.Table
+        result.validate(full=True)
+        assert result.schema.names == ["v"]
+        assert result.column(0).type == pa.string()
+        assert result.column(0).to_pylist() == texts
+    else:
+        assert result is None
+
+
[email protected]("error_kind", ["arrow", "overflow"])
+def test_arrow_string_simulated_cast_failures_decline(monkeypatch, error_kind):
+    """Exercise exception handling without allocating a greater-than-2-GiB 
column."""
+    import importlib
+    import sys
+    from types import ModuleType
+
+    _pandas_input = importlib.import_module("pyspark.sql._pandas_input")
+    frame = pd.DataFrame({"v": pd.array(["original", None], 
dtype="string[pyarrow]")})
+    schema = T.StructType([T.StructField("v", T.StringType())])
+    dtypes = frame.dtypes
+    array = frame.iloc[:, 0].array
+    values = array.__arrow_array__()
+    error_class = pa.ArrowInvalid if error_kind == "arrow" else OverflowError
+    calls = []
+
+    class SimulatedCastFailure:
+        type = values.type
+
+        def __len__(self):
+            return len(values)
+
+        def validate(self, *, full):
+            calls.append(("validate", full))
+            values.validate(full=full)
+
+        def cast(self, target):
+            assert target == pa.string()
+            calls.append(("cast", target))
+            raise error_class("simulated Arrow-string cast failure")
+
+    # PyArrow extension types are immutable. This local module proxy 
substitutes
+    # the exact recognized storage class only for this simulated cast branch;
+    # no real ChunkedArray subclass or capacity behavior is being tested.
+    proxy = ModuleType("pyarrow")
+    vars(proxy).update(vars(pa))
+    proxy.ChunkedArray = SimulatedCastFailure
+    with monkeypatch.context() as simulated:
+        simulated.setattr(array, "_pa_array", SimulatedCastFailure())
+        simulated.setitem(sys.modules, "pyarrow", proxy)
+        assert _pandas_input._arrow_string_table(frame, schema, dtypes) is None
+    assert calls == [("validate", True), ("cast", pa.string())]
+
+
+def test_arrow_string_object_labels_dtype_error_matches_native_rows(protocol, 
monkeypatch):
+    import importlib
+
+    _pandas_input = importlib.import_module("pyspark.sql._pandas_input")
+    frames = []
+    for _ in range(3):
+        frame = pd.DataFrame({"v": pd.array(["original", None] * 5000,
+                                          dtype="string[pyarrow]")})
+        frame.columns = pd.Index(["v"], dtype=object)
+        frames.append(frame)
+    schema = T.StructType([T.StructField("v", T.StringType())])
+    array_class = type(frames[0].iloc[:, 0].array)
+    calls = []
+
+    class HookError(Exception):
+        pass
+
+    def customized(self):
+        assert type(self) is array_class
+        assert len(self) == 10000
+        calls.append("dtype")
+        raise HookError("dtype sentinel")
+
+    monkeypatch.setattr(array_class, "dtype", property(customized))
+    # The generic route can authenticate the data class before reading its 
dtype.
+    assert not _pandas_input.accepts_pandas_table(frames[0], schema)
+    assert calls == []
+    errors = []
+    with pytest.raises(HookError, match="^dtype sentinel$") as error:
+        _pandas_input.prepare_string_input(frames[0], schema)
+    errors.append((type(error.value), str(error.value)))
+    assert calls == ["dtype"]
+    for row_control, frame in zip((False, True), frames[1:]):
+        calls.clear()
+        with monkeypatch.context() as control:
+            if row_control:
+                _force_native_pandas_rows(control)
+            with _arrow_input_service(protocol, config_error=True) as (spark, 
state):
+                with pytest.raises(HookError, match="^dtype sentinel$") as 
error:
+                    spark.createDataFrame(frame, schema)
+                errors.append((type(error.value), str(error.value)))
+                assert state["config_requests"] == []
+                assert state["uploads"] == 0
+                assert state["input_tables"] == []
+        assert calls == ["dtype"]
+    assert errors[0] == errors[1] == errors[2]
+
+
+def test_arrow_string_custom_label_astype_keeps_native_row_values(protocol, 
monkeypatch):
+    import importlib
+
+    _pandas_input = importlib.import_module("pyspark.sql._pandas_input")
+
+    texts = ["original", None] * 5000
+    frame = pd.DataFrame({"v": pd.array(texts, dtype="string[pyarrow]")})
+    frame.columns = pd.Index(pd.array(["v"], dtype="string[pyarrow]"))
+    schema = T.StructType([T.StructField("v", T.StringType())])
+    calls = []
+
+    def customized(*args, **kwargs):
+        calls.append("label astype")
+        raise AssertionError("native row conversion does not cast column 
labels")
+
+    monkeypatch.setattr(frame.columns._values, "astype", customized)
+    assert _pandas_input.prepare_string_input(frame, schema) is None
+    assert not _pandas_input.accepts_pandas_table(frame, schema)
+    assert calls == []
+    tables = []
+    for row_control in (False, True):
+        with monkeypatch.context() as control:
+            if row_control:
+                _force_native_pandas_rows(control)
+            with _arrow_input_service(protocol, config_error=True) as (spark, 
state):
+                spark.createDataFrame(frame, schema).toArrow()
+                tables.append(state["input_tables"][0])
+                assert state["config_requests"] == []
+                assert state["uploads"] == 0
+    assert calls == []
+    assert tables[0].column(0).to_pylist() == texts
+    assert tables[0].equals(tables[1], check_metadata=True)
+
+
+def test_arrow_string_null_labels_decline_without_getitem_callback(protocol, 
monkeypatch):
+    import importlib
+
+    from pandas.core.arrays.arrow.array import ArrowExtensionArray
+
+    _pandas_input = importlib.import_module("pyspark.sql._pandas_input")
+    # Index uniqueness can cast null labels and invoke indexed masking. Native
+    # normalization may also slice labels while aligning DataFrame.where; 
guards
+    # must decline without an early callback and preserve that native outcome.
+    expected = {"a": ["first"] * 10000, "b": ["second"] * 10000}
+    frames = []
+    for _ in range(2):
+        frame = pd.DataFrame({name: pd.array(values, dtype="string[pyarrow]")
+                              for name, values in expected.items()})
+        frame.columns = pd.Index(pd.array(["a", None], 
dtype="string[pyarrow]"))
+        frames.append(frame)
+    schema = T.StructType([T.StructField(name, T.StringType()) for name in 
expected])
+    calls = []
+
+    class HookError(Exception):
+        pass
+
+    def customized(*args, **kwargs):
+        calls.append("label getitem")
+        raise HookError("label getitem sentinel")
+
+    monkeypatch.setattr(ArrowExtensionArray, "__getitem__", customized)
+    for frame in frames:
+        assert _pandas_input.prepare_string_input(frame, schema) is None
+        assert not _pandas_input.accepts_pandas_table(frame, schema)
+        assert calls == []
+
+    outcomes = []
+    tables = []
+    # Observe native behavior first, using a fresh equal frame for each route.
+    for row_control, frame in zip((True, False), frames):
+        calls.clear()
+        with monkeypatch.context() as control:
+            if row_control:
+                _force_native_pandas_rows(control)
+            with _arrow_input_service(protocol, config_error=True) as (spark, 
state):
+                try:
+                    spark.createDataFrame(frame, schema).toArrow()
+                except HookError as error:
+                    assert type(error) is HookError
+                    assert str(error) == "label getitem sentinel"
+                    assert calls
+                    outcome = (type(error), str(error))
+                    assert state["input_tables"] == []
+                else:
+                    assert len(state["input_tables"]) == 1
+                    table = state["input_tables"][0]
+                    assert table.to_pydict() == expected
+                    tables.append(table)
+                    outcome = None
+                outcomes.append((outcome, tuple(calls)))
+                assert state["config_requests"] == []
+                assert state["uploads"] == 0
+    assert outcomes[0] == outcomes[1]
+    if tables:
+        assert len(tables) == 2
+        assert tables[0].equals(tables[1], check_metadata=True)
+
+
[email protected]("label_storage", ["object", "pyarrow"])
[email protected]("method", ["astype", "__array__", "__getitem__", 
"__init__"])
[email protected]("storage", ["pyarrow", "numpy_semantics"])
+def test_arrow_string_super_row_hooks_keep_native_errors(
+    protocol, monkeypatch, label_storage, method, storage
+):
+    import importlib
+
+    from pandas.core.arrays.arrow.array import ArrowExtensionArray
+
+    _pandas_input = importlib.import_module("pyspark.sql._pandas_input")
+    frames = []
+    for _ in range(2):
+        frame = pd.DataFrame({"v": _arrow_string_column(["original", None] * 
5000, storage)})
+        frame.columns = pd.Index(["v"], dtype=object) if label_storage == 
"object" else pd.Index(
+            pd.array(["v"], dtype="string[pyarrow]"))
+        frames.append(frame)
+    schema = T.StructType([T.StructField("v", T.StringType())])
+    calls = []
+    for frame in frames:
+        assert _pandas_input.prepare_string_input(frame, schema) is not None
+        assert _pandas_input.accepts_pandas_table(frame, schema)
+        assert _pandas_input.accepts_pandas_frame(frame)
+
+    class HookError(Exception):
+        pass
+
+    def customized(self, *args, **kwargs):
+        if method == "__init__":
+            # The receiver has no storage yet. Null-subarray construction 
passes
+            # the 5,000 non-null data values; labels have only one value.
+            values = args[0]
+            assert values.to_pylist() == ["original"] * 5000
+        else:
+            assert len(self) == 10000  # Preflight must not cast the 
length-one labels.
+        calls.append(method)
+        raise HookError(f"super {method} sentinel")
+
+    monkeypatch.setattr(ArrowExtensionArray, method, customized)
+    for frame in frames:
+        assert _pandas_input.prepare_string_input(frame, schema) is None
+        assert not _pandas_input.accepts_pandas_table(frame, schema)
+        assert not _pandas_input.accepts_pandas_frame(frame)
+    assert calls == []
+    outcomes = []
+    tables = []
+    # The native control determines whether this pandas layout calls __init__.
+    # pandas 3.1's null-subarray factory instead allocates through __new__.
+    for row_control, frame in zip((True, False), frames):
+        calls.clear()
+        with monkeypatch.context() as control:
+            if row_control:
+                _force_native_pandas_rows(control)
+            with _arrow_input_service(protocol, config_error=True) as (spark, 
state):
+                try:
+                    spark.createDataFrame(frame, schema).toArrow()
+                except HookError as error:
+                    assert type(error) is HookError
+                    assert str(error) == f"super {method} sentinel"
+                    assert calls == [method]
+                    assert state["input_tables"] == []
+                    outcome = (type(error), str(error))
+                else:
+                    assert method == "__init__"
+                    assert calls == []
+                    assert len(state["input_tables"]) == 1
+                    table = state["input_tables"][0]
+                    table.validate(full=True)
+                    assert table.to_pydict() == {"v": ["original", None] * 
5000}
+                    assert table.schema.names == ["v"]
+                    assert table.schema.field(0).type == pa.string()
+                    assert table.schema.field(0).nullable
+                    tables.append(table)
+                    outcome = None
+                outcomes.append((outcome, tuple(calls)))
+                assert state["config_requests"] == []
+                assert state["uploads"] == 0
+    assert outcomes[0] == outcomes[1]
+    if tables:
+        assert len(tables) == 2
+        assert tables[0].equals(tables[1], check_metadata=True)
 
 
 @pytest.mark.parametrize("reason", [


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to