This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 46a8c3a  [SPARK-60676][CONNECT] Use Arrow columns for signed-int64 
pandas input
46a8c3a is described below

commit 46a8c3a48f1aebf48ffbbfa79a666493ebd1dc4a
Author: DB Tsai <[email protected]>
AuthorDate: Fri Nov 6 17:52:54 2026 +0900

    [SPARK-60676][CONNECT] Use Arrow columns for signed-int64 pandas input
    
    ### What changes were proposed in this pull request?
    
    `createDataFrame(pandas_df)` now converts a pandas DataFrame whose columns 
are all NumPy `int64` or pandas nullable `Int64` through Arrow: 
`python/pyspark/sql/_pandas_input.py` builds one Arrow `int64` array per column 
(with its null mask) and hands the table to the existing Arrow importer, 
instead of materializing Python rows and converting each value. The schema can 
be inferred, a list of names (applied with `toDF`, as in PySpark Connect), or a 
`StructType` of `LongType` fields, who [...]
    
    Everything else keeps the row path: other dtypes, DDL and atomic schemas, 
schemas that convert types, duplicate, non-string or NUL column names, 
subclasses of `DataFrame`, `Series` or the column arrays, and customized pandas 
or `pyspark.sql.types` classes. Frames of fewer than 1,000 rows with only NumPy 
columns keep the row path too, as the importer's configuration RPC costs more 
than their row conversion; nullable `Int64` frames use Arrow at any size.
    
    ### Why are the changes needed?
    
    [SPARK-60676](https://issues.apache.org/jira/browse/SPARK-60676): integer 
columns are the most common pandas input, and converting each value through 
Python rows dominates the cost of `createDataFrame` for them. It also never 
coerces a mixed frame to a common NumPy dtype, which can round integers above 
2^53. Measured on macOS against a local Spark 4.3.0 server, as the client CPU 
time of `createDataFrame` (median of 5):
    
    | Input | Master | This change |
    | --- | ---: | ---: |
    | 1,000,000 rows x 3 `int64` columns | 987 ms | 126 ms (-87%) |
    | 1,000,000 rows x 3 `Int64` columns with NAs | 1,087 ms | 135 ms (-88%) |
    | 5,000 rows x 3 `int64` columns | 5.6 ms | 0.8 ms (-86%) |
    
    ### Does this PR introduce _any_ user-facing change?
    
    No, other than speed. Against a Spark 4.3.0 server, `int64` and `Int64` 
frames (values above 2^53, the int64 bounds and NAs included) with inferred, 
list and `StructType` schemas give the same schemas, nullability and rows as 
official PySpark 4.3.0.
    
    ### How was this patch tested?
    
    - New tests in `python/tests/test_arrow_handoff.py`, against the mock Arrow 
input service, check the IPC values, null masks, names, metadata and 
nullability for each schema form, row counts up to 1,000,000, the 1,000-row 
threshold, that supported input materializes no Python rows, and that each 
unsupported or customized input keeps the row path (with the errors of the row 
path, before any RPC).
    - Against the current master, merged into this branch, and a local Spark 
4.3.0 server: the lib and every integration test of `apache-spark-connect` and 
`apache-spark-connect-core`, e2e included (775 passed), the build of all tests 
with `wasm-udf`, the offline Python suites (1,095 passed, 9 skipped), the live 
drop-in and Arrow suites (81 passed) and the live cases (10 passed). `cargo fmt 
--all --check` passes.
    - In the merge with master, a test that required a second row conversion to 
read `Index._data` exactly as often as the first failed: master's row path now 
reads it fewer times on later calls, as pandas caches some reads. It now checks 
that the Arrow path declines without more reads than the row path.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex (GPT-6); the merge with master, the review 
follow-ups and the measurement by Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #121 from dbtsai/codex/spark-60676-columnar-pandas.
    
    Lead-authored-by: DB Tsai <[email protected]>
    Co-authored-by: DB Tsai <[email protected]>
    Co-authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/session.rs    |  23 +
 docs/architecture.md                |  10 +
 python/pyspark/sql/_pandas_input.py | 278 ++++++++++++
 python/tests/test_arrow_handoff.py  | 836 +++++++++++++++++++++++++++++++++++-
 4 files changed, 1145 insertions(+), 2 deletions(-)

diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index dd8009f..8dd873b 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -812,6 +812,29 @@ impl PySparkSession {
         }
         let spec = schema_spec(py, schema)?;
         if is_loaded_instance(py, data, "pandas", "DataFrame")? {
+            // Guarded integer columns share the existing Arrow importer. 
Schema
+            // validation above and unsupported pandas conversions keep 
master's
+            // normal ordering and row fallback.
+            let table = py
+                .import("pyspark.sql._pandas_input")?
+                .getattr("prepare_int64_table")?
+                .call1((data, schema))?;
+            if !table.is_none() {
+                let frame =
+                    crate::pyarrow_import::create_dataframe(py, &self.session, 
&table, schema)?;
+                // pandas applies explicit field names with toDF even though 
the
+                // local relation already carries them. Empty input stays 
local.
+                return Ok(match &spec {
+                    Spec::Struct(fields) if !fields.is_empty() && table.len()? 
> 0 => {
+                        PyDataFrame::new(
+                            frame
+                                .dataframe
+                                .to_df(fields.iter().map(|field| 
field.name.as_str()).collect()),
+                        )
+                    }
+                    _ => frame,
+                });
+            }
             return self.create_from_pandas(py, data, spec);
         }
         // Preserve schema parsing, ignored-argument warnings, and 
pandas/Arrow routing.
diff --git a/docs/architecture.md b/docs/architecture.md
index 5cce878..425f131 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -142,6 +142,16 @@ timestamp conversion helpers normalize the table, then 
arrow-rs imports its
 those batches to IPC for the local relation sent to Spark. No Python rows are
 materialized; casts, buffer alignment, and IPC serialization may still copy.
 Empty tables require an explicit schema, matching PySpark Connect.
+Pandas inputs with only NumPy `int64` or pandas nullable `Int64` columns also
+use this importer, preserving integer precision and null masks without Python
+rows. Small frames (fewer than 1,000 rows) with only NumPy columns retain the
+row path to avoid the configuration RPC of the importer; nullable `Int64` 
columns
+use Arrow at any size.
+This initial path accepts inferred schemas, complete string column names,
+and explicit `StructType` schemas whose fields are all `LongType`. Other 
dtypes,
+DDL/atomic schemas, schema conversions, NUL names, and duplicate/non-string 
input names
+retain the existing row conversion path. `verifySchema` remains ignored for
+pandas Arrow input, matching Spark Connect.
 Inputs at or above `spark.sql.session.localRelationCacheThreshold` are 
serialized
 into independent IPC streams and uploaded as session-scoped cache artifacts.
 The plan carries `ChunkedCachedLocalRelation` hashes instead of embedding the
diff --git a/python/pyspark/sql/_pandas_input.py 
b/python/pyspark/sql/_pandas_input.py
new file mode 100644
index 0000000..a95f5c0
--- /dev/null
+++ b/python/pyspark/sql/_pandas_input.py
@@ -0,0 +1,278 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+
+"""Conservative columnar pandas input; unsupported conversions use the row 
path."""
+
+
+def _has_canonical_columns(columns):
+    """Decline label containers whose row-path iteration can run custom 
hooks."""
+    import sys
+    from types import FunctionType
+
+    import numpy as np
+    import pandas as pd
+    from pandas.core import base as base_module
+    from pandas.core.indexes import base as index_module
+
+    Index = index_module.Index
+    if (pd.Index is not Index or type(columns) is not Index
+            or Index.__getattribute__ is not object.__getattribute__
+            or "__iter__" in vars(columns)):
+        return False
+
+    def ordinary(method, name, module):
+        return (type(method) is FunctionType and method.__name__ == name
+                and method.__module__ == module.__name__
+                and method.__globals__ is vars(module))
+
+    # The native row path iterates columns before normalizing any cells.
+    # Decline overrides without invoking them, so fallback preserves that 
order.
+    if (Index.__iter__ is not base_module.IndexOpsMixin.__iter__
+            or not ordinary(Index.__iter__, "__iter__", base_module)):
+        return False
+    values = vars(Index).get("_values")
+    if (type(values) is not property
+            or not ordinary(values.fget, "_values", index_module)):
+        return False
+    if any("_data" in vars(base) for base in Index.__mro__):
+        return False
+    backing = vars(columns).get("_data")
+    if type(backing) is np.ndarray:
+        return True
+
+    # Arrow-backed labels already hold PyArrow storage. Do not import optional
+    # Arrow modules just to decide whether a fallback input is eligible.
+    pa = sys.modules.get("pyarrow")
+    if pa is None:
+        return False
+
+    try:
+        from pandas.core.arrays import numpy_ as numpy_module, string_ as 
string_module
+        from pandas.core.arrays import base as array_base_module, _mixins as 
mixins_module
+        from pandas._libs.arrays import NDArrayBacked
+        from pandas.core.arrays import string_arrow as string_arrow_module
+        from pandas.core.arrays.arrow import array as arrow_module
+    except ImportError:
+        # Older pandas releases may lack these concrete string storage types.
+        return False
+
+    python_strings = tuple(vars(string_module).get(name) for name in
+                           ("StringArray", "StringArrayNumpySemantics"))
+    arrow_strings = tuple(vars(string_arrow_module).get(name) for name in
+                          ("ArrowStringArray", 
"ArrowStringArrayNumpySemantics"))
+    cls = type(backing)
+    if not any(cls is candidate for candidate in python_strings + 
arrow_strings):
+        return False
+    if (cls.__getattribute__ is not object.__getattribute__
+            or any(name in vars(backing) for name in
+                   ("__iter__", "__getitem__", "__len__", "_box_func",
+                    "tolist", "to_numpy", "isna", "fillna", "_hasna"))
+            or getattr(cls, "tolist", None) is not getattr(
+                vars(string_module).get("BaseStringArray"), "tolist", None)
+            or not ordinary(getattr(cls, "tolist", None), "tolist", 
string_module)):
+        return False
+    if any(cls is candidate for candidate in python_strings):
+        owner, module = vars(numpy_module).get("NumpyExtensionArray"), 
numpy_module
+        string_array = vars(string_module).get("StringArray")
+        if owner is None or string_array is None:
+            return False
+        try:
+            storage = object.__getattribute__(backing, "_ndarray")
+        except AttributeError:
+            return False
+        if type(storage) is not np.ndarray:
+            return False
+        # ExtensionArray iteration dispatches through len and indexed boxing.
+        for name, hook_module in (("__iter__", array_base_module),
+                                  ("__getitem__", mixins_module), 
("_box_func", mixins_module)):
+            if not ordinary(getattr(cls, name, None), name, hook_module):
+                return False
+        if (getattr(cls, "__len__", None) is not NDArrayBacked.__len__
+                or getattr(cls, "ndim", None) is not NDArrayBacked.ndim
+                or any("_ndarray" in vars(base) for base in cls.__mro__
+                       if base is not NDArrayBacked)):
+            return False
+        isna = vars(string_array).get("isna", getattr(owner, "isna", None))
+        isna_module = string_module if "isna" in vars(string_array) else module
+    else:
+        owner, module = vars(arrow_module).get("ArrowExtensionArray"), 
arrow_module
+        if owner is None:
+            return False
+        try:
+            storage = object.__getattribute__(backing, "_pa_array")
+        except AttributeError:
+            return False
+        if (type(storage) is not pa.ChunkedArray
+                or not ordinary(getattr(cls, "__iter__", None), "__iter__", 
module)):
+            return False
+        isna, isna_module = getattr(owner, "isna", None), module
+        hasna = getattr(cls, "_hasna", None)
+        if (type(hasna) is not property
+                or not ordinary(hasna.fget, "_hasna", module)
+                or getattr(cls, "fillna", None) is not getattr(owner, 
"fillna", None)
+                or not ordinary(getattr(cls, "fillna", None), "fillna", 
module)):
+            return False
+    # Keep the existing storage/conversion authentication as well as the
+    # iterator hooks: these objects also supply labels to pandas normalization.
+    return (getattr(cls, "to_numpy", None) is getattr(owner, "to_numpy", None)
+            and ordinary(getattr(cls, "to_numpy", None), "to_numpy", module)
+            and getattr(cls, "isna", None) is isna
+            and ordinary(getattr(cls, "isna", None), "isna", isna_module))
+
+
+def prepare_int64_table(frame, schema):
+    """Return an Arrow table only when each column is already a signed 64-bit 
integer.
+
+    In particular, never coerce a mixed pandas frame to a common NumPy dtype:
+    doing so can round integers above 2**53. Other dtypes and schema 
conversions
+    deliberately retain the existing row path until their semantics are 
covered.
+    ``verifySchema`` is ignored for pandas Arrow input, as in Spark Connect.
+    """
+    import sys
+    from types import GetSetDescriptorType, WrapperDescriptorType
+
+    import numpy as np
+    import pandas as pd
+    try:
+        from pandas.core.arrays.integer import Int64Dtype, IntegerArray
+        from pandas.core.arrays.numpy_ import NumpyExtensionArray
+        from pandas.core.frame import DataFrame
+        from pandas.core.series import Series
+    except ImportError:
+        # Older supported pandas versions call the NumPy wrapper PandasArray.
+        # Keep their existing row conversion when these concrete types are 
absent.
+        return None
+
+    from pyspark._pyspark import (
+        AtomicType, BooleanType, ByteType, ShortType, IntegerType, LongType, 
StructField, StructType,
+    )
+    from pyspark.sql import types as sql_types
+
+    # A subclass or custom extension array can change the values exposed to 
Arrow.
+    # Check concrete identities before consulting dtype equality or conversion 
hooks.
+    if pd.DataFrame is not DataFrame or type(frame) is not DataFrame:
+        return None
+
+    if not _has_canonical_columns(frame.columns):
+        return None
+    width = len(frame.columns)
+    if not width or not frame.columns.is_unique or not all(
+        type(name) is str and "\x00" not in name for name in frame.columns
+    ):
+        return None
+    dtypes = frame.dtypes
+    numpy_dtype = np.dtype("int64")
+    if not all((type(dtype) is type(numpy_dtype) and dtype == numpy_dtype)
+               or type(dtype) is Int64Dtype
+               for dtype in dtypes):
+        return None
+    # The importer's configuration RPC outweighs row conversion for small 
NumPy frames.
+    # Nullable Int64 frames use Arrow at any size, which keeps their null 
masks.
+    if len(frame) < 1000 and not any(type(dtype) is Int64Dtype for dtype in 
dtypes):
+        return None
+
+    # The native row path reads native schema fields directly. The Arrow 
converter
+    # instead uses Python descriptors and imported class aliases, including 
aliases
+    # captured before this helper was loaded. Customized schemas retain the 
row path.
+    if sys.modules.get("pyspark.sql.types") is not sql_types:
+        return None
+    importer = sys.modules.get("pyspark.sql._arrow_input")
+    if vars(sql_types).get("AtomicType") is not AtomicType:
+        return None
+    if importer is not None:
+        if getattr(importer, "StructType", None) is not StructType:
+            return None
+        # The importer uses this alias while validating schema widths.
+        if vars(importer).get("AtomicType") is not AtomicType:
+            return None
+    converter = sys.modules.get("pyspark.sql.pandas.types")
+    namespaces = [vars(sql_types)]
+    if converter is not None:
+        namespaces.append(vars(converter))
+    if importer is not None:
+        # An importer may still hold functions from a removed converter module.
+        for name in ("from_arrow_schema", "to_arrow_schema"):
+            # Later importers resolve these converters locally inside 
prepare_table.
+            if name not in vars(importer):
+                continue
+            namespace = getattr(getattr(importer, name, None), "__globals__", 
None)
+            if type(namespace) is not dict:
+                return None
+            namespaces.append(namespace)
+    for name, cls in (
+        ("BooleanType", BooleanType), ("ByteType", ByteType), ("ShortType", 
ShortType),
+        ("IntegerType", IntegerType), ("LongType", LongType),
+        ("StructField", StructField), ("StructType", StructType),
+    ):
+        if any(namespace.get(name) is not cls for namespace in namespaces):
+            return None
+    for cls in (LongType, StructField, StructType):
+        if (cls.__getattribute__ is not object.__getattribute__
+                or any("__class__" in vars(base) for base in cls.__mro__ if 
base is not object)):
+            return None
+    for cls, names, descriptor_type in (
+        (StructType, ("fields", "names"), GetSetDescriptorType),
+        (StructType, ("__iter__",), WrapperDescriptorType),
+        (StructField, ("name", "dataType", "nullable", "metadata"), 
GetSetDescriptorType),
+    ):
+        for name in names:
+            descriptor = vars(cls).get(name)
+            if (type(descriptor) is not descriptor_type
+                    or descriptor.__objclass__ is not cls or 
descriptor.__name__ != name):
+                return None
+
+    names = list(frame.columns)
+    fields = None
+    if type(schema) is StructType:
+        fields = schema.fields
+        if len(fields) != width or not all(
+            type(field) is StructField and type(field.dataType) is LongType
+            and type(field.name) is str and "\x00" not in field.name
+            for field in fields
+        ):
+            return None
+        names = schema.names
+        if len(set(names)) != width:
+            return None
+    elif schema is not None:
+        if type(schema) not in (list, tuple) or len(schema) != width or not 
all(
+            type(name) is str and "\x00" not in name for name in schema
+        ):
+            return None
+        # As PySpark Connect, the Arrow table keeps the pandas names, and the 
list is
+        # applied with toDF.
+
+    columns = [frame.iloc[:, i] for i in range(width)]
+    for i, (column, dtype) in enumerate(zip(columns, dtypes)):
+        if type(column) is not Series:
+            return None
+        expected = IntegerArray if type(dtype) is Int64Dtype else 
NumpyExtensionArray
+        if type(column.array) is not expected:
+            return None
+        # The native row path rejects invalid nullability before any Config 
RPC.
+        if (fields is not None and not fields[i].nullable
+                and type(dtype) is Int64Dtype and column.hasnans):
+            return None
+
+    import pyarrow as pa
+
+    # Arrow consumes each Series' integer buffer and nullable mask 
independently.
+    return pa.Table.from_arrays(
+        [pa.array(column, type=pa.int64(), from_pandas=True) for column in 
columns],
+        names=names,
+    )
diff --git a/python/tests/test_arrow_handoff.py 
b/python/tests/test_arrow_handoff.py
index e04bd8a..5f8a74b 100644
--- a/python/tests/test_arrow_handoff.py
+++ b/python/tests/test_arrow_handoff.py
@@ -26,17 +26,19 @@ import shutil
 import subprocess
 from concurrent.futures import ThreadPoolExecutor
 from contextlib import contextmanager
-from datetime import datetime, timezone
+from datetime import date, datetime, timezone
 from decimal import Decimal
 from pathlib import Path
 from urllib.parse import quote
 
 import grpc
+import numpy as np
 import pandas as pd
 import pyarrow as pa
 import pytest
 from google.protobuf import descriptor_pb2, descriptor_pool, message_factory
-from pyspark.sql import SparkSession
+from pyspark.sql import SparkSession, types as T
+from pyspark.sql._pandas_input import prepare_int64_table
 
 
 @pytest.fixture(scope="module")
@@ -403,6 +405,7 @@ def _arrow_input_service(
     protocol,
     *,
     upload_error=None,
+    config_error=False,
     user_id=None,
     user_agent=None,
     allow_file_artifacts=False,
@@ -447,6 +450,8 @@ def _arrow_input_service(
         operation = request.operation.WhichOneof("op_type")
         keys = list(getattr(request.operation, operation).keys)
         state["config_requests"].append((operation, keys))
+        if config_error:
+            context.abort(grpc.StatusCode.UNAVAILABLE, "configuration 
unavailable")
         if error := state.pop("config_error", None):
             context.abort(error, "configuration read failed")
         # Deliberately reverse responses and omit missing keys to test key 
mapping.
@@ -1036,3 +1041,830 @@ def 
test_add_artifacts_names_each_artifact_as_pyspark(protocol, tmp_path):
             "archives/b.zip#alias": b"b.zip",
             "files/n.txt": b"n.txt",
         }
+
+
+# Signed-int64 pandas input uses the same Arrow importer and mock service.
+
[email protected]("storage", ["object", "default", "python", "pyarrow", 
"python_nan", "pyarrow_nan"])
[email protected]("dtype", ["int64", "Int64"])
+def test_integer_canonical_columns_storage_stays_columnar(protocol, 
monkeypatch, storage, dtype):
+    from pyspark.sql._pandas_input import _has_canonical_columns
+
+    values = [2**53 + 1, -(2**63), 2**63 - 1, 0] * 250
+    if dtype == "Int64":
+        values[1] = None
+    frame = pd.DataFrame({"original": pd.Series(values, dtype=dtype)})
+    if storage == "object":
+        frame.columns = pd.Index(["original"], dtype=object)
+    elif storage != "default":
+        kind = storage.removesuffix("_nan")
+        label_dtype = (pd.StringDtype(storage=kind, na_value=np.nan)
+                       if storage.endswith("_nan") else 
pd.StringDtype(storage=kind))
+        frame.columns = pd.Index(pd.array(["original"], dtype=label_dtype))
+    assert _has_canonical_columns(frame.columns)
+    assert prepare_int64_table(frame, None).column(0).to_pylist() == values
+
+    def no_rows(*args, **kwargs):
+        raise AssertionError("ordinary columns must retain integer columnar 
construction")
+
+    monkeypatch.setattr(pd.DataFrame, "astype", no_rows)
+    with _arrow_input_service(protocol) as (spark, state):
+        spark.createDataFrame(frame).toArrow()
+        assert state["input_tables"][0].column(0).to_pylist() == values
+        assert state["config_requests"]
+
+
[email protected]("dtype", ["int64", "Int64"])
[email protected]("schema_kind", ["inferred", "names", "explicit"])
[email protected]("location", [
+    "subclass", "class", "owner", "backing_subclass_item", 
"backing_class_iter",
+    "backing_class_getitem", "backing_instance_box",
+])
[email protected]("behavior", ["raise", "mutate"])
+def test_integer_columns_iterator_hooks_keep_native_order(
+    protocol, monkeypatch, dtype, schema_kind, location, behavior
+):
+    from pandas.core.base import IndexOpsMixin
+
+    values = [2**53 + 1, -(2**63), 2**63 - 1, 0] * 250
+    if dtype == "Int64":
+        values[1] = None
+    frame = pd.DataFrame({"original": pd.Series(values, dtype=dtype)})
+    schema = None
+    expected_name = "original"
+    nullable = True
+    metadata = {}
+    if schema_kind == "names":
+        schema = ["renamed"]
+        expected_name = "renamed"
+    elif schema_kind == "explicit":
+        nullable, metadata = dtype == "Int64", {"source": "native"}
+        schema = T.StructType([T.StructField("renamed", T.LongType(), 
nullable, metadata)])
+        expected_name = "renamed"
+    original_iter = pd.Index.__iter__
+    original_notna, original_astype = frame.notna, frame.astype
+    calls = []
+    invoked = False
+
+    class ColumnsError(RuntimeError):
+        pass
+
+    def notna(*args, **kwargs):
+        calls.append("notna")
+        return original_notna(*args, **kwargs)
+
+    def astype(*args, **kwargs):
+        calls.append("astype")
+        return original_astype(*args, **kwargs)
+
+    def before_hook():
+        nonlocal invoked
+        if invoked:
+            return
+        invoked = True
+        calls.append("columns iterator")
+        if behavior == "raise":
+            raise ColumnsError("columns hook precedes pandas normalization")
+        # Install spies only when native iteration reaches the columns hook.
+        # Before eligibility, normalization methods are the canonical methods.
+        monkeypatch.setitem(vars(frame), "notna", notna)
+        monkeypatch.setitem(vars(frame), "astype", astype)
+        frame.iat[0, 0] = 2**53 + 7
+        frame.iat[-1, 0] = None if dtype == "Int64" else -(2**63) + 1
+
+    def iterate(index):
+        if index is columns:
+            before_hook()
+        return original_iter(index)
+
+    if location == "subclass":
+        class Columns(pd.Index):
+            pass
+        monkeypatch.setattr(Columns, "__iter__", iterate)
+        labels = pd.Index(["original"], dtype=object)
+        frame.columns = Columns._simple_new(labels._values, name=None)
+        assert type(frame.columns) is Columns
+    elif location == "backing_subclass_item":
+        # IndexOpsMixin iteration calls ndarray.item, rather than 
ndarray.__iter__.
+        class Names(np.ndarray):
+            def item(self, *args):
+                before_hook()
+                return super().item(*args)
+        backing = np.asarray(["original"], dtype=object).view(Names)
+        frame.columns = pd.Index._simple_new(backing, name=None)
+        assert type(frame.columns._values) is Names
+    elif location.startswith("backing_"):
+        frame.columns = pd.Index(pd.array(["original"], 
dtype=pd.StringDtype(storage="python")))
+        backing = frame.columns._values
+        if location == "backing_class_iter":
+            original = type(backing).__iter__
+
+            def backing_iter(array):
+                if array is backing:
+                    before_hook()
+                return original(array)
+
+            monkeypatch.setattr(type(backing), "__iter__", backing_iter)
+        elif location == "backing_class_getitem":
+            original = type(backing).__getitem__
+
+            def backing_getitem(array, key):
+                if array is backing:
+                    before_hook()
+                return original(array, key)
+
+            monkeypatch.setattr(type(backing), "__getitem__", backing_getitem)
+        else:
+            original = backing._box_func
+
+            def box(value):
+                before_hook()
+                return original(value)
+
+            monkeypatch.setitem(vars(backing), "_box_func", box)
+    else:
+        monkeypatch.setattr(pd.Index if location == "class" else IndexOpsMixin,
+                            "__iter__", iterate)
+    columns = frame.columns
+
+    with _arrow_input_service(protocol) as (spark, state):
+        if behavior == "raise":
+            with pytest.raises(ColumnsError,
+                               match="columns hook precedes pandas 
normalization"):
+                spark.createDataFrame(frame, schema)
+            assert calls == ["columns iterator"]
+            assert state["config_requests"] == []
+            assert state["input_tables"] == []
+        else:
+            spark.createDataFrame(frame, schema).toArrow()
+            expected = values.copy()
+            expected[0], expected[-1] = 2**53 + 7, (None if dtype == "Int64" 
else -(2**63) + 1)
+            table = state["input_tables"][0]
+            assert table.num_rows == 1000
+            assert table.column_names == [expected_name]
+            assert table.column(0).type == pa.int64()
+            assert table.column(0).to_pylist() == expected
+            assert table.column(0).null_count == sum(value is None for value 
in expected)
+            assert table.schema.field(0).nullable is nullable
+            assert table.schema.field(0).metadata is None
+            import json
+            logical = json.loads(state["relations"][0].local_relation.schema)
+            assert logical == {"type": "struct", "fields": [{
+                "name": expected_name, "type": "long", "nullable": nullable,
+                "metadata": metadata,
+            }]}
+            assert state["config_requests"] == []
+            assert calls == ["columns iterator", "notna", "astype"]
+
+
+def test_integer_rebound_index_values_keeps_native_error(protocol, 
monkeypatch):
+    frame = pd.DataFrame({"original": np.arange(1000, dtype=np.int64)})
+    schema = T.StructType([T.StructField("renamed", T.LongType())])
+    columns = frame.columns
+    original = pd.Index._values.fget
+    calls = []
+
+    def values(index):
+        if index is columns:
+            calls.append("_values")
+            raise RuntimeError("columns backing hook precedes normalization")
+        return original(index)
+
+    monkeypatch.setattr(pd.Index, "_values", property(values))
+    with _arrow_input_service(protocol) as (spark, state):
+        with pytest.raises(RuntimeError,
+                           match="columns backing hook precedes 
normalization"):
+            spark.createDataFrame(frame, schema)
+        assert calls == ["_values"]
+        assert state["config_requests"] == []
+        assert state["input_tables"] == []
+
+
[email protected]("schema_kind", ["inferred", "names", "explicit"])
[email protected]("verify", [True, False])
+def test_integer_columns_preserve_values_without_python_rows(protocol, 
monkeypatch, schema_kind, verify):
+    frame = pd.DataFrame({
+        "a": pd.Series([2**53 + 1, None, -(2**63)], dtype="Int64"),
+        "b": np.array([2**63 - 1, -1, 0], dtype=np.int64),
+    })
+    schema = None
+    names = ["a", "b"]
+    if schema_kind == "names":
+        schema = names = ["renamed_a", "renamed_b"]
+    elif schema_kind == "explicit":
+        schema = T.StructType([T.StructField("a", T.LongType()),
+                               T.StructField("b", T.LongType(), False, 
{"source": "test"})])
+    import pyspark.sql._arrow_input as helper
+    original = helper.prepare_table
+    observed = []
+
+    def capture(*args):
+        result = original(*args)
+        observed.append(result)
+        return result
+
+    def no_rows(*args, **kwargs):
+        raise AssertionError("supported pandas input must not materialize 
Python rows")
+
+    monkeypatch.setattr(helper, "prepare_table", capture)
+    monkeypatch.setattr(pd.DataFrame, "astype", no_rows)
+    with _arrow_input_service(protocol) as (spark, state):
+        df = spark.createDataFrame(frame, schema=schema, verifySchema=verify)
+        # Every nonempty pandas input applies its names through toDF, including
+        # explicit struct schemas, just as the current row fallback does.
+        assert not df.isLocal()
+        assert df.toArrow().column(0).to_pylist() == [3]
+        assert state["column_names"] == names
+        assert state["result"] == (3, 2**53 + 1, -(2**63))
+    table, resolved, renamed = observed[0]
+    assert table.to_pydict() == {"a": [2**53 + 1, None, -(2**63)],
+                                "b": [2**63 - 1, -1, 0]}
+    assert (renamed or resolved.names) == names
+    if schema_kind == "explicit":
+        assert resolved == schema
+        assert not table.schema.field(1).nullable
+        assert resolved.fields[1].metadata == {"source": "test"}
+
+
[email protected]("dtype", ["int64", "Int64"])
+def test_empty_integer_frame_requires_explicit_schema(protocol, dtype):
+    frame = pd.DataFrame({"a": pd.Series([], dtype=dtype)})
+    with _arrow_input_service(protocol) as (spark, state):
+        with pytest.raises(ValueError):
+            spark.createDataFrame(frame)
+        df = spark.createDataFrame(frame, T.StructType([T.StructField("a", 
T.LongType())]))
+        assert df.isLocal()
+
+
[email protected]("values", [
+    [1.0, float("nan")], [True, False], ["x", None],
+    [datetime(2024, 1, 1), None],
+    [datetime(2024, 1, 1, tzinfo=timezone.utc), None],
+    [date(2024, 1, 1), None], [Decimal("1.20"), None],
+    [[1, 2], None], [{"a": 1}, None],
+])
+def test_other_dtypes_keep_row_fallback(values):
+    frame = pd.DataFrame({"a": values})
+    assert prepare_int64_table(frame, None) is None
+    assert prepare_int64_table(frame, T.StructType([T.StructField("a", 
T.LongType())])) is None
+
+
[email protected]("schema", [
+    "a long", T.LongType(), ["a", "b"], [1],
+    T.StructType([T.StructField("a", T.IntegerType())]),
+    T.StructType([T.StructField("a", T.DoubleType())]),
+    T.StructType([]),
+])
+def test_schema_conversions_keep_row_fallback(schema):
+    assert prepare_int64_table(pd.DataFrame({"a": [1, 2]}), schema) is None
+
+
+def test_duplicate_names_fallback_and_nullable_masks():
+    frame = pd.DataFrame([[2**53 + 1, 2**53 + 3]], columns=["x", "x"], 
dtype="int64")
+    assert prepare_int64_table(frame, None) is None
+    assert prepare_int64_table(pd.DataFrame({0: [1]}), None) is None
+    frame = pd.DataFrame({"a": pd.Series([None, None], dtype="Int64")})
+    assert prepare_int64_table(frame, None).column(0).to_pylist() == [None, 
None]
+
+
[email protected]("dtype", ["int32", "uint64", "Int32", "UInt64", 
"int64[pyarrow]"])
+def test_other_integer_dtypes_keep_row_fallback(dtype):
+    assert prepare_int64_table(pd.DataFrame({"a": pd.Series([1, 2], 
dtype=dtype)}), None) is None
+
+
[email protected]("rows", [100, 1000, 100000, 1000000])
[email protected]("explicit", [False, True])
+def test_integer_scales_reach_mock_service(protocol, monkeypatch, rows, 
explicit):
+    values = np.arange(rows, dtype=np.int64) + (2**53 + 1)
+    frame = pd.DataFrame({name: values for name in ("a", "b", "c")})
+    schema = T.StructType([T.StructField(name, T.LongType()) for name in 
frame.columns])
+
+    def no_rows(*args, **kwargs):
+        raise AssertionError("integer input must stay columnar at every scale")
+
+    if rows >= 1000:
+        monkeypatch.setattr(pd.DataFrame, "astype", no_rows)
+    with _arrow_input_service(protocol) as (spark, state):
+        spark.createDataFrame(frame, schema if explicit else None).toArrow()
+        assert state["result"] == (rows, 2**53 + 1, 2**53 + rows)
+        if frame.memory_usage(index=False).sum() > 1048576:
+            assert 
state["relations"][0].HasField("chunked_cached_local_relation")
+
+
+def test_nullable_inference_does_not_depend_on_first_value(protocol):
+    frame = pd.DataFrame({"a": pd.Series([None, 2**53 + 1], dtype="Int64")})
+    with _arrow_input_service(protocol) as (spark, state):
+        spark.createDataFrame(frame).toArrow()
+        assert state["result"] == (2, None, 2**53 + 1)
+
+
[email protected]("verify", [True, False])
[email protected]("config_error", [False, True])
+def test_non_nullable_schema_is_preserved(protocol, verify, config_error):
+    frame = pd.DataFrame({"a": pd.Series([1, None], dtype="Int64")})
+    schema = T.StructType([T.StructField("a", T.LongType(), False)])
+    assert prepare_int64_table(frame, schema) is None
+    with _arrow_input_service(protocol, config_error=config_error) as (spark, 
state):
+        with pytest.raises(RuntimeError, match="null"):
+            spark.createDataFrame(frame, schema, verifySchema=verify)
+        assert not state["relations"]
+        assert not state["uploads"]
+        assert state["config_requests"] == []
+
+
+def test_small_numpy_frames_avoid_importer_configuration_overhead():
+    assert prepare_int64_table(pd.DataFrame({"a": np.arange(999, 
dtype=np.int64)}), None) is None
+    assert prepare_int64_table(pd.DataFrame({"a": np.arange(1000, 
dtype=np.int64)}), None) is not None
+    assert prepare_int64_table(pd.DataFrame({"a": pd.Series([None], 
dtype="Int64")}), None) is not None
+
+
[email protected]("schema_kind", ["inferred", "names", "explicit"])
+def test_nul_names_keep_lossless_ipc_row_fallback(protocol, schema_kind):
+    import pyarrow as pa
+
+    name = "a\x00b"
+    frame = pd.DataFrame({name if schema_kind == "inferred" else "original":
+                          np.arange(1000, dtype=np.int64)})
+    schema = None
+    if schema_kind == "names":
+        schema = [name]
+    elif schema_kind == "explicit":
+        schema = T.StructType([T.StructField(name, T.LongType())])
+    # C Stream field names are NUL-terminated; IPC preserves the full string.
+    assert prepare_int64_table(frame, schema) is None
+    with _arrow_input_service(protocol) as (spark, state):
+        spark.createDataFrame(frame, schema).toArrow()
+        assert state["result"] == (1000, 0, 999)
+        payload = state["relations"][0].local_relation.data
+        assert pa.ipc.open_stream(payload).schema.names == [name]
+
+
[email protected]("nullable", [False, True])
[email protected]("rows", [999, 1000])
+def test_integer_explicit_renamed_schema_preserves_all_ipc_cells(protocol, 
monkeypatch, nullable, rows):
+    values = ([2**53 + 1, -(2**63), 2**63 - 1, 0] * 250)[:rows]
+    other = [None if nullable and i % 7 == 0 else value
+             for i, value in enumerate(reversed(values))]
+    frame = pd.DataFrame({"original": np.asarray(values, dtype="int64"),
+                          "second": pd.Series(other, dtype="Int64" if nullable 
else "int64")})
+    schema = T.StructType([
+        T.StructField("renamed", T.LongType(), False, {"source": "native", 
"id": 3}),
+        T.StructField("nullable", T.LongType(), True, {"tag": [1, 2]}),
+    ])
+
+    def no_rows(*args, **kwargs):
+        raise AssertionError("trusted integer columns must stay columnar")
+
+    if rows >= 1000 or nullable:
+        monkeypatch.setattr(pd.DataFrame, "astype", no_rows)
+    with _arrow_input_service(protocol) as (spark, state):
+        spark.createDataFrame(frame, schema).toArrow()
+        table = state["input_tables"][0]
+        assert table.column_names == ["renamed", "nullable"]
+        assert [column.to_pylist() for column in table.columns] == [values, 
other]
+        assert table.schema.field(0).type == pa.int64()
+        assert table.schema.field(0).nullable is False
+        assert table.schema.field(1).nullable is True
+        if rows >= 1000 or nullable:
+            from pyspark.sql.pandas.types import from_arrow_schema
+            assert from_arrow_schema(table.schema).json() == schema.json(), (
+                from_arrow_schema(table.schema).json(), schema.json(), 
table.schema)
+            assert state["config_requests"]
+        else:
+            assert state["config_requests"] == []
+
+
+def test_integer_dataframe_and_name_subclasses_keep_fallback(monkeypatch):
+    class Frame(pd.DataFrame):
+        pass
+
+    class Name(str):
+        pass
+
+    class Names(list):
+        pass
+
+    frame = pd.DataFrame({"a": np.arange(1000, dtype="int64")})
+    assert prepare_int64_table(Frame(frame), None) is None
+    assert prepare_int64_table(frame, Names(["a"])) is None
+    assert prepare_int64_table(frame, [Name("a")]) is None
+    # Pandas 3 normalizes constructor labels to plain strings by default.
+    # An object Index keeps the subclass this fallback regression exercises.
+    named_frame = frame.copy()
+    named_frame.columns = pd.Index([Name("a")], dtype=object)
+    assert type(named_frame.columns[0]) is Name
+    assert prepare_int64_table(named_frame, None) is None
+    monkeypatch.setattr(pd, "DataFrame", Frame)
+    assert prepare_int64_table(Frame(frame), None) is None
+
+
+def test_integer_missing_internal_pandas_array_class_keeps_fallback():
+    import sys
+
+    script = r'''
+import sys
+import types
+import numpy as np
+import pandas as pd
+from pyspark.sql._pandas_input import prepare_int64_table
+frame = pd.DataFrame({"a": np.arange(1000, dtype="int64")})
+nullable = pd.DataFrame({"a": pd.Series([None, 1], dtype="Int64")})
+# Older pandas exposes PandasArray rather than NumpyExtensionArray. Missing
+# optional internals must disable the optimization without breaking pandas 
input.
+sys.modules["pandas.core.arrays.numpy_"] = 
types.ModuleType("pandas.core.arrays.numpy_")
+assert prepare_int64_table(frame, None) is None
+assert prepare_int64_table(nullable, None) is None
+'''
+    result = subprocess.run([sys.executable, "-c", script],
+                            capture_output=True, text=True, timeout=30)
+    assert result.returncode == 0, result.stdout + result.stderr
+
+
[email protected]("class_spoof", [False, True])
+def test_integer_custom_extension_dtype_cannot_select_arrow(protocol, 
class_spoof):
+    from pandas.api.extensions import ExtensionArray, ExtensionDtype, take
+
+    arrow_calls = []
+
+    class Int64EqualDtype(ExtensionDtype):
+        name = "int64_equal"
+        type = int
+        kind = "i"
+        na_value = None
+
+        @classmethod
+        def construct_array_type(cls):
+            return DifferentArrowArray
+
+        def __eq__(self, other):
+            return (isinstance(other, Int64EqualDtype)
+                    or isinstance(other, np.dtype) and other == 
np.dtype("int64"))
+
+        def __hash__(self):
+            return hash(self.name)
+
+    class DifferentArrowArray(ExtensionArray):
+        def __init__(self, values):
+            self._values = np.asarray(values, dtype=object)
+
+        @property
+        def dtype(self):
+            return Int64EqualDtype()
+
+        @property
+        def nbytes(self):
+            return self._values.nbytes
+
+        def __len__(self):
+            return len(self._values)
+
+        def __getitem__(self, key):
+            result = self._values[key]
+            return type(self)(result) if isinstance(result, np.ndarray) else 
result
+
+        def isna(self):
+            return pd.isna(self._values)
+
+        def take(self, indices, allow_fill=False, fill_value=None):
+            return type(self)(take(self._values, indices, 
allow_fill=allow_fill,
+                                   fill_value=fill_value))
+
+        def copy(self):
+            return type(self)(self._values.copy())
+
+        @classmethod
+        def _from_sequence(cls, scalars, *, dtype=None, copy=False):
+            values = np.asarray(scalars, dtype=object)
+            return cls(values.copy() if copy else values)
+
+        def to_numpy(self, dtype=None, copy=False, na_value=None):
+            if dtype is not None:
+                return self._values.astype(dtype, copy=copy)
+            return self._values.copy() if copy else self._values
+
+        def __arrow_array__(self, type=None):
+            arrow_calls.append("arrow")
+            return pa.array([-7] * len(self), type=type)
+
+    values = [2**53 + 1, None, -(2**63), 0] * 250
+    frame = pd.DataFrame({"a": pd.Series(DifferentArrowArray(values), 
copy=False)})
+    if class_spoof:
+        Int64EqualDtype.__getattribute__ = lambda self, name: (
+            type(np.dtype("int64")) if name == "__class__"
+            else object.__getattribute__(self, name))
+    assert frame.dtypes.iloc[0] == np.dtype("int64")
+    assert isinstance(frame.dtypes.iloc[0], np.dtype) is class_spoof
+    schema = T.StructType([T.StructField("renamed", T.LongType())])
+    assert prepare_int64_table(frame, schema) is None
+    with _arrow_input_service(protocol) as (spark, state):
+        spark.createDataFrame(frame, schema).toArrow()
+        assert state["config_requests"] == []
+        assert state["input_tables"][0].column(0).to_pylist() == values
+    assert arrow_calls == []
+
+
[email protected]("custom", ["dtype", "array"])
+def test_integer_nullable_dtype_and_array_subclasses_keep_fallback(protocol, 
custom):
+    from pandas.core.arrays.integer import IntegerArray, Int64Dtype
+
+    arrow_calls = []
+
+    class CustomDtype(Int64Dtype):
+        pass
+
+    class CustomArray(IntegerArray):
+        @property
+        def dtype(self):
+            if custom == "dtype":
+                return CustomDtype()
+            return Int64Dtype()
+
+        def __arrow_array__(self, type=None):
+            arrow_calls.append("arrow")
+            return pa.array([-7] * len(self), type=type)
+
+    values = [2**53 + 1, 0, -(2**63), 7] * 250
+    array = CustomArray(np.asarray(values, dtype="int64"), np.zeros(1000, 
dtype=bool))
+    frame = pd.DataFrame({"a": pd.Series(array, copy=False)})
+    assert type(frame["a"].array) is CustomArray
+    schema = T.StructType([T.StructField("a", T.LongType())])
+    assert prepare_int64_table(frame, schema) is None
+    with _arrow_input_service(protocol) as (spark, state):
+        spark.createDataFrame(frame, schema).toArrow()
+        assert state["config_requests"] == []
+        assert state["input_tables"][0].column(0).to_pylist() == values
+    assert arrow_calls == []
+
+
+def _integer_guard_process(mode, argument, before_import=False):
+    """Class mutation and converter imports must be isolated from other 
tests."""
+    import sys
+
+    if mode == "captured_alias":
+        import pyspark.sql._arrow_input as importer
+        if not hasattr(importer, "to_arrow_schema"):
+            pytest.skip("this importer resolves converters locally and holds 
no captured aliases")
+
+    script = r'''
+import importlib
+import importlib.util
+import sys
+import tempfile
+from pathlib import Path
+import numpy as np
+import pandas as pd
+from pyspark.sql import types as T
+from pyspark import _pyspark as native
+
+spec = importlib.util.spec_from_file_location("integer_guard_capture", 
sys.argv[4])
+capture = importlib.util.module_from_spec(spec)
+sys.modules[spec.name] = capture
+spec.loader.exec_module(capture)
+values = [2**53 + 1, -(2**63), 2**63 - 1, 0] * 250
+frame = pd.DataFrame({"renamed": np.asarray(values, dtype="int64")})
+schema = T.StructType([T.StructField("renamed", T.LongType(), False, 
{"source": "native"})])
+mode, argument, before_import = sys.argv[1:4]
+if before_import == "True":
+    sys.modules.pop("pyspark.sql._pandas_input", None)
+if mode == "descriptor":
+    class_name, attribute = argument.split(":")
+    cls = getattr(native, class_name)
+    if attribute == "alias":
+        setattr(T, class_name, object)
+    elif attribute == "__getattribute__":
+        setattr(cls, attribute, lambda self, name: 
object.__getattribute__(self, name))
+    elif attribute == "__class__":
+        setattr(cls, "__getattribute__", lambda self, name: (
+            object if name == "__class__" else object.__getattribute__(self, 
name)))
+    elif attribute == "__iter__":
+        setattr(cls, attribute, lambda self: iter([T.StructField("changed", 
T.BooleanType())]))
+    else:
+        value = {"names": ["changed"], "fields": [], "name": "changed",
+                 "dataType": T.BooleanType(), "nullable": True,
+                 "metadata": {"changed": True}}[attribute]
+        setattr(cls, attribute, property(lambda self: value))
+elif mode in ("public_alias", "converter_alias", "captured_alias"):
+    replacement = native.BooleanType if argument == "LongType" else 
native.LongType
+    if mode == "public_alias":
+        sys.modules.pop("pyspark.sql.pandas.types", None)
+        assert "pyspark.sql.pandas.types" not in sys.modules
+        setattr(T, argument, replacement)
+    else:
+        converter = importlib.import_module("pyspark.sql.pandas.types")
+        if mode == "captured_alias":
+            importer = importlib.import_module("pyspark.sql._arrow_input")
+        setattr(converter, argument, replacement)
+        if mode == "captured_alias":
+            sys.modules.pop("pyspark.sql.pandas.types")
+            current = importlib.import_module("pyspark.sql.pandas.types")
+            assert current is not converter
+            assert importer.to_arrow_schema.__globals__ is vars(converter)
+elif mode == "atomic_alias":
+    scope, replacement_name = argument.split(":")
+    replacement = None if replacement_name == "none" else native.LongType
+    if scope == "public_before":
+        sys.modules.pop("pyspark.sql._arrow_input", None)
+        T.AtomicType = replacement
+    elif scope == "public_after":
+        importlib.import_module("pyspark.sql._arrow_input")
+        T.AtomicType = replacement
+    else:
+        importer = importlib.import_module("pyspark.sql._arrow_input")
+        if replacement_name == "missing":
+            del importer.AtomicType
+        else:
+            importer.AtomicType = replacement
+elif mode == "importer_alias":
+    importer = importlib.import_module("pyspark.sql._arrow_input")
+    importer.StructType = None if argument == "none" else native.LongType
+else:
+    raise AssertionError(mode)
+class Factory:
+    def mktemp(self, name):
+        return Path(tempfile.mkdtemp(prefix=name))
+with capture._arrow_input_service(capture.protocol.__wrapped__(Factory())) as 
(spark, state):
+    spark.createDataFrame(frame, schema).toArrow()
+    assert state["config_requests"] == [], state["config_requests"]
+    table = state["input_tables"][0]
+    assert table.schema.names == ["renamed"], table.schema
+    assert table.column(0).type == __import__("pyarrow").int64()
+    assert table.column(0).to_pylist() == values
+'''
+    result = subprocess.run([sys.executable, "-c", script, mode, argument,
+                             str(before_import), __file__],
+                            capture_output=True, text=True, timeout=30)
+    assert result.returncode == 0, result.stdout + result.stderr
+
+
[email protected]("class_name,attribute", [
+    ("StructType", "fields"), ("StructType", "names"), ("StructType", 
"__iter__"),
+    ("StructField", "name"), ("StructField", "dataType"), ("StructField", 
"nullable"),
+    ("StructField", "metadata"), ("StructType", "__getattribute__"),
+    ("StructField", "__getattribute__"), ("LongType", "__getattribute__"),
+    ("StructType", "__class__"), ("StructField", "__class__"), ("LongType", 
"__class__"),
+    ("StructType", "alias"), ("StructField", "alias"), ("LongType", "alias"),
+])
[email protected]("before_import", [False, True])
+def test_integer_custom_schema_descriptors_keep_native_row_path(class_name, 
attribute, before_import):
+    _integer_guard_process("descriptor", f"{class_name}:{attribute}", 
before_import)
+
+
[email protected]("class_name", ["BooleanType", "ByteType", 
"ShortType", "IntegerType", "LongType"])
[email protected]("mode", ["public_alias", "converter_alias", 
"captured_alias"])
+def test_integer_rebound_converter_aliases_keep_native_row_path(class_name, 
mode):
+    _integer_guard_process(mode, class_name)
+
+
[email protected]("replacement", ["none", "long"])
+def 
test_integer_rebound_importer_schema_alias_keeps_native_row_path(replacement):
+    _integer_guard_process("importer_alias", replacement)
+
+
[email protected]("scope", ["public_before", "public_after", 
"importer"])
[email protected]("replacement", ["none", "long"])
+def test_integer_rebound_atomic_alias_keeps_native_row_path(scope, 
replacement):
+    _integer_guard_process("atomic_alias", f"{scope}:{replacement}")
+
+
+def test_integer_deleted_importer_atomic_alias_keeps_native_row_path():
+    _integer_guard_process("atomic_alias", "importer:missing")
+
+
+def test_integer_warm_importer_without_global_converter_aliases(protocol, 
monkeypatch):
+    import pyspark.sql._arrow_input as importer
+
+    original = importer.prepare_table
+
+    def local_imports(table, schema, *configs):
+        # Newer importers import converters within prepare_table. Exercise that
+        # module shape even on the pinned PR118 dependency used by this branch.
+        from pyspark.sql.pandas.types import from_arrow_schema, to_arrow_schema
+        table, resolved, names = original(table, schema, *configs)
+        assert from_arrow_schema(table.schema).fields
+        assert to_arrow_schema(resolved).names == table.column_names
+        return table, resolved, names
+
+    frame = pd.DataFrame({"a": np.arange(1000, dtype="int64")})
+    schema = T.StructType([T.StructField("renamed", T.LongType())])
+    with _arrow_input_service(protocol) as (spark, state):
+        spark.createDataFrame(pa.table({"warm": [1, 2]})).toArrow()
+        # Keep original's module globals available in the dependency importer;
+        # the full-stack validation separately uses the actual local-import 
helper.
+        import types
+        substitute = types.ModuleType(importer.__name__)
+        substitute.StructType = T.StructType
+        substitute.AtomicType = T.AtomicType
+        substitute.prepare_table = local_imports
+        monkeypatch.setitem(__import__("sys").modules, importer.__name__, 
substitute)
+
+        def no_rows(*args, **kwargs):
+            raise AssertionError("warm integer input must stay columnar")
+
+        monkeypatch.setattr(pd.DataFrame, "astype", no_rows)
+        for _ in range(3):
+            spark.createDataFrame(frame, schema).toArrow()
+        for table in state["input_tables"][1:]:
+            assert table.column_names == ["renamed"]
+            assert table.column(0).to_pylist() == list(range(1000))
+
+
+
+
+
+
+
[email protected]("dtype", ["int64", "Int64"])
[email protected]("schema_kind", ["inferred", "names", "explicit"])
+def test_integer_rebound_index_data_keeps_native_read_order(
+    protocol, monkeypatch, dtype, schema_kind
+):
+    import json
+
+    values = [2**53 + 1, -(2**63), 2**63 - 1, 0] * 250
+    if dtype == "Int64":
+        values[1] = None
+    frame = pd.DataFrame({"original": pd.Series(values, dtype=dtype)})
+    schema, expected_name, nullable, metadata = None, "original", True, {}
+    if schema_kind == "names":
+        schema, expected_name = ["renamed"], "renamed"
+    elif schema_kind == "explicit":
+        expected_name, nullable, metadata = "renamed", dtype == "Int64", 
{"source": "native"}
+        schema = T.StructType([T.StructField(expected_name, T.LongType(), 
nullable, metadata)])
+    columns, backing = frame.columns, vars(frame.columns)["_data"]
+    original_notna, original_astype = frame.notna, frame.astype
+    calls = []
+    normalizing = False
+    allowed_reads = None
+
+    class ColumnsDataError(RuntimeError):
+        pass
+
+    def notna(*args, **kwargs):
+        nonlocal normalizing
+        normalizing = True
+        calls.append("notna")
+        return original_notna(*args, **kwargs)
+
+    def astype(*args, **kwargs):
+        calls.append("astype")
+        return original_astype(*args, **kwargs)
+
+    def data(index):
+        if index is columns:
+            if not normalizing:
+                calls.append("_data")
+                if allowed_reads is not None and calls.count("_data") > 
allowed_reads:
+                    raise ColumnsDataError("eligibility added a columns 
storage read")
+                # Install spies only when the native columns storage hook runs.
+                # Canonical frame methods remain intact before eligibility.
+                monkeypatch.setitem(vars(frame), "notna", notna)
+                monkeypatch.setitem(vars(frame), "astype", astype)
+            return backing
+        return vars(index)["_data"]
+
+    def set_data(index, value):
+        # Preserve ordinary storage initialization for indexes pandas creates
+        # while normalizing values; only reads of the input columns are hooked.
+        vars(index)["_data"] = value
+
+    monkeypatch.setattr(pd.Index, "_data", property(data, set_data), 
raising=False)
+    import pyspark.sql._pandas_input as helper
+
+    with _arrow_input_service(protocol) as (spark, state):
+        # Record current master's fallback rather than assuming the old tolist
+        # path's single read. Index iteration reads NumPy-backed _values three
+        # times, and extension-backed _values twice, before normalization.
+        with monkeypatch.context() as baseline:
+            baseline.setattr(helper, "prepare_int64_table", lambda *_: None)
+            spark.createDataFrame(frame, schema).toArrow()
+        baseline_calls = calls.copy()
+        allowed_reads = baseline_calls.count("_data")
+        assert allowed_reads > 0 and "notna" in baseline_calls
+        calls.clear()
+        normalizing = False
+        # Restore canonical methods before eligibility, just as for the first
+        # call. The storage hook installs them when native iteration starts.
+        monkeypatch.delitem(vars(frame), "notna")
+        monkeypatch.delitem(vars(frame), "astype")
+        spark.createDataFrame(frame, schema).toArrow()
+        # The rebound storage declines the Arrow path without reading the 
columns more
+        # than the row path does. pandas caches some of these reads, so a 
later row
+        # conversion of the same frame can read them fewer times than the 
first.
+        assert "notna" in calls
+        assert calls.count("_data") <= allowed_reads
+        assert not state["config_requests"]
+        assert len(state["input_tables"]) == 2
+        table = state["input_tables"][1]
+        assert table.equals(state["input_tables"][0], check_metadata=True)
+        assert table.column(0).to_pylist() == values
+        assert table.column(0).null_count == sum(value is None for value in 
values)
+        assert table.schema.equals(
+            pa.schema([pa.field(expected_name, pa.int64(), 
nullable=nullable)]),
+            check_metadata=True,
+        )
+        assert json.loads(state["relations"][0].local_relation.schema) == {
+            "type": "struct", "fields": [
+                {"name": expected_name, "type": "long", "nullable": nullable,
+                 "metadata": metadata}
+            ],
+        }


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to