This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new e0163c1b [SPARK-61227][CONNECT][PYTHON] Convert a pandas frame with 
default column labels through Arrow, as PySpark does, not on the row path
e0163c1b is described below

commit e0163c1b958639b72955085db16042162127a42b
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Nov 9 17:30:34 2026 +0900

    [SPARK-61227][CONNECT][PYTHON] Convert a pandas frame with default column 
labels through Arrow, as PySpark does, not on the row path
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61227](https://issues.apache.org/jira/browse/SPARK-61227): Convert a 
pandas frame with default column labels through Arrow, as PySpark does.
    
    `_has_canonical_columns` in `python/pyspark/sql/_pandas_input.py` accepts a 
default `pd.RangeIndex` of column labels, so the Arrow routes of 
`createDataFrame` (explicit struct schema, names or inferred schema, and the 
generic explicit route) take a plain frame whose labels are the default ones. 
The check for a RangeIndex has the same override rules as the check for an 
`Index`: the exact type, no override of `__getattribute__`, `__iter__` or 
`__len__` on the class or the instance, a pa [...]
    
    Official PySpark 4.3.0 converts these frames with Arrow. Across a matrix of 
RangeIndex frames (numeric frames with NaN under int, long and double schemas, 
object columns of Rows, dicts with and without 
`spark.sql.execution.pandas.inferPandasDictAsMap`, lists of naive datetimes and 
Decimals, strings, timestamps with time zones, empty frames, names lists and 
DDL schemas, and Arrow on and off), the results and errors now match official 
PySpark 4.3.0. The Rows under non-struct schemas, th [...]
    
    ### Why are the changes needed?
    
    `pd.DataFrame([[None], [3]])` and `pd.DataFrame(ndarray)` have default 
column labels, a RangeIndex, which the Arrow routes did not accept, so they 
took the native row path. There `spark.createDataFrame(pd.DataFrame([[None], 
[3]]), "x int")` raised `PySparkRuntimeError: Type mismatch in row data`, while 
PySpark 4.3.0 returns `[Row(x=None), Row(x=3)]`. The same frame with named or 
integer labels already worked.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. A pandas frame with default column labels now converts with Arrow 
under an explicit schema, as PySpark does, instead of failing or converting 
differently on the row path. Customized (subclassed) frames and frames with 
other labels are unchanged.
    
    Two existing tests in `test_dropin_offline.py` pinned the row path for 
these frames. `test_container_input_values_and_errors` and 
`test_container_input_subclass_numeric_precedence` now expect the Arrow 
conversion errors (`PySparkValueError`, "Exception thrown when converting 
pandas.Series") that official PySpark 4.3.0 raises for the same frames.
    
    ### How was this patch tested?
    
    - `test_default_range_index_frame_takes_arrow_route` (offline): the NaN of 
a float column under `x int`, `x long` and `a int, b int` converts to nulls, 
not a row-path error.
    - `test_default_range_index_labels_keep_row_path_for_overrides` (offline): 
the label check accepts a default RangeIndex and declines a subclass, an 
instance `__iter__` override and a class `__len__` override.
    - `test_default_range_index_pandas_frames_convert_as_pyspark` (live): six 
frames (NaN under int, Rows under an array schema, dicts, lists of naive 
datetimes and Decimals) give the schemas and values of PySpark 4.3.0.
    - The existing tests `test_container_input_values_and_errors` and 
`test_container_input_subclass_numeric_precedence` are updated as described 
above.
    - The lib tests of both crates (658 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2811 passed, 8 skipped), and the 
live drop-in and Arrow suites (230 passed) and live cases (12 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #406 from HyukjinKwon/SPARK-61227-fu-p9.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 python/pyspark/sql/_pandas_input.py | 19 +++++++++++
 python/tests/test_dropin_e2e.py     | 28 ++++++++++++++++
 python/tests/test_dropin_offline.py | 64 +++++++++++++++++++++++++++++++------
 3 files changed, 102 insertions(+), 9 deletions(-)

diff --git a/python/pyspark/sql/_pandas_input.py 
b/python/pyspark/sql/_pandas_input.py
index f177f336..538db6a2 100644
--- a/python/pyspark/sql/_pandas_input.py
+++ b/python/pyspark/sql/_pandas_input.py
@@ -145,6 +145,20 @@ def _has_canonical_arrow_string_array(array):
                    + ("astype", "dtype", "_hasna"))
 
 
+def _has_canonical_range_columns(columns, range_module):
+    """Accept the default RangeIndex of column labels, which iterates its own 
int range.
+
+    As for an Index, subclasses and instance overrides keep the row path.
+    """
+    RangeIndex = range_module.RangeIndex
+    if (RangeIndex.__getattribute__ is not object.__getattribute__
+            or "__iter__" in vars(columns) or "__len__" in vars(columns)
+            or not _defined_in(RangeIndex.__iter__, "__iter__", range_module)
+            or not _defined_in(RangeIndex.__len__, "__len__", range_module)):
+        return False
+    return type(vars(columns).get("_range")) is range
+
+
 def _has_canonical_columns(columns):
     """Decline label containers whose row-path iteration can run custom 
hooks."""
     import sys
@@ -153,6 +167,11 @@ def _has_canonical_columns(columns):
     import pandas as pd
     from pandas.core import base as base_module
     from pandas.core.indexes import base as index_module
+    from pandas.core.indexes import range as range_module
+
+    # A default RangeIndex (pd.DataFrame of a list or ndarray) takes the Arrow 
routes too.
+    if pd.RangeIndex is range_module.RangeIndex and type(columns) is 
pd.RangeIndex:
+        return _has_canonical_range_columns(columns, range_module)
 
     Index = index_module.Index
     if (pd.Index is not Index or type(columns) is not Index
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 0419b353..d68e38ea 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -24,6 +24,7 @@ import os
 import time
 from collections import namedtuple
 from datetime import date, datetime
+from decimal import Decimal
 
 import pyarrow as pa
 import pytest
@@ -3701,6 +3702,33 @@ def 
test_create_dataframe_raises_merge_errors_as_pyspark(session):
         assert error.value.getMessageParameters() == parameters, data
 
 
[email protected]("make, schema, expected_type, expected", [
+    (lambda pd, np: pd.DataFrame([[None], [3]]), "x int", "struct<x:int>", 
[(None,), (3,)]),
+    (lambda pd, np: pd.DataFrame(np.array([[1.0, np.nan], [2.0, 3.0]])), "a 
int, b int",
+     "struct<a:int,b:int>", [(1, None), (2, 3)]),
+    (lambda pd, np: pd.DataFrame([[Row(a=1, b=2)], [Row(a=3, b=4)]]), "x 
array<bigint>",
+     "struct<x:array<bigint>>", [([1, 2],), ([3, 4],)]),
+    (lambda pd, np: pd.DataFrame([[{"u": 1, "v": "a"}], [{"u": 2, "v": 
"b"}]]), None,
+     "struct<0:struct<u:bigint,v:string>>", [(Row(u=1, v="a"),), (Row(u=2, 
v="b"),)]),
+    (lambda pd, np: pd.DataFrame([[[datetime(2020, 1, 1, 12)]], 
[[datetime(2021, 1, 1)]]]), None,
+     "struct<0:array<timestamp_ntz>>", [([datetime(2020, 1, 1, 12)],), 
([datetime(2021, 1, 1)],)]),
+    (lambda pd, np: pd.DataFrame([[[Decimal("1.5"), Decimal("10.25")]], 
[[Decimal("2")]]]), None,
+     "struct<0:array<decimal(4,2)>>",
+     [([Decimal("1.50"), Decimal("10.25")],), ([Decimal("2.00")],)]),
+])
+def test_default_range_index_pandas_frames_convert_as_pyspark(
+    session, make, schema, expected_type, expected
+):
+    # Plain frames with default (RangeIndex) labels convert with Arrow, and 
give the results
+    # of PySpark 4.3.0 for these frames.
+    pd = pytest.importorskip("pandas")
+    np = pytest.importorskip("numpy")
+    data = make(pd, np)
+    df = session.createDataFrame(data) if schema is None else 
session.createDataFrame(data, schema)
+    assert df.schema.simpleString() == expected_type
+    assert [tuple(row) for row in df.collect()] == expected
+
+
 def test_samples_and_splits_take_a_seed_of_the_client_as_pyspark(session):
     # As PySpark's: without a seed, the client takes one, so every action of a 
sample returns
     # the same rows, and the splits of randomSplit are disjoint and have every 
row; and the
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 41c4f50a..388993a8 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -9364,12 +9364,18 @@ def 
test_container_input_values_and_errors(container_input_capture_input, contai
     value = container([container([1, None]), container([2]), container()])
     table = container_input_capture_input([(value,)], nested_schema)
     assert table.column(0)[0].as_py() == [[1, None], [2], []]
-    with pytest.raises(TypeError, match="unsupported value type"):
+    # A plain frame with default labels converts with Arrow, as PySpark does, 
not with the row
+    # path, so the error is the Arrow conversion's (PySpark gives the same 
error).
+    from pyspark.errors import PySparkValueError
+
+    with pytest.raises(PySparkValueError, match="Exception thrown when 
converting pandas.Series"):
         container_input_capture_input([(container([object()]),)], array_schema)
 
 
 @pytest.mark.parametrize("container", [list, tuple])
 def 
test_container_input_subclass_numeric_precedence(container_input_capture_input, 
container):
+    from pyspark.errors import PySparkValueError
+
     trace = []
 
     class Indexed(container):
@@ -9386,14 +9392,18 @@ def 
test_container_input_subclass_numeric_precedence(container_input_capture_inp
         def __iter__(self):
             raise AssertionError("existing built-in iterator must remain in 
use")
 
-    cases = [(Indexed([1, 2]), T.LongType(), 42),
-             (Floating([1, 2]), T.DoubleType(), 1.25),
-             (Iterated([1, 2]), T.ArrayType(T.LongType()), [1, 2])]
-    for value, dtype, expected in cases:
-        schema = T.StructType([T.StructField("value", dtype)])
-        table = container_input_capture_input([(value,)], schema)
-        assert table.column(0)[0].as_py() == expected
-    assert trace == ["index", "float"]
+    # The Arrow conversion of a plain frame, as PySpark's, reads __index__ of 
a long column,
+    # but neither __float__ of a double column nor __iter__ of an array column.
+    table = container_input_capture_input(
+        [(Indexed([1, 2]),)], T.StructType([T.StructField("value", 
T.LongType())]))
+    assert table.column(0)[0].as_py() == 42
+    with pytest.raises(PySparkValueError, match="Exception thrown when 
converting pandas.Series"):
+        container_input_capture_input(
+            [(Floating([1, 2]),)], T.StructType([T.StructField("value", 
T.DoubleType())]))
+    array_schema = T.StructType([T.StructField("value", 
T.ArrayType(T.LongType()))])
+    with pytest.raises(AssertionError, match="existing built-in iterator must 
remain in use"):
+        container_input_capture_input([(Iterated([1, 2]),)], array_schema)
+    assert trace == ["index"]
 
 
 def 
test_container_input_recursive_leaf_and_row_types(container_input_capture_input):
@@ -16100,6 +16110,42 @@ def 
test_print_schema_default_reuses_cached_schema(mock_connect, capsys):
     assert service.execute_calls == 0
 
 
[email protected]("frame, schema, expected", [
+    (lambda pd, np: pd.DataFrame([[None], [3]]), "x int", [[None, 3]]),
+    (lambda pd, np: pd.DataFrame([[None], [3]]), "x long", [[None, 3]]),
+    (lambda pd, np: pd.DataFrame(np.array([[1.0, np.nan], [2.0, 3.0]])), "a 
int, b int",
+     [[1, 2], [None, 3]]),
+])
+def test_default_range_index_frame_takes_arrow_route(mock_connect, frame, 
schema, expected):
+    # A plain frame with default (RangeIndex) column labels converts with 
Arrow, as PySpark
+    # does: the NaN of a float column is a null, not a "Type mismatch in row 
data" error.
+    pd = pytest.importorskip("pandas")
+    np = pytest.importorskip("numpy")
+    data = frame(pd, np)
+    assert isinstance(data.columns, pd.RangeIndex)
+    table = _native_number_input_table(mock_connect, data, schema)
+    assert [table.column(i).to_pylist() for i in range(table.num_columns)] == 
expected
+
+
+def test_default_range_index_labels_keep_row_path_for_overrides(monkeypatch):
+    from pyspark.sql._pandas_input import _has_canonical_columns
+
+    pd = pytest.importorskip("pandas")
+    columns = pd.DataFrame([[1, 2]]).columns
+    assert _has_canonical_columns(columns)
+    # A subclass, or an instance or class override of iteration, keeps the row 
path as for an Index.
+
+    class RangeSubclass(pd.RangeIndex):
+        pass
+
+    assert not _has_canonical_columns(RangeSubclass(2))
+    overridden = pd.RangeIndex(2)
+    vars(overridden)["__iter__"] = lambda: iter(())
+    assert not _has_canonical_columns(overridden)
+    monkeypatch.setattr(pd.RangeIndex, "__len__", lambda self: 2)
+    assert not _has_canonical_columns(pd.RangeIndex(2))
+
+
 @pytest.mark.parametrize("level", [None, []])
 def test_print_schema_falsey_level_uses_default(mock_connect, capsys, level):
     spark, service = mock_connect


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to