This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new e0163c1b [SPARK-61227][CONNECT][PYTHON] Convert a pandas frame with
default column labels through Arrow, as PySpark does, not on the row path
e0163c1b is described below
commit e0163c1b958639b72955085db16042162127a42b
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Nov 9 17:30:34 2026 +0900
[SPARK-61227][CONNECT][PYTHON] Convert a pandas frame with default column
labels through Arrow, as PySpark does, not on the row path
### What changes were proposed in this pull request?
[SPARK-61227](https://issues.apache.org/jira/browse/SPARK-61227): Convert a
pandas frame with default column labels through Arrow, as PySpark does.
`_has_canonical_columns` in `python/pyspark/sql/_pandas_input.py` accepts a
default `pd.RangeIndex` of column labels, so the Arrow routes of
`createDataFrame` (explicit struct schema, names or inferred schema, and the
generic explicit route) take a plain frame whose labels are the default ones.
The check for a RangeIndex has the same override rules as the check for an
`Index`: the exact type, no override of `__getattribute__`, `__iter__` or
`__len__` on the class or the instance, a pa [...]
Official PySpark 4.3.0 converts these frames with Arrow. Across a matrix of
RangeIndex frames (numeric frames with NaN under int, long and double schemas,
object columns of Rows, dicts with and without
`spark.sql.execution.pandas.inferPandasDictAsMap`, lists of naive datetimes and
Decimals, strings, timestamps with time zones, empty frames, names lists and
DDL schemas, and Arrow on and off), the results and errors now match official
PySpark 4.3.0. The Rows under non-struct schemas, th [...]
### Why are the changes needed?
`pd.DataFrame([[None], [3]])` and `pd.DataFrame(ndarray)` have default
column labels, a RangeIndex, which the Arrow routes did not accept, so they
took the native row path. There `spark.createDataFrame(pd.DataFrame([[None],
[3]]), "x int")` raised `PySparkRuntimeError: Type mismatch in row data`, while
PySpark 4.3.0 returns `[Row(x=None), Row(x=3)]`. The same frame with named or
integer labels already worked.
### Does this PR introduce _any_ user-facing change?
Yes. A pandas frame with default column labels now converts with Arrow
under an explicit schema, as PySpark does, instead of failing or converting
differently on the row path. Customized (subclassed) frames and frames with
other labels are unchanged.
Two existing tests in `test_dropin_offline.py` pinned the row path for
these frames. `test_container_input_values_and_errors` and
`test_container_input_subclass_numeric_precedence` now expect the Arrow
conversion errors (`PySparkValueError`, "Exception thrown when converting
pandas.Series") that official PySpark 4.3.0 raises for the same frames.
### How was this patch tested?
- `test_default_range_index_frame_takes_arrow_route` (offline): the NaN of
a float column under `x int`, `x long` and `a int, b int` converts to nulls,
not a row-path error.
- `test_default_range_index_labels_keep_row_path_for_overrides` (offline):
the label check accepts a default RangeIndex and declines a subclass, an
instance `__iter__` override and a class `__len__` override.
- `test_default_range_index_pandas_frames_convert_as_pyspark` (live): six
frames (NaN under int, Rows under an array schema, dicts, lists of naive
datetimes and Decimals) give the schemas and values of PySpark 4.3.0.
- The existing tests `test_container_input_values_and_errors` and
`test_container_input_subclass_numeric_precedence` are updated as described
above.
- The lib tests of both crates (658 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2811 passed, 8 skipped), and the
live drop-in and Arrow suites (230 passed) and live cases (12 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #406 from HyukjinKwon/SPARK-61227-fu-p9.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
python/pyspark/sql/_pandas_input.py | 19 +++++++++++
python/tests/test_dropin_e2e.py | 28 ++++++++++++++++
python/tests/test_dropin_offline.py | 64 +++++++++++++++++++++++++++++++------
3 files changed, 102 insertions(+), 9 deletions(-)
diff --git a/python/pyspark/sql/_pandas_input.py
b/python/pyspark/sql/_pandas_input.py
index f177f336..538db6a2 100644
--- a/python/pyspark/sql/_pandas_input.py
+++ b/python/pyspark/sql/_pandas_input.py
@@ -145,6 +145,20 @@ def _has_canonical_arrow_string_array(array):
+ ("astype", "dtype", "_hasna"))
+def _has_canonical_range_columns(columns, range_module):
+ """Accept the default RangeIndex of column labels, which iterates its own
int range.
+
+ As for an Index, subclasses and instance overrides keep the row path.
+ """
+ RangeIndex = range_module.RangeIndex
+ if (RangeIndex.__getattribute__ is not object.__getattribute__
+ or "__iter__" in vars(columns) or "__len__" in vars(columns)
+ or not _defined_in(RangeIndex.__iter__, "__iter__", range_module)
+ or not _defined_in(RangeIndex.__len__, "__len__", range_module)):
+ return False
+ return type(vars(columns).get("_range")) is range
+
+
def _has_canonical_columns(columns):
"""Decline label containers whose row-path iteration can run custom
hooks."""
import sys
@@ -153,6 +167,11 @@ def _has_canonical_columns(columns):
import pandas as pd
from pandas.core import base as base_module
from pandas.core.indexes import base as index_module
+ from pandas.core.indexes import range as range_module
+
+ # A default RangeIndex (pd.DataFrame of a list or ndarray) takes the Arrow
routes too.
+ if pd.RangeIndex is range_module.RangeIndex and type(columns) is
pd.RangeIndex:
+ return _has_canonical_range_columns(columns, range_module)
Index = index_module.Index
if (pd.Index is not Index or type(columns) is not Index
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 0419b353..d68e38ea 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -24,6 +24,7 @@ import os
import time
from collections import namedtuple
from datetime import date, datetime
+from decimal import Decimal
import pyarrow as pa
import pytest
@@ -3701,6 +3702,33 @@ def
test_create_dataframe_raises_merge_errors_as_pyspark(session):
assert error.value.getMessageParameters() == parameters, data
[email protected]("make, schema, expected_type, expected", [
+ (lambda pd, np: pd.DataFrame([[None], [3]]), "x int", "struct<x:int>",
[(None,), (3,)]),
+ (lambda pd, np: pd.DataFrame(np.array([[1.0, np.nan], [2.0, 3.0]])), "a
int, b int",
+ "struct<a:int,b:int>", [(1, None), (2, 3)]),
+ (lambda pd, np: pd.DataFrame([[Row(a=1, b=2)], [Row(a=3, b=4)]]), "x
array<bigint>",
+ "struct<x:array<bigint>>", [([1, 2],), ([3, 4],)]),
+ (lambda pd, np: pd.DataFrame([[{"u": 1, "v": "a"}], [{"u": 2, "v":
"b"}]]), None,
+ "struct<0:struct<u:bigint,v:string>>", [(Row(u=1, v="a"),), (Row(u=2,
v="b"),)]),
+ (lambda pd, np: pd.DataFrame([[[datetime(2020, 1, 1, 12)]],
[[datetime(2021, 1, 1)]]]), None,
+ "struct<0:array<timestamp_ntz>>", [([datetime(2020, 1, 1, 12)],),
([datetime(2021, 1, 1)],)]),
+ (lambda pd, np: pd.DataFrame([[[Decimal("1.5"), Decimal("10.25")]],
[[Decimal("2")]]]), None,
+ "struct<0:array<decimal(4,2)>>",
+ [([Decimal("1.50"), Decimal("10.25")],), ([Decimal("2.00")],)]),
+])
+def test_default_range_index_pandas_frames_convert_as_pyspark(
+ session, make, schema, expected_type, expected
+):
+ # Plain frames with default (RangeIndex) labels convert with Arrow, and
give the results
+ # of PySpark 4.3.0 for these frames.
+ pd = pytest.importorskip("pandas")
+ np = pytest.importorskip("numpy")
+ data = make(pd, np)
+ df = session.createDataFrame(data) if schema is None else
session.createDataFrame(data, schema)
+ assert df.schema.simpleString() == expected_type
+ assert [tuple(row) for row in df.collect()] == expected
+
+
def test_samples_and_splits_take_a_seed_of_the_client_as_pyspark(session):
# As PySpark's: without a seed, the client takes one, so every action of a
sample returns
# the same rows, and the splits of randomSplit are disjoint and have every
row; and the
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 41c4f50a..388993a8 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -9364,12 +9364,18 @@ def
test_container_input_values_and_errors(container_input_capture_input, contai
value = container([container([1, None]), container([2]), container()])
table = container_input_capture_input([(value,)], nested_schema)
assert table.column(0)[0].as_py() == [[1, None], [2], []]
- with pytest.raises(TypeError, match="unsupported value type"):
+ # A plain frame with default labels converts with Arrow, as PySpark does,
not with the row
+ # path, so the error is the Arrow conversion's (PySpark gives the same
error).
+ from pyspark.errors import PySparkValueError
+
+ with pytest.raises(PySparkValueError, match="Exception thrown when
converting pandas.Series"):
container_input_capture_input([(container([object()]),)], array_schema)
@pytest.mark.parametrize("container", [list, tuple])
def
test_container_input_subclass_numeric_precedence(container_input_capture_input,
container):
+ from pyspark.errors import PySparkValueError
+
trace = []
class Indexed(container):
@@ -9386,14 +9392,18 @@ def
test_container_input_subclass_numeric_precedence(container_input_capture_inp
def __iter__(self):
raise AssertionError("existing built-in iterator must remain in
use")
- cases = [(Indexed([1, 2]), T.LongType(), 42),
- (Floating([1, 2]), T.DoubleType(), 1.25),
- (Iterated([1, 2]), T.ArrayType(T.LongType()), [1, 2])]
- for value, dtype, expected in cases:
- schema = T.StructType([T.StructField("value", dtype)])
- table = container_input_capture_input([(value,)], schema)
- assert table.column(0)[0].as_py() == expected
- assert trace == ["index", "float"]
+ # The Arrow conversion of a plain frame, as PySpark's, reads __index__ of
a long column,
+ # but neither __float__ of a double column nor __iter__ of an array column.
+ table = container_input_capture_input(
+ [(Indexed([1, 2]),)], T.StructType([T.StructField("value",
T.LongType())]))
+ assert table.column(0)[0].as_py() == 42
+ with pytest.raises(PySparkValueError, match="Exception thrown when
converting pandas.Series"):
+ container_input_capture_input(
+ [(Floating([1, 2]),)], T.StructType([T.StructField("value",
T.DoubleType())]))
+ array_schema = T.StructType([T.StructField("value",
T.ArrayType(T.LongType()))])
+ with pytest.raises(AssertionError, match="existing built-in iterator must
remain in use"):
+ container_input_capture_input([(Iterated([1, 2]),)], array_schema)
+ assert trace == ["index"]
def
test_container_input_recursive_leaf_and_row_types(container_input_capture_input):
@@ -16100,6 +16110,42 @@ def
test_print_schema_default_reuses_cached_schema(mock_connect, capsys):
assert service.execute_calls == 0
[email protected]("frame, schema, expected", [
+ (lambda pd, np: pd.DataFrame([[None], [3]]), "x int", [[None, 3]]),
+ (lambda pd, np: pd.DataFrame([[None], [3]]), "x long", [[None, 3]]),
+ (lambda pd, np: pd.DataFrame(np.array([[1.0, np.nan], [2.0, 3.0]])), "a
int, b int",
+ [[1, 2], [None, 3]]),
+])
+def test_default_range_index_frame_takes_arrow_route(mock_connect, frame,
schema, expected):
+ # A plain frame with default (RangeIndex) column labels converts with
Arrow, as PySpark
+ # does: the NaN of a float column is a null, not a "Type mismatch in row
data" error.
+ pd = pytest.importorskip("pandas")
+ np = pytest.importorskip("numpy")
+ data = frame(pd, np)
+ assert isinstance(data.columns, pd.RangeIndex)
+ table = _native_number_input_table(mock_connect, data, schema)
+ assert [table.column(i).to_pylist() for i in range(table.num_columns)] ==
expected
+
+
+def test_default_range_index_labels_keep_row_path_for_overrides(monkeypatch):
+ from pyspark.sql._pandas_input import _has_canonical_columns
+
+ pd = pytest.importorskip("pandas")
+ columns = pd.DataFrame([[1, 2]]).columns
+ assert _has_canonical_columns(columns)
+ # A subclass, or an instance or class override of iteration, keeps the row
path as for an Index.
+
+ class RangeSubclass(pd.RangeIndex):
+ pass
+
+ assert not _has_canonical_columns(RangeSubclass(2))
+ overridden = pd.RangeIndex(2)
+ vars(overridden)["__iter__"] = lambda: iter(())
+ assert not _has_canonical_columns(overridden)
+ monkeypatch.setattr(pd.RangeIndex, "__len__", lambda self: 2)
+ assert not _has_canonical_columns(pd.RangeIndex(2))
+
+
@pytest.mark.parametrize("level", [None, []])
def test_print_schema_falsey_level_uses_default(mock_connect, capsys, level):
spark, service = mock_connect
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]