This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 79ac194  [SPARK-60745][CONNECT] Keep Arrow-backed pandas string input 
columnar
79ac194 is described below

commit 79ac1944d330ae0ae2695eba577cf0412617e6e1
Author: DB Tsai <[email protected]>
AuthorDate: Sat Nov 7 11:22:45 2026 +0900

    [SPARK-60745][CONNECT] Keep Arrow-backed pandas string input columnar
    
    ### What changes were proposed in this pull request?
    
    This replaces #145 by DB Tsai, which does not allow edits by maintainers, 
with its commits plus the merge with master and the review changes below.
    
    `createDataFrame(pandas_df, schema)`, for a pandas DataFrame of at least 
10,000 rows whose columns are all kept in pandas' Arrow string storage 
(`StringDtype` with `pyarrow` storage, as `string[pyarrow]`, including the 
NumPy-semantics variant of pandas 2.2 and the `str` dtype of pandas 3), with an 
explicit `StructType` of `StringType` (UTF8_BINARY) fields, now passes that 
storage to the Arrow importer as Arrow string columns (`_arrow_string_table` in 
`python/pyspark/sql/_pandas_input. [...]
    
    Everything else keeps the row path: `string[python]` storage, mixed dtypes, 
a null in a non-nullable field, subclasses of pandas' Arrow string arrays, 
smaller frames, other schemas, and the cases that the guards of #208 
(SPARK-60720) already decline.
    
    Review changes on top of #145:
    - The storage is read through pandas' public API (`Series.array`, 
`__arrow_array__`), and only pandas' own array classes are trusted, instead of 
walking pandas' block manager (`_mgr`, `iget_values`, `blknos`, `blklocs`) and 
reading `_pa_array` behind about 200 lines of checks of pandas internals.
    - The table goes to the Arrow importer, as the int64 and date tables do, so 
it is uploaded as cached artifacts at 
`spark.sql.session.localRelationCacheThreshold`; #145 sent it inline, and a 
large inline relation fails (an HTTP/2 error above about 128 MB).
    - It works on pandas 2.2, the lower bound of the supported range, whose 
`StringDtype` has a `pyarrow_numpy` storage instead of `na_value`. 
`test_arrow_string_input.py`, which required pandas 2.3, is replaced by tests 
in `test_arrow_handoff.py`.
    - The architecture docs describe the path.
    
    ### Why are the changes needed?
    
    [SPARK-60745](https://issues.apache.org/jira/browse/SPARK-60745): 
converting Arrow-backed strings into Python rows and back to Arrow dominates 
`createDataFrame` of such frames. Measured on macOS against a local Spark 4.3.0 
server, as the client CPU time of `createDataFrame` of 200,000 rows of three 
`string[pyarrow]` columns (ASCII, empty, NUL, accents, CJK, emoji, combining 
marks and nulls) with an explicit schema (median of 5): 266 ms on the current 
master, 26 ms with this change (of [...]
    
    ### Does this PR introduce _any_ user-facing change?
    
    No, other than speed.
    
    ### How was this patch tested?
    
    - New tests in `python/tests/test_arrow_handoff.py`, against the mock Arrow 
input service: every cell, field name, nullability and metadata of 
`string[pyarrow]`, NumPy-semantics, `large_string`-backed and sliced columns, 
inline below the relation cache threshold and uploaded at it, without 
materializing rows; the same table as the row path; and each unsupported input 
keeps the row path.
    - Against the current master and a local Spark 4.3.0 server: the full 
suites (results below the merge).
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex (GPT-6); the merge with master, the review 
changes and the measurement by Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #249 from HyukjinKwon/SPARK-60745-arrow-string.
    
    Lead-authored-by: DB Tsai <[email protected]>
    Co-authored-by: DB Tsai <[email protected]>
    Co-authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/session.rs    |  16 ++++++
 docs/architecture.md                |   4 ++
 python/pyspark/sql/_pandas_input.py |  48 +++++++++++++++-
 python/tests/test_arrow_handoff.py  | 107 +++++++++++++++++++++++++++++++++++-
 4 files changed, 171 insertions(+), 4 deletions(-)

diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index a5d8cdd..8ec0452 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -1052,6 +1052,22 @@ impl PySparkSession {
                 .import("pyspark.sql._pandas_input")?
                 .getattr("prepare_string_input")?
                 .call1((data, schema))?;
+            // Columns kept in pandas' Arrow string storage come as an Arrow 
table, which the
+            // importer sends, or uploads at the relation cache threshold.
+            if !ipc.is_none() && !ipc.is_instance_of::<pyo3::types::PyBytes>() 
{
+                let frame =
+                    crate::pyarrow_import::create_dataframe(py, &self.session, 
&ipc, schema)?;
+                return Ok(match &spec {
+                    Spec::Struct(fields) if !fields.is_empty() && ipc.len()? > 
0 => {
+                        PyDataFrame::new(
+                            frame
+                                .dataframe
+                                .to_df(fields.iter().map(|field| 
field.name.as_str()).collect()),
+                        )
+                    }
+                    _ => frame,
+                });
+            }
             if !ipc.is_none() {
                 let fields = match &spec {
                     Spec::Struct(fields) => fields,
diff --git a/docs/architecture.md b/docs/architecture.md
index ee734f0..0c7b7be 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -159,6 +159,10 @@ Pandas frames of at least 100,000 rows whose columns are 
all NumPy `float64`,
 with an explicit `StructType` of `DoubleType` fields, are encoded to Arrow IPC
 from their columns, with pandas' missing values as nulls, and sent inline as 
the
 row path sends them, without the importer's configuration RPC.
+Pandas frames of at least 10,000 rows whose columns are all kept in pandas' 
Arrow
+string storage (`string[pyarrow]`, and the `str` dtype of pandas 3), with an
+explicit `StructType` of `StringType` fields, pass that storage to this 
importer
+as Arrow string columns, without Python rows.
 Inputs at or above `spark.sql.session.localRelationCacheThreshold` are 
serialized
 into independent IPC streams and uploaded as session-scoped cache artifacts.
 The plan carries `ChunkedCachedLocalRelation` hashes instead of embedding the
diff --git a/python/pyspark/sql/_pandas_input.py 
b/python/pyspark/sql/_pandas_input.py
index 8cf6d98..1077dd4 100644
--- a/python/pyspark/sql/_pandas_input.py
+++ b/python/pyspark/sql/_pandas_input.py
@@ -533,7 +533,8 @@ def prepare_float64_input(frame, schema):
 
 
 def prepare_string_input(frame, schema):
-    """Prepare inline IPC for explicit columns of exact strings or None.
+    """Prepare inline IPC for explicit columns of exact strings or None, or an 
Arrow table for
+    columns that pandas keeps in Arrow string storage (see 
_arrow_string_table).
 
     Inspect every object before Arrow conversion: str subclasses can override
     conversion hooks, and pandas treats several other objects as missing 
values.
@@ -598,9 +599,10 @@ def prepare_string_input(frame, schema):
         for field in schema.fields
     ) or len(set(schema.names)) != width:
         return None
+    dtypes = frame.dtypes
     if not all(type(dtype) is type(np.dtype("object")) and dtype == 
np.dtype("object")
-               for dtype in frame.dtypes):
-        return None
+               for dtype in dtypes):
+        return _arrow_string_table(frame, schema, dtypes)
     # A custom ExtensionArray may report a genuine NumPy dtype while its
     # Arrow hook emits different values from to_numpy(). Trust only pandas'
     # concrete NumPy backing array, including its name in older pandas 
releases.
@@ -631,6 +633,46 @@ def prepare_string_input(frame, schema):
     return _pandas_string_ipc(arrays, schema)
 
 
+def _arrow_string_table(frame, schema, dtypes):
+    """The columns of a frame kept in pandas' Arrow string storage 
(string[pyarrow], and the
+    str dtype of pandas 3), as an Arrow table of the schema, or None.
+
+    For the row path, pandas converts such a column from that storage, with 
its missing
+    values as None, so the storage holds the same values. Only pandas' own 
array classes are
+    trusted to keep them there: a subclass could convert differently. The 
caller's guards of
+    prepare_string_input (frame, schema, names, size) have passed.
+    """
+    import pandas as pd
+    import pyarrow as pa
+    from pandas.core.arrays import string_arrow
+
+    classes = {pd.arrays.ArrowStringArray,
+               getattr(string_arrow, "ArrowStringArrayNumpySemantics", None)} 
- {None}
+    if not all(type(dtype) is pd.StringDtype and dtype.storage in ("pyarrow", 
"pyarrow_numpy")
+               for dtype in dtypes):
+        return None
+    arrays = []
+    for (_, column), field in zip(frame.items(), schema.fields):
+        array = column.array
+        if type(array) not in classes:
+            return None
+        values = pa.chunked_array(array.__arrow_array__())
+        if values.type not in (pa.string(), pa.large_string()):
+            return None
+        try:
+            # Malformed offsets or UTF-8 keep the row path and its error.
+            values.validate(full=True)
+            values = values.cast(pa.string())
+        except (pa.ArrowException, OverflowError):
+            return None
+        if not field.nullable and values.null_count:
+            return None
+        arrays.append(values)
+    # The importer applies the nullability and metadata of the schema, and 
uploads the table
+    # when it is at least spark.sql.session.localRelationCacheThreshold.
+    return pa.Table.from_arrays(arrays, names=schema.names)
+
+
 def prepare_date_table(frame, schema):
     """Accept only explicit date columns whose cells are exact dates or None.
 
diff --git a/python/tests/test_arrow_handoff.py 
b/python/tests/test_arrow_handoff.py
index 9424ac5..738c3fb 100644
--- a/python/tests/test_arrow_handoff.py
+++ b/python/tests/test_arrow_handoff.py
@@ -3968,8 +3968,113 @@ def 
test_string_column_guard_preserves_row_fallback(reason):
         frame.iat[-1, -1] = value
         if reason in ("late_unsupported", "late_surrogate"):
             assert frame.iat[-1, -1] is value
-    assert prepare_string_input(frame, schema) is None
+    result = prepare_string_input(frame, schema)
+    if reason == "arrow_string_dtype":
+        # Arrow string storage has a path of its own, which returns an Arrow 
table.
+        assert type(result) is pa.Table
+    else:
+        assert result is None
+
+
+
+# Columns kept in pandas' Arrow string storage reach the importer as Arrow, 
without Python rows.
+
+def _arrow_string_column(texts, storage):
+    import pyarrow as pa
+    if storage == "numpy_semantics":
+        try:
+            dtype = pd.StringDtype("pyarrow", na_value=float("nan"))
+        except TypeError:  # pandas 2.2
+            dtype = pd.StringDtype("pyarrow_numpy")
+        return pd.array(texts, dtype=dtype)
+    if storage == "large_string":
+        return pd.arrays.ArrowStringArray(pa.chunked_array(
+            [pa.array(texts[:7], pa.large_string()), pa.array(texts[7:], 
pa.large_string())]))
+    return pd.array(texts, dtype="string[pyarrow]")
+
+
[email protected]("cached", [False, True])
[email protected]("storage", ["pyarrow", "numpy_semantics", 
"large_string", "sliced"])
+def test_arrow_string_columns_preserve_every_ipc_cell(protocol, monkeypatch, 
cached, storage):
+    from pyspark.sql.pandas.types import from_arrow_metadata
+
+    texts = ["", "a", "\x00", "café", "中文", "\U0001f600", "é", None] * 2000
+    other = [None if i % 3 == 0 else str(i) for i in range(len(texts))]
+    frame = pd.DataFrame({
+        "original": _arrow_string_column(texts, storage),
+        "b": _arrow_string_column(other, storage),
+        "c": _arrow_string_column([None] * len(texts), storage),
+    })
+    expected = [texts, other, [None] * len(texts)]
+    if storage == "sliced":
+        frame = frame.iloc[3:-5]
+        expected = [values[3:-5] for values in expected]
+    schema = T.StructType([
+        T.StructField("renamed.`text", T.StringType(), True, {"source": 
"string"}),
+        T.StructField("other", T.StringType()),
+        T.StructField("all_null", T.StringType()),
+    ])
+
+    def no_rows(*args, **kwargs):
+        raise AssertionError("Arrow string storage must not materialize Python 
rows")
+
+    for method in ("to_numpy", "iterrows", "itertuples"):
+        monkeypatch.setattr(pd.DataFrame, method, no_rows)
+    monkeypatch.setattr(pd.DataFrame, "values", property(no_rows))
+    monkeypatch.setattr(pd.Series, "tolist", no_rows)
+    overrides = {"spark.sql.session.localRelationCacheThreshold": "1" if 
cached else "999999999"}
+    with _arrow_input_service(protocol, **overrides) as (spark, state):
+        df = spark.createDataFrame(frame, schema)
+        df.toArrow()
+        assert state["column_names"] == schema.names
+        table = state["input_tables"][0]
+        assert table.schema.names == schema.names
+        for column, values, field in zip(table.columns, expected, 
schema.fields):
+            assert column.type == pa.string()
+            assert column.to_pylist() == values
+            actual_field = table.schema.field(field.name)
+            assert actual_field.nullable == field.nullable
+            assert from_arrow_metadata(actual_field.metadata) == 
(field.metadata or None)
+        assert bool(state["uploads"]) == cached
+
+
[email protected]("reason", [
+    "python_storage", "mixed", "nonnullable_null", "subclass", "small", 
"atomic_schema",
+])
+def test_arrow_string_columns_keep_the_row_path(reason):
+    from pyspark.sql._pandas_input import prepare_string_input
+
+    texts = ["a", None, "b"] * 4000
+    columns = {"a": _arrow_string_column(texts, "pyarrow"), "b": 
_arrow_string_column(texts, "pyarrow")}
+    fields = [T.StructField("a", T.StringType()), T.StructField("b", 
T.StringType())]
+    if reason == "python_storage":
+        columns["b"] = pd.array(texts, dtype="string[python]")
+    elif reason == "mixed":
+        columns["b"] = pd.Series(texts, dtype=object)
+    elif reason == "nonnullable_null":
+        fields[1] = T.StructField("b", T.StringType(), False)
+    elif reason == "subclass":
+        class Custom(pd.arrays.ArrowStringArray):
+            pass
 
+        columns["b"] = Custom(pa.chunked_array([pa.array(texts)]))
+    elif reason == "small":
+        columns = {name: values[:9999] for name, values in columns.items()}
+    elif reason == "atomic_schema":
+        fields[1] = T.StructField("b", T.IntegerType())
+    frame = pd.DataFrame(columns)
+    assert prepare_string_input(frame, T.StructType(fields)) is None
+
+
+def test_arrow_string_columns_match_the_row_path(protocol):
+    texts = ["x" * (i % 5) if i % 4 else None for i in range(12000)]
+    schema = T.StructType([T.StructField("v", T.StringType())])
+    tables = []
+    for dtype in ("string[pyarrow]", object):
+        with _arrow_input_service(protocol) as (spark, state):
+            spark.createDataFrame(pd.DataFrame({"v": pd.array(texts, 
dtype=dtype)}), schema).toArrow()
+            tables.append(state["input_tables"][0])
+    assert tables[0].equals(tables[1])
 
 
 @pytest.mark.parametrize("reason", [


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to