This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 79ac194 [SPARK-60745][CONNECT] Keep Arrow-backed pandas string input
columnar
79ac194 is described below
commit 79ac1944d330ae0ae2695eba577cf0412617e6e1
Author: DB Tsai <[email protected]>
AuthorDate: Sat Nov 7 11:22:45 2026 +0900
[SPARK-60745][CONNECT] Keep Arrow-backed pandas string input columnar
### What changes were proposed in this pull request?
This replaces #145 by DB Tsai, which does not allow edits by maintainers,
with its commits plus the merge with master and the review changes below.
`createDataFrame(pandas_df, schema)`, for a pandas DataFrame of at least
10,000 rows whose columns are all kept in pandas' Arrow string storage
(`StringDtype` with `pyarrow` storage, as `string[pyarrow]`, including the
NumPy-semantics variant of pandas 2.2 and the `str` dtype of pandas 3), with an
explicit `StructType` of `StringType` (UTF8_BINARY) fields, now passes that
storage to the Arrow importer as Arrow string columns (`_arrow_string_table` in
`python/pyspark/sql/_pandas_input. [...]
Everything else keeps the row path: `string[python]` storage, mixed dtypes,
a null in a non-nullable field, subclasses of pandas' Arrow string arrays,
smaller frames, other schemas, and the cases that the guards of #208
(SPARK-60720) already decline.
Review changes on top of #145:
- The storage is read through pandas' public API (`Series.array`,
`__arrow_array__`), and only pandas' own array classes are trusted, instead of
walking pandas' block manager (`_mgr`, `iget_values`, `blknos`, `blklocs`) and
reading `_pa_array` behind about 200 lines of checks of pandas internals.
- The table goes to the Arrow importer, as the int64 and date tables do, so
it is uploaded as cached artifacts at
`spark.sql.session.localRelationCacheThreshold`; #145 sent it inline, and a
large inline relation fails (an HTTP/2 error above about 128 MB).
- It works on pandas 2.2, the lower bound of the supported range, whose
`StringDtype` has a `pyarrow_numpy` storage instead of `na_value`.
`test_arrow_string_input.py`, which required pandas 2.3, is replaced by tests
in `test_arrow_handoff.py`.
- The architecture docs describe the path.
### Why are the changes needed?
[SPARK-60745](https://issues.apache.org/jira/browse/SPARK-60745):
converting Arrow-backed strings into Python rows and back to Arrow dominates
`createDataFrame` of such frames. Measured on macOS against a local Spark 4.3.0
server, as the client CPU time of `createDataFrame` of 200,000 rows of three
`string[pyarrow]` columns (ASCII, empty, NUL, accents, CJK, emoji, combining
marks and nulls) with an explicit schema (median of 5): 266 ms on the current
master, 26 ms with this change (of [...]
### Does this PR introduce _any_ user-facing change?
No, other than speed.
### How was this patch tested?
- New tests in `python/tests/test_arrow_handoff.py`, against the mock Arrow
input service: every cell, field name, nullability and metadata of
`string[pyarrow]`, NumPy-semantics, `large_string`-backed and sliced columns,
inline below the relation cache threshold and uploaded at it, without
materializing rows; the same table as the row path; and each unsupported input
keeps the row path.
- Against the current master and a local Spark 4.3.0 server: the full
suites (results below the merge).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6); the merge with master, the review
changes and the measurement by Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #249 from HyukjinKwon/SPARK-60745-arrow-string.
Lead-authored-by: DB Tsai <[email protected]>
Co-authored-by: DB Tsai <[email protected]>
Co-authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/session.rs | 16 ++++++
docs/architecture.md | 4 ++
python/pyspark/sql/_pandas_input.py | 48 +++++++++++++++-
python/tests/test_arrow_handoff.py | 107 +++++++++++++++++++++++++++++++++++-
4 files changed, 171 insertions(+), 4 deletions(-)
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index a5d8cdd..8ec0452 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -1052,6 +1052,22 @@ impl PySparkSession {
.import("pyspark.sql._pandas_input")?
.getattr("prepare_string_input")?
.call1((data, schema))?;
+ // Columns kept in pandas' Arrow string storage come as an Arrow
table, which the
+ // importer sends, or uploads at the relation cache threshold.
+ if !ipc.is_none() && !ipc.is_instance_of::<pyo3::types::PyBytes>()
{
+ let frame =
+ crate::pyarrow_import::create_dataframe(py, &self.session,
&ipc, schema)?;
+ return Ok(match &spec {
+ Spec::Struct(fields) if !fields.is_empty() && ipc.len()? >
0 => {
+ PyDataFrame::new(
+ frame
+ .dataframe
+ .to_df(fields.iter().map(|field|
field.name.as_str()).collect()),
+ )
+ }
+ _ => frame,
+ });
+ }
if !ipc.is_none() {
let fields = match &spec {
Spec::Struct(fields) => fields,
diff --git a/docs/architecture.md b/docs/architecture.md
index ee734f0..0c7b7be 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -159,6 +159,10 @@ Pandas frames of at least 100,000 rows whose columns are
all NumPy `float64`,
with an explicit `StructType` of `DoubleType` fields, are encoded to Arrow IPC
from their columns, with pandas' missing values as nulls, and sent inline as
the
row path sends them, without the importer's configuration RPC.
+Pandas frames of at least 10,000 rows whose columns are all kept in pandas'
Arrow
+string storage (`string[pyarrow]`, and the `str` dtype of pandas 3), with an
+explicit `StructType` of `StringType` fields, pass that storage to this
importer
+as Arrow string columns, without Python rows.
Inputs at or above `spark.sql.session.localRelationCacheThreshold` are
serialized
into independent IPC streams and uploaded as session-scoped cache artifacts.
The plan carries `ChunkedCachedLocalRelation` hashes instead of embedding the
diff --git a/python/pyspark/sql/_pandas_input.py
b/python/pyspark/sql/_pandas_input.py
index 8cf6d98..1077dd4 100644
--- a/python/pyspark/sql/_pandas_input.py
+++ b/python/pyspark/sql/_pandas_input.py
@@ -533,7 +533,8 @@ def prepare_float64_input(frame, schema):
def prepare_string_input(frame, schema):
- """Prepare inline IPC for explicit columns of exact strings or None.
+ """Prepare inline IPC for explicit columns of exact strings or None, or an
Arrow table for
+ columns that pandas keeps in Arrow string storage (see
_arrow_string_table).
Inspect every object before Arrow conversion: str subclasses can override
conversion hooks, and pandas treats several other objects as missing
values.
@@ -598,9 +599,10 @@ def prepare_string_input(frame, schema):
for field in schema.fields
) or len(set(schema.names)) != width:
return None
+ dtypes = frame.dtypes
if not all(type(dtype) is type(np.dtype("object")) and dtype ==
np.dtype("object")
- for dtype in frame.dtypes):
- return None
+ for dtype in dtypes):
+ return _arrow_string_table(frame, schema, dtypes)
# A custom ExtensionArray may report a genuine NumPy dtype while its
# Arrow hook emits different values from to_numpy(). Trust only pandas'
# concrete NumPy backing array, including its name in older pandas
releases.
@@ -631,6 +633,46 @@ def prepare_string_input(frame, schema):
return _pandas_string_ipc(arrays, schema)
+def _arrow_string_table(frame, schema, dtypes):
+ """The columns of a frame kept in pandas' Arrow string storage
(string[pyarrow], and the
+ str dtype of pandas 3), as an Arrow table of the schema, or None.
+
+ For the row path, pandas converts such a column from that storage, with
its missing
+ values as None, so the storage holds the same values. Only pandas' own
array classes are
+ trusted to keep them there: a subclass could convert differently. The
caller's guards of
+ prepare_string_input (frame, schema, names, size) have passed.
+ """
+ import pandas as pd
+ import pyarrow as pa
+ from pandas.core.arrays import string_arrow
+
+ classes = {pd.arrays.ArrowStringArray,
+ getattr(string_arrow, "ArrowStringArrayNumpySemantics", None)}
- {None}
+ if not all(type(dtype) is pd.StringDtype and dtype.storage in ("pyarrow",
"pyarrow_numpy")
+ for dtype in dtypes):
+ return None
+ arrays = []
+ for (_, column), field in zip(frame.items(), schema.fields):
+ array = column.array
+ if type(array) not in classes:
+ return None
+ values = pa.chunked_array(array.__arrow_array__())
+ if values.type not in (pa.string(), pa.large_string()):
+ return None
+ try:
+ # Malformed offsets or UTF-8 keep the row path and its error.
+ values.validate(full=True)
+ values = values.cast(pa.string())
+ except (pa.ArrowException, OverflowError):
+ return None
+ if not field.nullable and values.null_count:
+ return None
+ arrays.append(values)
+ # The importer applies the nullability and metadata of the schema, and
uploads the table
+ # when it is at least spark.sql.session.localRelationCacheThreshold.
+ return pa.Table.from_arrays(arrays, names=schema.names)
+
+
def prepare_date_table(frame, schema):
"""Accept only explicit date columns whose cells are exact dates or None.
diff --git a/python/tests/test_arrow_handoff.py
b/python/tests/test_arrow_handoff.py
index 9424ac5..738c3fb 100644
--- a/python/tests/test_arrow_handoff.py
+++ b/python/tests/test_arrow_handoff.py
@@ -3968,8 +3968,113 @@ def
test_string_column_guard_preserves_row_fallback(reason):
frame.iat[-1, -1] = value
if reason in ("late_unsupported", "late_surrogate"):
assert frame.iat[-1, -1] is value
- assert prepare_string_input(frame, schema) is None
+ result = prepare_string_input(frame, schema)
+ if reason == "arrow_string_dtype":
+ # Arrow string storage has a path of its own, which returns an Arrow
table.
+ assert type(result) is pa.Table
+ else:
+ assert result is None
+
+
+
+# Columns kept in pandas' Arrow string storage reach the importer as Arrow,
without Python rows.
+
+def _arrow_string_column(texts, storage):
+ import pyarrow as pa
+ if storage == "numpy_semantics":
+ try:
+ dtype = pd.StringDtype("pyarrow", na_value=float("nan"))
+ except TypeError: # pandas 2.2
+ dtype = pd.StringDtype("pyarrow_numpy")
+ return pd.array(texts, dtype=dtype)
+ if storage == "large_string":
+ return pd.arrays.ArrowStringArray(pa.chunked_array(
+ [pa.array(texts[:7], pa.large_string()), pa.array(texts[7:],
pa.large_string())]))
+ return pd.array(texts, dtype="string[pyarrow]")
+
+
[email protected]("cached", [False, True])
[email protected]("storage", ["pyarrow", "numpy_semantics",
"large_string", "sliced"])
+def test_arrow_string_columns_preserve_every_ipc_cell(protocol, monkeypatch,
cached, storage):
+ from pyspark.sql.pandas.types import from_arrow_metadata
+
+ texts = ["", "a", "\x00", "café", "中文", "\U0001f600", "é", None] * 2000
+ other = [None if i % 3 == 0 else str(i) for i in range(len(texts))]
+ frame = pd.DataFrame({
+ "original": _arrow_string_column(texts, storage),
+ "b": _arrow_string_column(other, storage),
+ "c": _arrow_string_column([None] * len(texts), storage),
+ })
+ expected = [texts, other, [None] * len(texts)]
+ if storage == "sliced":
+ frame = frame.iloc[3:-5]
+ expected = [values[3:-5] for values in expected]
+ schema = T.StructType([
+ T.StructField("renamed.`text", T.StringType(), True, {"source":
"string"}),
+ T.StructField("other", T.StringType()),
+ T.StructField("all_null", T.StringType()),
+ ])
+
+ def no_rows(*args, **kwargs):
+ raise AssertionError("Arrow string storage must not materialize Python
rows")
+
+ for method in ("to_numpy", "iterrows", "itertuples"):
+ monkeypatch.setattr(pd.DataFrame, method, no_rows)
+ monkeypatch.setattr(pd.DataFrame, "values", property(no_rows))
+ monkeypatch.setattr(pd.Series, "tolist", no_rows)
+ overrides = {"spark.sql.session.localRelationCacheThreshold": "1" if
cached else "999999999"}
+ with _arrow_input_service(protocol, **overrides) as (spark, state):
+ df = spark.createDataFrame(frame, schema)
+ df.toArrow()
+ assert state["column_names"] == schema.names
+ table = state["input_tables"][0]
+ assert table.schema.names == schema.names
+ for column, values, field in zip(table.columns, expected,
schema.fields):
+ assert column.type == pa.string()
+ assert column.to_pylist() == values
+ actual_field = table.schema.field(field.name)
+ assert actual_field.nullable == field.nullable
+ assert from_arrow_metadata(actual_field.metadata) ==
(field.metadata or None)
+ assert bool(state["uploads"]) == cached
+
+
[email protected]("reason", [
+ "python_storage", "mixed", "nonnullable_null", "subclass", "small",
"atomic_schema",
+])
+def test_arrow_string_columns_keep_the_row_path(reason):
+ from pyspark.sql._pandas_input import prepare_string_input
+
+ texts = ["a", None, "b"] * 4000
+ columns = {"a": _arrow_string_column(texts, "pyarrow"), "b":
_arrow_string_column(texts, "pyarrow")}
+ fields = [T.StructField("a", T.StringType()), T.StructField("b",
T.StringType())]
+ if reason == "python_storage":
+ columns["b"] = pd.array(texts, dtype="string[python]")
+ elif reason == "mixed":
+ columns["b"] = pd.Series(texts, dtype=object)
+ elif reason == "nonnullable_null":
+ fields[1] = T.StructField("b", T.StringType(), False)
+ elif reason == "subclass":
+ class Custom(pd.arrays.ArrowStringArray):
+ pass
+ columns["b"] = Custom(pa.chunked_array([pa.array(texts)]))
+ elif reason == "small":
+ columns = {name: values[:9999] for name, values in columns.items()}
+ elif reason == "atomic_schema":
+ fields[1] = T.StructField("b", T.IntegerType())
+ frame = pd.DataFrame(columns)
+ assert prepare_string_input(frame, T.StructType(fields)) is None
+
+
+def test_arrow_string_columns_match_the_row_path(protocol):
+ texts = ["x" * (i % 5) if i % 4 else None for i in range(12000)]
+ schema = T.StructType([T.StructField("v", T.StringType())])
+ tables = []
+ for dtype in ("string[pyarrow]", object):
+ with _arrow_input_service(protocol) as (spark, state):
+ spark.createDataFrame(pd.DataFrame({"v": pd.array(texts,
dtype=dtype)}), schema).toArrow()
+ tables.append(state["input_tables"][0])
+ assert tables[0].equals(tables[1])
@pytest.mark.parametrize("reason", [
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]