This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 4bb3857 [SPARK-60720][CONNECT] Encode guarded pandas string columns
directly in Rust
4bb3857 is described below
commit 4bb3857ef8650265490eb0e81432a9e751f3cb6c
Author: DB Tsai <[email protected]>
AuthorDate: Fri Nov 6 22:58:54 2026 +0900
[SPARK-60720][CONNECT] Encode guarded pandas string columns directly in Rust
### What changes were proposed in this pull request?
This replaces #144 by DB Tsai, which does not allow edits by maintainers,
with its commits plus the merge with master and the review changes below. It is
stacked on #207 (SPARK-60727), with which it shares the inline IPC bridge and
the dispatcher; until #207 merges, its diff includes #207's.
`createDataFrame(pandas_df, schema)`, for a pandas DataFrame of at least
10,000 rows whose columns are all object columns of exact `str` values and
`None`, with an explicit `StructType` of `StringType` (UTF8_BINARY) fields, now
encodes the Arrow IPC in Rust (`_pandas_string_ipc` in
`crates/pyspark-rs/src/pandas_input.rs`) from the NumPy object arrays, and
sends it inline as the row path does, instead of materializing Python rows.
Everything else keeps the row path, with its errors: subclasses of `str`,
other objects (including `NaN`, `pd.NA` and numbers), `None` in a non-nullable
field, strings with lone surrogates (which cannot be encoded to UTF-8), other
collations, smaller frames, other dtypes and schemas, and customized pandas or
`pyspark.sql.types` classes, as for #121 (SPARK-60676).
Review changes on top of #144:
- `_pandas_string_ipc` encoded every string to UTF-8 twice, in a validation
pass and while building; it now builds in one pass and falls back on the first
string that cannot be encoded.
- Dropped the checks of the aliases of the Arrow importer and of
`pyspark.sql.pandas.types`: the path calls the Rust encoder with the native
schema and never calls them. The two tests that only exercised them went with
them.
- A test that required a second row conversion to read `Index._data`
exactly as often as the first now checks that the path declines without more
reads than the row path, as for #121.
### Why are the changes needed?
[SPARK-60720](https://issues.apache.org/jira/browse/SPARK-60720):
converting each string through Python rows dominates `createDataFrame` of large
text frames. Measured on macOS against a local Spark 4.3.0 server, as the
client CPU time of `createDataFrame` of 200,000 rows of three string columns
(ASCII, empty, NUL, accents, CJK, emoji, combining marks and nulls) with an
explicit schema (median of 5): 216 ms on the current master, 60 ms with #144 as
it was, and 43 ms with this change ( [...]
### Does this PR introduce _any_ user-facing change?
No, other than speed.
### How was this patch tested?
- New tests in `python/tests/test_arrow_handoff.py`, against the mock Arrow
input service, check every IPC cell (Unicode, NUL, empty strings, nulls,
all-null columns), field names, nullability and metadata, the inline transport
with any relation-cache setting, that no configuration is read, that surrogates
fall back before any configuration read, and that each unsupported or
customized input keeps the row path with its errors.
- Against the current master, merged into #207 and this branch, and a local
Spark 4.3.0 server: the lib and every integration test of
`apache-spark-connect` and `apache-spark-connect-core`, e2e included (789
passed), the build of all tests with `wasm-udf`, the offline Python suites
(1,614 passed, 8 skipped), the live drop-in and Arrow suites (86 passed) and
the live cases (10 passed). `cargo fmt --all --check` passes.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6); the merge with master, the review
changes and the measurement by Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #208 from HyukjinKwon/SPARK-60720-pandas-strings.
Lead-authored-by: DB Tsai <[email protected]>
Co-authored-by: DB Tsai <[email protected]>
Co-authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/lib.rs | 3 +
crates/pyspark-rs/src/pandas_input.rs | 98 ++++
crates/pyspark-rs/src/session.rs | 25 +
python/pyspark/sql/_pandas_input.py | 99 ++++
python/tests/test_arrow_handoff.py | 839 ++++++++++++++++++++++++++++++++++
5 files changed, 1064 insertions(+)
diff --git a/crates/pyspark-rs/src/lib.rs b/crates/pyspark-rs/src/lib.rs
index 7f632c3..549565d 100644
--- a/crates/pyspark-rs/src/lib.rs
+++ b/crates/pyspark-rs/src/lib.rs
@@ -36,6 +36,7 @@ mod group;
mod infer;
mod ml;
mod observation;
+mod pandas_input;
mod pipelines;
mod profiler;
mod pyarrow_export;
@@ -130,6 +131,8 @@ fn _pyspark(_py: Python<'_>, m: &Bound<'_, PyModule>) ->
PyResult<()> {
m.add_function(wrap_pyfunction!(types::parse_datatype_json, m)?)?;
m.add_function(wrap_pyfunction!(column::column_reference_name, m)?)?;
+ m.add_function(wrap_pyfunction!(pandas_input::_pandas_string_ipc, m)?)?;
+
// Spark Declarative Pipelines (SDP) command execution
m.add_class::<pipelines::PyPipelineRunStream>()?;
m.add_function(wrap_pyfunction!(
diff --git a/crates/pyspark-rs/src/pandas_input.rs
b/crates/pyspark-rs/src/pandas_input.rs
new file mode 100644
index 0000000..4d02adb
--- /dev/null
+++ b/crates/pyspark-rs/src/pandas_input.rs
@@ -0,0 +1,98 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Fused validation and inline encoding of guarded pandas string columns.
+
+use std::sync::Arc;
+
+use arrow::array::{ArrayRef, StringBuilder};
+use arrow::datatypes::{DataType as ArrowType, Field, Schema};
+use arrow::ipc::writer::StreamWriter;
+use arrow::record_batch::RecordBatch;
+use pyo3::prelude::*;
+use pyo3::types::{PyBytes, PyString};
+
+use crate::types::PyStructType;
+
+/// The Python eligibility helper authenticates pandas storage and native
schema.
+/// Validate each object before accepting it: extraction must never call a
custom
+/// numeric/string conversion hook or turn a pandas missing sentinel into null.
+#[pyfunction]
+pub(crate) fn _pandas_string_ipc<'py>(
+ py: Python<'py>,
+ columns: Vec<Bound<'py, PyAny>>,
+ schema: PyRef<'py, PyStructType>,
+) -> PyResult<Option<Bound<'py, PyBytes>>> {
+ if columns.len() != schema.fields.len() {
+ return Ok(None);
+ }
+ // Reject unsupported objects before copying any string data. Otherwise a
+ // late subclass would pay for most of an Arrow build before the row path.
+ // Keep this pass to exact type/null checks; UTF-8 extraction stays below.
+ for (column, field) in columns.iter().zip(&schema.fields) {
+ if !matches!(&field.data_type, spark_connect::types::DataType::String
{ collation }
+ if collation == "UTF8_BINARY")
+ {
+ return Ok(None);
+ }
+ for value in column.try_iter()? {
+ let value = value?;
+ if !(value.is_none() && field.nullable ||
value.is_exact_instance_of::<PyString>()) {
+ return Ok(None);
+ }
+ }
+ }
+ // Encode each string once, while building. A lone surrogate cannot be
encoded: it
+ // falls back to the row path, which drops the partial Arrow buffers. The
type pass
+ // above keeps a late subclass from paying for the build.
+ let mut arrays: Vec<ArrayRef> = Vec::with_capacity(columns.len());
+ let mut fields = Vec::with_capacity(columns.len());
+ for (column, field) in columns.iter().zip(&schema.fields) {
+ let mut builder = StringBuilder::with_capacity(column.len()?,
column.len()? * 8);
+ for value in column.try_iter()? {
+ let value = value?;
+ if value.is_none() && field.nullable {
+ builder.append_null();
+ } else if let Ok(value) = value.cast_exact::<PyString>() {
+ let Ok(bytes) = value.encode_utf8() else {
+ return Ok(None);
+ };
+ let Ok(text) = std::str::from_utf8(bytes.as_bytes()) else {
+ return Ok(None);
+ };
+ // append_value copies while the bytes object and GIL are live.
+ // This avoids to_cow's owned String on the abi3-py39 build.
+ builder.append_value(text);
+ } else {
+ return Ok(None);
+ }
+ }
+ arrays.push(Arc::new(builder.finish()));
+ // Like the row encoder, Spark metadata lives in the logical schema;
+ // Arrow fields carry the name, standard string type and nullability.
+ fields.push(Field::new(&field.name, ArrowType::Utf8, field.nullable));
+ }
+ let bytes = py
+ .detach(move || -> Result<Vec<u8>, arrow::error::ArrowError> {
+ let batch = RecordBatch::try_new(Arc::new(Schema::new(fields)),
arrays)?;
+ let mut writer = StreamWriter::try_new(Vec::new(),
&batch.schema())?;
+ writer.write(&batch)?;
+ writer.into_inner()
+ })
+ .map_err(|error|
pyo3::exceptions::PyRuntimeError::new_err(error.to_string()))?;
+ Ok(Some(PyBytes::new(py, &bytes)))
+}
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 88eacdb..96a092a 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -1003,6 +1003,31 @@ impl PySparkSession {
frame.to_df(fields.iter().map(|field|
field.name.as_str()).collect()),
));
}
+ let ipc = py
+ .import("pyspark.sql._pandas_input")?
+ .getattr("prepare_string_input")?
+ .call1((data, schema))?;
+ if !ipc.is_none() {
+ let fields = match &spec {
+ Spec::Struct(fields) => fields,
+ _ => {
+ return Err(pyo3::exceptions::PyValueError::new_err(
+ "string input requires a struct schema",
+ ));
+ }
+ };
+ let schema = DataType::Struct {
+ fields: fields.clone(),
+ };
+ let frame = self
+ .session
+ .create_dataframe_from_inline_arrow_ipc(ipc.extract()?,
schema);
+ // Match pandas row input: explicit nonempty field names travel
+ // through toDF as well as the local relation's logical schema.
+ return Ok(PyDataFrame::new(
+ frame.to_df(fields.iter().map(|field|
field.name.as_str()).collect()),
+ ));
+ }
return self.create_from_pandas(py, data, spec);
}
// Preserve schema parsing, ignored-argument warnings, and
pandas/Arrow routing.
diff --git a/python/pyspark/sql/_pandas_input.py
b/python/pyspark/sql/_pandas_input.py
index 2c97e5f..df89d96 100644
--- a/python/pyspark/sql/_pandas_input.py
+++ b/python/pyspark/sql/_pandas_input.py
@@ -530,3 +530,102 @@ def prepare_float64_input(frame, schema):
with pa.ipc.new_stream(sink, table.schema) as writer:
writer.write_table(table)
return sink.getvalue().to_pybytes()
+
+
+def prepare_string_input(frame, schema):
+ """Prepare inline IPC for explicit columns of exact strings or None.
+
+ Inspect every object before Arrow conversion: str subclasses can override
+ conversion hooks, and pandas treats several other objects as missing
values.
+ Those cases must keep the existing row conversion and its error ordering.
+ """
+ import numpy as np
+ import pandas as pd
+ from pandas.core import frame as frame_module, generic as generic_module
+ from pandas.core.frame import DataFrame
+ from pandas.core.series import Series
+ from types import FunctionType, GetSetDescriptorType,
MethodDescriptorType, WrapperDescriptorType
+
+ from pyspark._pyspark import StringType, StructField, StructType as
NativeStructType
+ from pyspark.sql import types as sql_types
+
+ # Keep small frames on the established row path.
+ if (pd.DataFrame is not DataFrame or type(frame) is not DataFrame
+ or type(schema) is not NativeStructType
+ or len(frame) < 10000
+ or any(name in vars(frame) for name in ("astype", "notna",
"where"))):
+ return None
+ # Preserve the direct normalization hooks used by the existing row path.
+ for name, module in (("astype", generic_module), ("notna", frame_module),
+ ("where", generic_module)):
+ method = getattr(DataFrame, name)
+ if (type(method) is not FunctionType or method.__name__ != name
+ or method.__module__ != module.__name__ or method.__globals__
is not vars(module)):
+ return None
+ if (DataFrame.astype is not generic_module.NDFrame.astype
+ or DataFrame.where is not generic_module.NDFrame.where):
+ return None
+ # The row path reads native fields directly. Class attributes are mutable,
+ # so customized Python schema access must not select a different Arrow
path.
+ for name, cls in (("StringType", StringType), ("StructField", StructField),
+ ("StructType", NativeStructType)):
+ if (getattr(sql_types, name, None) is not cls
+ or cls.__getattribute__ is not object.__getattribute__
+ or any("__class__" in vars(base) for base in cls.__mro__ if
base is not object)):
+ return None
+ for cls, names, descriptor_type in (
+ (NativeStructType, ("fields", "names"), GetSetDescriptorType),
+ (NativeStructType, ("__iter__",), WrapperDescriptorType),
+ (StructField, ("name", "dataType", "nullable", "metadata"),
GetSetDescriptorType),
+ (StringType, ("isUTF8BinaryCollation",), MethodDescriptorType),
+ ):
+ for name in names:
+ descriptor = vars(cls).get(name)
+ if (type(descriptor) is not descriptor_type
+ or descriptor.__objclass__ is not cls or
descriptor.__name__ != name):
+ return None
+ if not _has_canonical_columns(frame.columns):
+ return None
+ width = len(frame.columns)
+ if not width or len(schema.fields) != width or not frame.columns.is_unique:
+ return None
+ if not all(type(name) is str and "\x00" not in name for name in
frame.columns):
+ return None
+ if not all(
+ type(field) is StructField and type(field.dataType) is StringType
+ and field.dataType.isUTF8BinaryCollation()
+ and type(field.name) is str and "\x00" not in field.name
+ for field in schema.fields
+ ) or len(set(schema.names)) != width:
+ return None
+ if not all(type(dtype) is type(np.dtype("object")) and dtype ==
np.dtype("object")
+ for dtype in frame.dtypes):
+ return None
+ # A custom ExtensionArray may report a genuine NumPy dtype while its
+ # Arrow hook emits different values from to_numpy(). Trust only pandas'
+ # concrete NumPy backing array, including its name in older pandas
releases.
+ try:
+ from pandas.core.arrays.numpy_ import NumpyExtensionArray
+ except ImportError:
+ try:
+ from pandas.core.arrays.numpy_ import PandasArray as
NumpyExtensionArray
+ except ImportError:
+ return None
+ columns = [frame.iloc[:, i] for i in range(width)]
+ if any(type(column) is not Series or type(column.array) is not
NumpyExtensionArray
+ for column in columns):
+ return None
+
+ from pyspark._pyspark import _pandas_string_ipc
+
+ # Preflight exact object types before the native Arrow buffer traversal.
+ # Unsupported values, non-nullable None, and surrogates return to the row
+ # path before serialization. Keep its inline transport and configuration
+ # behavior rather than selecting the public Arrow import/upload policy.
+ arrays = [column.array._ndarray for column in columns]
+ if any(type(array) is not np.ndarray or array.ndim != 1
+ or type(array.dtype) is not type(np.dtype("object"))
+ or array.dtype != np.dtype("object") or len(array) != len(frame)
+ for array in arrays):
+ return None
+ return _pandas_string_ipc(arrays, schema)
diff --git a/python/tests/test_arrow_handoff.py
b/python/tests/test_arrow_handoff.py
index eebb17c..ca77f9c 100644
--- a/python/tests/test_arrow_handoff.py
+++ b/python/tests/test_arrow_handoff.py
@@ -3494,3 +3494,842 @@ def
test_float64_canonical_columns_storage_stays_columnar(protocol, monkeypatch,
with _arrow_input_service(protocol) as (spark, state):
spark.createDataFrame(frame, schema).toArrow()
_assert_float64_columns_guard_payload(state, schema, values)
+
+
[email protected]("location", [
+ "subclass", "class", "owner", "backing_subclass_item",
"backing_class_iter",
+ "backing_class_getitem", "backing_instance_box", "arrow_class_iter",
+])
[email protected]("behavior", ["raise", "mutate"])
+def test_string_columns_iterator_hooks_keep_native_order(
+ protocol, monkeypatch, location, behavior
+):
+ from pandas.core.base import IndexOpsMixin
+
+ values = ["plain"] * 10000
+ frame = pd.DataFrame({"original": values}, dtype=object)
+ schema = T.StructType([T.StructField("renamed", T.StringType())])
+ original_iter = pd.Index.__iter__
+ original_notna, original_astype = frame.notna, frame.astype
+ calls = []
+ invoked = False
+
+ class ColumnsError(RuntimeError):
+ pass
+
+ def notna(*args, **kwargs):
+ calls.append("notna")
+ return original_notna(*args, **kwargs)
+
+ def astype(*args, **kwargs):
+ calls.append("astype")
+ return original_astype(*args, **kwargs)
+
+ def before_hook():
+ nonlocal invoked
+ if invoked:
+ return
+ invoked = True
+ calls.append("columns iterator")
+ if behavior == "raise":
+ raise ColumnsError("columns hook precedes pandas normalization")
+ # Install spies only when native iteration reaches the columns hook.
+ # Before eligibility, normalization methods are the canonical methods.
+ monkeypatch.setitem(vars(frame), "notna", notna)
+ monkeypatch.setitem(vars(frame), "astype", astype)
+ frame.iat[0, 0], frame.iat[-1, 0] = "changed", None
+
+ def iterate(index):
+ if index is columns:
+ before_hook()
+ return original_iter(index)
+
+ if location == "subclass":
+ class Columns(pd.Index):
+ pass
+ monkeypatch.setattr(Columns, "__iter__", iterate)
+ labels = pd.Index(["original"], dtype=object)
+ frame.columns = Columns._simple_new(labels._values, name=None)
+ assert type(frame.columns) is Columns
+ elif location == "backing_subclass_item":
+ # IndexOpsMixin iteration uses ndarray.item, not ndarray.__iter__.
+ class Names(np.ndarray):
+ def item(self, *args):
+ before_hook()
+ return super().item(*args)
+ backing = np.asarray(["original"], dtype=object).view(Names)
+ frame.columns = pd.Index._simple_new(backing, name=None)
+ assert type(frame.columns._values) is Names
+ elif location.startswith(("backing_", "arrow_")):
+ storage = "pyarrow" if location.startswith("arrow_") else "python"
+ frame.columns = pd.Index(pd.array(["original"],
dtype=pd.StringDtype(storage=storage)))
+ backing = frame.columns._values
+ if location.endswith("class_iter"):
+ original = type(backing).__iter__
+
+ def backing_iter(array):
+ if array is backing:
+ before_hook()
+ return original(array)
+
+ monkeypatch.setattr(type(backing), "__iter__", backing_iter)
+ elif location.endswith("class_getitem"):
+ original = type(backing).__getitem__
+
+ def backing_getitem(array, key):
+ if array is backing:
+ before_hook()
+ return original(array, key)
+
+ monkeypatch.setattr(type(backing), "__getitem__", backing_getitem)
+ else:
+ original = backing._box_func
+
+ def box(value):
+ before_hook()
+ return original(value)
+
+ monkeypatch.setitem(vars(backing), "_box_func", box)
+ else:
+ monkeypatch.setattr(pd.Index if location == "class" else IndexOpsMixin,
+ "__iter__", iterate)
+ columns = frame.columns
+
+ with _arrow_input_service(protocol) as (spark, state):
+ if behavior == "raise":
+ with pytest.raises(ColumnsError,
+ match="columns hook precedes pandas
normalization"):
+ spark.createDataFrame(frame, schema)
+ assert calls == ["columns iterator"]
+ assert state["config_requests"] == []
+ assert state["input_tables"] == []
+ assert state["uploads"] == 0
+ else:
+ spark.createDataFrame(frame, schema).toArrow()
+ expected = values.copy()
+ expected[0], expected[-1] = "changed", None
+ import json
+
+ table = state["input_tables"][0]
+ assert table.schema.names == ["renamed"]
+ assert table.column(0).type == pa.string()
+ assert table.column(0).to_pylist() == expected
+ assert table.column(0).is_null().to_pylist() == [False] * 9999 +
[True]
+ assert table.schema.equals(pa.schema([pa.field("renamed",
pa.string())]),
+ check_metadata=True)
+ assert json.loads(state["relations"][0].local_relation.schema) ==
json.loads(schema.json())
+ assert state["config_requests"] == []
+ assert state["uploads"] == 0
+ assert calls == ["columns iterator", "notna", "astype"]
+
+
+
+def test_string_rebound_index_values_keeps_native_error(protocol, monkeypatch):
+
+ frame = pd.DataFrame({"original": ["plain"] * 10000}, dtype=object)
+ schema = T.StructType([T.StructField("renamed", T.StringType())])
+ columns = frame.columns
+ original = pd.Index._values.fget
+ calls = []
+
+ def values(index):
+ if index is columns:
+ calls.append("_values")
+ raise RuntimeError("columns backing hook precedes normalization")
+ return original(index)
+
+ monkeypatch.setattr(pd.Index, "_values", property(values))
+ assert calls == []
+ with _arrow_input_service(protocol) as (spark, state):
+ with pytest.raises(RuntimeError,
+ match="columns backing hook precedes
normalization"):
+ spark.createDataFrame(frame, schema)
+ assert calls == ["_values"]
+ assert state["config_requests"] == []
+ assert state["input_tables"] == []
+
+
+
[email protected]("nullable", [False, True])
+def test_string_rebound_index_data_keeps_native_read_order(protocol,
monkeypatch, nullable):
+ import json
+
+ values = ["plain", "é", "漢字", ""] * 2500
+ if nullable:
+ values[1] = None
+ frame = pd.DataFrame({"original": values}, dtype=object)
+ schema = T.StructType([
+ T.StructField("renamed", T.StringType(), nullable, {"source":
"native"})
+ ])
+ columns, backing = frame.columns, vars(frame.columns)["_data"]
+ original_notna, original_astype = frame.notna, frame.astype
+ calls = []
+ normalizing = False
+ allowed_reads = None
+
+ class ColumnsDataError(RuntimeError):
+ pass
+
+ def notna(*args, **kwargs):
+ nonlocal normalizing
+ normalizing = True
+ calls.append("notna")
+ return original_notna(*args, **kwargs)
+
+ def astype(*args, **kwargs):
+ calls.append("astype")
+ return original_astype(*args, **kwargs)
+
+ def data(index):
+ if index is columns:
+ if not normalizing:
+ calls.append("_data")
+ if allowed_reads is not None and calls.count("_data") >
allowed_reads:
+ raise ColumnsDataError("eligibility added a columns
storage read")
+ # Canonical normalization remains intact before eligibility.
+ monkeypatch.setitem(vars(frame), "notna", notna)
+ monkeypatch.setitem(vars(frame), "astype", astype)
+ return backing
+ return vars(index)["_data"]
+
+ def set_data(index, value):
+ # Keep ordinary storage initialization for indexes created by pandas.
+ vars(index)["_data"] = value
+
+ monkeypatch.setattr(pd.Index, "_data", property(data, set_data),
raising=False)
+ import pyspark.sql._pandas_input as helper
+
+ with _arrow_input_service(protocol) as (spark, state):
+ # Measure the current native fallback's storage reads, rather than the
+ # former tolist path's single-read contract.
+ with monkeypatch.context() as baseline:
+ baseline.setattr(helper, "prepare_string_input", lambda *_: None)
+ spark.createDataFrame(frame, schema).toArrow()
+ baseline_calls = calls.copy()
+ allowed_reads = baseline_calls.count("_data")
+ assert allowed_reads > 0 and "notna" in baseline_calls
+ calls.clear()
+ normalizing = False
+ # Restore canonical methods before the candidate eligibility check;
+ # native label iteration will install the spies through the storage
hook.
+ monkeypatch.delitem(vars(frame), "notna")
+ monkeypatch.delitem(vars(frame), "astype")
+ spark.createDataFrame(frame, schema).toArrow()
+ # The rebound storage declines the string path without reading the
columns
+ # more than the row path does. pandas caches some of these reads, so a
later
+ # row conversion of the same frame can read them fewer times than the
first.
+ assert "notna" in calls
+ assert calls.count("_data") <= allowed_reads
+ assert not state["config_requests"]
+ assert state["uploads"] == 0
+ assert len(state["input_tables"]) == 2
+ assert len(state["relations"]) == 2
+ table = state["input_tables"][1]
+ assert table.equals(state["input_tables"][0], check_metadata=True)
+ assert table.column(0).to_pylist() == values
+ assert table.column(0).is_null().to_pylist() == [value is None for
value in values]
+ assert table.schema.equals(
+ pa.schema([pa.field("renamed", pa.string(), nullable=nullable)]),
+ check_metadata=True,
+ )
+ assert json.loads(state["relations"][1].local_relation.schema) == {
+ "type": "struct", "fields": [
+ {"name": "renamed", "type": "string", "nullable": nullable,
+ "metadata": {"source": "native"}}
+ ],
+ }
+
+
+
[email protected]("kind", [
+ "object", "default", "python", "pyarrow", "python_numpy", "pyarrow_numpy",
+])
+def test_string_canonical_column_storage_stays_eligible(kind):
+ from pyspark.sql._pandas_input import _has_canonical_columns,
prepare_string_input
+
+ frame = pd.DataFrame({"original": ["plain"] * 10000}, dtype=object)
+ schema = T.StructType([T.StructField("renamed", T.StringType())])
+ if kind == "object":
+ frame.columns = pd.Index(np.asarray(["original"], dtype=object),
dtype=object)
+ elif kind == "default":
+ frame.columns = pd.Index(["original"])
+ elif kind.endswith("_numpy"):
+ from pandas.core.arrays import string_ as strings, string_arrow as
arrow_strings
+
+ module = strings if kind.startswith("python") else arrow_strings
+ name = ("StringArrayNumpySemantics" if kind.startswith("python")
+ else "ArrowStringArrayNumpySemantics")
+ cls = getattr(module, name, None)
+ if cls is None:
+ pytest.skip("older infer-string storage class is absent")
+ frame.columns = pd.Index._simple_new(cls._from_sequence(["original"]),
name=None)
+ else:
+ frame.columns = pd.Index(pd.array(["original"],
dtype=pd.StringDtype(storage=kind)))
+ assert _has_canonical_columns(frame.columns)
+ assert type(prepare_string_input(frame, schema)) is bytes
+
+
+
[email protected]("missing", ["module", "owner", "base"])
+def test_string_absent_private_columns_storage_keeps_row_fallback(monkeypatch,
missing):
+ import builtins
+ from pandas.core.arrays import numpy_ as numpy_module, string_ as
string_module
+ from pyspark.sql._pandas_input import _has_canonical_columns
+
+ columns = pd.Index(pd.array(["original"],
dtype=pd.StringDtype(storage="python")))
+ if missing == "module":
+ original = builtins.__import__
+
+ def without_arrow(name, *args, **kwargs):
+ if name == "pandas.core.arrays.arrow":
+ raise ImportError("private storage module is absent")
+ return original(name, *args, **kwargs)
+
+ monkeypatch.setattr(builtins, "__import__", without_arrow)
+ elif missing == "owner":
+ monkeypatch.delattr(numpy_module, "NumpyExtensionArray")
+ else:
+ monkeypatch.delattr(string_module, "BaseStringArray")
+ assert not _has_canonical_columns(columns)
+
+
+
[email protected]("cached", [False, True])
[email protected]("verify", [False, True])
[email protected]("large", ["false", "true"])
+def test_string_columns_preserve_every_ipc_cell(protocol, monkeypatch, cached,
verify, large):
+ import json
+
+ from pyspark.sql.pandas.types import from_arrow_metadata
+
+ strings = ["", "plain", "é", "e\u0301", "\x00embedded\x00", "漢字", "🙂",
+ "\U0010ffff", "\uffff", "\u202eright-to-left"] * 1000
+ nullable = [None if i % 7 == 0 else value for i, value in
enumerate(reversed(strings))]
+ expected = [strings, nullable, [None] * len(strings)]
+ frame = pd.DataFrame(dict(zip(["original", "b", "c"], expected)),
dtype=object)
+ schema = T.StructType([
+ T.StructField("renamed.`text", T.StringType(), False, {"source":
"string", "id": 3}),
+ T.StructField("nullable", T.StringType(), True, {"tag": [1, 2]}),
+ T.StructField("all_null", T.StringType()),
+ ])
+
+ def no_rows(*args, **kwargs):
+ raise AssertionError("string fast path must not materialize Python
rows")
+
+ for method in ("to_numpy", "iterrows", "itertuples"):
+ monkeypatch.setattr(pd.DataFrame, method, no_rows)
+ monkeypatch.setattr(pd.DataFrame, "values", property(no_rows))
+ monkeypatch.setattr(pd.Series, "tolist", no_rows)
+ overrides = {
+ "spark.sql.session.localRelationCacheThreshold": "1" if cached else
"999999999",
+ "spark.sql.execution.arrow.useLargeVarTypes": large,
+ "spark.sql.session.localRelationChunkSizeRows": "997",
+ }
+ with _arrow_input_service(protocol, **overrides) as (spark, state):
+ result = spark.createDataFrame(frame, schema, verifySchema=verify)
+ assert not result.isLocal()
+ result.toArrow()
+ assert state["column_names"] == schema.names
+ table = state["input_tables"][0]
+ assert table.num_rows == len(strings)
+ assert table.schema.names == schema.names
+ for column, values, field in zip(table.columns, expected,
schema.fields):
+ assert column.type == pa.string()
+ assert column.to_pylist() == values
+ actual_field = table.schema.field(field.name)
+ assert actual_field.nullable == field.nullable
+ assert from_arrow_metadata(actual_field.metadata) is None
+ spark_field =
json.loads(state["relations"][0].local_relation.schema)["fields"][
+ schema.names.index(field.name)]
+ assert spark_field["metadata"] == field.metadata
+ assert state["config_requests"] == []
+ assert state["uploads"] == 0
+
+
+
[email protected]("reason", [
+ "small", "empty", "inferred", "names", "ddl", "cast", "atomic",
"mixed_schema",
+ "width", "duplicate_input", "duplicate_schema", "input_nul", "schema_nul",
+ "input_number", "datetime", "date", "nan", "pd_na", "nat", "mixed",
"bytes",
+ "nonnullable_null", "frame_subclass", "string_subclass", "string_dtype",
+ "arrow_string_dtype", "late_unsupported", "surrogate", "late_surrogate",
"collation",
+])
+def test_string_column_guard_preserves_row_fallback(reason):
+ from pyspark.sql._pandas_input import prepare_string_input
+
+ class Frame(pd.DataFrame):
+ pass
+
+ class Text(str):
+ def __str__(self):
+ raise AssertionError("guard must not call conversion hooks")
+
+ class Unsupported:
+ def __str__(self):
+ raise AssertionError("guard must not stringify unsupported
objects")
+
+ values = ["plain"] * 10000
+ frame = pd.DataFrame({"text": values}, dtype=object)
+ schema = T.StructType([T.StructField("text", T.StringType())])
+ if reason == "small":
+ frame = frame.iloc[:9999]
+ elif reason == "empty":
+ frame = frame.iloc[:0]
+ elif reason in ("inferred", "names", "ddl", "cast", "atomic"):
+ schema = {"inferred": None, "names": ["text"], "ddl": "text STRING",
+ "cast": T.StructType([T.StructField("text", T.LongType())]),
+ "atomic": T.StringType()}[reason]
+ elif reason == "mixed_schema":
+ frame["long"] = np.arange(10000)
+ schema.add("long", T.LongType())
+ elif reason == "width":
+ schema.add("extra", T.StringType())
+ elif reason in ("duplicate_input", "duplicate_schema"):
+ frame["other"] = values
+ schema.add("other", T.StringType())
+ if reason == "duplicate_input":
+ frame.columns = ["text", "text"]
+ else:
+ schema = T.StructType([T.StructField("text", T.StringType())] * 2)
+ elif reason in ("input_nul", "input_number"):
+ frame.columns = ["text\x00" if reason == "input_nul" else 1]
+ elif reason == "schema_nul":
+ schema = T.StructType([T.StructField("text\x00", T.StringType())])
+ elif reason == "frame_subclass":
+ frame = Frame(frame)
+ elif reason in ("string_dtype", "arrow_string_dtype"):
+ frame = pd.DataFrame({"text": pd.Series(
+ values, dtype=pd.StringDtype(storage="pyarrow" if reason ==
"arrow_string_dtype"
+ else "python"))})
+ elif reason == "collation":
+ schema = T.StructType([T.StructField("text",
T.StringType("UTF8_LCASE"))])
+ else:
+ value = {
+ "datetime": datetime(2020, 1, 1), "date": date(2020, 1, 1),
+ "nan": float("nan"), "pd_na": pd.NA, "nat": pd.NaT, "mixed": 42,
+ "bytes": b"plain", "nonnullable_null": None, "string_subclass":
Text("plain"),
+ "late_unsupported": Unsupported(), "surrogate": "\ud800",
"late_surrogate": "\udfff",
+ }[reason]
+ if reason == "nonnullable_null":
+ schema = T.StructType([T.StructField("text", T.StringType(),
False)])
+ if reason in ("late_unsupported", "late_surrogate"):
+ # pandas 3 infers StringDtype for a newly assigned list column.
+ # Keep these unsupported-object fixtures on concrete object
storage.
+ frame["other"] = pd.Series(values, dtype=object)
+ schema.add("other", T.StringType())
+ backing = frame["other"].array._ndarray
+ assert type(backing) is np.ndarray and backing.dtype ==
np.dtype("object")
+ frame.iat[-1, -1] = value
+ if reason in ("late_unsupported", "late_surrogate"):
+ assert frame.iat[-1, -1] is value
+ assert prepare_string_input(frame, schema) is None
+
+
+
[email protected]("reason", [
+ "string_subclass", "frame_subclass", "nan", "pd_na", "nat", "string_dtype",
+ "nonnullable_null", "inferred", "names", "ddl", "small", "collation",
"cast", "mixed",
+])
+def test_string_public_fallback_keeps_row_semantics(protocol, monkeypatch,
reason):
+ calls = []
+
+ class Text(str):
+ def __str__(self):
+ calls.append("str")
+ return "changed"
+
+ class Frame(pd.DataFrame):
+ pass
+
+ values = ["plain"] * (9999 if reason == "small" else 10000)
+ if reason in ("nan", "pd_na", "nat", "nonnullable_null", "mixed"):
+ values[-1] = {"nan": float("nan"), "pd_na": pd.NA, "nat": pd.NaT,
+ "nonnullable_null": None, "mixed": 42}[reason]
+ elif reason == "string_subclass":
+ values[-1] = Text("plain")
+ frame_type = Frame if reason == "frame_subclass" else pd.DataFrame
+ frame = frame_type({"text": values}, dtype="string" if reason ==
"string_dtype" else object)
+ schema = T.StructType([T.StructField("text", T.StringType(), reason !=
"nonnullable_null")])
+ if reason == "inferred":
+ schema = None
+ elif reason == "names":
+ schema = ["renamed"]
+ elif reason == "ddl":
+ schema = "text STRING"
+ elif reason == "collation":
+ schema = T.StructType([T.StructField("text",
T.StringType("UTF8_LCASE"))])
+ elif reason == "cast":
+ schema = T.StructType([T.StructField("text", T.LongType())])
+ astype = pd.DataFrame.astype
+
+ def record_rows(self, *args, **kwargs):
+ calls.append("astype")
+ return astype(self, *args, **kwargs)
+
+ monkeypatch.setattr(pd.DataFrame, "astype", record_rows)
+ with _arrow_input_service(protocol) as (spark, state):
+ if reason in ("nonnullable_null", "cast", "mixed"):
+ from pyspark.errors import PySparkRuntimeError
+
+ message = ("Column 'text' is declared as non-nullable but contains
null values"
+ if reason == "nonnullable_null" else "Type mismatch in
row data")
+ with pytest.raises(PySparkRuntimeError, match=message):
+ spark.createDataFrame(frame, schema)
+ assert calls == ["astype"]
+ assert state["config_requests"] == []
+ assert state["relations"] == []
+ return
+ spark.createDataFrame(frame, schema).toArrow()
+ assert calls == ["astype"] # Native extraction keeps str.__str__
overrides uncalled.
+ table = state["input_tables"][0]
+ assert table.schema.names == ["renamed" if reason == "names" else
"text"]
+ expected = ["plain"] * len(values)
+ if reason in ("nan", "pd_na", "nat"):
+ expected[-1] = None
+ assert table.column(0).to_pylist() == expected
+ assert state["config_requests"] == []
+
+
+
[email protected]("surrogate", ["\ud800", "\udfff"])
+def test_string_surrogate_fallback_precedes_importer_configuration(protocol,
monkeypatch, surrogate):
+ values = ["plain"] * 10000
+ values[-1] = surrogate
+ frame = pd.DataFrame({"text": values}, dtype=object)
+ schema = T.StructType([T.StructField("text", T.StringType())])
+ astype = pd.DataFrame.astype
+ calls = []
+
+ def record_rows(self, *args, **kwargs):
+ calls.append("astype")
+ return astype(self, *args, **kwargs)
+
+ monkeypatch.setattr(pd.DataFrame, "astype", record_rows)
+ with _arrow_input_service(protocol) as (spark, state):
+ state["config_error"] = grpc.StatusCode.INVALID_ARGUMENT
+ with pytest.raises(TypeError, match="unsupported value type for
createDataFrame: str"):
+ spark.createDataFrame(frame, schema)
+ assert calls == ["astype"]
+ assert state["config_requests"] == []
+ assert state["relations"] == []
+ assert state["uploads"] == 0
+
+
+
[email protected]("class_name,attribute", [
+ ("StructType", "names"), ("StructType", "fields"), ("StructType",
"__iter__"),
+ ("StructField", "name"), ("StructField", "dataType"), ("StructField",
"nullable"),
+ ("StructField", "metadata"), ("StringType", "isUTF8BinaryCollation"),
+ ("StructType", "__getattribute__"), ("StructField", "__getattribute__"),
+ ("StringType", "__getattribute__"), ("StructType", "__class__"),
+ ("StructField", "__class__"), ("StringType", "__class__"),
+ ("StructType", "alias"), ("StructField", "alias"), ("StringType", "alias"),
+])
[email protected]("before_import", [False, True])
+def test_string_custom_schema_descriptors_keep_native_row_path(
+ protocol, class_name, attribute, before_import
+):
+ import sys
+
+ script = r'''
+import sys
+import tempfile
+from pathlib import Path
+import pandas as pd
+from pyspark.sql import types as T
+
+import importlib.util
+spec = importlib.util.spec_from_file_location("string_schema_capture",
sys.argv[4])
+capture = importlib.util.module_from_spec(spec)
+sys.modules[spec.name] = capture
+spec.loader.exec_module(capture)
+_arrow_input_service, protocol = capture._arrow_input_service, capture.protocol
+frame = pd.DataFrame({"text": ["plain"] * 10000}, dtype=object)
+schema = T.StructType([T.StructField("text", T.StringType(), True, {"source":
"native"})])
+class_name, attribute, before_import = sys.argv[1:4]
+if before_import == "True":
+ # The capture module imports prepare_int64_table; force the public call to
+ # import a new helper after the class or public alias has been customized.
+ sys.modules.pop("pyspark.sql._pandas_input", None)
+assert ("pyspark.sql._pandas_input" in sys.modules) == (before_import ==
"False")
+cls = getattr(T, class_name)
+if attribute == "alias":
+ setattr(T, class_name, object)
+else:
+ if attribute == "__getattribute__":
+ replacement = lambda self, name: object.__getattribute__(self, name)
+ elif attribute == "__iter__":
+ replacement = lambda self: iter([T.StructField("changed",
T.StringType())])
+ elif attribute == "__class__":
+ attribute = "__getattribute__"
+ replacement = lambda self, name: object if name == "__class__" else (
+ object.__getattribute__(self, name))
+ elif attribute == "isUTF8BinaryCollation":
+ replacement = lambda self: False
+ else:
+ value = {"names": ["changed"], "fields": [], "name": "changed",
+ "dataType": T.StringType(), "nullable": False, "metadata":
{"changed": True}}[attribute]
+ replacement = property(lambda self: value)
+ setattr(cls, attribute, replacement)
+class Factory:
+ def mktemp(self, name):
+ return Path(tempfile.mkdtemp(prefix=name))
+with _arrow_input_service(protocol.__wrapped__(Factory())) as (spark, state):
+ spark.createDataFrame(frame, schema).toArrow()
+ assert state["config_requests"] == [], state["config_requests"]
+ table = state["input_tables"][0]
+ assert table.schema.names == ["text"], table.schema
+ assert table.column(0).to_pylist() == ["plain"] * 10000
+'''
+ result = subprocess.run([sys.executable, "-c", script, class_name,
attribute,
+ str(before_import), __file__],
+ capture_output=True, text=True, timeout=30)
+ assert result.returncode == 0, result.stdout + result.stderr
+
+
+
+def test_string_rebound_pandas_alias_keeps_subclass_fallback(monkeypatch):
+ from pyspark.sql._pandas_input import prepare_string_input
+
+ class Frame(pd.DataFrame):
+ pass
+
+ frame = Frame({"text": ["plain"] * 10000}, dtype=object)
+ schema = T.StructType([T.StructField("text", T.StringType())])
+ monkeypatch.setattr(pd, "DataFrame", Frame)
+ assert prepare_string_input(frame, schema) is None
+
+
+
[email protected]("dtype_mode", ["equal", "class_spoof", "exact_numpy"])
+def
test_string_custom_extension_dtype_cannot_impersonate_numpy_object(protocol,
dtype_mode):
+ from pandas.api.extensions import ExtensionArray, ExtensionDtype, take
+ from pyspark.sql._pandas_input import prepare_string_input
+
+ arrow_calls = []
+
+ class ObjectEqualDtype(ExtensionDtype):
+ name = "object_equal_string"
+ type = str
+ kind = "O"
+ na_value = None
+
+ @classmethod
+ def construct_array_type(cls):
+ return DifferentArrowArray
+
+ def __eq__(self, other):
+ return (isinstance(other, ObjectEqualDtype)
+ or isinstance(other, np.dtype) and other.kind == "O")
+
+ def __hash__(self):
+ return hash(self.name)
+
+ class DifferentArrowArray(ExtensionArray):
+ def __init__(self, values):
+ self._values = np.asarray(values, dtype=object)
+
+ @property
+ def dtype(self):
+ return ObjectEqualDtype()
+
+ @property
+ def nbytes(self):
+ return self._values.nbytes
+
+ def __len__(self):
+ return len(self._values)
+
+ def __getitem__(self, key):
+ result = self._values[key]
+ return type(self)(result) if isinstance(result, np.ndarray) else
result
+
+ def isna(self):
+ return pd.isna(self._values)
+
+ def take(self, indices, allow_fill=False, fill_value=None):
+ return type(self)(take(self._values, indices,
allow_fill=allow_fill,
+ fill_value=fill_value))
+
+ def copy(self):
+ return type(self)(self._values.copy())
+
+ @classmethod
+ def _from_sequence(cls, scalars, *, dtype=None, copy=False):
+ values = np.asarray(scalars, dtype=object)
+ return cls(values.copy() if copy else values)
+
+ def to_numpy(self, dtype=None, copy=False, na_value=None):
+ if dtype is not None:
+ return self._values.astype(dtype, copy=copy)
+ return self._values.copy() if copy else self._values
+
+ def __arrow_array__(self, type=None):
+ arrow_calls.append("arrow")
+ # A scan of to_numpy() cannot establish the values this hook emits.
+ return pa.array(["different"] * len(self), type=type)
+
+ array = DifferentArrowArray(["plain"] * 10000)
+ frame = pd.DataFrame({"text": pd.Series(array, copy=False)})
+ schema = T.StructType([T.StructField("text", T.StringType())])
+ if dtype_mode == "exact_numpy":
+ # An ExtensionArray may change its advertised dtype after construction.
+ # The dtype and NumPy scan both pass while its Arrow hook changes
cells.
+ DifferentArrowArray.dtype = property(lambda self: np.dtype("object"))
+ DifferentArrowArray.__array__ = lambda self, dtype=None, copy=None:
self._values
+ elif dtype_mode == "class_spoof":
+ def spoof_class(self, name):
+ if name == "__class__":
+ return type(np.dtype("object"))
+ return object.__getattribute__(self, name)
+
+ ObjectEqualDtype.__getattribute__ = spoof_class
+ dtype = frame.dtypes.iloc[0]
+ assert type(frame) is pd.DataFrame
+ assert type(frame.iloc[:, 0].array) is DifferentArrowArray
+ assert type(dtype) is (type(np.dtype("object")) if dtype_mode ==
"exact_numpy"
+ else ObjectEqualDtype)
+ assert isinstance(dtype, np.dtype) is (dtype_mode != "equal")
+ assert dtype == np.dtype("object")
+ assert prepare_string_input(frame, schema) is None
+ with _arrow_input_service(protocol) as (spark, state):
+ if dtype_mode == "exact_numpy":
+ # This deliberately inconsistent EA already fails pandas' row-path
+ # astype(object). Preserve that error before importer
configuration.
+ with pytest.raises(TypeError, match="cannot set astype"):
+ spark.createDataFrame(frame, schema)
+ assert state["input_tables"] == []
+ else:
+ spark.createDataFrame(frame, schema).toArrow()
+ assert state["input_tables"][0].column(0).to_pylist() == ["plain"]
* 10000
+ assert state["config_requests"] == []
+ assert arrow_calls == []
+
+
+
[email protected]("method", ["astype", "notna"])
+def test_string_instance_conversion_hooks_keep_row_fallback(protocol, method):
+ frame = pd.DataFrame({"text": ["plain"] * 10000}, dtype=object)
+ schema = T.StructType([T.StructField("text", T.StringType())])
+ original = getattr(frame, method)
+ calls = []
+ expected = ["plain"] * 10000
+ if method == "astype":
+ expected[0] = "changed"
+
+ def customized(*args, **kwargs):
+ calls.append(method)
+ result = original(*args, **kwargs)
+ result.iat[0, 0] = "changed"
+ return result
+ else:
+ expected[0] = None
+
+ def customized(*args, **kwargs):
+ calls.append(method)
+ result = original(*args, **kwargs)
+ result.iat[0, 0] = False
+ return result
+ setattr(frame, method, customized)
+ with _arrow_input_service(protocol) as (spark, state):
+ spark.createDataFrame(frame, schema).toArrow()
+ assert calls == [method]
+ assert state["config_requests"] == []
+ assert state["uploads"] == 0
+ assert state["input_tables"][0].column(0).to_pylist() == expected
+
+
+
[email protected]("method", ["astype", "notna", "where"])
[email protected]("before_import", [False, True])
+def test_string_class_conversion_hooks_keep_row_fallback(
+ protocol, monkeypatch, method, before_import
+):
+ import importlib
+ import sys
+
+ name = "pyspark.sql._pandas_input"
+ if before_import:
+ monkeypatch.delitem(sys.modules, name, raising=False)
+ else:
+ importlib.import_module(name)
+ frame = pd.DataFrame({"text": ["plain"] * 10000}, dtype=object)
+ schema = T.StructType([T.StructField("text", T.StringType())])
+ original = getattr(pd.DataFrame, method)
+ calls = []
+ expected = ["plain"] * 10000
+ expected[0] = None if method == "notna" else "changed"
+
+ def customized(self, *args, **kwargs):
+ calls.append(method)
+ result = original(self, *args, **kwargs)
+ result.iat[0, 0] = False if method == "notna" else "changed"
+ return result
+
+ monkeypatch.setattr(pd.DataFrame, method, customized)
+ with _arrow_input_service(protocol) as (spark, state):
+ spark.createDataFrame(frame, schema).toArrow()
+ assert calls == [method]
+ assert state["config_requests"] == []
+ assert state["uploads"] == 0
+ assert state["input_tables"][0].column(0).to_pylist() == expected
+
+
+
+def test_string_instance_where_override_keeps_existing_row_behavior(protocol):
+ from pyspark.sql._pandas_input import prepare_string_input
+
+ frame = pd.DataFrame({"text": ["plain"] * 10000}, dtype=object)
+ schema = T.StructType([T.StructField("text", T.StringType())])
+
+ def customized(*args, **kwargs):
+ raise AssertionError("row normalization calls where on the astype
result")
+
+ frame.where = customized
+ assert prepare_string_input(frame, schema) is None
+ with _arrow_input_service(protocol) as (spark, state):
+ spark.createDataFrame(frame, schema).toArrow()
+ assert state["config_requests"] == []
+ assert state["input_tables"][0].column(0).to_pylist() == ["plain"] *
10000
+
+
+
+def test_string_native_backing_does_not_add_series_conversion_hooks(protocol,
monkeypatch):
+ frame = pd.DataFrame({"text": ["plain"] * 10000}, dtype=object)
+ schema = T.StructType([T.StructField("text", T.StringType())])
+
+ def customized(*args, **kwargs):
+ raise AssertionError("native string input must not call
Series.to_numpy")
+
+ monkeypatch.setattr(pd.Series, "to_numpy", customized)
+ with _arrow_input_service(protocol) as (spark, state):
+ spark.createDataFrame(frame, schema).toArrow()
+ assert state["config_requests"] == []
+ assert state["input_tables"][0].column(0).to_pylist() == ["plain"] *
10000
+
+
+
+def test_string_large_inline_input_keeps_row_transport_and_schema(protocol):
+ values = ["漢字-🙂-" + str(i) for i in range(100000)]
+ frame = pd.DataFrame({"text": values}, dtype=object)
+ schema = T.StructType([T.StructField("renamed", T.StringType(), False,
+ {"source": "inline-string", "size":
len(values)})])
+ with _arrow_input_service(protocol, **{
+ "spark.sql.session.localRelationCacheThreshold": "1",
+ "spark.sql.session.localRelationSizeLimit": "1",
+ "spark.sql.execution.arrow.useLargeVarTypes": "true",
+ }) as (spark, state):
+ result = spark.createDataFrame(frame, schema)
+ assert not result.isLocal()
+ result.toArrow()
+ assert state["column_names"] == schema.names
+ assert state["config_requests"] == []
+ assert state["uploads"] == 0
+ table = state["input_tables"][0]
+ assert table.schema.names == ["renamed"]
+ assert table.column(0).type == pa.string()
+ assert table.schema.field(0).nullable is False
+ assert table.schema.field(0).metadata is None
+ assert table.column(0).to_pylist() == values
+ assert
__import__("json").loads(state["relations"][0].local_relation.schema) == (
+ __import__("json").loads(schema.json()))
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]