This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 03451b7 [SPARK-61104][CONNECT][PYTHON] Convert pandas input without a
schema as PySpark's Arrow conversion does in the Python client
03451b7 is described below
commit 03451b7d36f34f146957ce88623fe77448369c64
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 19:59:30 2026 +0900
[SPARK-61104][CONNECT][PYTHON] Convert pandas input without a schema as
PySpark's Arrow conversion does in the Python client
### What changes were proposed in this pull request?
[SPARK-61104](https://issues.apache.org/jira/browse/SPARK-61104):
`spark.createDataFrame` of a pandas DataFrame without a schema, or with a list
of names, now converts it as PySpark Connect does.
This follows
[SPARK-61079](https://issues.apache.org/jira/browse/SPARK-61079), which did the
same for an explicit schema. Each column goes through the vendored official
`create_arrow_table_from_pandas`, with the type that Arrow infers. The
exceptions are datetimes, which become timestamps in the session time zone, and
timedeltas, which become day-time intervals.
- The table is sent without a schema, so the server infers its types from
Arrow's.
- `toDF` names the columns: by their labels, as strings, or by the given
names, with `_n` for any columns past them.
- With `spark.sql.execution.pandas.inferPandasDictAsMap`, a column of dicts
is a map; otherwise it is a struct.
This is `create_from_pandas_inferred` in
`crates/pyspark-rs/src/session.rs`, and `prepare_inferred_pandas_table` in
`python/pyspark/sql/_pandas_input.py`. The fast path for int64 columns still
runs first. A customized frame, and any frame when pyarrow is not installed,
keep the row path, as with an explicit schema. The check of the frame is split
out of `accepts_pandas_table` as `accepts_pandas_frame`.
It reuses `pyarrow_import::table_batches` of
[SPARK-61103](https://issues.apache.org/jira/browse/SPARK-61103).
### Why are the changes needed?
The client converted such a frame row by row, typing each column by its
first value. On a local Spark 4.3.0 server, with a client in Asia/Seoul and a
session in UTC, 32 of 92 cases of pandas input differed from PySpark 4.3.0:
| Input | master | This change and PySpark 4.3.0 |
| --- | --- | --- |
| an `int8`, `int16` or `int32` column | `bigint` | `tinyint`, `smallint`,
`int` |
| a `float32` column | `double` | `float` |
| a column of dicts | `map<string,bigint>` | `struct<k:bigint>` |
| dicts of other keys | `PySparkRuntimeError` | a struct of all the keys |
| a naive `2020-01-01 01:02:03` | collects as 01:02:03 | collects as
10:02:03 (01:02:03 UTC) |
| an int and a float in an object column | `PySparkRuntimeError: Type
mismatch in row data` | a `double` |
| an int and a str in a column | `PySparkRuntimeError: Type mismatch in row
data` | `PySparkValueError` of the Arrow conversion |
| a timestamp with nanoseconds | truncated silently | `PySparkValueError`
of the Arrow conversion |
| a column of sets, periods or intervals | `TypeError` | an array, a
bigint, a struct |
| a `uint8`, `uint64` or `float16` column | a bigint or a double |
`UNSUPPORTED_ARROWTYPE` |
| a sparse column | a bigint | `PySparkTypeError` of the Arrow conversion |
### Does this PR introduce _any_ user-facing change?
Yes. pandas input without a schema converts, or fails, as in PySpark 4.3.0.
### How was this patch tested?
- A new live test checks the types of an `int8`, a dict and a `float32`
column, and the time of a naive datetime. It passes with official PySpark 4.3.0
and fails without this change.
- New tests in `python/tests/test_arrow_handoff.py`, against the mock Arrow
input service, check:
- the Arrow table and its names for no schema, names, and fewer names
- `inferPandasDictAsMap`
- the session time zone for two client time zones
- the errors
- Existing tests that pinned the row path for a plain frame without a
schema now expect PySpark's behavior:
- the IPC column names `_0`, `_1`, ..., named by `toDF`, and no schema on
the relation
- the configuration request first
- the errors of the Arrow conversion
- Four tests created such frames on a session with no server, which the
configuration request now needs. They now use the mock Connect server.
- The tests of the row path's converter (`py_to_value`) now build a frame
of a `pd.DataFrame` subclass, which keeps that path, so they still cover it.
- The mock servers now serve
`spark.sql.execution.pandas.inferPandasDictAsMap`, as a real server does.
- On a local server, 92 cases of pandas input without a schema give the
same results as official PySpark 4.3.0. They cover each dtype, object columns
of each kind of value, time zones, column labels and names, empty frames, large
frames and the Arrow settings.
- The lib tests of both crates (590 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2068 passed, 8 skipped), and the
live drop-in and Arrow suites (162 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #308 from HyukjinKwon/SPARK-61104-pandas-inferred-arrow.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/session.rs | 108 +++++++++++++++++++++++++++++++++++
python/pyspark/sql/_pandas_input.py | 99 +++++++++++++++++++++++++++++++-
python/tests/_mock_connect_server.py | 1 +
python/tests/input_name_cases.py | 15 +++--
python/tests/integer_tuple_cases.py | 1 +
python/tests/interval_input_cases.py | 3 +-
python/tests/test_arrow_handoff.py | 67 +++++++++++++++++++++-
python/tests/test_dropin_e2e.py | 16 ++++++
python/tests/test_dropin_offline.py | 74 +++++++++++++++---------
9 files changed, 349 insertions(+), 35 deletions(-)
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 88fe6f3..bc11999 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -712,6 +712,111 @@ impl PySparkSession {
))))
}
+ /// A DataFrame of a pandas DataFrame without a schema, or of names, as
PySpark Connect
+ /// converts one: each column by PySpark's Arrow conversion, of the type
that Arrow infers
+ /// (`prepare_inferred_pandas_table`), sent without a schema, for the
server to infer the
+ /// types of the Arrow data. The columns are named with `toDF`: by their
labels, as strings,
+ /// or by the names, and `_n` for the n-th column past them.
+ ///
+ /// None for a frame that the row path converts: an empty one, a
customized one
+ /// (`accepts_pandas_frame`), and any without pyarrow, which the row path
does not need.
+ fn create_from_pandas_inferred(
+ &self,
+ py: Python<'_>,
+ data: &Bound<'_, PyAny>,
+ spec: &Spec,
+ ) -> PyResult<Option<PyDataFrame>> {
+ let names = match spec {
+ Spec::None => None,
+ Spec::Names(names) => Some(names),
+ Spec::Struct(_) | Spec::Atomic(_) => return Ok(None),
+ };
+ if data.len()? == 0 {
+ return Ok(None);
+ }
+ let helper = py.import("pyspark.sql._pandas_input")?;
+ if !helper.getattr("pyarrow_available")?.call0()?.is_truthy()?
+ || !helper
+ .getattr("accepts_pandas_frame")?
+ .call1((data,))?
+ .is_truthy()?
+ {
+ return Ok(None);
+ }
+ let width = data.getattr("columns")?.len()?;
+ let mut columns: Vec<String> = match names {
+ Some(names) => names.clone(),
+ None => {
+ let mut labels = Vec::with_capacity(width);
+ for label in data.getattr("columns")?.try_iter()? {
+ let label = label?;
+ labels.push(match label.cast::<pyo3::types::PyString>() {
+ Ok(label) => label.to_cow()?.into_owned(),
+ Err(_) => label.str()?.to_cow()?.into_owned(),
+ });
+ }
+ if labels.is_empty() {
+ return Err(crate::column::raise_pyspark(
+ py,
+ "PySparkValueError",
+ "CANNOT_INFER_EMPTY_SCHEMA",
+ &[],
+ ));
+ }
+ labels
+ }
+ };
+ if columns.len() < width {
+ columns.extend((columns.len()..width).map(|i| format!("_{}", i +
1)));
+ }
+ let options = py.detach(||
self.session.arrow_input_options().to_pyerr())?;
+ let conf = |key: &'static str| {
+ py.detach(|| self.session.conf().get(key))
+ .to_pyerr()
+ .map(|value| value.as_deref() == Some("true"))
+ };
+ let safecheck =
conf("spark.sql.execution.pandas.convertToArrowArraySafely")?;
+ let infer_dict_as_map =
+ names.is_none() &&
conf("spark.sql.execution.pandas.inferPandasDictAsMap")?;
+ let result = helper.getattr("prepare_inferred_pandas_table")?.call1((
+ data,
+ options.timezone().unwrap_or("UTC"),
+ safecheck,
+ options.use_large_var_types() == Some("true"),
+ infer_dict_as_map,
+ options.timestamp_type() == Some("TIMESTAMP_NTZ"),
+ ))?;
+ let table = result.get_item(0)?;
+ if let Some(struct_names) =
result.get_item(1)?.extract::<Option<Vec<String>>>()? {
+ columns = struct_names;
+ }
+ let actual: usize = table.getattr("num_columns")?.extract()?;
+ if columns.len() != actual {
+ return Err(crate::column::raise_pyspark(
+ py,
+ "PySparkValueError",
+ "AXIS_LENGTH_MISMATCH",
+ &[
+ ("expected_length", &columns.len().to_string()),
+ ("actual_length", &actual.to_string()),
+ ],
+ ));
+ }
+ let batches = crate::pyarrow_import::table_batches(&table)?;
+ let df = py
+ .detach(|| {
+ self.session
+ .create_dataframe_from_arrow_with_options(batches, None,
&options)
+ .map_err(Box::new)
+ })
+ .to_pyerr()?;
+ Ok(Some(PyDataFrame::new(if columns.is_empty() {
+ df
+ } else {
+ df.to_df(columns.iter().map(String::as_str).collect())
+ })))
+ }
+
/// A DataFrame of a numpy array, as PySpark Connect's `createDataFrame`
builds it: an Arrow
/// table of a column of each column of the array, sent with the schema,
if any, for the
/// server to cast to. The columns are named by the given names, else
`value` for a
@@ -1429,6 +1534,9 @@ impl PySparkSession {
return Ok(frame);
}
}
+ if let Some(frame) = self.create_from_pandas_inferred(py, data,
&spec)? {
+ return Ok(frame);
+ }
return self.create_from_pandas(py, data, spec);
}
// Preserve schema parsing, ignored-argument warnings, and
pandas/Arrow routing.
diff --git a/python/pyspark/sql/_pandas_input.py
b/python/pyspark/sql/_pandas_input.py
index fcbc9a0..fa92a3f 100644
--- a/python/pyspark/sql/_pandas_input.py
+++ b/python/pyspark/sql/_pandas_input.py
@@ -909,6 +909,12 @@ def accepts_pandas_table(frame, schema):
names by lookup, so each namespace must still hold the native classes. The
schema is
the caller's object, or a DDL string, which is parsed.
"""
+ return accepts_pandas_frame(frame) and (type(schema) is str or
_canonical_struct(schema))
+
+
+def accepts_pandas_frame(frame):
+ """Whether PySpark's Arrow conversion may convert this frame, as
`accepts_pandas_table`
+ checks it, but for a schema."""
import sys
import pandas as pd
@@ -973,7 +979,7 @@ def accepts_pandas_table(frame, schema):
and vars(importer)[name] is not getattr(converter, name,
None))
):
return False
- return type(schema) is str or _canonical_struct(schema)
+ return True
def prepare_pandas_table(frame, schema, timezone, safecheck,
prefers_large_types):
@@ -1034,3 +1040,94 @@ def arrow_decimal_column(values, dtype, name):
raise PySparkTypeError(message)
return array.type.precision, array.type.scale, array.to_pylist()
+
+def pyarrow_available():
+ """Whether pyarrow can be imported, checked without importing it. Without
it, a frame
+ without a schema keeps the row path, which does not need it."""
+ import importlib.util
+
+ try:
+ return importlib.util.find_spec("pyarrow") is not None
+ except (ImportError, ValueError):
+ return False
+
+
+def prepare_inferred_pandas_table(frame, timezone, safecheck,
prefers_large_types,
+ infer_dict_as_map, prefer_timestamp_ntz):
+ """Convert a pandas frame without a schema, or of names, as PySpark
Connect does.
+
+ Each column goes through PySpark's own Arrow conversion, of the type that
Arrow infers,
+ except that datetimes are timestamps, of the session time zone, and
timedeltas day-time
+ intervals, for the server to infer the types of the table. With
`inferPandasDictAsMap`
+ (`infer_dict_as_map`), the frame converts under the struct of the types
that Arrow infers,
+ where a column of dicts is a map, and the table takes the names and the
Arrow types of the
+ struct. Returns the table and the names of that struct, or None. The
caller checks the
+ frame with `accepts_pandas_frame`.
+ """
+ import pandas as pd
+ import pyarrow as pa
+ from pandas.api.types import is_datetime64_dtype, is_timedelta64_dtype
+
+ from pyspark.errors import PySparkValueError
+ from pyspark.sql.pandas.conversion import create_arrow_table_from_pandas
+ from pyspark.sql.pandas.types import (
+ _deduplicate_field_names,
+ from_arrow_type,
+ to_arrow_schema,
+ )
+ from pyspark.sql.types import (
+ DayTimeIntervalType,
+ MapType,
+ StringType,
+ StructType,
+ TimestampType,
+ )
+
+ struct = None
+ if infer_dict_as_map:
+ struct = StructType()
+ for field in pa.Schema.from_pandas(frame):
+ field_type = field.type
+ if isinstance(field_type, pa.StructType):
+ if len(field_type) == 0:
+ raise PySparkValueError(
+ errorClass="CANNOT_INFER_EMPTY_SCHEMA",
+ messageParameters={},
+ )
+ spark_type = MapType(
+ StringType(), from_arrow_type(field_type.field(0).type,
prefer_timestamp_ntz)
+ )
+ else:
+ spark_type = from_arrow_type(field_type, prefer_timestamp_ntz)
+ struct = struct.add(field.name, spark_type,
nullable=field.nullable)
+ deduplicated = _deduplicate_field_names(struct)
+ spark_types = [field.dataType for field in deduplicated.fields]
+ else:
+ spark_types = [
+ (
+ TimestampType()
+ if is_datetime64_dtype(t) or isinstance(t, pd.DatetimeTZDtype)
+ else DayTimeIntervalType()
+ if is_timedelta64_dtype(t)
+ else None
+ )
+ for t in frame.dtypes
+ ]
+ if len(frame.columns) == 0:
+ # pa.RecordBatch.from_pandas keeps the number of rows of a frame of no
columns.
+ table = pa.Table.from_batches([pa.RecordBatch.from_pandas(frame)])
+ else:
+ table = create_arrow_table_from_pandas(
+ [(c, st) for (_, c), st in zip(frame.items(), spark_types)],
+ timezone=timezone,
+ safecheck=safecheck,
+ prefers_large_types=prefers_large_types,
+ )
+ if struct is None:
+ return table, None
+ if len(struct.fields) > 0:
+ arrow_schema = to_arrow_schema(
+ deduplicated, timezone="UTC",
prefers_large_types=prefers_large_types
+ )
+ table = table.rename_columns(deduplicated.names).cast(arrow_schema)
+ return table, [str(name) for name in struct.fieldNames()]
diff --git a/python/tests/_mock_connect_server.py
b/python/tests/_mock_connect_server.py
index ec2c4da..0d4beee 100644
--- a/python/tests/_mock_connect_server.py
+++ b/python/tests/_mock_connect_server.py
@@ -354,6 +354,7 @@ def serve(connection):
"spark.sql.timestampType": "TIMESTAMP_LTZ",
"spark.sql.execution.arrow.useLargeVarTypes": "false",
"spark.sql.execution.pandas.convertToArrowArraySafely": "true",
+ "spark.sql.execution.pandas.inferPandasDictAsMap": "false",
"spark.sql.session.localRelationCacheThreshold": "1048576",
"spark.sql.session.localRelationSizeLimit": "3221225472",
"spark.sql.session.localRelationChunkSizeRows": "10000",
diff --git a/python/tests/input_name_cases.py b/python/tests/input_name_cases.py
index 6462b00..00d9c5a 100644
--- a/python/tests/input_name_cases.py
+++ b/python/tests/input_name_cases.py
@@ -112,6 +112,7 @@ class InputNameTests(unittest.TestCase):
b"spark.sql.session.localRelationChunkSizeBytes": str(1 <<
40).encode(),
b"spark.sql.session.localRelationBatchOfChunksSizeBytes":
str(1 << 40).encode(),
b"spark.sql.execution.pandas.convertToArrowArraySafely":
b"true",
+ b"spark.sql.execution.pandas.inferPandasDictAsMap": b"false",
}
pairs = b"".join(_bytes(2, _bytes(1, key) + _bytes(2, values[key]))
for key in keys)
@@ -143,8 +144,13 @@ class InputNameTests(unittest.TestCase):
frame.toArrow()
self.assertEqual(len(self.captured), before + 1)
local, renames = self.captured[-1]
- schema = json.loads(local[2][0])
- payload_names = [field["name"] for field in schema["fields"]]
+ if 2 in local:
+ schema = json.loads(local[2][0])
+ payload_names = [field["name"] for field in schema["fields"]]
+ else:
+ # As PySpark's Arrow conversion of pandas input without a schema,
which sends the
+ # table without one: the names of the table.
+ payload_names =
pa.ipc.open_stream(local[1][0]).read_all().schema.names
if 1 in local:
table = pa.ipc.open_stream(local[1][0]).read_all()
table.validate(full=True)
@@ -168,10 +174,11 @@ class InputNameTests(unittest.TestCase):
([Row(z=7, a=8), Row(z=9, a=10)], ["x", "y"], ["x", "y"], ["z",
"a"]),
([{"z": 8, "a": 7}, {"z": 10, "a": 9}], None, ["a", "z"], ["a",
"z"]),
([{"z": 8, "a": 7}, {"z": 10, "a": 9}], ["x", "y"], ["x", "y"],
["a", "z"]),
+ # The Arrow conversion of pandas input names its table's columns
_0, _1, ... for toDF.
(pd.DataFrame([[7, 8], [9, 10]], columns=["café", "雪"]),
- None, ["café", "雪"], ["café", "雪"]),
+ None, ["café", "雪"], ["_0", "_1"]),
(pd.DataFrame([[7, 8], [9, 10]], columns=["old", "names"]),
- ["x", "y"], ["x", "y"], ["x", "y"]),
+ ["x", "y"], ["x", "y"], ["_0", "_1"]),
]
for data, schema, names, ipc_names in cases:
with self.subTest(names=names, source=type(data).__name__):
diff --git a/python/tests/integer_tuple_cases.py
b/python/tests/integer_tuple_cases.py
index 2a6bd3f..ed49ae2 100644
--- a/python/tests/integer_tuple_cases.py
+++ b/python/tests/integer_tuple_cases.py
@@ -111,6 +111,7 @@ class IntegerTupleTests(unittest.TestCase):
b"spark.sql.session.localRelationChunkSizeBytes": str(1 <<
40).encode(),
b"spark.sql.session.localRelationBatchOfChunksSizeBytes":
str(1 << 40).encode(),
b"spark.sql.execution.pandas.convertToArrowArraySafely":
b"true",
+ b"spark.sql.execution.pandas.inferPandasDictAsMap": b"false",
}
pairs = b"".join(_bytes(2, _bytes(1, key) + _bytes(2, values[key]))
for key in keys)
diff --git a/python/tests/interval_input_cases.py
b/python/tests/interval_input_cases.py
index ee41a1d..c38d7c2 100644
--- a/python/tests/interval_input_cases.py
+++ b/python/tests/interval_input_cases.py
@@ -137,7 +137,8 @@ class IntervalInputConversionTests(unittest.TestCase):
self.assertEqual(table.column(0).cast(pa.int64()).to_pylist(),
values)
for value in (timedelta.min, timedelta.max):
overflow = pd.DataFrame({"interval": pd.Series([value],
dtype=object)})
- with self.assertRaises(OverflowError):
+ # As PySpark's Arrow conversion: ArrowInvalid, a ValueError,
of the duration.
+ with self.assertRaises(ValueError):
spark.createDataFrame(overflow)
self.assertEqual(service.execute_calls, 1)
diff --git a/python/tests/test_arrow_handoff.py
b/python/tests/test_arrow_handoff.py
index c88dd5e..3911ad8 100644
--- a/python/tests/test_arrow_handoff.py
+++ b/python/tests/test_arrow_handoff.py
@@ -1596,7 +1596,13 @@ def
test_nul_names_keep_lossless_ipc_row_fallback(protocol, schema_kind):
spark.createDataFrame(frame, schema).toArrow()
assert state["result"] == (1000, 0, 999)
payload = state["relations"][0].local_relation.data
- assert pa.ipc.open_stream(payload).schema.names == [name]
+ if schema_kind == "explicit":
+ assert pa.ipc.open_stream(payload).schema.names == [name]
+ else:
+ # As PySpark Connect's, the Arrow conversion of a frame without a
schema names its
+ # columns _0, _1, ..., which toDF names.
+ assert pa.ipc.open_stream(payload).schema.names == ["_0"]
+ assert state["column_names"] == [name]
@pytest.mark.parametrize("nullable", [False, True])
@@ -3047,10 +3053,14 @@ def
test_float64_public_fallback_preserves_every_ipc_cell(protocol, reason):
frame.columns = ["x\x00original"]
elif reason == "schema_nul":
schema = expected_schema = T.StructType([T.StructField("x\x00renamed",
T.DoubleType())])
+ if reason in ("inferred", "names"):
+ # As PySpark Connect's, the Arrow conversion of a frame without a
schema names its
+ # columns _0, _1, ..., which toDF names.
+ expected_schema = T.StructType([T.StructField("_0", T.DoubleType())])
with _arrow_input_service(protocol) as (spark, state):
spark.createDataFrame(frame, schema).toArrow()
- # Official converts explicit schemas with Arrow, reading the
configuration first.
- native = reason in ("inferred", "names", "frame_subclass",
"schema_nul")
+ # Official converts pandas input with Arrow, reading the configuration
first.
+ native = reason in ("frame_subclass", "schema_nul")
assert state["config_requests"][:1] == (
[] if native else [("get", _NORMALIZATION_KEYS + _UPLOAD_KEYS)])
_assert_float64_ipc(state["input_tables"][0], columns, expected_schema)
@@ -4838,6 +4848,57 @@ def
test_pandas_explicit_nonnullable_null_raises_as_pyspark(protocol):
assert state["input_tables"] == []
[email protected]("schema", [None, ["p", "q"], ["p"]], ids=["inferred",
"names", "fewer"])
+def test_pandas_inferred_types_are_those_of_arrow_as_pyspark(protocol, schema):
+ # As PySpark Connect's: without a schema, each column converts with
PySpark's Arrow
+ # conversion, of the type that Arrow infers, and the table goes without a
schema, of
+ # columns _0, _1, ..., which toDF names, by the labels, or the names and
_n past them.
+ frame = pd.DataFrame({"a": np.array([1, 2], dtype=np.int8), "b": [{"k":
1}, {"k": 2}]})
+ with _arrow_input_service(protocol) as (spark, state):
+ spark.createDataFrame(frame, schema).toArrow()
+ table = state["input_tables"][0]
+ assert table.schema.names == ["_0", "_1"]
+ assert table.schema.types == [pa.int8(), pa.struct([("k",
pa.int64())])]
+ assert table.column("_1").to_pylist() == [{"k": 1}, {"k": 2}]
+ assert state["spark_schema"] is None
+ assert state["column_names"] == {None: ["a", "b"], 2: ["p", "q"], 1:
["p", "_2"]}[
+ None if schema is None else len(schema)]
+
+
+def test_pandas_inferred_dicts_are_maps_of_infer_pandas_dict_as_map(protocol):
+ # As PySpark Connect's: of
spark.sql.execution.pandas.inferPandasDictAsMap, a column of dicts
+ # is a map of the type of the first key, and the table takes the names of
the columns.
+ frame = pd.DataFrame({"a": [{"k": 1}, {"j": 2}], "b": ["x", "y"]})
+ table = _pandas_table(protocol, frame, None, **{
+ "spark.sql.execution.pandas.inferPandasDictAsMap": "true"})
+ assert table.schema.names == ["a", "b"]
+ assert table.schema.field("a").type == pa.map_(pa.string(), pa.int64())
+ assert table.column("a").to_pylist() == [[("k", 1)], [("j", 2)]]
+
+
[email protected]("host", ["America/Los_Angeles", "UTC"])
+def test_pandas_inferred_naive_timestamp_uses_session_zone(protocol, host):
+ # As PySpark Connect's: a naive time is a time of the session zone,
whatever the client's.
+ frame = pd.DataFrame({"t": [pd.Timestamp("2020-07-01 12:00")]})
+ table = _in_host_zone(host, lambda: _pandas_table(
+ protocol, frame, None, **{"spark.sql.session.timeZone":
"America/Los_Angeles"}))
+ assert table.column("_0").to_pylist() == [datetime(2020, 7, 1, 19,
tzinfo=timezone.utc)]
+
+
+def test_pandas_inferred_conversion_errors_are_those_of_pyspark(protocol):
+ from pyspark.errors import PySparkValueError
+
+ with _arrow_input_service(protocol) as (spark, state):
+ # As PySpark Connect's: Arrow's error of a column of values of several
types.
+ with pytest.raises(PySparkValueError, match="Exception thrown when
converting "
+ "pandas.Series \\(object\\) with name 'a' to Arrow
Array"):
+ spark.createDataFrame(pd.DataFrame({"a": [1, "x"]}))
+ with pytest.raises(PySparkValueError) as error:
+ spark.createDataFrame(pd.DataFrame({"a": [1]}), ["p", "q"])
+ assert error.value.getCondition() == "AXIS_LENGTH_MISMATCH"
+ assert error.value.getMessageParameters() == {"expected_length": "2",
"actual_length": "1"}
+
+
@pytest.mark.parametrize("host", ["America/Los_Angeles", "UTC"])
def test_pandas_naive_timestamp_uses_session_zone(protocol, host):
frame = pd.DataFrame({"t": [pd.Timestamp("2020-07-01 12:00")]})
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 1123a2c..4d4b4e9 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1633,6 +1633,22 @@ def
test_variant_spatial_and_user_defined_values_round_trip_as_pyspark(session):
assert geometry == Geometry.fromWKB(wkb, 0)
+def
test_create_dataframe_of_pandas_infers_the_types_of_arrow_as_pyspark(session):
+ # As PySpark Connect's: without a schema, the columns of a pandas
DataFrame convert with
+ # PySpark's Arrow conversion, of the types that Arrow infers, and a naive
datetime is a time
+ # of the session time zone.
+ import numpy as np
+ import pandas as pd
+
+ frame = pd.DataFrame({"a": np.array([1, 2], dtype=np.int8), "b": [{"k":
1}, {"k": 2}],
+ "c": np.array([1.5, 2.5], dtype=np.float32)})
+ df = session.createDataFrame(frame)
+ assert df.schema.simpleString() ==
"struct<a:tinyint,b:struct<k:bigint>,c:float>"
+ assert [row.b.k for row in df.orderBy("a").collect()] == [1, 2]
+ naive = pd.DataFrame({"t": [pd.Timestamp("2020-07-01 12:00")]})
+ rows = session.createDataFrame(naive).selectExpr("CAST(t AS STRING) AS
s").collect()
+ assert rows == [Row(s="2020-07-01 12:00:00")]
+
def test_create_dataframe_types_pandas_columns_as_pyspark(session):
from decimal import Decimal
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 9de9f77..7f3bf3f 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -5745,18 +5745,20 @@ for schema in schemas:
"""], check=True)
-def test_create_dataframe_types_pandas_columns_as_arrow():
+def test_create_dataframe_types_pandas_columns_as_arrow(mock_connect):
pd = pytest.importorskip("pandas")
np = pytest.importorskip("numpy")
from pyspark.errors import PySparkTypeError
# As PySpark, whose Arrow conversion types a column without values by its
dtype, rather
- # than raising CANNOT_DETERMINE_TYPE (the live suite checks the types).
+ # than raising CANNOT_DETERMINE_TYPE (the live suite checks the types).
The conversion
+ # reads the server's Arrow settings first.
+ spark, _ = mock_connect
for column in [[np.nan, np.nan], [None, None], pd.to_datetime([None,
None])]:
- _local_create_dataframe(pd.DataFrame({"id": [1, 2], "s": column}))
+ spark.createDataFrame(pd.DataFrame({"id": [1, 2], "s": column}))
with pytest.raises(PySparkTypeError, match=re.escape(
"Exception thrown when converting pandas.Series (object) with name
'm' to Arrow")):
- _local_create_dataframe(pd.DataFrame({"m": [{None: 1}]}))
+ spark.createDataFrame(pd.DataFrame({"m": [{None: 1}]}))
def test_timedelta_values_convert_as_pyspark():
@@ -5813,15 +5815,15 @@ def
test_create_dataframe_rows_is_local_as_pyspark(data, schema, expected):
@pytest.mark.parametrize("schema", [None, ["c", "d"], ["c"], "a long, b
string",
T.StructType([T.StructField("a",
T.LongType()),
T.StructField("b",
T.StringType())])])
-def test_create_dataframe_pandas_is_not_local_as_pyspark(schema):
+def test_create_dataframe_pandas_is_not_local_as_pyspark(mock_connect, schema):
pd = pytest.importorskip("pandas")
- if schema is not None:
- # Explicit-schema pandas input reads the server's Arrow settings
first, as PySpark
- # does, so it needs a live server; the live-server suites cover it.
- pytest.skip("explicit-schema pandas input reads the server
configuration")
- # pyspark applies the columns, or the field names of the schema, with toDF.
- df = _local_create_dataframe(pd.DataFrame({"a": [1, 2], "b": ["x", "y"]}),
schema)
- assert not df.isLocal()
+ # pyspark applies the columns, or the field names of the schema, with
toDF, so the result
+ # is not local. pandas input reads the server's Arrow settings first, as
PySpark does.
+ spark, service = mock_connect
+ spark.createDataFrame(pd.DataFrame({"a": [1, 2], "b": ["x", "y"]}),
schema).collect()
+ # ExecutePlanRequest.plan=3, Plan.root=1, Relation.to_df=18.
+ plan = parse_message(parse_message(service.execute_requests[-1])[3][0])
+ assert 18 in parse_message(plan[1][0])
def test_create_dataframe_empty_pandas_is_local_as_pyspark():
@@ -5862,9 +5864,6 @@ def test_create_dataframe_pandas_errors_as_pyspark():
_local_create_dataframe(pdf, schema)
assert error.value.getCondition() == "UNSUPPORTED_DATA_TYPE_FOR_ARROW"
assert error.value.getMessageParameters() == {"data_type": data_type}
- with pytest.raises(PySparkValueError) as error:
- _local_create_dataframe(pdf, ["c", "d", "e"])
- assert error.value.getCondition() == "AXIS_LENGTH_MISMATCH"
for data, schema in [(pd.DataFrame(index=[0, 1]), None), ([], None), ([],
["a"]),
(pdf.iloc[:0], None), (pdf.iloc[:0], ["a", "b"])]:
with pytest.raises(PySparkValueError) as error:
@@ -6189,14 +6188,21 @@ def
test_create_dataframe_raises_recursion_error_for_a_value_that_contains_itsel
# py_to_value converts the cells of object-dtype pandas input, as well as the
values of
# fillna, pivot and Row conversion; local rows go through infer.convert
instead, so the
# "local" source is a control for values that the exact-number fast path must
not change.
+def _RowPathFrame(pd):
+ """A subclass of pandas.DataFrame: createDataFrame converts a customized
frame with its
+ row path (py_to_value), where it converts a plain one with PySpark's Arrow
conversion."""
+ return type("RowPathFrame", (pd.DataFrame,), {})
+
+
def _native_number_input_table(mock_connect, data, schema=None, *,
source="local"):
import pyarrow as pa
if source == "pandas":
pd = pytest.importorskip("pandas")
- # Local rows now use infer.convert. Object-backed pandas input retains
- # py_to_value, which is the generic converter optimized by this patch.
- data = pd.DataFrame({"value": [row[0] for row in data]}, dtype=object)
+ # Local rows now use infer.convert. Object-backed pandas input of a
customized frame, of
+ # a subclass, retains py_to_value, the converter of the row path
optimized by this
+ # patch; a plain frame converts with PySpark's Arrow conversion.
+ data = _RowPathFrame(pd)({"value": [row[0] for row in data]},
dtype=object)
spark, service = mock_connect
spark.createDataFrame(data, schema).toArrow()
request = parse_message(service.execute_requests[-1])
@@ -6454,6 +6460,7 @@ def _container_input_capture_server(connection):
"spark.sql.session.localRelationChunkSizeBytes": str(1 << 40),
"spark.sql.session.localRelationBatchOfChunksSizeBytes": str(1
<< 40),
"spark.sql.execution.pandas.convertToArrowArraySafely": "true",
+ "spark.sql.execution.pandas.inferPandasDictAsMap": "false",
}
return message_field(1, request[1][0]) + b"".join(
message_field(2, message_field(1, key) + message_field(2,
defaults[key]))
@@ -6613,9 +6620,10 @@ def _exact_date_converted_value(mock_connect, value):
pd = pytest.importorskip("pandas")
spark, service = mock_connect
- # py_to_value remains the generic pandas converter. Local rows now use
+ # py_to_value remains the pandas converter of the row path, which a
customized frame, of a
+ # subclass, keeps; a plain frame converts with PySpark's Arrow conversion.
Local rows use
# infer.convert, which intentionally has different subclass/type semantics.
- data = pd.DataFrame({"value": [value]}, dtype=object)
+ data = _RowPathFrame(pd)({"value": [value]}, dtype=object)
before = len(service.execute_requests)
spark.createDataFrame(data).toArrow()
requests = service.execute_requests
@@ -6711,11 +6719,17 @@ def
test_exact_date_nested_unrelated_values(mock_connect):
assert (actual["number"], actual["flag"], actual["label"]) == (42, True,
"date")
-def test_exact_date_named_impostor_rejected():
+def test_exact_date_named_impostor_rejected(mock_connect):
pd = pytest.importorskip("pandas")
+ from pyspark.errors import PySparkValueError
+
+ # As PySpark's Arrow conversion of a frame without a schema, which does
not take an object
+ # that is not a date as one.
+ spark, _ = mock_connect
impostor = type("date", (), {"toordinal": lambda self: 719163})
- with pytest.raises(TypeError, match="unsupported value type"):
- _local_create_dataframe(pd.DataFrame({"value": [impostor()]},
dtype=object))
+ with pytest.raises(PySparkValueError, match=re.escape(
+ "Exception thrown when converting pandas.Series (object) with name
'value'")):
+ spark.createDataFrame(pd.DataFrame({"value": [impostor()]},
dtype=object))
def _exact_date_fields(data):
@@ -6775,6 +6789,7 @@ def _exact_date_capture_server(connection):
"spark.sql.session.localRelationChunkSizeBytes": str(1 << 40),
"spark.sql.session.localRelationBatchOfChunksSizeBytes": str(1 << 40),
"spark.sql.execution.pandas.convertToArrowArraySafely": "true",
+ "spark.sql.execution.pandas.inferPandasDictAsMap": "false",
}
def _exact_date_config(request, context):
@@ -6902,6 +6917,7 @@ def config(raw, context):
'spark.sql.session.localRelationBatchOfChunksSizeBytes': str(64 << 20),
# As the default of a server.
'spark.sql.execution.pandas.convertToArrowArraySafely': 'true',
+ 'spark.sql.execution.pandas.inferPandasDictAsMap': 'false',
}
operation = request.operation
if operation.HasField('get'):
@@ -7141,7 +7157,10 @@ def test_decimal_scale_input_ipc(decimal_ipc_session,
kind, schema_mode, texts):
context.traps[Inexact] = True
spark.createDataFrame(data, schema).collect()
table = pa.ipc.open_stream(payload.read_bytes()).read_all()
- expected = pa.table({name: pa.array(values, type=decimal_type)})
+ # As PySpark's Arrow conversion of a pandas frame without a schema, whose
table names its
+ # columns _0, _1, ..., which toDF names.
+ ipc_name = "_0" if kind == "pandas" and schema_mode != "explicit" else name
+ expected = pa.table({ipc_name: pa.array(values, type=decimal_type)})
table.validate(full=True)
assert table.schema == expected.schema
assert table.equals(expected)
@@ -7191,7 +7210,9 @@ def
test_decimal_scale_avoids_coefficient_tuple(decimal_ipc_session, schema):
pd = pytest.importorskip("pandas")
spark, _ = decimal_ipc_session
tuple_constructor = type(Decimal("1.20").as_tuple()).__new__.__code__
- values = pd.DataFrame({"v": [Decimal("123.4500"), Decimal("-2.3000"),
None]}, dtype=object)
+ # Of the converter of the row path, which a customized frame keeps
(_RowPathFrame).
+ values = _RowPathFrame(pd)(
+ {"v": [Decimal("123.4500"), Decimal("-2.3000"), None]}, dtype=object)
profile = cProfile.Profile()
profile.enable()
try:
@@ -7208,7 +7229,8 @@ def
test_decimal_scale_of_non_finite_values(decimal_ipc_session, text):
pd = pytest.importorskip("pandas")
spark, payload = decimal_ipc_session
- data = pd.DataFrame({"v": [Decimal(text), Decimal("1.5")]}, dtype=object)
+ # Of the converter of the row path, which a customized frame keeps
(_RowPathFrame).
+ data = _RowPathFrame(pd)({"v": [Decimal(text), Decimal("1.5")]},
dtype=object)
with pytest.raises(ValueError, match="has no precision to type a column
by"):
spark.createDataFrame(data, ["v"])
assert not payload.exists()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]