This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 56e9d18 [SPARK-60954][CONNECT] Type all-null and Decimal columns of
pandas input as PySpark does, and reject null map keys
56e9d18 is described below
commit 56e9d18e9f877f8517b2518a9044b500cbef2ef6
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 13:13:38 2026 +0900
[SPARK-60954][CONNECT] Type all-null and Decimal columns of pandas input as
PySpark does, and reject null map keys
### What changes were proposed in this pull request?
[SPARK-60954](https://issues.apache.org/jira/browse/SPARK-60954):
`createDataFrame` of a pandas DataFrame now types these columns as PySpark's
Arrow conversion does, and the Arrow encoder rejects a null map key with an
error.
- **A column without values** (all NaN, NaT or None) takes the type of its
dtype, as `pa.Schema.from_pandas` gives it:
- `double` or `float` for floats;
- `timestamp` for datetimes;
- the integer, boolean or string type of a nullable dtype;
- `void` for an object column.
- **A column of Decimals** takes the precision and scale that Arrow infers
from all its values: the most integer digits and the largest scale. For
example, `1.5` and `2.25` give `decimal(3,2)`.
- **Decimal strings:** a Decimal is now written without an exponent, so a
value such as `Decimal("1E+2")` encodes.
- **Null map keys:**
- A pandas column with a map of a null key raises PySpark's
`PySparkTypeError`: `Exception thrown when converting pandas.Series (object)
with name '<column>' to Arrow Array (None).`
- In the Rust encoder, a null map key is an error, and the map arrays are
built with `try_new`, so that `SparkSession::create_dataframe` cannot panic on
one.
Other pandas columns are still typed by their first non-null value.
### Why are the changes needed?
Every column was typed by its first non-null value:
- `pd.DataFrame({"id": [1, 2], "s": [np.nan, np.nan]})` raised
`CANNOT_DETERMINE_TYPE`, where PySpark gives a `double` column, and before #149
this worked too.
- A column of `Decimal("1.5")` and `Decimal("2.25")` became
`decimal(38,1)`, and `2.25` was silently stored as `2.2`. A Decimal written
with an exponent failed to encode.
- A dict with a `None` key panicked in Rust (`PanicException`), which
`except Exception` does not catch.
### Does this PR introduce _any_ user-facing change?
Yes, these pandas columns convert as in PySpark 4.3.0.
### How was this patch tested?
- Against a local Spark 4.3.0 server, a probe of seven pandas inputs gives
output identical to official PySpark 4.3.0:
- all-NaN, all-None and all-NaT columns;
- Decimals of different scales, with an exponent and with None;
- a map with a None key.
- A new unit test checks that a null map key is an error of the encoder.
- A new offline test checks the columns without values and the error.
- A new live test checks the types and values, and it also passes with
official PySpark 4.3.0.
- The lib tests and every integration test, e2e included, against the local
server (644 passed), the build of all tests with `wasm-udf`, the offline Python
suites (614 passed, 8 live cases skipped), and the live drop-in and Arrow
suites (58 passed) and live cases (10 passed). Clippy reports no diagnostics in
the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #177 from HyukjinKwon/SPARK-60954-pandas.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/session.rs | 123 ++++++++++++++++++++++++++++++++----
crates/spark-connect/src/session.rs | 24 ++++++-
python/tests/test_dropin_e2e.py | 24 +++++++
python/tests/test_dropin_offline.py | 15 +++++
4 files changed, 171 insertions(+), 15 deletions(-)
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 4f13fcb..433926d 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -1,7 +1,7 @@
//! PyO3 wrapper for Spark Connect sessions.
use pyo3::prelude::*;
-use pyo3::types::{PyDict, PyList};
+use pyo3::types::{PyDict, PyList, PyString};
use spark_connect::row::{Row, StructValue, Value};
use spark_connect::session::SparkSession;
use spark_connect::types::{DataType, StructField};
@@ -444,6 +444,11 @@ impl PySparkSession {
for label in data.getattr("columns")?.try_iter()? {
columns.push(label?.str()?.extract::<String>()?);
}
+ // The dtype of each column, which types a column without values, as
Arrow does.
+ let mut dtypes = Vec::new();
+ for dtype in data.getattr("dtypes")?.try_iter()? {
+ dtypes.push(dtype?.str()?.extract::<String>()?);
+ }
// df.astype(object).where(df.notna(), None).values.tolist() -> rows
with None for NA.
let notna = data.call_method0("notna")?;
let obj = data.call_method1("astype", ("object",))?;
@@ -455,6 +460,8 @@ impl PySparkSession {
crate::column::raise_pyspark(py, "PySparkValueError",
"CANNOT_INFER_EMPTY_SCHEMA", &[])
};
// The given fields, and the names that toDF applies, which empty data
does not get.
+ // The labels, which errors of a column name, as PySpark names its
pandas.Series.
+ let labels = columns.clone();
let (given, names) = match spec {
Spec::Atomic(atomic) if !empty => {
// pyspark reports str() of the parsed type, which is its repr.
@@ -507,27 +514,44 @@ impl PySparkSession {
rows.push(Row::new(row_names.clone(), values));
}
// A given struct wins; otherwise each field takes the type of its
first non-null
- // value (empty input has a struct schema, checked above).
+ // value (empty input has a struct schema, checked above), but for
these, which are
+ // typed as Arrow types a pandas column, as PySpark does: a column of
Decimals takes
+ // the precision and scale of all its values, and a column without
values the type of
+ // its dtype.
let fields = match given {
Some(fields) => fields,
None => {
let mut fields: Vec<StructField> =
Vec::with_capacity(row_names.len());
for (i, name) in row_names.iter().enumerate() {
- let Some(value) = rows
+ // As PySpark, whose Arrow conversion of the column fails
for a null map key.
+ if rows
.iter()
.filter_map(|row| row.get(i))
- .find(|value| !matches!(value, Value::Null))
- else {
- return Err(crate::column::raise_pyspark(
- py,
- "PySparkValueError",
- "CANNOT_DETERMINE_TYPE",
- &[],
+ .any(has_null_map_key)
+ {
+ return Err(PyErr::from_value(
+ py.import("pyspark.errors")?
+ .getattr("PySparkTypeError")?
+ .call1((format!(
+ "Exception thrown when converting
pandas.Series ({}) with name \
+ '{}' to Arrow Array (None).",
+ dtypes.get(i).map_or("object",
String::as_str),
+ labels.get(i).unwrap_or(name),
+ ),))?,
));
+ }
+ let first = rows
+ .iter()
+ .filter_map(|row| row.get(i))
+ .find(|value| !matches!(value, Value::Null));
+ let data_type = match first {
+ Some(Value::Decimal { .. }) =>
pandas_decimal_type(&data, i)?,
+ Some(value) => value_to_datatype(value),
+ None =>
pandas_dtype_type(dtypes.get(i).map_or("object", String::as_str)),
};
fields.push(StructField {
name: name.clone(),
- data_type: value_to_datatype(value),
+ data_type,
nullable: true,
metadata: Default::default(),
});
@@ -1263,6 +1287,78 @@ impl PyConnectClientStub {
fn _cleanup_ml_cache(&self) {}
}
+/// Whether `value` has a map with a null key, at any depth.
+fn has_null_map_key(value: &Value) -> bool {
+ match value {
+ Value::Map(entries) => entries
+ .iter()
+ .any(|(key, value)| matches!(key, Value::Null) ||
has_null_map_key(value)),
+ Value::List(items) => items.iter().any(has_null_map_key),
+ Value::Struct(row) => row.values().iter().any(has_null_map_key),
+ _ => false,
+ }
+}
+
+/// The type of a pandas column without values, by its dtype, as Arrow types it
+/// (`pa.Schema.from_pandas`): `void` for an object column of None, as for one
of another
+/// dtype that Spark has no type for.
+fn pandas_dtype_type(dtype: &str) -> DataType {
+ match dtype {
+ "float64" | "Float64" => DataType::Double,
+ "float32" | "Float32" => DataType::Float,
+ "int64" | "Int64" => DataType::Long,
+ "int32" | "Int32" => DataType::Integer,
+ "int16" | "Int16" => DataType::Short,
+ "int8" | "Int8" => DataType::Byte,
+ "bool" | "boolean" => DataType::Boolean,
+ "string" | "str" => crate::infer::string_type(),
+ dtype if dtype.starts_with("datetime64") => DataType::Timestamp,
+ dtype if dtype.starts_with("timedelta64") => DataType::DayTimeInterval
{
+ start_field: 0,
+ end_field: 3,
+ },
+ _ => DataType::Null,
+ }
+}
+
+/// The decimal type of column `i` of pandas rows, as Arrow infers it from all
its Decimals:
+/// the most integer digits and the largest scale of the values.
+fn pandas_decimal_type(rows: &Bound<'_, PyList>, i: usize) ->
PyResult<DataType> {
+ let (mut integer_digits, mut scale) = (0i64, 0i64);
+ for row in rows.iter() {
+ let value = row.get_item(i)?;
+ if value.is_none() {
+ continue;
+ }
+ let parts = value.call_method0("as_tuple")?;
+ let digits = parts.getattr("digits")?.len()? as i64;
+ // NaN and infinities have no exponent of digits, which Arrow cannot
type either.
+ let Ok(exponent) = parts.getattr("exponent")?.extract::<i64>() else {
+ return Err(pyo3::exceptions::PyValueError::new_err(format!(
+ "Decimal {} has no precision to type a column by",
+ value.str()?
+ )));
+ };
+ let (precision, value_scale) = if exponent < 0 {
+ (digits.max(-exponent), -exponent)
+ } else {
+ (digits + exponent, 0)
+ };
+ integer_digits = integer_digits.max(precision - value_scale);
+ scale = scale.max(value_scale);
+ }
+ let precision = integer_digits + scale;
+ if precision > 38 {
+ return Err(pyo3::exceptions::PyValueError::new_err(format!(
+ "Decimal precision {precision} is out of the range of DecimalType,
up to 38"
+ )));
+ }
+ Ok(DataType::Decimal {
+ precision: precision.max(1) as i32,
+ scale: scale as i32,
+ })
+}
+
/// Infer the Spark `DataType` for a `createDataFrame` column from its (first)
value.
/// Nested collections are inferred recursively; nulls and empty collections
use
/// String as a fallback. Interval input conversion is tracked in SPARK-60659.
@@ -1492,7 +1588,10 @@ fn decimal_to_value(obj: &Bound<'_, PyAny>) ->
PyResult<Option<Value>> {
if !obj.is_instance(&decimal_cls)? {
return Ok(None);
}
- let dec_str: String = obj.str()?.extract()?;
+ // Without an exponent, as `1E+2` is written, which the Arrow encoder
parses.
+ let dec_str: String = PyString::new(py, "{:f}")
+ .call_method1("format", (obj,))?
+ .extract()?;
let as_tuple = obj.call_method0("as_tuple")?;
let mut scale: Option<i32> = None;
if let Ok(exp_obj) = as_tuple.get_item(2) {
diff --git a/crates/spark-connect/src/session.rs
b/crates/spark-connect/src/session.rs
index 67e8870..ab57f64 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -1191,6 +1191,9 @@ fn values_to_arrow(values: &[Value], dt: &DataType) ->
Result<Arc<dyn arrow::arr
Value::Map(m) => {
valid.append(true);
for (k, val) in m {
+ if matches!(k, Value::Null) {
+ return Err(SparkError::connect_msg("A map key
cannot be null"));
+ }
keys.push(k.clone());
vals.push(val.clone());
}
@@ -1210,19 +1213,22 @@ fn values_to_arrow(values: &[Value], dt: &DataType) ->
Result<Arc<dyn arrow::arr
),
]
.into();
- let entries = StructArray::new(entry_fields.clone(), vec![key_arr,
val_arr], None);
+ let entries = StructArray::try_new(entry_fields.clone(),
vec![key_arr, val_arr], None)
+ .map_err(|e| SparkError::connect_msg(format!("map build:
{e}")))?;
let entries_field = Arc::new(Field::new(
"entries",
arrow::datatypes::DataType::Struct(entry_fields),
false,
));
- Ok(Arc::new(MapArray::new(
+ MapArray::try_new(
entries_field,
OffsetBuffer::new(offsets.into()),
entries,
Some(NullBuffer::new(valid.finish())),
false,
- )))
+ )
+ .map(|array| Arc::new(array) as Arc<dyn Array>)
+ .map_err(|e| SparkError::connect_msg(format!("map build: {e}")))
}
other => Err(SparkError::connect_msg(format!(
"Unsupported type for Arrow createDataFrame conversion: {other:?}"
@@ -1827,6 +1833,18 @@ mod tests {
}
}
+ #[test]
+ fn a_null_map_key_is_an_error() {
+ let map = DataType::Map {
+ key_type: Box::new(DataType::Long),
+ value_type: Box::new(DataType::Long),
+ value_contains_null: true,
+ };
+ let value = Value::Map(vec![(Value::Null, Value::Long(1))]);
+ let error = values_to_arrow(&[value], &map).unwrap_err().to_string();
+ assert!(error.contains("A map key cannot be null"), "{error}");
+ }
+
#[test]
fn struct_values_of_the_declared_fields_are_taken_by_position() {
// Fields of the same name keep their own values.
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 4adf970..833b357 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -501,6 +501,30 @@ def
test_apply_in_pandas_with_state_runs_as_pyspark(session, tmp_path):
session.conf.unset("spark.sql.shuffle.partitions")
+def test_create_dataframe_types_pandas_columns_as_pyspark(session):
+ from decimal import Decimal
+
+ pd = pytest.importorskip("pandas")
+ np = pytest.importorskip("numpy")
+ cases = [
+ # A column without values takes the type of its dtype.
+ ({"id": [1, 2], "s": [np.nan, np.nan]}, "struct<id:bigint,s:double>",
[(1, None), (2, None)]),
+ ({"id": [1, 2], "s": [None, None]}, "struct<id:bigint,s:void>", [(1,
None), (2, None)]),
+ ({"id": [1, 2], "t": pd.to_datetime([None, None])},
"struct<id:bigint,t:timestamp>",
+ [(1, None), (2, None)]),
+ # A column of Decimals takes the precision and scale of all its values.
+ ({"d": [Decimal("1.5"), Decimal("2.25")]}, "struct<d:decimal(3,2)>",
+ [(Decimal("1.50"),), (Decimal("2.25"),)]),
+ ({"d": [Decimal("1E+2"), Decimal("0.001")]}, "struct<d:decimal(6,3)>",
+ [(Decimal("100.000"),), (Decimal("0.001"),)]),
+ ({"d": [Decimal("1.5"), None]}, "struct<d:decimal(2,1)>",
[(Decimal("1.5"),), (None,)]),
+ ]
+ for data, schema, rows in cases:
+ df = session.createDataFrame(pd.DataFrame(data))
+ assert df.schema.simpleString() == schema, data
+ assert [tuple(r) for r in df.collect()] == rows, data
+
+
def test_collected_rows_behave_as_pyspark_rows(session):
import pickle
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 2c0a9d0..b7b8683 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -8,6 +8,7 @@ that require a live server live in test_dropin_e2e.py and are
gated on SPARK_REM
import array
import datetime
+import re
import decimal
import json
import pickle
@@ -3572,6 +3573,20 @@ for schema in schemas:
"""], check=True)
+def test_create_dataframe_types_pandas_columns_as_arrow():
+ pd = pytest.importorskip("pandas")
+ np = pytest.importorskip("numpy")
+ from pyspark.errors import PySparkTypeError
+
+ # As PySpark, whose Arrow conversion types a column without values by its
dtype, rather
+ # than raising CANNOT_DETERMINE_TYPE (the live suite checks the types).
+ for column in [[np.nan, np.nan], [None, None], pd.to_datetime([None,
None])]:
+ _local_create_dataframe(pd.DataFrame({"id": [1, 2], "s": column}))
+ with pytest.raises(PySparkTypeError, match=re.escape(
+ "Exception thrown when converting pandas.Series (object) with name
'm' to Arrow")):
+ _local_create_dataframe(pd.DataFrame({"m": [{None: 1}]}))
+
+
def _local_create_dataframe(*args):
# The local relation and its toDF are built without a server.
return _offline_session().createDataFrame(*args)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]