This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 56e9d18  [SPARK-60954][CONNECT] Type all-null and Decimal columns of 
pandas input as PySpark does, and reject null map keys
56e9d18 is described below

commit 56e9d18e9f877f8517b2518a9044b500cbef2ef6
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 13:13:38 2026 +0900

    [SPARK-60954][CONNECT] Type all-null and Decimal columns of pandas input as 
PySpark does, and reject null map keys
    
    ### What changes were proposed in this pull request?
    
    [SPARK-60954](https://issues.apache.org/jira/browse/SPARK-60954): 
`createDataFrame` of a pandas DataFrame now types these columns as PySpark's 
Arrow conversion does, and the Arrow encoder rejects a null map key with an 
error.
    
    - **A column without values** (all NaN, NaT or None) takes the type of its 
dtype, as `pa.Schema.from_pandas` gives it:
      - `double` or `float` for floats;
      - `timestamp` for datetimes;
      - the integer, boolean or string type of a nullable dtype;
      - `void` for an object column.
    - **A column of Decimals** takes the precision and scale that Arrow infers 
from all its values: the most integer digits and the largest scale. For 
example, `1.5` and `2.25` give `decimal(3,2)`.
    - **Decimal strings:** a Decimal is now written without an exponent, so a 
value such as `Decimal("1E+2")` encodes.
    - **Null map keys:**
      - A pandas column with a map of a null key raises PySpark's 
`PySparkTypeError`: `Exception thrown when converting pandas.Series (object) 
with name '<column>' to Arrow Array (None).`
      - In the Rust encoder, a null map key is an error, and the map arrays are 
built with `try_new`, so that `SparkSession::create_dataframe` cannot panic on 
one.
    
    Other pandas columns are still typed by their first non-null value.
    
    ### Why are the changes needed?
    
    Every column was typed by its first non-null value:
    
    - `pd.DataFrame({"id": [1, 2], "s": [np.nan, np.nan]})` raised 
`CANNOT_DETERMINE_TYPE`, where PySpark gives a `double` column, and before #149 
this worked too.
    - A column of `Decimal("1.5")` and `Decimal("2.25")` became 
`decimal(38,1)`, and `2.25` was silently stored as `2.2`. A Decimal written 
with an exponent failed to encode.
    - A dict with a `None` key panicked in Rust (`PanicException`), which 
`except Exception` does not catch.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, these pandas columns convert as in PySpark 4.3.0.
    
    ### How was this patch tested?
    
    - Against a local Spark 4.3.0 server, a probe of seven pandas inputs gives 
output identical to official PySpark 4.3.0:
      - all-NaN, all-None and all-NaT columns;
      - Decimals of different scales, with an exponent and with None;
      - a map with a None key.
    - A new unit test checks that a null map key is an error of the encoder.
    - A new offline test checks the columns without values and the error.
    - A new live test checks the types and values, and it also passes with 
official PySpark 4.3.0.
    - The lib tests and every integration test, e2e included, against the local 
server (644 passed), the build of all tests with `wasm-udf`, the offline Python 
suites (614 passed, 8 live cases skipped), and the live drop-in and Arrow 
suites (58 passed) and live cases (10 passed). Clippy reports no diagnostics in 
the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #177 from HyukjinKwon/SPARK-60954-pandas.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/session.rs    | 123 ++++++++++++++++++++++++++++++++----
 crates/spark-connect/src/session.rs |  24 ++++++-
 python/tests/test_dropin_e2e.py     |  24 +++++++
 python/tests/test_dropin_offline.py |  15 +++++
 4 files changed, 171 insertions(+), 15 deletions(-)

diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 4f13fcb..433926d 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -1,7 +1,7 @@
 //! PyO3 wrapper for Spark Connect sessions.
 
 use pyo3::prelude::*;
-use pyo3::types::{PyDict, PyList};
+use pyo3::types::{PyDict, PyList, PyString};
 use spark_connect::row::{Row, StructValue, Value};
 use spark_connect::session::SparkSession;
 use spark_connect::types::{DataType, StructField};
@@ -444,6 +444,11 @@ impl PySparkSession {
         for label in data.getattr("columns")?.try_iter()? {
             columns.push(label?.str()?.extract::<String>()?);
         }
+        // The dtype of each column, which types a column without values, as 
Arrow does.
+        let mut dtypes = Vec::new();
+        for dtype in data.getattr("dtypes")?.try_iter()? {
+            dtypes.push(dtype?.str()?.extract::<String>()?);
+        }
         // df.astype(object).where(df.notna(), None).values.tolist() -> rows 
with None for NA.
         let notna = data.call_method0("notna")?;
         let obj = data.call_method1("astype", ("object",))?;
@@ -455,6 +460,8 @@ impl PySparkSession {
             crate::column::raise_pyspark(py, "PySparkValueError", 
"CANNOT_INFER_EMPTY_SCHEMA", &[])
         };
         // The given fields, and the names that toDF applies, which empty data 
does not get.
+        // The labels, which errors of a column name, as PySpark names its 
pandas.Series.
+        let labels = columns.clone();
         let (given, names) = match spec {
             Spec::Atomic(atomic) if !empty => {
                 // pyspark reports str() of the parsed type, which is its repr.
@@ -507,27 +514,44 @@ impl PySparkSession {
             rows.push(Row::new(row_names.clone(), values));
         }
         // A given struct wins; otherwise each field takes the type of its 
first non-null
-        // value (empty input has a struct schema, checked above).
+        // value (empty input has a struct schema, checked above), but for 
these, which are
+        // typed as Arrow types a pandas column, as PySpark does: a column of 
Decimals takes
+        // the precision and scale of all its values, and a column without 
values the type of
+        // its dtype.
         let fields = match given {
             Some(fields) => fields,
             None => {
                 let mut fields: Vec<StructField> = 
Vec::with_capacity(row_names.len());
                 for (i, name) in row_names.iter().enumerate() {
-                    let Some(value) = rows
+                    // As PySpark, whose Arrow conversion of the column fails 
for a null map key.
+                    if rows
                         .iter()
                         .filter_map(|row| row.get(i))
-                        .find(|value| !matches!(value, Value::Null))
-                    else {
-                        return Err(crate::column::raise_pyspark(
-                            py,
-                            "PySparkValueError",
-                            "CANNOT_DETERMINE_TYPE",
-                            &[],
+                        .any(has_null_map_key)
+                    {
+                        return Err(PyErr::from_value(
+                            py.import("pyspark.errors")?
+                                .getattr("PySparkTypeError")?
+                                .call1((format!(
+                                    "Exception thrown when converting 
pandas.Series ({}) with name \
+                                     '{}' to Arrow Array (None).",
+                                    dtypes.get(i).map_or("object", 
String::as_str),
+                                    labels.get(i).unwrap_or(name),
+                                ),))?,
                         ));
+                    }
+                    let first = rows
+                        .iter()
+                        .filter_map(|row| row.get(i))
+                        .find(|value| !matches!(value, Value::Null));
+                    let data_type = match first {
+                        Some(Value::Decimal { .. }) => 
pandas_decimal_type(&data, i)?,
+                        Some(value) => value_to_datatype(value),
+                        None => 
pandas_dtype_type(dtypes.get(i).map_or("object", String::as_str)),
                     };
                     fields.push(StructField {
                         name: name.clone(),
-                        data_type: value_to_datatype(value),
+                        data_type,
                         nullable: true,
                         metadata: Default::default(),
                     });
@@ -1263,6 +1287,78 @@ impl PyConnectClientStub {
     fn _cleanup_ml_cache(&self) {}
 }
 
+/// Whether `value` has a map with a null key, at any depth.
+fn has_null_map_key(value: &Value) -> bool {
+    match value {
+        Value::Map(entries) => entries
+            .iter()
+            .any(|(key, value)| matches!(key, Value::Null) || 
has_null_map_key(value)),
+        Value::List(items) => items.iter().any(has_null_map_key),
+        Value::Struct(row) => row.values().iter().any(has_null_map_key),
+        _ => false,
+    }
+}
+
+/// The type of a pandas column without values, by its dtype, as Arrow types it
+/// (`pa.Schema.from_pandas`): `void` for an object column of None, as for one 
of another
+/// dtype that Spark has no type for.
+fn pandas_dtype_type(dtype: &str) -> DataType {
+    match dtype {
+        "float64" | "Float64" => DataType::Double,
+        "float32" | "Float32" => DataType::Float,
+        "int64" | "Int64" => DataType::Long,
+        "int32" | "Int32" => DataType::Integer,
+        "int16" | "Int16" => DataType::Short,
+        "int8" | "Int8" => DataType::Byte,
+        "bool" | "boolean" => DataType::Boolean,
+        "string" | "str" => crate::infer::string_type(),
+        dtype if dtype.starts_with("datetime64") => DataType::Timestamp,
+        dtype if dtype.starts_with("timedelta64") => DataType::DayTimeInterval 
{
+            start_field: 0,
+            end_field: 3,
+        },
+        _ => DataType::Null,
+    }
+}
+
+/// The decimal type of column `i` of pandas rows, as Arrow infers it from all 
its Decimals:
+/// the most integer digits and the largest scale of the values.
+fn pandas_decimal_type(rows: &Bound<'_, PyList>, i: usize) -> 
PyResult<DataType> {
+    let (mut integer_digits, mut scale) = (0i64, 0i64);
+    for row in rows.iter() {
+        let value = row.get_item(i)?;
+        if value.is_none() {
+            continue;
+        }
+        let parts = value.call_method0("as_tuple")?;
+        let digits = parts.getattr("digits")?.len()? as i64;
+        // NaN and infinities have no exponent of digits, which Arrow cannot 
type either.
+        let Ok(exponent) = parts.getattr("exponent")?.extract::<i64>() else {
+            return Err(pyo3::exceptions::PyValueError::new_err(format!(
+                "Decimal {} has no precision to type a column by",
+                value.str()?
+            )));
+        };
+        let (precision, value_scale) = if exponent < 0 {
+            (digits.max(-exponent), -exponent)
+        } else {
+            (digits + exponent, 0)
+        };
+        integer_digits = integer_digits.max(precision - value_scale);
+        scale = scale.max(value_scale);
+    }
+    let precision = integer_digits + scale;
+    if precision > 38 {
+        return Err(pyo3::exceptions::PyValueError::new_err(format!(
+            "Decimal precision {precision} is out of the range of DecimalType, 
up to 38"
+        )));
+    }
+    Ok(DataType::Decimal {
+        precision: precision.max(1) as i32,
+        scale: scale as i32,
+    })
+}
+
 /// Infer the Spark `DataType` for a `createDataFrame` column from its (first) 
value.
 /// Nested collections are inferred recursively; nulls and empty collections 
use
 /// String as a fallback. Interval input conversion is tracked in SPARK-60659.
@@ -1492,7 +1588,10 @@ fn decimal_to_value(obj: &Bound<'_, PyAny>) -> 
PyResult<Option<Value>> {
     if !obj.is_instance(&decimal_cls)? {
         return Ok(None);
     }
-    let dec_str: String = obj.str()?.extract()?;
+    // Without an exponent, as `1E+2` is written, which the Arrow encoder 
parses.
+    let dec_str: String = PyString::new(py, "{:f}")
+        .call_method1("format", (obj,))?
+        .extract()?;
     let as_tuple = obj.call_method0("as_tuple")?;
     let mut scale: Option<i32> = None;
     if let Ok(exp_obj) = as_tuple.get_item(2) {
diff --git a/crates/spark-connect/src/session.rs 
b/crates/spark-connect/src/session.rs
index 67e8870..ab57f64 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -1191,6 +1191,9 @@ fn values_to_arrow(values: &[Value], dt: &DataType) -> 
Result<Arc<dyn arrow::arr
                     Value::Map(m) => {
                         valid.append(true);
                         for (k, val) in m {
+                            if matches!(k, Value::Null) {
+                                return Err(SparkError::connect_msg("A map key 
cannot be null"));
+                            }
                             keys.push(k.clone());
                             vals.push(val.clone());
                         }
@@ -1210,19 +1213,22 @@ fn values_to_arrow(values: &[Value], dt: &DataType) -> 
Result<Arc<dyn arrow::arr
                 ),
             ]
             .into();
-            let entries = StructArray::new(entry_fields.clone(), vec![key_arr, 
val_arr], None);
+            let entries = StructArray::try_new(entry_fields.clone(), 
vec![key_arr, val_arr], None)
+                .map_err(|e| SparkError::connect_msg(format!("map build: 
{e}")))?;
             let entries_field = Arc::new(Field::new(
                 "entries",
                 arrow::datatypes::DataType::Struct(entry_fields),
                 false,
             ));
-            Ok(Arc::new(MapArray::new(
+            MapArray::try_new(
                 entries_field,
                 OffsetBuffer::new(offsets.into()),
                 entries,
                 Some(NullBuffer::new(valid.finish())),
                 false,
-            )))
+            )
+            .map(|array| Arc::new(array) as Arc<dyn Array>)
+            .map_err(|e| SparkError::connect_msg(format!("map build: {e}")))
         }
         other => Err(SparkError::connect_msg(format!(
             "Unsupported type for Arrow createDataFrame conversion: {other:?}"
@@ -1827,6 +1833,18 @@ mod tests {
         }
     }
 
+    #[test]
+    fn a_null_map_key_is_an_error() {
+        let map = DataType::Map {
+            key_type: Box::new(DataType::Long),
+            value_type: Box::new(DataType::Long),
+            value_contains_null: true,
+        };
+        let value = Value::Map(vec![(Value::Null, Value::Long(1))]);
+        let error = values_to_arrow(&[value], &map).unwrap_err().to_string();
+        assert!(error.contains("A map key cannot be null"), "{error}");
+    }
+
     #[test]
     fn struct_values_of_the_declared_fields_are_taken_by_position() {
         // Fields of the same name keep their own values.
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 4adf970..833b357 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -501,6 +501,30 @@ def 
test_apply_in_pandas_with_state_runs_as_pyspark(session, tmp_path):
         session.conf.unset("spark.sql.shuffle.partitions")
 
 
+def test_create_dataframe_types_pandas_columns_as_pyspark(session):
+    from decimal import Decimal
+
+    pd = pytest.importorskip("pandas")
+    np = pytest.importorskip("numpy")
+    cases = [
+        # A column without values takes the type of its dtype.
+        ({"id": [1, 2], "s": [np.nan, np.nan]}, "struct<id:bigint,s:double>", 
[(1, None), (2, None)]),
+        ({"id": [1, 2], "s": [None, None]}, "struct<id:bigint,s:void>", [(1, 
None), (2, None)]),
+        ({"id": [1, 2], "t": pd.to_datetime([None, None])}, 
"struct<id:bigint,t:timestamp>",
+         [(1, None), (2, None)]),
+        # A column of Decimals takes the precision and scale of all its values.
+        ({"d": [Decimal("1.5"), Decimal("2.25")]}, "struct<d:decimal(3,2)>",
+         [(Decimal("1.50"),), (Decimal("2.25"),)]),
+        ({"d": [Decimal("1E+2"), Decimal("0.001")]}, "struct<d:decimal(6,3)>",
+         [(Decimal("100.000"),), (Decimal("0.001"),)]),
+        ({"d": [Decimal("1.5"), None]}, "struct<d:decimal(2,1)>", 
[(Decimal("1.5"),), (None,)]),
+    ]
+    for data, schema, rows in cases:
+        df = session.createDataFrame(pd.DataFrame(data))
+        assert df.schema.simpleString() == schema, data
+        assert [tuple(r) for r in df.collect()] == rows, data
+
+
 def test_collected_rows_behave_as_pyspark_rows(session):
     import pickle
 
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 2c0a9d0..b7b8683 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -8,6 +8,7 @@ that require a live server live in test_dropin_e2e.py and are 
gated on SPARK_REM
 
 import array
 import datetime
+import re
 import decimal
 import json
 import pickle
@@ -3572,6 +3573,20 @@ for schema in schemas:
 """], check=True)
 
 
+def test_create_dataframe_types_pandas_columns_as_arrow():
+    pd = pytest.importorskip("pandas")
+    np = pytest.importorskip("numpy")
+    from pyspark.errors import PySparkTypeError
+
+    # As PySpark, whose Arrow conversion types a column without values by its 
dtype, rather
+    # than raising CANNOT_DETERMINE_TYPE (the live suite checks the types).
+    for column in [[np.nan, np.nan], [None, None], pd.to_datetime([None, 
None])]:
+        _local_create_dataframe(pd.DataFrame({"id": [1, 2], "s": column}))
+    with pytest.raises(PySparkTypeError, match=re.escape(
+            "Exception thrown when converting pandas.Series (object) with name 
'm' to Arrow")):
+        _local_create_dataframe(pd.DataFrame({"m": [{None: 1}]}))
+
+
 def _local_create_dataframe(*args):
     # The local relation and its toDF are built without a server.
     return _offline_session().createDataFrame(*args)


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to