This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new d678130  [SPARK-60902][CONNECT] Apply column-name lists and pandas 
columns in createDataFrame as PySpark does
d678130 is described below

commit d6781304ab228db9569b016999445eba0b5d1b1d
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Thu Nov 5 18:46:25 2026 +0900

    [SPARK-60902][CONNECT] Apply column-name lists and pandas columns in 
createDataFrame as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-60902](https://issues.apache.org/jira/browse/SPARK-60902): the 
Rust-backed Python client's `createDataFrame` now applies column names as 
PySpark Connect 4.3.0 does.
    
    - A list or tuple of column names is applied with `toDF`, and so are the 
columns of a pandas DataFrame or the field names of its schema. The server 
resolves `toDF` into a `Project`, so `isLocal()` is False for these DataFrames, 
as in PySpark.
    - Without a schema, plain rows are named `_1`, `_2`, ... instead of `col0`, 
`col1`, .... A list of names shorter than the plain rows or the pandas 
DataFrame is extended the same way, and an empty list gives `_1`, `_2`, ....
    - Dict rows, and other objects through their `__dict__`, are read by name, 
as in PySpark. A struct schema takes their values by its field names, so 
missing keys are null and other keys are ignored; without one, the fields are 
the rows' sorted keys in the order first seen. They used to be matched by 
position after sorting their keys.
    - A namedtuple row is named by its `_fields`, as a Row is, rather than 
`_1`, `_2`, ....
    - Each inferred field takes the type of its first non-null value, where 
only the first row counted, and a field that is null in every row raises 
`CANNOT_DETERMINE_TYPE`, as in PySpark.
    - Plain rows and Rows that do not match the width of the schema raise 
`AXIS_LENGTH_MISMATCH`, with PySpark's expected and actual lengths. A pandas 
DataFrame with an atomic schema, given as a type or as DDL, raises 
`UNSUPPORTED_DATA_TYPE_FOR_ARROW` with `str()` of the parsed type. Empty input 
without a struct schema raises `CANNOT_INFER_EMPTY_SCHEMA` instead of a plain 
`ValueError`.
    
    Two pandas cases still differ, on purpose. With a StructType narrower than 
the DataFrame, PySpark silently drops the extra columns (its conversion zips 
the columns with the fields), and with a wider one it raises PyArrow's 
`ArrowInvalid`; this client raises `AXIS_LENGTH_MISMATCH` for both, as it 
raised an error before. The new `dev/parity/known-differences.md` records this, 
NumPy ndarray input (SPARK-60772), and how PySpark merges the inferred types of 
all rows (SPARK-60908).
    
    ### Why are the changes needed?
    
    `createDataFrame([(1, 2)])` had different column names from PySpark, 
`createDataFrame([(1, 2)], ["a"])` failed where PySpark returns columns `a` and 
`_2`, and `isLocal()` and the error classes differed from PySpark's.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, these now match PySpark Connect 4.3.0:
    
    - `createDataFrame([(1, 2)]).columns` is `['_1', '_2']`, previously 
`['col0', 'col1']`.
    - `createDataFrame([(1, 2)], ["a"])` has the columns `['a', '_2']`; it 
raised an error before.
    - `isLocal()` is False for `createDataFrame([(1, 2)], ["a", "b"])` and for 
`createDataFrame(pandas_df)`, with or without a schema; it was True.
    - `createDataFrame([{"b": 1, "a": "x"}], "b long, a string")` puts `1` in 
`b`; it put `"x"` there. A dict missing a key of the schema gets a null for it, 
where it raised an error.
    - `createDataFrame([(None, 1), (2, 3)])` infers `bigint` for both fields; 
it raised an error.
    - The error classes above, where errors had no class before.
    
    ### How was this patch tested?
    
    - Against a local Spark 4.3.0 server, probes over 49 inputs give output 
identical to official PySpark 4.3.0, including the column names, collected 
rows, `isLocal()`, and the errors and their parameters. The inputs cover rows 
with no schema, a list or tuple of names, Rows, namedtuples, dicts and other 
objects, and StructType, DDL, and atomic schemas; pandas DataFrames with no 
schema, names, StructType, DDL, atomic, and atomic DDL schemas, and empty ones; 
dicts with reordered, missing,  [...]
    - New offline tests check `isLocal()` for rows and pandas input, and the 
error classes and parameters. The existing width test now expects PySpark's 
`AXIS_LENGTH_MISMATCH`, still a `PySparkValueError`.
    - New live tests check the column names and rows for rows, namedtuples, 
dicts and other objects, and pandas input, and the types inferred past nulls. 
They also pass with official PySpark 4.3.0.
    - The offline Python suites (447 passed, 8 live cases skipped), the live 
drop-in and Arrow suites (45 passed), and the live cases (9 passed). Clippy 
reports no new diagnostics in the changed files.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #149 from HyukjinKwon/SPARK-60902-createdataframe-todf.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/session.rs    | 318 ++++++++++++++++++++++++++----------
 crates/spark-connect/src/session.rs |   6 +-
 dev/parity/known-differences.md     |  20 +++
 python/tests/test_dropin_e2e.py     |  88 +++++++++-
 python/tests/test_dropin_offline.py |  98 ++++++++++-
 5 files changed, 435 insertions(+), 95 deletions(-)

diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index b8f6d5f..f302757 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -3,7 +3,7 @@
 use pyo3::prelude::*;
 use pyo3::types::{PyDict, PyList};
 use spark_connect::row::{Row, Value};
-use spark_connect::session::{row_width_mismatch, SparkSession};
+use spark_connect::session::SparkSession;
 use spark_connect::types::{DataType, StructField};
 use spark_connect::udf::{CommonInlineUserDefinedFunctionExpression, 
PythonUDFPayload};
 
@@ -378,11 +378,14 @@ impl PySparkSession {
         Ok(PyDataFrame::new(df))
     }
 
-    /// Create a DataFrame from a list of tuples/lists and an optional schema. 
A pandas
-    /// DataFrame or a pyarrow Table is accepted too, as in pyspark.
+    /// Create a DataFrame from local data (a list of rows, a pandas 
DataFrame, or a pyarrow
+    /// Table) and an optional schema, as in pyspark.
     ///
-    /// If schema is a list of strings, those are used as column names (all 
types inferred as string).
-    /// If schema is None, columns are named col0, col1, etc. and all types 
are string.
+    /// Column names follow pyspark: without a schema, plain rows are named 
`_1`, `_2`, ...
+    /// and Rows and dicts keep their own names. A list of names, extended 
with `_2`, `_3`,
+    /// ... for wider plain rows, and the columns of a pandas DataFrame (or 
the field names
+    /// of its schema) are applied with `toDF`. Without a struct schema, the 
field types are
+    /// inferred from the first row.
     #[pyo3(signature = (data, schema=None, samplingRatio=None, 
verifySchema=None))]
     fn createDataFrame(
         &self,
@@ -406,10 +409,8 @@ impl PySparkSession {
             return crate::pyarrow_import::create_dataframe(py, &self.session, 
data, schema);
         }
 
-        // Accept a pandas DataFrame (as pyspark does): convert it to a list 
of row lists
-        // with NaN/NaT mapped to None, and take its column names as the 
default schema.
-        // pandas-on-Spark always passes an explicit `schema`, so this mainly 
needs to
-        // deliver the row values; the list path below handles the rest.
+        // A pandas DataFrame (as pyspark does): its rows as lists with 
NaN/NaT mapped to None,
+        // and its column labels, as strings, which are applied below.
         let mut pandas_columns: Option<Vec<String>> = None;
         let is_pandas = py
             .import("pandas")
@@ -418,7 +419,11 @@ impl PySparkSession {
             .unwrap_or(false);
         let data_owned: Bound<'_, PyList>;
         let data: &Bound<'_, PyList> = if is_pandas {
-            pandas_columns = 
Some(data.getattr("columns")?.call_method0("tolist")?.extract()?);
+            let mut labels = Vec::new();
+            for label in data.getattr("columns")?.try_iter()? {
+                labels.push(label?.str()?.extract::<String>()?);
+            }
+            pandas_columns = Some(labels);
             // df.astype(object).where(df.notna(), None).values.tolist() -> 
rows with None for NA.
             let notna = data.call_method0("notna")?;
             let obj = data.call_method1("astype", ("object",))?;
@@ -437,6 +442,7 @@ impl PySparkSession {
             Names(Vec<String>),
             Struct(Vec<StructField>),
         }
+        let mut atomic_schema: Option<DataType> = None;
         let spec = match schema {
             None => Spec::None,
             Some(s) => {
@@ -449,104 +455,228 @@ impl PySparkSession {
                     // column (matching pyspark createDataFrame(data, 
AtomicType())).
                     match crate::types::py_to_data_type(s)? {
                         DataType::Struct { fields } => Spec::Struct(fields),
-                        atomic => 
Spec::Struct(vec![spark_connect::types::StructField {
-                            name: "value".to_string(),
-                            data_type: atomic,
-                            nullable: true,
-                            metadata: std::collections::BTreeMap::new(),
-                        }]),
+                        atomic => {
+                            atomic_schema = Some(atomic.clone());
+                            
Spec::Struct(vec![spark_connect::types::StructField {
+                                name: "value".to_string(),
+                                data_type: atomic,
+                                nullable: true,
+                                metadata: std::collections::BTreeMap::new(),
+                            }])
+                        }
                     }
                 }
             }
         };
 
-        let spec_names: Option<Vec<String>> = match &spec {
-            Spec::Names(n) => Some(n.clone()),
-            Spec::Struct(f) => Some(f.iter().map(|x| 
x.name.clone()).collect()),
-            Spec::None => None,
-        };
+        // As pyspark, empty input needs a schema to type its relation, and 
gets no toDF.
+        let empty = data.is_empty();
+        if empty && !matches!(spec, Spec::Struct(_)) {
+            return Err(crate::column::raise_pyspark(
+                py,
+                "PySparkValueError",
+                "CANNOT_INFER_EMPTY_SCHEMA",
+                &[],
+            ));
+        }
+        if !empty {
+            if let Some(columns) = &pandas_columns {
+                if let Some(atomic) = &atomic_schema {
+                    // pyspark reports str() of the parsed type, which is its 
repr.
+                    let data_type = crate::types::data_type_to_py(py, atomic)?
+                        .bind(py)
+                        .repr()?
+                        .extract::<String>()?;
+                    return Err(crate::column::raise_pyspark(
+                        py,
+                        "PySparkTypeError",
+                        "UNSUPPORTED_DATA_TYPE_FOR_ARROW",
+                        &[("data_type", &data_type)],
+                    ));
+                }
+                if matches!(spec, Spec::None) && columns.is_empty() {
+                    return Err(crate::column::raise_pyspark(
+                        py,
+                        "PySparkValueError",
+                        "CANNOT_INFER_EMPTY_SCHEMA",
+                        &[],
+                    ));
+                }
+            }
+        }
 
-        // Parse each item into values; a list/tuple carries no names, a Row 
carries its own.
-        let mut rows: Vec<Row> = vec![];
+        // How a row carries its values, as in pyspark: positionally (lists, 
tuples and
+        // scalars), positionally with names of its own (Rows, and namedtuples 
by their
+        // `_fields`), or by name (dicts, and other objects through their 
`__dict__`, whose
+        // keys are sorted as `_infer_schema` does).
+        enum Item {
+            Plain(Vec<Value>),
+            Named(Vec<String>, Vec<Value>),
+            Record(Vec<(String, Value)>),
+        }
+        let record = |dict: &Bound<'_, PyDict>| -> PyResult<Item> {
+            let mut pairs: Vec<(String, Value)> = 
Vec::with_capacity(dict.len());
+            for (k, v) in dict.iter() {
+                pairs.push((k.str()?.to_string(), py_to_value(&v)?));
+            }
+            pairs.sort_by(|a, b| a.0.cmp(&b.0));
+            Ok(Item::Record(pairs))
+        };
+        let mut items: Vec<Item> = Vec::with_capacity(data.len());
         for item in data.iter() {
-            let (values, row_names): (Vec<Value>, Option<Vec<String>>) =
-                if let Ok(pyrow) = item.extract::<PyRef<PyRow>>() {
-                    (
-                        pyrow.row.values().to_vec(),
-                        Some(pyrow.row.fields().to_vec()),
-                    )
-                } else if let Ok(lst) = item.cast::<PyList>() {
-                    (
-                        lst.iter()
-                            .map(|v| py_to_value(&v))
-                            .collect::<PyResult<_>>()?,
-                        None,
-                    )
-                } else if let Ok(tuple) = item.cast::<pyo3::types::PyTuple>() {
-                    (
-                        tuple
-                            .iter()
-                            .map(|v| py_to_value(&v))
-                            .collect::<PyResult<_>>()?,
-                        None,
-                    )
-                } else if let Ok(dict) = item.cast::<PyDict>() {
-                    // A dict row: field names are its keys (sorted, mirroring 
pyspark's
-                    // `_infer_schema` for dicts), values in that key order.
-                    let mut pairs: Vec<(String, Bound<'_, PyAny>)> = 
Vec::with_capacity(dict.len());
-                    for (k, v) in dict.iter() {
-                        pairs.push((k.str()?.to_string(), v));
+            items.push(if let Ok(dict) = item.cast::<PyDict>() {
+                record(dict)?
+            } else if let Ok(pyrow) = item.extract::<PyRef<PyRow>>() {
+                Item::Named(pyrow.row.fields().to_vec(), 
pyrow.row.values().to_vec())
+            } else if let Ok(lst) = item.cast::<PyList>() {
+                Item::Plain(
+                    lst.iter()
+                        .map(|v| py_to_value(&v))
+                        .collect::<PyResult<_>>()?,
+                )
+            } else if let Ok(tuple) = item.cast::<pyo3::types::PyTuple>() {
+                let values = tuple
+                    .iter()
+                    .map(|v| py_to_value(&v))
+                    .collect::<PyResult<_>>()?;
+                match tuple
+                    .getattr("_fields")
+                    .and_then(|fields| fields.extract::<Vec<String>>())
+                {
+                    Ok(fields) => Item::Named(fields, values),
+                    Err(_) => Item::Plain(values),
+                }
+            } else if let Some(attributes) = item
+                .getattr("__dict__")
+                .ok()
+                .and_then(|attributes| attributes.cast_into::<PyDict>().ok())
+            {
+                record(&attributes)?
+            } else {
+                // A scalar row (int/str/float/...): a single-field row, 
matching
+                // `createDataFrame([1, 2, 3], IntegerType())`. The field name 
comes from
+                // the schema when given, else defaults below.
+                Item::Plain(vec![py_to_value(&item)?])
+            });
+        }
+
+        // Plain rows are as wide as the widest of them: pyspark's schema 
inference names
+        // them `_1`, `_2`, ..., and extends a shorter list of names the same 
way.
+        let plain_width = items
+            .iter()
+            .filter_map(|item| match item {
+                Item::Plain(values) => Some(values.len()),
+                _ => None,
+            })
+            .max()
+            .unwrap_or(0);
+        // The keys of the dict rows, in the order pyspark merges their 
schemas: each row's
+        // sorted keys that no earlier row had.
+        let mut record_names: Vec<String> = Vec::new();
+        let mut seen = std::collections::HashSet::new();
+        for item in &items {
+            if let Item::Record(pairs) = item {
+                for (key, _) in pairs {
+                    if seen.insert(key.as_str()) {
+                        record_names.push(key.clone());
                     }
-                    pairs.sort_by(|a, b| a.0.cmp(&b.0));
-                    let names: Vec<String> = pairs.iter().map(|(k, _)| 
k.clone()).collect();
-                    let values: Vec<Value> = pairs
-                        .iter()
-                        .map(|(_, v)| py_to_value(v))
-                        .collect::<PyResult<_>>()?;
-                    (values, Some(names))
-                } else {
-                    // A scalar row (int/str/float/...): a single-field row, 
matching
-                    // `createDataFrame([1, 2, 3], IntegerType())`. The field 
name comes from
-                    // the schema when given, else defaults below.
-                    (vec![py_to_value(&item)?], None)
-                };
-            let names = spec_names
-                .clone()
-                .or(row_names)
-                .or_else(|| pandas_columns.clone())
-                .unwrap_or_else(|| (0..values.len()).map(|i| format!("col{}", 
i)).collect());
+                }
+            }
+        }
+        let extend = |names: &[String], width: usize| -> Vec<String> {
+            let mut names = names.to_vec();
+            names.extend((names.len()..width).map(|i| format!("_{}", i + 1)));
+            names
+        };
+        // The names pyspark applies with toDF: a list of names, and the 
columns of a pandas
+        // DataFrame or the field names of its schema.
+        let to_df_names: Option<Vec<String>> = match (&spec, &pandas_columns) {
+            _ if empty => None,
+            (Spec::Names(names), Some(columns)) => Some(extend(names, 
columns.len())),
+            (Spec::Names(names), None) => Some(extend(names, plain_width)),
+            (Spec::None, Some(columns)) => Some(columns.clone()),
+            (Spec::Struct(fields), Some(_)) => {
+                Some(fields.iter().map(|f| f.name.clone()).collect())
+            }
+            (_, None) => None,
+        };
+        let default_names = extend(&[], plain_width);
+        // A dict row's value for `name`, as pyspark looks it up: null when 
the key is missing.
+        // The pairs are sorted by key, and the first of equal keys is taken.
+        let lookup = |pairs: &[(String, Value)], name: &String| {
+            let index = pairs.partition_point(|(key, _)| key < name);
+            match pairs.get(index) {
+                Some((key, value)) if key == name => value.clone(),
+                _ => Value::Null,
+            }
+        };
+
+        let mut rows: Vec<Row> = Vec::with_capacity(items.len());
+        for item in items {
+            let (names, values) = match (&spec, &to_df_names, item) {
+                // A struct schema takes dict values by its field names, so 
that missing
+                // keys are null and other keys are ignored, and other rows by 
position.
+                (Spec::Struct(fields), _, item) => {
+                    let names: Vec<String> = fields.iter().map(|f| 
f.name.clone()).collect();
+                    let values = match item {
+                        Item::Record(pairs) => names.iter().map(|n| 
lookup(&pairs, n)).collect(),
+                        Item::Plain(values) | Item::Named(_, values) => values,
+                    };
+                    (names, values)
+                }
+                (_, names, Item::Record(pairs)) => (
+                    names.clone().unwrap_or_else(|| record_names.clone()),
+                    record_names.iter().map(|n| lookup(&pairs, n)).collect(),
+                ),
+                (_, Some(names), Item::Plain(values) | Item::Named(_, values)) 
=> {
+                    (names.clone(), values)
+                }
+                (_, None, Item::Named(names, values)) => (names, values),
+                (_, None, Item::Plain(values)) => (default_names.clone(), 
values),
+            };
             if names.len() != values.len() {
-                return Err(row_width_mismatch(rows.len(), values.len(), 
names.len())).to_pyerr();
+                // As pyspark, which checks every row against the width of the 
schema.
+                return Err(crate::column::raise_pyspark(
+                    py,
+                    "PySparkValueError",
+                    "AXIS_LENGTH_MISMATCH",
+                    &[
+                        ("expected_length", &names.len().to_string()),
+                        ("actual_length", &values.len().to_string()),
+                    ],
+                ));
             }
             rows.push(Row::new(names, values));
         }
 
-        // An explicit struct schema wins; otherwise infer field types from 
the first row
-        // (which requires at least one row -- an empty dataset needs an 
explicit schema,
-        // matching pyspark which creates an empty DataFrame from schema + [] 
data).
+        // An explicit struct schema wins; otherwise each field takes the type 
of its first
+        // non-null value (empty input has a struct schema, checked above). 
pyspark merges
+        // the types of all rows, which gives the same type when they agree, 
and cannot
+        // type a field that is null in every row.
         let schema_dtype = match spec {
             Spec::Struct(fields) => DataType::Struct { fields },
             _ => {
-                if rows.is_empty() {
-                    return Err(PyErr::new::<pyo3::exceptions::PyValueError, _>(
-                        "can not infer schema from empty dataset",
-                    ));
-                }
-                let fields: Vec<StructField> = rows[0]
-                    .fields()
-                    .iter()
-                    .enumerate()
-                    .map(|(i, name)| StructField {
+                let mut fields: Vec<StructField> = 
Vec::with_capacity(rows[0].len());
+                for (i, name) in rows[0].fields().iter().enumerate() {
+                    let Some(value) = rows
+                        .iter()
+                        .filter_map(|row| row.get(i))
+                        .find(|value| !matches!(value, Value::Null))
+                    else {
+                        return Err(crate::column::raise_pyspark(
+                            py,
+                            "PySparkValueError",
+                            "CANNOT_DETERMINE_TYPE",
+                            &[],
+                        ));
+                    };
+                    fields.push(StructField {
                         name: name.clone(),
-                        data_type: 
rows[0].get(i).map(value_to_datatype).unwrap_or(
-                            DataType::String {
-                                collation: "UTF8_BINARY".to_string(),
-                            },
-                        ),
+                        data_type: value_to_datatype(value),
                         nullable: true,
                         metadata: Default::default(),
-                    })
-                    .collect();
+                    });
+                }
                 DataType::Struct { fields }
             }
         };
@@ -554,7 +684,15 @@ impl PySparkSession {
         let df = py
             .detach(|| self.session.create_dataframe(rows, schema_dtype))
             .to_pyerr()?;
-        Ok(PyDataFrame::new(df))
+        // As pyspark, column names are applied with toDF. The local relation 
already carries
+        // them, so this only changes the plan, which the server resolves into 
a Project that
+        // is not local; it keeps isLocal() as in pyspark and is not redundant.
+        Ok(PyDataFrame::new(match to_df_names {
+            Some(names) if !names.is_empty() => {
+                df.to_df(names.iter().map(String::as_str).collect())
+            }
+            _ => df,
+        }))
     }
 
     /// Get the catalog API (`spark.catalog`).
diff --git a/crates/spark-connect/src/session.rs 
b/crates/spark-connect/src/session.rs
index 1493ae0..7a3cdc6 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -647,9 +647,9 @@ impl Default for SparkSessionBuilder {
     }
 }
 
-/// Construct the shared local-input width error for Rust and Python callers.
-#[doc(hidden)]
-pub fn row_width_mismatch(row_index: usize, actual: usize, expected: usize) -> 
SparkError {
+/// The error for a row of `create_dataframe` whose width differs from the 
schema's. The
+/// Python client checks widths itself, raising pyspark's 
`AXIS_LENGTH_MISMATCH`.
+fn row_width_mismatch(row_index: usize, actual: usize, expected: usize) -> 
SparkError {
     SparkError::value_msg(format!(
         "Row {row_index} has {actual} values, but the schema has {expected} 
fields"
     ))
diff --git a/dev/parity/known-differences.md b/dev/parity/known-differences.md
new file mode 100644
index 0000000..a07fada
--- /dev/null
+++ b/dev/parity/known-differences.md
@@ -0,0 +1,20 @@
+# Known behavior differences from PySpark Connect
+
+Behaviors of the Rust-backed Python client that differ from PySpark Connect 
4.3.0, kept
+here so that a version upgrade does not rediscover them. Each entry says 
whether the
+difference is intentional or tracked for a fix.
+
+## `createDataFrame`
+
+- **A pandas DataFrame with a StructType of another width** (intentional, 
SPARK-60902).
+  PySpark's pandas conversion zips the DataFrame's columns with the schema's 
fields. With
+  fewer fields than columns it silently drops the extra columns, and with more 
fields it
+  raises PyArrow's `ArrowInvalid`. This client raises `AXIS_LENGTH_MISMATCH` 
for both
+  rather than drop data.
+- **Type inference across rows** (tracked by SPARK-60908). PySpark infers each 
row's type
+  and merges them: other atomic types merge into a string type, whose values 
they become;
+  nested values null in every row raise `CANNOT_DETERMINE_TYPE`; other 
conflicts raise
+  `CANNOT_MERGE_TYPE` or `CANNOT_INFER_TYPE_FOR_FIELD`; and a `Decimal` value 
infers
+  `decimal(38,18)`. This client takes each top-level field's type from its 
first non-null
+  value, and a nested type from the first non-null element or map value.
+- **NumPy ndarray input** (tracked by SPARK-60772). Not accepted yet.
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 2f72056..c0b59b8 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -21,12 +21,13 @@ values here are those of the reference client; the tests 
pass with it as well.
 """
 
 import os
+from collections import namedtuple
 from datetime import date, datetime
 
 import pyarrow as pa
 import pytest
 from pyspark.errors import PySparkValueError
-from pyspark.sql import SparkSession
+from pyspark.sql import Row, SparkSession
 from pyspark.sql import functions as F
 from pyspark.sql.types import (
     ArrayType,
@@ -172,6 +173,91 @@ def test_create_dataframe_from_empty_arrow_table(session):
         assert error.value.getCondition() == "CANNOT_INFER_EMPTY_SCHEMA"
 
 
+_Pair = namedtuple("_Pair", ["x", "y"])
+
+
[email protected](
+    "data, schema, columns, rows",
+    [
+        ([(1, 2)], None, ["_1", "_2"], [(1, 2)]),
+        ([[1, 2]], ["a"], ["a", "_2"], [(1, 2)]),
+        ([(1, 2), (3, 4)], ["a"], ["a", "_2"], [(1, 2), (3, 4)]),
+        ([(1, 2)], [], ["_1", "_2"], [(1, 2)]),
+        ([(1, 2)], ("a", "b"), ["a", "b"], [(1, 2)]),
+        ([Row(x=1, y=2)], ["a", "b"], ["a", "b"], [(1, 2)]),
+        ([{"y": 2, "x": 1}], ["a", "b"], ["a", "b"], [(1, 2)]),
+        # A namedtuple is named by its fields, as a Row is.
+        ([_Pair(1, 2), _Pair(3, 4)], None, ["x", "y"], [(1, 2), (3, 4)]),
+        ([_Pair(1, 2)], ["a", "b"], ["a", "b"], [(1, 2)]),
+    ],
+)
+def test_create_dataframe_rows_column_names(session, data, schema, columns, 
rows):
+    df = session.createDataFrame(data, schema)
+    assert df.columns == columns
+    assert [tuple(r) for r in df.collect()] == rows
+    assert df.isLocal() is (schema is None)
+
+
[email protected](
+    "frame, schema, columns",
+    [
+        ({"a": [1, 2], "b": ["x", "y"]}, None, ["a", "b"]),
+        ({"a": [1, 2], "b": ["x", "y"]}, ["c"], ["c", "_2"]),
+        ({"a": [1, 2], "b": ["x", "y"]}, "a long, b string", ["a", "b"]),
+        ({0: [1, 2], 1: ["x", "y"]}, None, ["0", "1"]),
+    ],
+)
+def test_create_dataframe_pandas_column_names(session, frame, schema, columns):
+    pd = pytest.importorskip("pandas")
+    df = session.createDataFrame(pd.DataFrame(frame), schema)
+    assert df.columns == columns
+    assert [tuple(r) for r in df.collect()] == [(1, "x"), (2, "y")]
+    assert not df.isLocal()
+
+
+class _Point:
+    def __init__(self, b, a):
+        self.b = b
+        self.a = a
+
+
[email protected](
+    "data, schema, columns, rows",
+    [
+        # As pyspark, a dict row is read by name: missing keys are null, 
others ignored.
+        ([{"b": 1, "a": "x"}], "b long, a string", ["b", "a"], [(1, "x")]),
+        ([{"a": 1}], "a long, b string", ["a", "b"], [(1, None)]),
+        ([{"a": 1, "c": 3}], "a long", ["a"], [(1,)]),
+        # Without a schema, the fields are the rows' sorted keys, in the order 
first seen.
+        ([{"a": 1}, {"b": 2}], None, ["a", "b"], [(1, None), (None, 2)]),
+        ([{"b": 2, "a": 1}], None, ["a", "b"], [(1, 2)]),
+        ([{"b": 2, "a": 1}], ["x", "y"], ["x", "y"], [(1, 2)]),
+        ([{"a": 1}, {"a": 2, "b": 3}], ["x", "y"], ["x", "y"], [(1, None), (2, 
3)]),
+        # Other objects are read through their __dict__.
+        ([_Point(2, "x")], None, ["a", "b"], [("x", 2)]),
+        ([_Point(2, "x")], "a string, b long", ["a", "b"], [("x", 2)]),
+    ],
+)
+def test_create_dataframe_dict_rows_by_name(session, data, schema, columns, 
rows):
+    df = session.createDataFrame(data, schema)
+    assert df.columns == columns
+    assert [tuple(r) for r in df.collect()] == rows
+
+
[email protected](
+    "data, simple_string, rows",
+    [
+        ([(None, 1), (2, 3)], "struct<_1:bigint,_2:bigint>", [(None, 1), (2, 
3)]),
+        ([{"a": None}, {"a": 1}], "struct<a:bigint>", [(None,), (1,)]),
+    ],
+)
+def test_create_dataframe_infers_types_past_nulls(session, data, 
simple_string, rows):
+    # A field takes the type of its first non-null value.
+    df = session.createDataFrame(data)
+    assert df.schema.simpleString() == simple_string
+    assert [tuple(r) for r in df.collect()] == rows
+
+
 def test_collect_reads_a_result_across_stream_ends(session):
     # Takes several response streams when the server's senderMaxStreamSize is 
small.
     n = 300_000
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 5da4658..a6a35fd 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -8,6 +8,7 @@ that require a live server live in test_dropin_e2e.py and are 
gated on SPARK_REM
 
 import pickle
 import time
+from collections import namedtuple
 from concurrent.futures import ThreadPoolExecutor
 from threading import Barrier
 
@@ -3052,13 +3053,108 @@ for schema in schemas:
     try:
         spark.createDataFrame([(1, 2), (3,)], schema=schema)
     except PySparkValueError as error:
+        # The same error as pyspark, for every kind of schema.
         assert type(error) is PySparkValueError, type(error)
-        assert str(error) == "Row 1 has 1 values, but the schema has 2 
fields", error
+        assert error.getCondition() == "AXIS_LENGTH_MISMATCH", error
+        assert error.getMessageParameters() == {
+            "expected_length": "2", "actual_length": "1"}, error
     else:
         raise AssertionError("expected a width validation error")
 """], check=True)
 
 
+def _local_create_dataframe(*args):
+    # The local relation and its toDF are built without a server.
+    return _offline_session().createDataFrame(*args)
+
+
[email protected]("data,schema,expected", [
+    # Column names are applied with toDF, as in pyspark, so the result is not 
local.
+    ([(1, 2)], ["a", "b"], False),
+    ([(1, 2)], ("a", "b"), False),
+    ([(1, 2)], ["a"], False),
+    ([(1, 2)], [], False),
+    ([T.Row(x=1, y=2)], ["a", "b"], False),
+    ([{"x": 1, "y": 2}], ["a", "b"], False),
+    # Without names to apply, the local relation itself is local.
+    ([(1, 2)], None, True),
+    ([T.Row(x=1, y=2)], None, True),
+    ([(1, 2)], "a long, b long", True),
+    ([1, 2], T.LongType(), True),
+    ([], "a long", True),
+])
+def test_create_dataframe_rows_is_local_as_pyspark(data, schema, expected):
+    assert _local_create_dataframe(data, schema).isLocal() is expected
+
+
[email protected]("schema", [None, ["c", "d"], ["c"], "a long, b 
string",
+                                    T.StructType([T.StructField("a", 
T.LongType()),
+                                                  T.StructField("b", 
T.StringType())])])
+def test_create_dataframe_pandas_is_not_local_as_pyspark(schema):
+    pd = pytest.importorskip("pandas")
+    # pyspark applies the columns, or the field names of the schema, with toDF.
+    df = _local_create_dataframe(pd.DataFrame({"a": [1, 2], "b": ["x", "y"]}), 
schema)
+    assert not df.isLocal()
+
+
+def test_create_dataframe_empty_pandas_is_local_as_pyspark():
+    pd = pytest.importorskip("pandas")
+    # Empty input returns the relation of its schema, without toDF.
+    assert _local_create_dataframe(pd.DataFrame({"a": [], "b": []}), "a long, 
b string").isLocal()
+
+
+_Pair = namedtuple("_Pair", ["x", "y"])
+
+
[email protected]("data,schema,expected,actual", [
+    ([(1,)], ["a", "b"], "2", "1"),
+    ([(1,), (2,)], ["a", "b"], "2", "1"),
+    ([(1, 2), (3, 4, 5)], ["a"], "3", "2"),
+    ([T.Row(x=1, y=2)], ["a"], "1", "2"),
+    # A namedtuple, as a Row, does not extend the names.
+    ([_Pair(1, 2)], ["a"], "1", "2"),
+])
+def test_create_dataframe_name_width_errors_as_pyspark(data, schema, expected, 
actual):
+    from pyspark.errors import PySparkValueError
+
+    with pytest.raises(PySparkValueError) as error:
+        _local_create_dataframe(data, schema)
+    assert error.value.getCondition() == "AXIS_LENGTH_MISMATCH"
+    assert error.value.getMessageParameters() == {
+        "expected_length": expected, "actual_length": actual}
+
+
+def test_create_dataframe_pandas_errors_as_pyspark():
+    pd = pytest.importorskip("pandas")
+    from pyspark.errors import PySparkTypeError, PySparkValueError
+
+    pdf = pd.DataFrame({"a": [1, 2], "b": ["x", "y"]})
+    # pyspark reports the parsed type, also for an atomic DDL string.
+    for schema, data_type in [(T.LongType(), "LongType()"), ("int", 
"IntegerType()")]:
+        with pytest.raises(PySparkTypeError) as error:
+            _local_create_dataframe(pdf, schema)
+        assert error.value.getCondition() == "UNSUPPORTED_DATA_TYPE_FOR_ARROW"
+        assert error.value.getMessageParameters() == {"data_type": data_type}
+    with pytest.raises(PySparkValueError) as error:
+        _local_create_dataframe(pdf, ["c", "d", "e"])
+    assert error.value.getCondition() == "AXIS_LENGTH_MISMATCH"
+    for data, schema in [(pd.DataFrame(index=[0, 1]), None), ([], None), ([], 
["a"]),
+                         (pdf.iloc[:0], None), (pdf.iloc[:0], ["a", "b"])]:
+        with pytest.raises(PySparkValueError) as error:
+            _local_create_dataframe(data, schema)
+        assert error.value.getCondition() == "CANNOT_INFER_EMPTY_SCHEMA"
+
+
[email protected]("data", [[(None,)], [(1, None)], [{"a": None}], 
[{"a": 1}, {"b": None}]])
+def test_create_dataframe_cannot_type_an_all_null_field_as_pyspark(data):
+    from pyspark.errors import PySparkValueError
+
+    # A field without a non-null value in any row has no type to infer.
+    with pytest.raises(PySparkValueError) as error:
+        _local_create_dataframe(data, None)
+    assert error.value.getCondition() == "CANNOT_DETERMINE_TYPE"
+
+
 # --------------------------------------------------------------------------- 
schema cache
 
 


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to