This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 42c5b10  [SPARK-60908][CONNECT] Infer and convert local data in 
createDataFrame as PySpark does
42c5b10 is described below

commit 42c5b101b3ceb89f054557323b6d836c5d7e351b
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 08:39:24 2026 +0900

    [SPARK-60908][CONNECT] Infer and convert local data in createDataFrame as 
PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-60908](https://issues.apache.org/jira/browse/SPARK-60908): 
`createDataFrame` of local rows now infers and converts its data as PySpark 
Connect does.
    
    - A new module, `crates/pyspark-rs/src/infer.rs`, ports PySpark 4.3.0's 
`_infer_type`, `_infer_schema`, `_merge_type` and `_has_nulltype`, and the 
converters of `LocalDataToArrowConversion`, keeping their order of checks and 
their errors. The schema is the merge of every row's type. A string type 
absorbs another atomic type, whose values then become strings (`True` as 
`'true'`), a null type takes the other type, and types of different classes 
raise `CANNOT_MERGE_TYPE`, or `CANNOT_INFE [...]
    - The path of local rows follows PySpark Connect's. Dict rows are sorted by 
their keys, a row of another value, such as `1`, is a row of that value, and 
every row is taken apart and then converted column by column. A list of names 
names the inferred fields, and inference extends it with `_2`, `_3`, ...; the 
number of names must then equal the number of fields.
    - The four configurations that inference depends on 
(`spark.sql.pyspark.inferNestedDictAsStruct.enabled`, the two 
`spark.sql.pyspark.legacy.infer...` ones and `spark.sql.timestampType`) are 
read in one request, and only when a value depends on them: a dict, a list of 
more than one element, or a naive datetime. Other data is still inferred 
without a request. PySpark's inference of list elements, which passes 
`prefer_timestamp_ntz` for `infer_map_from_first_pair` and leaves the former f 
[...]
    - As in PySpark Connect:
      - A schema that is neither a list of names nor a struct or atomic type 
raises `NOT_EXPECTED_TYPE`.
      - A DataFrame as data raises `INVALID_TYPE`.
      - `samplingRatio` and `verifySchema` warn that they are ignored.
      - Data that is not a list, such as a tuple or a generator, is accepted.
    - A TIMESTAMP value is the exact instant of its datetime, and a 
TIMESTAMP_NTZ value its wall-clock time.
    - In the Rust core:
      - `RuntimeConf::get_many` reads several keys in one request.
      - A struct value whose fields are the declared ones, in order, is encoded 
by position, so that fields of the same name keep their own values.
      - A batch without columns keeps its number of rows, and a struct without 
fields its length, as PySpark's `createDataFrame([()])` gives a row without 
columns.
    - A pandas DataFrame is converted as before.
    
    ### Why are the changes needed?
    
    Each field took the type of its first non-null value:
    
    - `createDataFrame([(1,), ("a",)])` and `[([1, "a"],)]` raised an 
unclassified error rather than giving a string column of `'1'` and `'a'`, and 
`array<string>`.
    - `[(1,), (2.5,)]` and `[([1, 2.5],)]` raised an unclassified error rather 
than `CANNOT_MERGE_TYPE` and `CANNOT_INFER_TYPE_FOR_FIELD`.
    - `[([None],)]` and `[({"k": None},)]` inferred `array<string>` and 
`map<string,string>` rather than raising `CANNOT_DETERMINE_TYPE`, and 
`[([None],), ([1],)]` raised an error rather than inferring `array<bigint>`.
    - A `Decimal` inferred `decimal(38,s)` with the scale of its value, rather 
than `decimal(38,18)`.
    - Tuples, namedtuples and objects in a field, `array.array`, the inference 
configurations, and the errors for an unsupported value or row were not as in 
PySpark.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, `createDataFrame` of local rows infers, converts and fails as in 
PySpark Connect 4.3.0.
    
    The tests that #127 (SPARK-60684) added now check PySpark's results:
    
    - A string and an integer in two rows share a string column rather than 
raising.
    - A lone surrogate raises `UnicodeEncodeError`, as PyArrow does.
    - An instance of a `str` subclass, which PySpark takes by its `__dict__`, 
is an empty struct.
    
    Some values still cannot be sent, and raise `TypeError` as before: 
`datetime.time`, `timedelta` (whose input 
[SPARK-60659](https://issues.apache.org/jira/browse/SPARK-60659) adds in #116, 
which can now convert it in `infer::convert`), a value of a user-defined type, 
`VariantVal`, and `Geometry` and `Geography`. Their types are inferred as in 
PySpark.
    
    ### How was this patch tested?
    
    - Against a local Spark 4.3.0 server, a probe of forty inference cases 
gives output identical to official PySpark 4.3.0, apart from the `time` and 
`timedelta` values above. The probes of #149 (thirty-one `createDataFrame` 
cases of names, schemas and rows, nine of dict rows, two of null-first rows and 
seven of namedtuples) and the sixteen map cases of #161 are still identical.
    - New offline tests check the inference errors, the schema and data 
arguments, and the warnings. Their cases also agree with official PySpark 4.3.0 
against the server.
    - New live tests check the merged types and values, the merge errors, and 
inference under each of the four configurations. They also pass with official 
PySpark 4.3.0.
    - New unit tests check that a struct value of the declared fields is 
encoded by position, and that a struct without fields keeps its length, which 
encoding one used to panic on.
    - The lib tests and every integration test, e2e included, against the local 
server (641 passed), the build of all tests with `wasm-udf`, the offline Python 
suites (611 passed, 8 live cases skipped), and the live drop-in and Arrow 
suites (55 passed) and live cases (10 passed). Clippy reports no diagnostics in 
the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #167 from HyukjinKwon/SPARK-60908-infer-merge.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/errors.rs     |    8 +
 crates/pyspark-rs/src/infer.rs      | 1098 +++++++++++++++++++++++++++++++++++
 crates/pyspark-rs/src/lib.rs        |    1 +
 crates/pyspark-rs/src/row.rs        |    9 +
 crates/pyspark-rs/src/session.rs    |  643 ++++++++++----------
 crates/spark-connect/src/conf.rs    |    6 +
 crates/spark-connect/src/session.rs |   78 ++-
 python/tests/test_dropin_e2e.py     |   91 +++
 python/tests/test_dropin_offline.py |   96 ++-
 9 files changed, 1703 insertions(+), 327 deletions(-)

diff --git a/crates/pyspark-rs/src/errors.rs b/crates/pyspark-rs/src/errors.rs
index 1da4078..4ea7a0c 100644
--- a/crates/pyspark-rs/src/errors.rs
+++ b/crates/pyspark-rs/src/errors.rs
@@ -193,3 +193,11 @@ impl<T> ResultExt<T> for Result<T, SparkError> {
         self.map_err(spark_error_to_py_exception)
     }
 }
+
+/// For a boxed error, as a closure that runs without the GIL can return one, 
which keeps
+/// its result small.
+impl<T> ResultExt<T> for Result<T, Box<SparkError>> {
+    fn to_pyerr(self) -> PyResult<T> {
+        self.map_err(|e| spark_error_to_py_exception(*e))
+    }
+}
diff --git a/crates/pyspark-rs/src/infer.rs b/crates/pyspark-rs/src/infer.rs
new file mode 100644
index 0000000..4d3386b
--- /dev/null
+++ b/crates/pyspark-rs/src/infer.rs
@@ -0,0 +1,1098 @@
+//! Schema inference and conversion of local data for `createDataFrame`, as 
PySpark 4.3.0
+//! does them: `_infer_schema`, `_infer_type`, `_merge_type` and 
`_has_nulltype` of
+//! `pyspark.sql.types`, and `LocalDataToArrowConversion` of 
`pyspark.sql.conversion`, with
+//! which Spark Connect converts local data. The checks run in PySpark's order 
and raise its
+//! errors, so that the same data gives the same schema, values and errors.
+
+use std::cell::OnceCell;
+use std::collections::{HashMap, HashSet};
+
+use pyo3::exceptions::{PyAssertionError, PyTypeError, PyValueError};
+use pyo3::intern;
+use pyo3::prelude::*;
+use pyo3::sync::PyOnceLock;
+use pyo3::types::{
+    IntoPyDict, PyBool, PyByteArray, PyBytes, PyDict, PyFloat, PyInt, PyList, 
PyString, PyTuple,
+    PyType,
+};
+use spark_connect::conf::RuntimeConf;
+use spark_connect::row::{StructValue, Value};
+use spark_connect::types::{DataType, GeographyType, GeometryType, StructField};
+
+use crate::column::raise_pyspark;
+use crate::errors::ResultExt;
+use crate::row::PyRow;
+use crate::values::{PyGeography, PyGeometry, PyVariantVal};
+
+static DECIMAL: PyOnceLock<Py<PyType>> = PyOnceLock::new();
+static DATE: PyOnceLock<Py<PyType>> = PyOnceLock::new();
+static DATETIME: PyOnceLock<Py<PyType>> = PyOnceLock::new();
+static TIME: PyOnceLock<Py<PyType>> = PyOnceLock::new();
+static TIMEDELTA: PyOnceLock<Py<PyType>> = PyOnceLock::new();
+static ARRAY: PyOnceLock<Py<PyType>> = PyOnceLock::new();
+static SORTED: PyOnceLock<Py<PyAny>> = PyOnceLock::new();
+
+/// The session configurations that inference reads, as PySpark reads them for
+/// `createDataFrame`.
+struct Confs {
+    dict_as_struct: bool,
+    array_from_first_element: bool,
+    map_from_first_pair: bool,
+    prefer_timestamp_ntz: bool,
+}
+
+/// The configurations, read from the server, in one request, the first time 
that inference
+/// needs one. Only dicts, lists of more than one element and naive datetimes 
depend on them,
+/// so other data is inferred without a request, as without a server.
+pub(crate) struct LazyConfs {
+    conf: RuntimeConf,
+    confs: OnceCell<Confs>,
+}
+
+impl LazyConfs {
+    pub(crate) fn new(conf: RuntimeConf) -> Self {
+        LazyConfs {
+            conf,
+            confs: OnceCell::new(),
+        }
+    }
+
+    fn get(&self, py: Python<'_>) -> PyResult<&Confs> {
+        if let Some(confs) = self.confs.get() {
+            return Ok(confs);
+        }
+        let keys = [
+            "spark.sql.pyspark.inferNestedDictAsStruct.enabled",
+            "spark.sql.pyspark.legacy.inferArrayTypeFromFirstElement.enabled",
+            "spark.sql.pyspark.legacy.inferMapTypeFromFirstPair.enabled",
+            "spark.sql.timestampType",
+        ];
+        let conf = &self.conf;
+        let values = py
+            .detach(|| conf.get_many(&keys).map_err(Box::new))
+            .to_pyerr()?;
+        let is =
+            |i: usize, expected: &str| values.get(i).and_then(|v| 
v.as_deref()) == Some(expected);
+        Ok(self.confs.get_or_init(|| Confs {
+            dict_as_struct: is(0, "true"),
+            array_from_first_element: is(1, "true"),
+            map_from_first_pair: is(2, "true"),
+            prefer_timestamp_ntz: is(3, "TIMESTAMP_NTZ"),
+        }))
+    }
+}
+
+/// Where an option of inference takes its value from.
+#[derive(Clone, Copy)]
+enum Source {
+    /// False, where PySpark leaves the option at its default.
+    Off,
+    /// `spark.sql.pyspark.legacy.inferMapTypeFromFirstPair.enabled`.
+    MapFromFirstPair,
+    /// `spark.sql.timestampType` being `TIMESTAMP_NTZ`.
+    PreferTimestampNtz,
+}
+
+/// The options that inference passes down, as PySpark does. 
`infer_dict_as_struct` and
+/// `infer_array_from_first_element` always come from the configurations, and 
the other two
+/// shift where PySpark passes them on.
+#[derive(Clone, Copy)]
+pub(crate) struct Options<'c> {
+    confs: &'c LazyConfs,
+    map_from_first_pair: Source,
+    prefer_timestamp_ntz: Source,
+}
+
+impl<'c> Options<'c> {
+    pub(crate) fn new(confs: &'c LazyConfs) -> Self {
+        Options {
+            confs,
+            map_from_first_pair: Source::MapFromFirstPair,
+            prefer_timestamp_ntz: Source::PreferTimestampNtz,
+        }
+    }
+
+    /// For the elements of a list. PySpark passes the options to 
`_infer_type` by position
+    /// without `infer_map_from_first_pair`, so that it takes the value of
+    /// `prefer_timestamp_ntz`, which is left false.
+    fn for_elements(self) -> Self {
+        Options {
+            map_from_first_pair: self.prefer_timestamp_ntz,
+            prefer_timestamp_ntz: Source::Off,
+            ..self
+        }
+    }
+
+    /// For a row in a value, such as a tuple, a Row or an object, which 
PySpark infers
+    /// without `infer_map_from_first_pair`.
+    fn for_nested_rows(self) -> Self {
+        Options {
+            map_from_first_pair: Source::Off,
+            ..self
+        }
+    }
+
+    fn value(&self, py: Python<'_>, source: Source) -> PyResult<bool> {
+        Ok(match source {
+            Source::Off => false,
+            Source::MapFromFirstPair => 
self.confs.get(py)?.map_from_first_pair,
+            Source::PreferTimestampNtz => 
self.confs.get(py)?.prefer_timestamp_ntz,
+        })
+    }
+}
+
+pub(crate) fn string_type() -> DataType {
+    DataType::String {
+        collation: "UTF8_BINARY".to_string(),
+    }
+}
+
+fn nullable_field(name: String, data_type: DataType) -> StructField {
+    StructField {
+        name,
+        data_type,
+        nullable: true,
+        metadata: Default::default(),
+    }
+}
+
+fn map_type(key_type: DataType, value_type: DataType) -> DataType {
+    DataType::Map {
+        key_type: Box::new(key_type),
+        value_type: Box::new(value_type),
+        value_contains_null: true,
+    }
+}
+
+pub(crate) fn type_name(obj: &Bound<'_, PyAny>) -> PyResult<String> {
+    Ok(obj.get_type().name()?.to_string())
+}
+
+/// A nullable field of `name`, which PySpark's `StructField` asserts is a 
string.
+fn field_of(name: &Bound<'_, PyAny>, data_type: DataType) -> 
PyResult<StructField> {
+    match name.cast::<PyString>() {
+        Ok(name) => Ok(nullable_field(name.to_cow()?.into_owned(), data_type)),
+        Err(_) => Err(PyAssertionError::new_err(format!(
+            "field name {} should be a string",
+            name.str()?
+        ))),
+    }
+}
+
+/// The type of `obj`, as PySpark's `_infer_type`.
+pub(crate) fn infer_type(obj: &Bound<'_, PyAny>, o: Options<'_>) -> 
PyResult<DataType> {
+    let py = obj.py();
+    if obj.is_none() {
+        return Ok(DataType::Null);
+    }
+    // The types that PySpark maps by `type(obj)`, before it looks for 
`__UDT__`, which none
+    // of them can have: neither these classes nor their instances take new 
attributes.
+    if obj.is_exact_instance_of::<PyBool>() {
+        return Ok(DataType::Boolean);
+    }
+    if obj.is_exact_instance_of::<PyInt>() {
+        return Ok(DataType::Long);
+    }
+    if obj.is_exact_instance_of::<PyFloat>() {
+        return Ok(DataType::Double);
+    }
+    if obj.is_exact_instance_of::<PyString>() {
+        return Ok(string_type());
+    }
+    if obj.is_exact_instance_of::<PyBytes>() || 
obj.is_exact_instance_of::<PyByteArray>() {
+        return Ok(DataType::Binary);
+    }
+    let ty = obj.get_type();
+    if ty.is(DECIMAL.import(py, "decimal", "Decimal")?) {
+        // As PySpark: the precision and scale of a value may differ from row 
to row.
+        return Ok(DataType::Decimal {
+            precision: 38,
+            scale: 18,
+        });
+    }
+    if ty.is(DATETIME.import(py, "datetime", "datetime")?) {
+        let naive = obj.getattr(intern!(py, "tzinfo"))?.is_none();
+        return Ok(if naive && o.value(py, o.prefer_timestamp_ntz)? {
+            DataType::TimestampNtz
+        } else {
+            DataType::Timestamp
+        });
+    }
+    if ty.is(DATE.import(py, "datetime", "date")?) {
+        return Ok(DataType::Date);
+    }
+    if ty.is(TIME.import(py, "datetime", "time")?) {
+        return Ok(DataType::Time { precision: 6 });
+    }
+    if ty.is(TIMEDELTA.import(py, "datetime", "timedelta")?) {
+        // `DayTimeIntervalType()`, from days to seconds.
+        return Ok(DataType::DayTimeInterval {
+            start_field: 0,
+            end_field: 3,
+        });
+    }
+    if obj.is_exact_instance_of::<PyGeometry>() {
+        let srid = obj.call_method0(intern!(py, "getSrid"))?.extract()?;
+        return Ok(DataType::Geometry { srid });
+    }
+    if obj.is_exact_instance_of::<PyGeography>() {
+        let srid = obj.call_method0(intern!(py, "getSrid"))?.extract()?;
+        return Ok(DataType::Geography { srid });
+    }
+    if obj.hasattr(intern!(py, "__UDT__"))? {
+        return crate::types::py_to_data_type(&obj.getattr(intern!(py, 
"__UDT__"))?);
+    }
+    if let Ok(dict) = obj.cast::<PyDict>() {
+        return infer_dict(dict, o);
+    }
+    if let Ok(list) = obj.cast::<PyList>() {
+        return infer_list(list, o);
+    }
+    if obj.is_instance(ARRAY.import(py, "array", "array")?.as_any())? {
+        let typecode: String = obj.getattr(intern!(py, 
"typecode"))?.extract()?;
+        return match array_element_type(&typecode) {
+            Some(element_type) => Ok(DataType::Array {
+                element_type: Box::new(element_type),
+                contains_null: false,
+            }),
+            None => Err(raise_pyspark(
+                py,
+                "PySparkTypeError",
+                "UNSUPPORTED_DATA_TYPE",
+                &[("data_type", &format!("array({typecode})"))],
+            )),
+        };
+    }
+    if obj.is_instance_of::<PyVariantVal>() {
+        return Ok(DataType::Variant);
+    }
+    match infer_schema(obj, None, o.for_nested_rows()) {
+        Err(e) if e.is_instance_of::<PyTypeError>(py) => Err(raise_pyspark(
+            py,
+            "PySparkTypeError",
+            "UNSUPPORTED_DATA_TYPE",
+            &[("data_type", &type_name(obj)?)],
+        )),
+        result => result,
+    }
+}
+
+/// A dict: a struct of its keys with non-null values, the map type of its 
first pair with
+/// neither null, or, by default, the merged types of its keys and of its 
values.
+fn infer_dict(dict: &Bound<'_, PyDict>, o: Options<'_>) -> PyResult<DataType> {
+    let py = dict.py();
+    if o.confs.get(py)?.dict_as_struct {
+        let mut fields = Vec::new();
+        for (key, value) in dict.iter() {
+            if !key.is_none() && !value.is_none() {
+                let data_type = infer_type(&value, o)?;
+                fields.push(field_of(&key, data_type)?);
+            }
+        }
+        return Ok(DataType::Struct { fields });
+    }
+    if o.value(py, o.map_from_first_pair)? {
+        for (key, value) in dict.iter() {
+            if !key.is_none() && !value.is_none() {
+                return Ok(map_type(infer_type(&key, o)?, infer_type(&value, 
o)?));
+            }
+        }
+        return Ok(map_type(DataType::Null, DataType::Null));
+    }
+    let (mut key_type, mut value_type) = (DataType::Null, DataType::Null);
+    for (key, value) in dict.iter() {
+        if !key.is_none() {
+            key_type = merge_type(py, key_type, infer_type(&key, o)?)?;
+        }
+        if !value.is_none() {
+            value_type = merge_type(py, value_type, infer_type(&value, o)?)?;
+        }
+    }
+    Ok(map_type(key_type, value_type))
+}
+
+/// A list: an array of the merged types of its elements, or of its first 
element's type.
+fn infer_list(list: &Bound<'_, PyList>, o: Options<'_>) -> PyResult<DataType> {
+    let py = list.py();
+    let elements = o.for_elements();
+    let element_type = match list.len() {
+        0 => DataType::Null,
+        // One element has its own type either way, without reading the 
configuration.
+        1 => infer_type(&list.get_item(0)?, elements)?,
+        _ if o.confs.get(py)?.array_from_first_element => 
infer_type(&list.get_item(0)?, elements)?,
+        _ => {
+            let mut merged = DataType::Null;
+            for (i, value) in list.iter().enumerate() {
+                let data_type = infer_type(&value, elements)?;
+                merged = if i == 0 {
+                    data_type
+                } else {
+                    merge_type(py, merged, data_type)?
+                };
+            }
+            merged
+        }
+    };
+    Ok(DataType::Array {
+        element_type: Box::new(element_type),
+        contains_null: true,
+    })
+}
+
+/// The element type of an `array.array` of `typecode`, as PySpark maps the 
typecodes by the
+/// sizes of their C types; the unsigned ones take one more bit, and none 
needs more than 64.
+fn array_element_type(typecode: &str) -> Option<DataType> {
+    use std::ffi::{c_int, c_long, c_schar, c_short, c_uchar, c_uint, c_ulong, 
c_ushort};
+    use std::mem::size_of;
+    let integral = |size: usize, unsigned: bool| match size * 8 + 
usize::from(unsigned) {
+        0..=8 => Some(DataType::Byte),
+        9..=16 => Some(DataType::Short),
+        17..=32 => Some(DataType::Integer),
+        33..=64 => Some(DataType::Long),
+        _ => None,
+    };
+    match typecode {
+        "f" => Some(DataType::Float),
+        "d" => Some(DataType::Double),
+        "u" => Some(string_type()),
+        "b" => integral(size_of::<c_schar>(), false),
+        "h" => integral(size_of::<c_short>(), false),
+        "i" => integral(size_of::<c_int>(), false),
+        "l" => integral(size_of::<c_long>(), false),
+        "B" => integral(size_of::<c_uchar>(), true),
+        "H" => integral(size_of::<c_ushort>(), true),
+        "I" => integral(size_of::<c_uint>(), true),
+        "L" => integral(size_of::<c_ulong>(), true),
+        _ => None,
+    }
+}
+
+type Items<'py> = Vec<(Bound<'py, PyAny>, Bound<'py, PyAny>)>;
+
+/// `sorted(mapping.items())`, as PySpark orders the fields of a dict or of an 
object.
+fn sorted_items<'py>(mapping: &Bound<'py, PyAny>) -> PyResult<Items<'py>> {
+    let py = mapping.py();
+    let sorted = SORTED.import(py, "builtins", "sorted")?;
+    let items = sorted.call1((mapping.call_method0(intern!(py, "items"))?,))?;
+    items
+        .try_iter()?
+        .map(|item| {
+            let item = item?;
+            Ok((item.get_item(0)?, item.get_item(1)?))
+        })
+        .collect()
+}
+
+/// The values of a tuple or list, named by `names` as `_infer_schema` zips 
them. Without
+/// names, they are `_1`, `_2`, and so on; a shorter list of names is extended 
so, in place.
+pub(crate) fn positional<'py>(
+    py: Python<'py>,
+    values: Vec<Bound<'py, PyAny>>,
+    names: Option<&mut Vec<String>>,
+) -> Items<'py> {
+    let defaults: Vec<String>;
+    let names: &[String] = match names {
+        Some(names) => {
+            let width = values.len();
+            if names.len() < width {
+                let start = names.len() + 1;
+                names.extend((start..=width).map(|i| format!("_{i}")));
+            }
+            names
+        }
+        None => {
+            defaults = (1..=values.len()).map(|i| format!("_{i}")).collect();
+            &defaults
+        }
+    };
+    names
+        .iter()
+        .map(|name| PyString::new(py, name).into_any())
+        .zip(values)
+        .collect()
+}
+
+/// The (name, value) pairs of a row, as `_infer_schema` takes them: a dict by 
its sorted
+/// keys, a Row or a namedtuple by its field names, another tuple or a list by 
`names`, and
+/// another object by the sorted names of its `__dict__`.
+fn row_items<'py>(
+    row: &Bound<'py, PyAny>,
+    names: Option<&mut Vec<String>>,
+) -> PyResult<Items<'py>> {
+    let py = row.py();
+    if row.is_instance_of::<PyDict>() {
+        return sorted_items(row);
+    }
+    if let Ok(pyrow) = row.cast::<PyRow>() {
+        let pyrow = pyrow.borrow();
+        let (fields, values) = pyrow.parts(py)?;
+        return Ok(match fields {
+            Some(fields) => fields
+                .iter()
+                .map(|f| PyString::new(py, f).into_any())
+                .zip(values.iter())
+                .collect(),
+            None => positional(py, values.iter().collect(), names),
+        });
+    }
+    if row.is_instance_of::<PyTuple>() || row.is_instance_of::<PyList>() {
+        let fields = if row.hasattr(intern!(py, "__fields__"))? {
+            Some(row.getattr(intern!(py, "__fields__"))?)
+        } else if row.hasattr(intern!(py, "_fields"))? {
+            Some(row.getattr(intern!(py, "_fields"))?)
+        } else {
+            None
+        };
+        let values = row.try_iter()?.collect::<PyResult<Vec<_>>>()?;
+        return match fields {
+            Some(fields) => Ok(fields
+                .try_iter()?
+                .collect::<PyResult<Vec<_>>>()?
+                .into_iter()
+                .zip(values)
+                .collect()),
+            None => Ok(positional(py, values, names)),
+        };
+    }
+    if row.hasattr(intern!(py, "__dict__"))? {
+        return sorted_items(&row.getattr(intern!(py, "__dict__"))?);
+    }
+    Err(raise_pyspark(
+        py,
+        "PySparkTypeError",
+        "CANNOT_INFER_SCHEMA_FOR_TYPE",
+        &[("data_type", &type_name(row)?)],
+    ))
+}
+
+/// The struct type of (name, value) pairs. A `TypeError` in inferring a 
value, as from a
+/// type that cannot be inferred or merged, becomes 
`CANNOT_INFER_TYPE_FOR_FIELD`.
+pub(crate) fn infer_fields(py: Python<'_>, items: Items<'_>, o: Options<'_>) 
-> PyResult<DataType> {
+    let mut fields = Vec::with_capacity(items.len());
+    for (name, value) in items {
+        let data_type = match infer_type(&value, o) {
+            Err(e) if e.is_instance_of::<PyTypeError>(py) => {
+                return Err(raise_pyspark(
+                    py,
+                    "PySparkTypeError",
+                    "CANNOT_INFER_TYPE_FOR_FIELD",
+                    &[("field_name", &name.str()?.to_cow()?)],
+                ));
+            }
+            result => result?,
+        };
+        fields.push(field_of(&name, data_type)?);
+    }
+    Ok(DataType::Struct { fields })
+}
+
+/// The struct type of a row, as PySpark's `_infer_schema`.
+pub(crate) fn infer_schema(
+    row: &Bound<'_, PyAny>,
+    names: Option<&mut Vec<String>>,
+    o: Options<'_>,
+) -> PyResult<DataType> {
+    let items = row_items(row, names)?;
+    infer_fields(row.py(), items, o)
+}
+
+/// Whether PySpark's class of `dt` is an `AtomicType`.
+pub(crate) fn is_atomic(dt: &DataType) -> bool {
+    !matches!(
+        dt,
+        DataType::Null
+            | DataType::CalendarInterval
+            | DataType::Array { .. }
+            | DataType::Map { .. }
+            | DataType::Struct { .. }
+            | DataType::Udt { .. }
+            | DataType::Unparsed { .. }
+    )
+}
+
+/// The name of PySpark's class of `dt`, by which its errors name a type.
+pub(crate) fn class_name(dt: &DataType) -> String {
+    let name = match dt {
+        DataType::Null => "NullType",
+        DataType::Boolean => "BooleanType",
+        DataType::Byte => "ByteType",
+        DataType::Short => "ShortType",
+        DataType::Integer => "IntegerType",
+        DataType::Long => "LongType",
+        DataType::Float => "FloatType",
+        DataType::Double => "DoubleType",
+        DataType::Decimal { .. } => "DecimalType",
+        DataType::String { .. } => "StringType",
+        DataType::Char { .. } => "CharType",
+        DataType::Varchar { .. } => "VarcharType",
+        DataType::Binary => "BinaryType",
+        DataType::Date => "DateType",
+        DataType::Timestamp => "TimestampType",
+        DataType::TimestampNtz => "TimestampNTZType",
+        DataType::Time { .. } => "TimeType",
+        DataType::TimestampNtzNanos { .. } => "TimestampNTZNanosType",
+        DataType::TimestampLtzNanos { .. } => "TimestampLTZNanosType",
+        DataType::CalendarInterval => "CalendarIntervalType",
+        DataType::YearMonthInterval { .. } => "YearMonthIntervalType",
+        DataType::DayTimeInterval { .. } => "DayTimeIntervalType",
+        DataType::Array { .. } => "ArrayType",
+        DataType::Map { .. } => "MapType",
+        DataType::Struct { .. } => "StructType",
+        DataType::Variant => "VariantType",
+        DataType::Geometry { .. } => "GeometryType",
+        DataType::Geography { .. } => "GeographyType",
+        DataType::Unparsed { .. } => "UnparsedDataType",
+        DataType::Udt { python_class, .. } => {
+            // A user-defined type is its own class, which `pyClass` names in 
full.
+            let class = python_class.as_deref().unwrap_or("UserDefinedType");
+            return class.rsplit('.').next().unwrap_or(class).to_string();
+        }
+    };
+    name.to_string()
+}
+
+/// Whether two types are of the same PySpark class, as `_merge_type` compares 
them.
+fn same_class(a: &DataType, b: &DataType) -> bool {
+    match (a, b) {
+        (DataType::Udt { .. }, DataType::Udt { .. }) => class_name(a) == 
class_name(b),
+        _ => std::mem::discriminant(a) == std::mem::discriminant(b),
+    }
+}
+
+/// The merge of two inferred types, as PySpark's `_merge_type`: a null type 
takes the
+/// other, a string absorbs another atomic type, and structs, arrays and maps 
merge by
+/// their parts. Other types must be of the same class, and the first one is 
taken.
+pub(crate) fn merge_type(py: Python<'_>, a: DataType, b: DataType) -> 
PyResult<DataType> {
+    use DataType as D;
+    match (&a, &b) {
+        (D::Null, _) => return Ok(b),
+        (_, D::Null) => return Ok(a),
+        (D::Timestamp, D::TimestampNtz) => return Ok(a),
+        (D::TimestampNtz, D::Timestamp) => return Ok(b),
+        (D::Geometry { srid: x }, D::Geometry { srid: y }) if x != y => {
+            return Ok(D::Geometry {
+                srid: GeometryType::MIXED_SRID,
+            });
+        }
+        (D::Geography { srid: x }, D::Geography { srid: y }) if x != y => {
+            return Ok(D::Geography {
+                srid: GeographyType::MIXED_SRID,
+            });
+        }
+        _ => {}
+    }
+    if is_atomic(&a) && matches!(b, D::String { .. }) {
+        return Ok(b);
+    }
+    if matches!(a, D::String { .. }) && is_atomic(&b) {
+        return Ok(a);
+    }
+    if !same_class(&a, &b) {
+        return Err(raise_pyspark(
+            py,
+            "PySparkTypeError",
+            "CANNOT_MERGE_TYPE",
+            &[
+                ("data_type1", &class_name(&a)),
+                ("data_type2", &class_name(&b)),
+            ],
+        ));
+    }
+    Ok(match (a, b) {
+        (D::Struct { fields: a }, D::Struct { fields: b }) => {
+            // `dict((f.name, f.dataType) for f in b.fields)`: each name at 
its first place,
+            // with the type of its last field.
+            let mut index: HashMap<String, usize> = 
HashMap::with_capacity(b.len());
+            let mut others: Vec<(String, DataType)> = 
Vec::with_capacity(b.len());
+            for f in b {
+                match index.get(&f.name) {
+                    Some(&i) => others[i].1 = f.data_type,
+                    None => {
+                        index.insert(f.name.clone(), others.len());
+                        others.push((f.name, f.data_type));
+                    }
+                }
+            }
+            let mut names: HashSet<String> = HashSet::with_capacity(a.len());
+            let mut fields = Vec::with_capacity(a.len() + others.len());
+            for f in a {
+                let other = match index.get(&f.name) {
+                    Some(&i) => others[i].1.clone(),
+                    None => D::Null,
+                };
+                names.insert(f.name.clone());
+                fields.push(nullable_field(f.name, merge_type(py, f.data_type, 
other)?));
+            }
+            for (name, data_type) in others {
+                if !names.contains(&name) {
+                    fields.push(nullable_field(name, data_type));
+                }
+            }
+            D::Struct { fields }
+        }
+        (
+            D::Array {
+                element_type: x, ..
+            },
+            D::Array {
+                element_type: y, ..
+            },
+        ) => D::Array {
+            element_type: Box::new(merge_type(py, *x, *y)?),
+            contains_null: true,
+        },
+        (
+            D::Map {
+                key_type: ka,
+                value_type: va,
+                ..
+            },
+            D::Map {
+                key_type: kb,
+                value_type: vb,
+                ..
+            },
+        ) => map_type(merge_type(py, *ka, *kb)?, merge_type(py, *va, *vb)?),
+        (a, _) => a,
+    })
+}
+
+/// Whether `dt` has a null type in it, which PySpark cannot infer data of.
+pub(crate) fn has_null_type(dt: &DataType) -> bool {
+    match dt {
+        DataType::Struct { fields } => fields.iter().any(|f| 
has_null_type(&f.data_type)),
+        DataType::Array { element_type, .. } => has_null_type(element_type),
+        DataType::Map {
+            key_type,
+            value_type,
+            ..
+        } => has_null_type(key_type) || has_null_type(value_type),
+        other => matches!(other, DataType::Null),
+    }
+}
+
+/// A PySpark error of `class` with a message alone, as PySpark raises some of 
them.
+fn pyspark_message(py: Python<'_>, class: &str, message: &str) -> PyErr {
+    let build = || -> PyResult<PyErr> {
+        let cls = py.import("pyspark.errors")?.getattr(class)?;
+        Ok(PyErr::from_value(cls.call1((message,))?))
+    };
+    build().unwrap_or_else(|e| e)
+}
+
+fn data_type_str(py: Python<'_>, dt: &DataType) -> PyResult<String> {
+    Ok(crate::types::data_type_to_py(py, dt)?
+        .bind(py)
+        .str()?
+        .to_cow()?
+        .into_owned())
+}
+
+fn must_not_be_none(py: Python<'_>, dt: &DataType) -> PyResult<PyErr> {
+    Ok(pyspark_message(
+        py,
+        "PySparkValueError",
+        &format!("input for {} must not be None", data_type_str(py, dt)?),
+    ))
+}
+
+pub(crate) fn axis_length_mismatch(py: Python<'_>, expected: usize, actual: 
usize) -> PyErr {
+    raise_pyspark(
+        py,
+        "PySparkValueError",
+        "AXIS_LENGTH_MISMATCH",
+        &[
+            ("expected_length", &expected.to_string()),
+            ("actual_length", &actual.to_string()),
+        ],
+    )
+}
+
+/// What PySpark's `assert` raises, with no message.
+fn assertion() -> PyErr {
+    PyAssertionError::new_err(())
+}
+
+/// The error of a value that this client cannot send in a column of its type 
yet.
+fn unsupported(obj: &Bound<'_, PyAny>) -> PyResult<PyErr> {
+    Ok(PyTypeError::new_err(format!(
+        "unsupported value type for createDataFrame: {}",
+        type_name(obj)?
+    )))
+}
+
+/// The error of a value that PyArrow cannot convert to the type it is named 
by.
+fn arrow_invalid(obj: &Bound<'_, PyAny>, arrow_type: &str) -> PyResult<PyErr> {
+    Ok(PyValueError::new_err(format!(
+        "Could not convert {} with type {}: tried to convert to {arrow_type}",
+        obj.repr()?,
+        type_name(obj)?
+    )))
+}
+
+/// Microseconds of a `timedelta`, exactly.
+fn delta_micros(delta: &Bound<'_, PyAny>) -> PyResult<i64> {
+    let py = delta.py();
+    let days: i64 = delta.getattr(intern!(py, "days"))?.extract()?;
+    let seconds: i64 = delta.getattr(intern!(py, "seconds"))?.extract()?;
+    let micros: i64 = delta.getattr(intern!(py, "microseconds"))?.extract()?;
+    Ok(days * 86_400_000_000 + seconds * 1_000_000 + micros)
+}
+
+/// Microseconds since the epoch of the instant of a datetime, which PySpark 
takes with
+/// `astimezone(timezone.utc)`: a naive datetime is in the local time zone.
+fn instant_micros(obj: &Bound<'_, PyAny>) -> PyResult<i64> {
+    let py = obj.py();
+    let datetime = py.import(intern!(py, "datetime"))?;
+    let utc = datetime
+        .getattr(intern!(py, "timezone"))?
+        .getattr(intern!(py, "utc"))?;
+    let aware = obj.call_method1(intern!(py, "astimezone"), (&utc,))?;
+    let epoch = DATETIME
+        .import(py, "datetime", "datetime")?
+        .call((1970, 1, 1), Some(&[("tzinfo", &utc)].into_py_dict(py)?))?;
+    delta_micros(&aware.sub(epoch)?)
+}
+
+/// Microseconds since 1970-01-01 00:00 of the wall-clock time of a naive 
datetime, as
+/// PyArrow takes a TIMESTAMP_NTZ value.
+fn wall_clock_micros(obj: &Bound<'_, PyAny>) -> PyResult<i64> {
+    let py = obj.py();
+    let epoch = DATETIME
+        .import(py, "datetime", "datetime")?
+        .call1((1970, 1, 1))?;
+    delta_micros(&obj.sub(epoch)?)
+}
+
+/// A field value of `dt` from `obj`, as PySpark's converters and then PyArrow 
convert it.
+pub(crate) fn convert(obj: &Bound<'_, PyAny>, dt: &DataType, nullable: bool) 
-> PyResult<Value> {
+    let py = obj.py();
+    if obj.is_none() {
+        // A null type takes None either way.
+        if !nullable && !matches!(dt, DataType::Null) {
+            return Err(must_not_be_none(py, dt)?);
+        }
+        return Ok(Value::Null);
+    }
+    match dt {
+        DataType::Null => Err(pyspark_message(
+            py,
+            "PySparkValueError",
+            &format!(
+                "input for {} must be None, but got {}",
+                data_type_str(py, dt)?,
+                obj.str()?
+            ),
+        )),
+        DataType::Struct { fields } => convert_struct(obj, fields),
+        DataType::Array {
+            element_type,
+            contains_null,
+        } => {
+            if !obj.is_instance_of::<PyList>()
+                && !obj.is_instance(ARRAY.import(py, "array", 
"array")?.as_any())?
+            {
+                return Err(assertion());
+            }
+            let mut items = Vec::with_capacity(obj.len()?);
+            for item in obj.try_iter()? {
+                items.push(convert(&item?, element_type, *contains_null)?);
+            }
+            Ok(Value::List(items))
+        }
+        DataType::Map {
+            key_type,
+            value_type,
+            value_contains_null,
+        } => {
+            let dict = obj.cast::<PyDict>().map_err(|_| assertion())?;
+            let mut entries = Vec::with_capacity(dict.len());
+            for (key, value) in dict.iter() {
+                entries.push((
+                    convert(&key, key_type, false)?,
+                    convert(&value, value_type, *value_contains_null)?,
+                ));
+            }
+            Ok(Value::Map(entries))
+        }
+        DataType::Binary => {
+            if let Ok(bytes) = obj.cast::<PyBytes>() {
+                Ok(Value::Binary(bytes.as_bytes().to_vec()))
+            } else if let Ok(bytes) = obj.cast::<PyByteArray>() {
+                Ok(Value::Binary(bytes.to_vec()))
+            } else {
+                Err(assertion())
+            }
+        }
+        DataType::Timestamp | DataType::TimestampLtzNanos { .. } => {
+            if !obj.is_instance(DATETIME.import(py, "datetime", 
"datetime")?.as_any())? {
+                return Err(assertion());
+            }
+            Ok(Value::Timestamp(instant_micros(obj)?))
+        }
+        DataType::TimestampNtz | DataType::TimestampNtzNanos { .. } => {
+            if !obj.is_instance(DATETIME.import(py, "datetime", 
"datetime")?.as_any())?
+                || !obj.getattr(intern!(py, "tzinfo"))?.is_none()
+            {
+                return Err(assertion());
+            }
+            Ok(Value::Timestamp(wall_clock_micros(obj)?))
+        }
+        DataType::Decimal { precision, scale } => {
+            let decimal = DECIMAL.import(py, "decimal", "Decimal")?;
+            if !obj.is_instance(decimal.as_any())? {
+                return Err(assertion());
+            }
+            if obj.call_method0(intern!(py, "is_nan"))?.is_truthy()? {
+                if !nullable {
+                    return Err(must_not_be_none(py, dt)?);
+                }
+                return Ok(Value::Null);
+            }
+            // `value.quantize(Decimal("1E-<scale>"), 
context=Context(prec=<precision>,
+            // rounding=ROUND_HALF_EVEN))`, written without an exponent.
+            let module = py.import(intern!(py, "decimal"))?;
+            let exp = decimal.call1((format!("1E-{scale}"),))?;
+            let context = module.getattr(intern!(py, "Context"))?.call(
+                (),
+                Some(
+                    &[
+                        ("prec", precision.into_pyobject(py)?.into_any()),
+                        ("rounding", module.getattr(intern!(py, 
"ROUND_HALF_EVEN"))?),
+                    ]
+                    .into_py_dict(py)?,
+                ),
+            )?;
+            let quantized = obj.call_method(
+                intern!(py, "quantize"),
+                (exp,),
+                Some(&[("context", context)].into_py_dict(py)?),
+            )?;
+            let text =
+                PyString::new(py, "{:f}").call_method1(intern!(py, "format"), 
(quantized,))?;
+            Ok(Value::Decimal {
+                value: text.extract()?,
+                precision: Some(*precision),
+                scale: Some(*scale),
+            })
+        }
+        DataType::String { .. } => {
+            let text = if obj.is_exact_instance_of::<PyString>() {
+                obj.cast::<PyString>()?.to_cow()?.into_owned()
+            } else if obj.is_instance_of::<PyBool>() {
+                // As PySpark Classic, which converts a bool to a string in 
the JVM.
+                (if obj.is_truthy()? { "true" } else { "false" }).to_string()
+            } else {
+                obj.str()?.to_cow()?.into_owned()
+            };
+            Ok(Value::String(text))
+        }
+        DataType::Char { .. } | DataType::Varchar { .. } => {
+            if let Ok(text) = obj.cast::<PyString>() {
+                Ok(Value::String(text.to_cow()?.into_owned()))
+            } else if let Ok(bytes) = obj.cast::<PyBytes>() {
+                Ok(Value::String(
+                    String::from_utf8(bytes.as_bytes().to_vec())
+                        .map_err(|e| PyValueError::new_err(e.to_string()))?,
+                ))
+            } else {
+                Err(PyTypeError::new_err(format!(
+                    "Expected bytes, got a '{}' object",
+                    type_name(obj)?
+                )))
+            }
+        }
+        DataType::Boolean => {
+            if obj.is_instance_of::<PyBool>() {
+                Ok(Value::Bool(obj.is_truthy()?))
+            } else {
+                Err(arrow_invalid(obj, "boolean")?)
+            }
+        }
+        DataType::Byte | DataType::Short | DataType::Integer | DataType::Long 
=> {
+            convert_integral(obj, dt)
+        }
+        DataType::Float | DataType::Double => {
+            // PyArrow takes a float, an int or a bool.
+            if obj.is_instance_of::<PyFloat>() || 
obj.is_instance_of::<PyInt>() {
+                let value: f64 = obj.extract()?;
+                Ok(match dt {
+                    DataType::Float => Value::Float(value as f32),
+                    _ => Value::Double(value),
+                })
+            } else {
+                let arrow_type = if matches!(dt, DataType::Float) {
+                    "float"
+                } else {
+                    "double"
+                };
+                Err(arrow_invalid(obj, arrow_type)?)
+            }
+        }
+        DataType::Date => {
+            // PyArrow takes a date, or the date of a datetime.
+            if !obj.is_instance(DATE.import(py, "datetime", 
"date")?.as_any())? {
+                return Err(arrow_invalid(obj, "date32")?);
+            }
+            let ordinal: i64 = obj.call_method0(intern!(py, 
"toordinal"))?.extract()?;
+            Ok(Value::Date((ordinal - 719_163) as i32))
+        }
+        _ => Err(unsupported(obj)?),
+    }
+}
+
+/// An integral value, as PyArrow converts one: an int in range, or a float 
truncated.
+fn convert_integral(obj: &Bound<'_, PyAny>, dt: &DataType) -> PyResult<Value> {
+    let (bits, arrow_type) = match dt {
+        DataType::Byte => (8, "int8"),
+        DataType::Short => (16, "int16"),
+        DataType::Integer => (32, "int32"),
+        _ => (64, "int64"),
+    };
+    if obj.is_instance_of::<PyBool>() {
+        return Err(PyTypeError::new_err("Expected integer, got bool"));
+    }
+    let value: i64 = if obj.is_instance_of::<PyInt>() {
+        obj.extract()?
+    } else if obj.is_instance_of::<PyFloat>() {
+        let value: f64 = obj.extract()?;
+        // The range of i64 is [-2^63, 2^63), both bounds exact as f64.
+        if !(i64::MIN as f64..i64::MAX as f64).contains(&value.trunc()) {
+            return Err(arrow_invalid(obj, arrow_type)?);
+        }
+        value.trunc() as i64
+    } else {
+        return Err(arrow_invalid(obj, arrow_type)?);
+    };
+    if bits < 64 && !(-(1i64 << (bits - 1))..(1i64 << (bits - 
1))).contains(&value) {
+        return Err(PyValueError::new_err(format!(
+            "Value {value} too large to fit in C integer type"
+        )));
+    }
+    Ok(match dt {
+        DataType::Byte => Value::Byte(value as i8),
+        DataType::Short => Value::Short(value as i16),
+        DataType::Integer => Value::Integer(value as i32),
+        _ => Value::Long(value),
+    })
+}
+
+/// The values of `fields` from a mapping, by their names, as 
`mapping.get(name)`.
+fn values_by_name<'py>(
+    mapping: &Bound<'py, PyAny>,
+    fields: &[StructField],
+) -> PyResult<Vec<Bound<'py, PyAny>>> {
+    let py = mapping.py();
+    if let Ok(dict) = mapping.cast::<PyDict>() {
+        return fields
+            .iter()
+            .map(|f| {
+                Ok(dict
+                    .get_item(&f.name)?
+                    .unwrap_or_else(|| py.None().into_bound(py)))
+            })
+            .collect();
+    }
+    fields
+        .iter()
+        .map(|f| mapping.call_method1(intern!(py, "get"), (&f.name,)))
+        .collect()
+}
+
+/// The values of a tuple or a Row, which must be as many as `fields`.
+fn values_by_position<'py>(
+    py: Python<'py>,
+    values: Vec<Bound<'py, PyAny>>,
+    fields: &[StructField],
+) -> PyResult<Vec<Bound<'py, PyAny>>> {
+    if values.len() == fields.len() {
+        Ok(values)
+    } else {
+        Err(axis_length_mismatch(py, fields.len(), values.len()))
+    }
+}
+
+/// A struct value, as PySpark converts one: a tuple or a Row by position, a 
dict by name,
+/// and another object by the names of its `__dict__`.
+fn convert_struct(obj: &Bound<'_, PyAny>, fields: &[StructField]) -> 
PyResult<Value> {
+    let py = obj.py();
+    let values = if let Ok(row) = obj.cast::<PyRow>() {
+        let (_, values) = row.borrow().parts(py)?;
+        values_by_position(py, values.iter().collect(), fields)?
+    } else if let Ok(tuple) = obj.cast::<PyTuple>() {
+        values_by_position(py, tuple.iter().collect(), fields)?
+    } else if obj.is_instance_of::<PyDict>() {
+        values_by_name(obj, fields)?
+    } else if let Ok(attributes) = obj.getattr(intern!(py, "__dict__")) {
+        values_by_name(&attributes, fields)?
+    } else {
+        // PySpark asserts that another value has a `__dict__`.
+        return Err(PyAssertionError::new_err(format!(
+            "{} {}",
+            obj.get_type().str()?,
+            obj.str()?
+        )));
+    };
+    struct_of(&values, fields)
+}
+
+fn struct_of(values: &[Bound<'_, PyAny>], fields: &[StructField]) -> 
PyResult<Value> {
+    let names: std::sync::Arc<[String]> = fields.iter().map(|f| 
f.name.clone()).collect();
+    let values = values
+        .iter()
+        .zip(fields)
+        .map(|(value, f)| convert(value, &f.data_type, f.nullable))
+        .collect::<PyResult<Vec<_>>>()?;
+    Ok(Value::Struct(StructValue::new(names, values)))
+}
+
+/// `dict(sorted(row.items()))`, as PySpark sorts dict rows to respect their 
inferred schema.
+pub(crate) fn sorted_dict<'py>(row: &Bound<'py, PyAny>) -> PyResult<Bound<'py, 
PyAny>> {
+    let py = row.py();
+    let sorted = SORTED.import(py, "builtins", "sorted")?;
+    let items = sorted.call1((row.call_method0(intern!(py, "items"))?,))?;
+    Ok(py.get_type::<PyDict>().call1((items,))?.into_any())
+}
+
+/// The field of the single column that data of an atomic schema has, as 
PySpark names it.
+pub(crate) fn value_field(data_type: DataType) -> StructField {
+    nullable_field("value".to_string(), data_type)
+}
+
+/// The values of a row in the order of `fields`, as PySpark's `to_row` takes 
them: nulls for
+/// None, a tuple or a Row by position, a dict by name, another object by its 
`__dict__`, and
+/// another value, such as a list, by position.
+pub(crate) fn row_values<'py>(
+    item: &Bound<'py, PyAny>,
+    fields: &[StructField],
+) -> PyResult<Vec<Bound<'py, PyAny>>> {
+    let py = item.py();
+    if item.is_none() {
+        return Ok(vec![py.None().into_bound(py); fields.len()]);
+    }
+    if let Ok(row) = item.cast::<PyRow>() {
+        let (_, values) = row.borrow().parts(py)?;
+        return values_by_position(py, values.iter().collect(), fields);
+    }
+    if let Ok(tuple) = item.cast::<PyTuple>() {
+        return values_by_position(py, tuple.iter().collect(), fields);
+    }
+    if item.is_instance_of::<PyDict>() {
+        return values_by_name(item, fields);
+    }
+    if item.is_instance_of::<PyVariantVal>() {
+        return Err(pyspark_message(
+            py,
+            "PySparkValueError",
+            "Rows cannot be of type VariantVal",
+        ));
+    }
+    if item.hasattr(intern!(py, "__dict__"))? {
+        return values_by_name(&item.getattr(intern!(py, "__dict__"))?, fields);
+    }
+    let length = item.len()?;
+    if length != fields.len() {
+        return Err(axis_length_mismatch(py, fields.len(), length));
+    }
+    item.try_iter()?.collect()
+}
diff --git a/crates/pyspark-rs/src/lib.rs b/crates/pyspark-rs/src/lib.rs
index 8f2ab6b..e5b3c6f 100644
--- a/crates/pyspark-rs/src/lib.rs
+++ b/crates/pyspark-rs/src/lib.rs
@@ -33,6 +33,7 @@ mod errors;
 mod eval_type;
 mod functions;
 mod group;
+mod infer;
 mod ml;
 mod observation;
 mod pipelines;
diff --git a/crates/pyspark-rs/src/row.rs b/crates/pyspark-rs/src/row.rs
index fe43303..53d8b21 100644
--- a/crates/pyspark-rs/src/row.rs
+++ b/crates/pyspark-rs/src/row.rs
@@ -90,6 +90,15 @@ impl PyRow {
         }
     }
 
+    /// The field names, if the row has them (`__fields__`), and the values, 
as the tuple
+    /// that a PySpark row is, for reading the row as PySpark's schema 
inference does.
+    pub(crate) fn parts<'py>(
+        &self,
+        py: Python<'py>,
+    ) -> PyResult<(Option<&[String]>, Bound<'py, PyTuple>)> {
+        Ok((self.named.then(|| self.fields()), self.values(py)?))
+    }
+
     /// The row with Spark values, for the rest of the client, which converts 
the values of a
     /// row made in Python as it converts those of a tuple.
     pub(crate) fn to_row(&self, py: Python<'_>) -> PyResult<Row> {
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 8807b8c..1432ff3 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -2,7 +2,7 @@
 
 use pyo3::prelude::*;
 use pyo3::types::{PyDict, PyList};
-use spark_connect::row::{Row, Value};
+use spark_connect::row::{Row, StructValue, Value};
 use spark_connect::session::SparkSession;
 use spark_connect::types::{DataType, StructField};
 use spark_connect::udf::{CommonInlineUserDefinedFunctionExpression, 
PythonUDFPayload};
@@ -278,6 +278,268 @@ impl PySparkSession {
     pub fn new(session: SparkSession) -> Self {
         PySparkSession { session }
     }
+
+    /// A DataFrame of local rows, as PySpark Connect creates one: the schema 
is given, or
+    /// inferred by merging the types of every row, and each value is 
converted to the type
+    /// of its field. A list of names names the inferred fields and is applied 
with `toDF`.
+    fn create_from_local(
+        &self,
+        py: Python<'_>,
+        data: &Bound<'_, PyAny>,
+        spec: Spec,
+    ) -> PyResult<PyDataFrame> {
+        use crate::infer;
+
+        let (given, mut cols) = match spec {
+            Spec::None => (None, None),
+            Spec::Names(names) => (None, Some(names)),
+            Spec::Struct(fields) => (Some(fields), None),
+            Spec::Atomic(atomic) => (Some(vec![infer::value_field(atomic)]), 
None),
+        };
+        // Empty data with a length needs a schema to type its relation, and 
gets no toDF.
+        if data.len().is_ok_and(|n| n == 0) {
+            let Some(fields) = given else {
+                return Err(crate::column::raise_pyspark(
+                    py,
+                    "PySparkValueError",
+                    "CANNOT_INFER_EMPTY_SCHEMA",
+                    &[],
+                ));
+            };
+            let schema = DataType::Struct { fields };
+            let df = py
+                .detach(|| {
+                    self.session
+                        .create_dataframe(Vec::new(), schema)
+                        .map_err(Box::new)
+                })
+                .to_pyerr()?;
+            return Ok(PyDataFrame::new(df));
+        }
+        let mut items: Vec<Bound<'_, PyAny>> = 
data.try_iter()?.collect::<PyResult<_>>()?;
+        let Some(first) = items.first().cloned() else {
+            // As PySpark takes the first row of data without a length.
+            return Err(pyo3::exceptions::PyIndexError::new_err(
+                "list index out of range",
+            ));
+        };
+        // As PySpark: dict rows are sorted by their keys, and rows that are 
none of a Row, a
+        // tuple, a list, a dict or an object, such as 1, are rows of that 
single value.
+        let single = if first.is_instance_of::<PyDict>() {
+            for item in items.iter_mut() {
+                if !item.is_none() {
+                    *item = infer::sorted_dict(item)?;
+                }
+            }
+            false
+        } else {
+            !(first.is_instance_of::<crate::row::PyRow>()
+                || first.is_instance_of::<pyo3::types::PyTuple>()
+                || first.is_instance_of::<PyList>()
+                || first.hasattr("__dict__")?)
+        };
+
+        let fields = match given {
+            Some(fields) => fields,
+            None => {
+                // The configurations are read only when a value depends on 
them.
+                let confs = infer::LazyConfs::new(self.session.conf());
+                let options = infer::Options::new(&confs);
+                let mut merged: Option<DataType> = None;
+                for item in &items {
+                    let row_type = if single {
+                        let items = infer::positional(py, vec![item.clone()], 
cols.as_mut());
+                        infer::infer_fields(py, items, options)?
+                    } else {
+                        infer::infer_schema(item, cols.as_mut(), options)?
+                    };
+                    merged = Some(match merged {
+                        None => row_type,
+                        Some(merged) => infer::merge_type(py, merged, 
row_type)?,
+                    });
+                }
+                match merged {
+                    Some(schema) if infer::has_null_type(&schema) => {
+                        return Err(crate::column::raise_pyspark(
+                            py,
+                            "PySparkValueError",
+                            "CANNOT_DETERMINE_TYPE",
+                            &[],
+                        ));
+                    }
+                    Some(DataType::Struct { fields }) => fields,
+                    _ => unreachable!("rows are inferred as structs"),
+                }
+            }
+        };
+
+        // As PySpark: every row is taken apart first, then each column is 
converted.
+        let mut parts: Vec<Vec<Bound<'_, PyAny>>> = 
Vec::with_capacity(items.len());
+        for item in &items {
+            parts.push(if single {
+                if fields.len() != 1 {
+                    return Err(infer::axis_length_mismatch(py, fields.len(), 
1));
+                }
+                vec![item.clone()]
+            } else {
+                infer::row_values(item, &fields)?
+            });
+        }
+        let mut values: Vec<Vec<Value>> = (0..parts.len())
+            .map(|_| Vec::with_capacity(fields.len()))
+            .collect();
+        for (i, field) in fields.iter().enumerate() {
+            for (row, out) in parts.iter().zip(values.iter_mut()) {
+                out.push(infer::convert(&row[i], &field.data_type, 
field.nullable)?);
+            }
+        }
+        // The names, with those that inference added, must name every column.
+        if let Some(cols) = &cols {
+            if cols.len() != fields.len() {
+                return Err(infer::axis_length_mismatch(py, cols.len(), 
fields.len()));
+            }
+        }
+        let names: std::sync::Arc<[String]> = fields.iter().map(|f| 
f.name.clone()).collect();
+        let rows: Vec<Row> = values
+            .into_iter()
+            .map(|values| StructValue::new(names.clone(), values).into())
+            .collect();
+        let schema = DataType::Struct { fields };
+        let df = py
+            .detach(|| {
+                self.session
+                    .create_dataframe(rows, schema)
+                    .map_err(Box::new)
+            })
+            .to_pyerr()?;
+        // As pyspark, column names are applied with toDF. The local relation 
already carries
+        // them, so this only changes the plan, which the server resolves into 
a Project that
+        // is not local; it keeps isLocal() as in pyspark and is not redundant.
+        Ok(PyDataFrame::new(match cols.take() {
+            Some(cols) if !cols.is_empty() => 
df.to_df(cols.iter().map(String::as_str).collect()),
+            _ => df,
+        }))
+    }
+
+    /// A DataFrame of a pandas DataFrame: its rows as lists with NaN and NaT 
as None, typed
+    /// by the first non-null value of each column, and its column labels, as 
strings, or the
+    /// given names, applied with `toDF`.
+    fn create_from_pandas(
+        &self,
+        py: Python<'_>,
+        data: &Bound<'_, PyAny>,
+        spec: Spec,
+    ) -> PyResult<PyDataFrame> {
+        let mut columns = Vec::new();
+        for label in data.getattr("columns")?.try_iter()? {
+            columns.push(label?.str()?.extract::<String>()?);
+        }
+        // df.astype(object).where(df.notna(), None).values.tolist() -> rows 
with None for NA.
+        let notna = data.call_method0("notna")?;
+        let obj = data.call_method1("astype", ("object",))?;
+        let replaced = obj.call_method1("where", (notna, py.None()))?;
+        let values = replaced.getattr("values")?.call_method0("tolist")?;
+        let data = values.cast_into::<PyList>()?;
+        let empty = data.is_empty();
+        let cannot_infer_empty = || {
+            crate::column::raise_pyspark(py, "PySparkValueError", 
"CANNOT_INFER_EMPTY_SCHEMA", &[])
+        };
+        // The given fields, and the names that toDF applies, which empty data 
does not get.
+        let (given, names) = match spec {
+            Spec::Atomic(atomic) if !empty => {
+                // pyspark reports str() of the parsed type, which is its repr.
+                let data_type = crate::types::data_type_to_py(py, &atomic)?
+                    .bind(py)
+                    .repr()?
+                    .extract::<String>()?;
+                return Err(crate::column::raise_pyspark(
+                    py,
+                    "PySparkTypeError",
+                    "UNSUPPORTED_DATA_TYPE_FOR_ARROW",
+                    &[("data_type", &data_type)],
+                ));
+            }
+            Spec::Atomic(atomic) => 
(Some(vec![crate::infer::value_field(atomic)]), None),
+            Spec::Struct(fields) => {
+                let names = fields.iter().map(|f| f.name.clone()).collect();
+                (Some(fields), (!empty).then_some(names))
+            }
+            _ if empty => return Err(cannot_infer_empty()),
+            Spec::None if columns.is_empty() => return 
Err(cannot_infer_empty()),
+            Spec::None => (None, Some(columns)),
+            Spec::Names(mut names) => {
+                // Extended with `_2`, `_3`, ... for more columns, as pyspark 
does.
+                let width = columns.len();
+                let start = names.len() + 1;
+                names.extend((start..=width).map(|i| format!("_{i}")));
+                (None, Some(names))
+            }
+        };
+        let row_names: Vec<String> = match (&given, &names) {
+            (Some(fields), _) => fields.iter().map(|f| 
f.name.clone()).collect(),
+            (None, Some(names)) => names.clone(),
+            (None, None) => Vec::new(),
+        };
+        let mut rows: Vec<Row> = Vec::with_capacity(data.len());
+        for row in data.iter() {
+            let values = row
+                .try_iter()?
+                .map(|v| py_to_value(&v?))
+                .collect::<PyResult<Vec<_>>>()?;
+            if values.len() != row_names.len() {
+                // As pyspark, which checks every row against the width of the 
schema.
+                return Err(crate::infer::axis_length_mismatch(
+                    py,
+                    row_names.len(),
+                    values.len(),
+                ));
+            }
+            rows.push(Row::new(row_names.clone(), values));
+        }
+        // A given struct wins; otherwise each field takes the type of its 
first non-null
+        // value (empty input has a struct schema, checked above).
+        let fields = match given {
+            Some(fields) => fields,
+            None => {
+                let mut fields: Vec<StructField> = 
Vec::with_capacity(row_names.len());
+                for (i, name) in row_names.iter().enumerate() {
+                    let Some(value) = rows
+                        .iter()
+                        .filter_map(|row| row.get(i))
+                        .find(|value| !matches!(value, Value::Null))
+                    else {
+                        return Err(crate::column::raise_pyspark(
+                            py,
+                            "PySparkValueError",
+                            "CANNOT_DETERMINE_TYPE",
+                            &[],
+                        ));
+                    };
+                    fields.push(StructField {
+                        name: name.clone(),
+                        data_type: value_to_datatype(value),
+                        nullable: true,
+                        metadata: Default::default(),
+                    });
+                }
+                fields
+            }
+        };
+        let schema = DataType::Struct { fields };
+        let df = py
+            .detach(|| {
+                self.session
+                    .create_dataframe(rows, schema)
+                    .map_err(Box::new)
+            })
+            .to_pyerr()?;
+        Ok(PyDataFrame::new(match names {
+            Some(names) if !names.is_empty() => {
+                df.to_df(names.iter().map(String::as_str).collect())
+            }
+            _ => df,
+        }))
+    }
 }
 
 /// Whether `obj` is an instance of `module.class`, without importing `module` 
when it is not
@@ -296,6 +558,59 @@ fn is_loaded_instance(
     }
 }
 
+/// A `createDataFrame` schema, as PySpark Connect reads one.
+enum Spec {
+    None,
+    /// Column names, which name the inferred fields and are applied with 
`toDF`.
+    Names(Vec<String>),
+    /// The fields of a struct type, or of a DDL string of one.
+    Struct(Vec<StructField>),
+    /// An atomic type, of the single column `value`.
+    Atomic(DataType),
+}
+
+/// The schema argument of `createDataFrame`: None, a list or tuple of names, 
or a struct or
+/// atomic type, given as a type or a DDL string. Other types raise 
`NOT_EXPECTED_TYPE`.
+fn schema_spec(py: Python<'_>, schema: Option<&Bound<'_, PyAny>>) -> 
PyResult<Spec> {
+    let Some(schema) = schema.filter(|s| !s.is_none()) else {
+        return Ok(Spec::None);
+    };
+    let not_expected = |arg_type: &str| {
+        crate::column::raise_pyspark(
+            py,
+            "PySparkTypeError",
+            "NOT_EXPECTED_TYPE",
+            &[
+                ("arg_name", "schema"),
+                ("expected_type", "list, None or StructType"),
+                ("arg_type", arg_type),
+            ],
+        )
+    };
+    if schema.is_instance_of::<PyList>() || 
schema.is_instance_of::<pyo3::types::PyTuple>() {
+        let mut names = Vec::new();
+        for name in schema.try_iter()? {
+            let name = name?;
+            names.push(match name.cast::<pyo3::types::PyString>() {
+                Ok(name) => name.to_cow()?.into_owned(),
+                // PySpark encodes a name that is not a string, which most 
values cannot do.
+                Err(_) => name.call_method1("encode", 
("utf-8",))?.str()?.to_string(),
+            });
+        }
+        return Ok(Spec::Names(names));
+    }
+    if !schema.is_instance_of::<pyo3::types::PyString>()
+        && !schema.is_instance_of::<crate::types::PyDataType>()
+    {
+        return Err(not_expected(&crate::infer::type_name(schema)?));
+    }
+    Ok(match crate::types::py_to_data_type(schema)? {
+        DataType::Struct { fields } => Spec::Struct(fields),
+        atomic if crate::infer::is_atomic(&atomic) => Spec::Atomic(atomic),
+        other => return Err(not_expected(&crate::infer::class_name(&other))),
+    })
+}
+
 #[pymethods]
 impl PySparkSession {
     /// The session builder, reached as `SparkSession.builder` (class 
attribute,
@@ -381,319 +696,53 @@ impl PySparkSession {
     /// Create a DataFrame from local data (a list of rows, a pandas 
DataFrame, or a pyarrow
     /// Table) and an optional schema, as in pyspark.
     ///
-    /// Column names follow pyspark: without a schema, plain rows are named 
`_1`, `_2`, ...
-    /// and Rows and dicts keep their own names. A list of names, extended 
with `_2`, `_3`,
-    /// ... for wider plain rows, and the columns of a pandas DataFrame (or 
the field names
-    /// of its schema) are applied with `toDF`. Without a struct schema, the 
field types are
-    /// inferred from the first row.
+    /// Local rows are inferred, converted and named as PySpark Connect does 
it, in
+    /// `crate::infer`. The columns of a pandas DataFrame, or the field names 
of its schema,
+    /// are applied with `toDF`, and its fields take the type of their first 
non-null value.
     #[pyo3(signature = (data, schema=None, samplingRatio=None, 
verifySchema=None))]
     fn createDataFrame(
         &self,
         py: Python<'_>,
         data: &Bound<'_, PyAny>,
         schema: Option<&Bound<'_, PyAny>>,
-        samplingRatio: Option<f64>,
-        verifySchema: Option<bool>,
+        samplingRatio: Option<&Bound<'_, PyAny>>,
+        verifySchema: Option<&Bound<'_, PyAny>>,
     ) -> PyResult<PyDataFrame> {
-        use crate::row::PyRow;
-
-        // `samplingRatio` only affects RDD-based schema inference in 
reference PySpark and is
-        // unused for local data over Spark Connect (reference ignores it here 
too).
-        // `verifySchema` toggles client-side row/type verification; this path 
builds the plan
-        // directly and does not perform that extra verification, so both are 
accepted for
-        // signature parity. See PARITY_GAP_ANALYSIS.md.
-        let _ = (samplingRatio, verifySchema);
-
-        // A caller holding a Table has already imported PyArrow.
-        if is_loaded_instance(py, data, "pyarrow", "Table")? {
-            return crate::pyarrow_import::create_dataframe(py, &self.session, 
data, schema);
-        }
-
-        // A pandas DataFrame (as pyspark does): its rows as lists with 
NaN/NaT mapped to None,
-        // and its column labels, as strings, which are applied below.
-        let mut pandas_columns: Option<Vec<String>> = None;
-        let is_pandas = py
-            .import("pandas")
-            .and_then(|m| m.getattr("DataFrame"))
-            .and_then(|c| data.is_instance(&c))
-            .unwrap_or(false);
-        let data_owned: Bound<'_, PyList>;
-        let data: &Bound<'_, PyList> = if is_pandas {
-            let mut labels = Vec::new();
-            for label in data.getattr("columns")?.try_iter()? {
-                labels.push(label?.str()?.extract::<String>()?);
-            }
-            pandas_columns = Some(labels);
-            // df.astype(object).where(df.notna(), None).values.tolist() -> 
rows with None for NA.
-            let notna = data.call_method0("notna")?;
-            let obj = data.call_method1("astype", ("object",))?;
-            let replaced = obj.call_method1("where", (notna, py.None()))?;
-            let values = replaced.getattr("values")?.call_method0("tolist")?;
-            data_owned = values.cast_into::<PyList>()?;
-            &data_owned
-        } else {
-            data.cast::<PyList>()?
-        };
-
-        // Resolve the schema: a list of column names (types inferred), a DDL 
string or
-        // a StructType (names + types explicit), or None (names from Row / 
default).
-        enum Spec {
-            None,
-            Names(Vec<String>),
-            Struct(Vec<StructField>),
+        if data.is_none() {
+            return Err(pyo3::exceptions::PyAssertionError::new_err(()));
         }
-        let mut atomic_schema: Option<DataType> = None;
-        let spec = match schema {
-            None => Spec::None,
-            Some(s) => {
-                if let Ok(names) = s.extract::<Vec<String>>() {
-                    Spec::Names(names)
-                } else {
-                    // Any DataType object (StructType, PyDataType) or a DDL 
string,
-                    // resolved through the shared converter. A struct becomes 
the
-                    // schema directly; a scalar AtomicType becomes a single 
"value"
-                    // column (matching pyspark createDataFrame(data, 
AtomicType())).
-                    match crate::types::py_to_data_type(s)? {
-                        DataType::Struct { fields } => Spec::Struct(fields),
-                        atomic => {
-                            atomic_schema = Some(atomic.clone());
-                            
Spec::Struct(vec![spark_connect::types::StructField {
-                                name: "value".to_string(),
-                                data_type: atomic,
-                                nullable: true,
-                                metadata: std::collections::BTreeMap::new(),
-                            }])
-                        }
-                    }
-                }
-            }
-        };
-
-        // As pyspark, empty input needs a schema to type its relation, and 
gets no toDF.
-        let empty = data.is_empty();
-        if empty && !matches!(spec, Spec::Struct(_)) {
+        if data.is_instance_of::<PyDataFrame>() {
             return Err(crate::column::raise_pyspark(
                 py,
-                "PySparkValueError",
-                "CANNOT_INFER_EMPTY_SCHEMA",
-                &[],
+                "PySparkTypeError",
+                "INVALID_TYPE",
+                &[("arg_name", "data"), ("arg_type", "DataFrame")],
             ));
         }
-        if !empty {
-            if let Some(columns) = &pandas_columns {
-                if let Some(atomic) = &atomic_schema {
-                    // pyspark reports str() of the parsed type, which is its 
repr.
-                    let data_type = crate::types::data_type_to_py(py, atomic)?
-                        .bind(py)
-                        .repr()?
-                        .extract::<String>()?;
-                    return Err(crate::column::raise_pyspark(
-                        py,
-                        "PySparkTypeError",
-                        "UNSUPPORTED_DATA_TYPE_FOR_ARROW",
-                        &[("data_type", &data_type)],
-                    ));
-                }
-                if matches!(spec, Spec::None) && columns.is_empty() {
-                    return Err(crate::column::raise_pyspark(
-                        py,
-                        "PySparkValueError",
-                        "CANNOT_INFER_EMPTY_SCHEMA",
-                        &[],
-                    ));
-                }
-            }
-        }
-
-        // How a row carries its values, as in pyspark: positionally (lists, 
tuples and
-        // scalars), positionally with names of its own (Rows, and namedtuples 
by their
-        // `_fields`), or by name (dicts, and other objects through their 
`__dict__`, whose
-        // keys are sorted as `_infer_schema` does).
-        enum Item {
-            Plain(Vec<Value>),
-            Named(Vec<String>, Vec<Value>),
-            Record(Vec<(String, Value)>),
-        }
-        let record = |dict: &Bound<'_, PyDict>| -> PyResult<Item> {
-            let mut pairs: Vec<(String, Value)> = 
Vec::with_capacity(dict.len());
-            for (k, v) in dict.iter() {
-                pairs.push((k.str()?.to_string(), py_to_value(&v)?));
+        // As PySpark Connect, which ignores both.
+        for (name, value) in [
+            ("samplingRatio", samplingRatio),
+            ("verifySchema", verifySchema),
+        ] {
+            if value.is_some_and(|v| !v.is_none()) {
+                py.import("warnings")?.call_method1(
+                    "warn",
+                    (format!(
+                        "'{name}' is ignored. It is not supported with Spark 
Connect."
+                    ),),
+                )?;
             }
-            pairs.sort_by(|a, b| a.0.cmp(&b.0));
-            Ok(Item::Record(pairs))
-        };
-        let mut items: Vec<Item> = Vec::with_capacity(data.len());
-        for item in data.iter() {
-            items.push(if let Ok(dict) = item.cast::<PyDict>() {
-                record(dict)?
-            } else if let Ok(pyrow) = item.extract::<PyRef<PyRow>>() {
-                let row = pyrow.to_row(py)?;
-                Item::Named(row.fields().to_vec(), row.values().to_vec())
-            } else if let Ok(lst) = item.cast::<PyList>() {
-                Item::Plain(
-                    lst.iter()
-                        .map(|v| py_to_value(&v))
-                        .collect::<PyResult<_>>()?,
-                )
-            } else if let Ok(tuple) = item.cast::<pyo3::types::PyTuple>() {
-                let values = tuple
-                    .iter()
-                    .map(|v| py_to_value(&v))
-                    .collect::<PyResult<_>>()?;
-                match tuple
-                    .getattr("_fields")
-                    .and_then(|fields| fields.extract::<Vec<String>>())
-                {
-                    Ok(fields) => Item::Named(fields, values),
-                    Err(_) => Item::Plain(values),
-                }
-            } else if let Some(attributes) = item
-                .getattr("__dict__")
-                .ok()
-                .and_then(|attributes| attributes.cast_into::<PyDict>().ok())
-            {
-                record(&attributes)?
-            } else {
-                // A scalar row (int/str/float/...): a single-field row, 
matching
-                // `createDataFrame([1, 2, 3], IntegerType())`. The field name 
comes from
-                // the schema when given, else defaults below.
-                Item::Plain(vec![py_to_value(&item)?])
-            });
         }
 
-        // Plain rows are as wide as the widest of them: pyspark's schema 
inference names
-        // them `_1`, `_2`, ..., and extends a shorter list of names the same 
way.
-        let plain_width = items
-            .iter()
-            .filter_map(|item| match item {
-                Item::Plain(values) => Some(values.len()),
-                _ => None,
-            })
-            .max()
-            .unwrap_or(0);
-        // The keys of the dict rows, in the order pyspark merges their 
schemas: each row's
-        // sorted keys that no earlier row had.
-        let mut record_names: Vec<String> = Vec::new();
-        let mut seen = std::collections::HashSet::new();
-        for item in &items {
-            if let Item::Record(pairs) = item {
-                for (key, _) in pairs {
-                    if seen.insert(key.as_str()) {
-                        record_names.push(key.clone());
-                    }
-                }
-            }
+        // A caller holding a Table, or a pandas DataFrame, has already 
imported its module.
+        if is_loaded_instance(py, data, "pyarrow", "Table")? {
+            return crate::pyarrow_import::create_dataframe(py, &self.session, 
data, schema);
         }
-        let extend = |names: &[String], width: usize| -> Vec<String> {
-            let mut names = names.to_vec();
-            names.extend((names.len()..width).map(|i| format!("_{}", i + 1)));
-            names
-        };
-        // The names pyspark applies with toDF: a list of names, and the 
columns of a pandas
-        // DataFrame or the field names of its schema.
-        let to_df_names: Option<Vec<String>> = match (&spec, &pandas_columns) {
-            _ if empty => None,
-            (Spec::Names(names), Some(columns)) => Some(extend(names, 
columns.len())),
-            (Spec::Names(names), None) => Some(extend(names, plain_width)),
-            (Spec::None, Some(columns)) => Some(columns.clone()),
-            (Spec::Struct(fields), Some(_)) => {
-                Some(fields.iter().map(|f| f.name.clone()).collect())
-            }
-            (_, None) => None,
-        };
-        let default_names = extend(&[], plain_width);
-        // A dict row's value for `name`, as pyspark looks it up: null when 
the key is missing.
-        // The pairs are sorted by key, and the first of equal keys is taken.
-        let lookup = |pairs: &[(String, Value)], name: &String| {
-            let index = pairs.partition_point(|(key, _)| key < name);
-            match pairs.get(index) {
-                Some((key, value)) if key == name => value.clone(),
-                _ => Value::Null,
-            }
-        };
-
-        let mut rows: Vec<Row> = Vec::with_capacity(items.len());
-        for item in items {
-            let (names, values) = match (&spec, &to_df_names, item) {
-                // A struct schema takes dict values by its field names, so 
that missing
-                // keys are null and other keys are ignored, and other rows by 
position.
-                (Spec::Struct(fields), _, item) => {
-                    let names: Vec<String> = fields.iter().map(|f| 
f.name.clone()).collect();
-                    let values = match item {
-                        Item::Record(pairs) => names.iter().map(|n| 
lookup(&pairs, n)).collect(),
-                        Item::Plain(values) | Item::Named(_, values) => values,
-                    };
-                    (names, values)
-                }
-                (_, names, Item::Record(pairs)) => (
-                    names.clone().unwrap_or_else(|| record_names.clone()),
-                    record_names.iter().map(|n| lookup(&pairs, n)).collect(),
-                ),
-                (_, Some(names), Item::Plain(values) | Item::Named(_, values)) 
=> {
-                    (names.clone(), values)
-                }
-                (_, None, Item::Named(names, values)) => (names, values),
-                (_, None, Item::Plain(values)) => (default_names.clone(), 
values),
-            };
-            if names.len() != values.len() {
-                // As pyspark, which checks every row against the width of the 
schema.
-                return Err(crate::column::raise_pyspark(
-                    py,
-                    "PySparkValueError",
-                    "AXIS_LENGTH_MISMATCH",
-                    &[
-                        ("expected_length", &names.len().to_string()),
-                        ("actual_length", &values.len().to_string()),
-                    ],
-                ));
-            }
-            rows.push(Row::new(names, values));
+        let spec = schema_spec(py, schema)?;
+        if is_loaded_instance(py, data, "pandas", "DataFrame")? {
+            return self.create_from_pandas(py, data, spec);
         }
-
-        // An explicit struct schema wins; otherwise each field takes the type 
of its first
-        // non-null value (empty input has a struct schema, checked above). 
pyspark merges
-        // the types of all rows, which gives the same type when they agree, 
and cannot
-        // type a field that is null in every row.
-        let schema_dtype = match spec {
-            Spec::Struct(fields) => DataType::Struct { fields },
-            _ => {
-                let mut fields: Vec<StructField> = 
Vec::with_capacity(rows[0].len());
-                for (i, name) in rows[0].fields().iter().enumerate() {
-                    let Some(value) = rows
-                        .iter()
-                        .filter_map(|row| row.get(i))
-                        .find(|value| !matches!(value, Value::Null))
-                    else {
-                        return Err(crate::column::raise_pyspark(
-                            py,
-                            "PySparkValueError",
-                            "CANNOT_DETERMINE_TYPE",
-                            &[],
-                        ));
-                    };
-                    fields.push(StructField {
-                        name: name.clone(),
-                        data_type: value_to_datatype(value),
-                        nullable: true,
-                        metadata: Default::default(),
-                    });
-                }
-                DataType::Struct { fields }
-            }
-        };
-
-        let df = py
-            .detach(|| self.session.create_dataframe(rows, schema_dtype))
-            .to_pyerr()?;
-        // As pyspark, column names are applied with toDF. The local relation 
already carries
-        // them, so this only changes the plan, which the server resolves into 
a Project that
-        // is not local; it keeps isLocal() as in pyspark and is not redundant.
-        Ok(PyDataFrame::new(match to_df_names {
-            Some(names) if !names.is_empty() => {
-                df.to_df(names.iter().map(String::as_str).collect())
-            }
-            _ => df,
-        }))
+        self.create_from_local(py, data, spec)
     }
 
     /// Get the catalog API (`spark.catalog`).
diff --git a/crates/spark-connect/src/conf.rs b/crates/spark-connect/src/conf.rs
index d43d611..3227f11 100644
--- a/crates/spark-connect/src/conf.rs
+++ b/crates/spark-connect/src/conf.rs
@@ -62,6 +62,12 @@ impl RuntimeConf {
         Ok(results.into_iter().next().flatten())
     }
 
+    /// Get the values of several configuration keys in one request, in the 
order of `keys`,
+    /// with `None` for a key that does not exist on the server.
+    pub fn get_many(&self, keys: &[&str]) -> Result<Vec<Option<String>>> {
+        block_on(self.client.get_configs(keys))
+    }
+
     /// Get a configuration value, returning `default` when the key is unset — 
instead of
     /// erroring as `get` does for an unknown key. Uses the server's 
`GetWithDefault`
     /// operation, mirroring `RuntimeConfig.get(key, default)`.
diff --git a/crates/spark-connect/src/session.rs 
b/crates/spark-connect/src/session.rs
index a66366f..a791e1f 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -724,8 +724,9 @@ fn rows_to_arrow_ipc(rows: &[Row], schema: &DataType) -> 
Result<Vec<u8>> {
         columns.push(values_to_arrow(&column, &field.data_type)?);
     }
 
-    // Create RecordBatch
-    let batch = RecordBatch::try_new(Arc::new(arrow_schema), columns)
+    // Create RecordBatch, with the number of rows, which a batch without 
columns needs.
+    let options = 
arrow::record_batch::RecordBatchOptions::new().with_row_count(Some(rows.len()));
+    let batch = RecordBatch::try_new_with_options(Arc::new(arrow_schema), 
columns, &options)
         .map_err(|e| SparkError::connect_msg(format!("Failed to create Arrow 
batch: {}", e)))?;
 
     // Write to IPC stream
@@ -1031,14 +1032,19 @@ fn values_to_arrow(values: &[Value], dt: &DataType) -> 
Result<Arc<dyn arrow::arr
                     }
                     Value::Struct(sf) => {
                         valid.append(true);
+                        // A struct of the declared fields in order, as the 
Python client
+                        // converts one, is taken by position, so that fields 
of the same
+                        // name keep their own values. Otherwise match by 
field name, else
+                        // by position.
+                        let in_order = sf.len() == fields.len()
+                            && sf.fields().iter().zip(fields).all(|(n, f)| *n 
== f.name);
                         for (i, f) in fields.iter().enumerate() {
-                            // Prefer match by field name, else positional.
-                            let val = sf
-                                .get_by_name(&f.name)
-                                .or_else(|| sf.get(i))
-                                .cloned()
-                                .unwrap_or(Value::Null);
-                            cols[i].push(val);
+                            let val = if in_order {
+                                sf.get(i)
+                            } else {
+                                sf.get_by_name(&f.name).or_else(|| sf.get(i))
+                            };
+                            cols[i].push(val.cloned().unwrap_or(Value::Null));
                         }
                     }
                     // A Python dict supplied for a struct field (e.g. 
array<struct> data)
@@ -1072,11 +1078,15 @@ fn values_to_arrow(values: &[Value], dt: &DataType) -> 
Result<Arc<dyn arrow::arr
                 })
                 .collect::<Result<Vec<_>>>()?
                 .into();
-            Ok(Arc::new(StructArray::new(
+            // A struct without fields takes its length from the values.
+            StructArray::try_new_with_length(
                 afields,
                 arrays,
                 Some(NullBuffer::new(valid.finish())),
-            )))
+                values.len(),
+            )
+            .map(|array| Arc::new(array) as Arc<dyn Array>)
+            .map_err(|e| SparkError::connect_msg(format!("struct build: {e}")))
         }
         DataType::Map {
             key_type,
@@ -1733,12 +1743,48 @@ mod tests {
     }
 
     #[test]
-    fn input_ipc_preserves_empty_input_and_zero_column_error() {
+    fn struct_values_of_the_declared_fields_are_taken_by_position() {
+        // Fields of the same name keep their own values.
+        let fields = vec![
+            struct_field("a", DataType::Long),
+            struct_field("a", DataType::Long),
+        ];
+        let names = vec!["a".to_string(), "a".to_string()];
+        let value = Value::Struct(Row::new(names, vec![Value::Long(1), 
Value::Long(2)]).into());
+        let array = values_to_arrow(&[value], &DataType::Struct { fields 
}).unwrap();
+        let array = array.as_any().downcast_ref::<StructArray>().unwrap();
+        let column = |i: usize| {
+            let column = array
+                .column(i)
+                .as_any()
+                .downcast_ref::<arrow::array::Int64Array>();
+            column.unwrap().value(0)
+        };
+        assert_eq!((column(0), column(1)), (1, 2));
+    }
+
+    #[test]
+    fn struct_values_without_fields_keep_their_length() {
+        let empty = Value::Struct(Row::empty().into());
+        let array = values_to_arrow(&[empty, Value::Null], &DataType::Struct { 
fields: vec![] });
+        let array = array.unwrap();
+        assert_eq!(array.len(), 2);
+        assert!(array.is_valid(0));
+        assert!(array.is_null(1));
+    }
+
+    #[test]
+    fn input_ipc_preserves_empty_input_and_rows_without_columns() {
         assert!(rows_to_arrow_ipc(&[], &DataType::Long).unwrap().is_empty());
-        let error = rows_to_arrow_ipc(&[Row::empty()], &schema(vec![]))
-            .unwrap_err()
-            .to_string();
-        assert!(error.contains("Failed to create Arrow batch"), "{error}");
+        // Rows without columns keep their number, as PySpark's table without 
columns does.
+        let data = rows_to_arrow_ipc(&[Row::empty(), Row::empty()], 
&schema(vec![])).unwrap();
+        let batches = StreamReader::try_new(Cursor::new(data), None)
+            .unwrap()
+            .collect::<std::result::Result<Vec<_>, _>>()
+            .unwrap();
+        assert_eq!(batches.len(), 1);
+        assert_eq!(batches[0].num_columns(), 0);
+        assert_eq!(batches[0].num_rows(), 2);
     }
 
     #[test]
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index ceb4569..d6ec5e0 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -381,3 +381,94 @@ def 
test_maps_keep_their_entries_in_order_with_keys_of_their_type(session):
     df = session.createDataFrame([({2: "b", 1: "a"},)])
     assert df.schema.simpleString() == "struct<_1:map<bigint,string>>"
     assert list(df.first()[0].items()) == [(2, "b"), (1, "a")]
+
+
+def test_create_dataframe_merges_the_types_of_every_row_as_pyspark(session):
+    import array
+    import decimal
+
+    Point = namedtuple("Point", ["x", "y"])
+
+    class Attributes:
+        def __init__(self):
+            self.b = 1
+            self.a = "x"
+
+    cases = [
+        # A string absorbs another atomic type, whose values become strings, 
and a null
+        # takes the type of another row.
+        ([(1, None), ("a", 2.5), (True, None)], "struct<_1:string,_2:double>",
+         [("1", None), ("a", 2.5), ("true", None)]),
+        ([([1, "a"],)], "struct<_1:array<string>>", [(["1", "a"],)]),
+        ([{"a": 1}, {"a": "x"}], "struct<a:string>", [("1",), ("x",)]),
+        ([({"a": [1]},), ({"a": [None]},)], 
"struct<_1:map<string,array<bigint>>>",
+         [({"a": [1]},), ({"a": [None]},)]),
+        # A decimal is decimal(38,18), a tuple, a namedtuple or an object a 
struct, and an
+        # array.array an array of its typecode.
+        ([(decimal.Decimal("1.5"),)], "struct<_1:decimal(38,18)>",
+         [(decimal.Decimal("1.500000000000000000"),)]),
+        ([((1, 2),)], "struct<_1:struct<_1:bigint,_2:bigint>>", [(Row(_1=1, 
_2=2),)]),
+        ([(Point(1, 2),)], "struct<_1:struct<x:bigint,y:bigint>>", [(Row(x=1, 
y=2),)]),
+        ([(Attributes(),)], "struct<_1:struct<a:string,b:bigint>>", 
[(Row(a="x", b=1),)]),
+        ([(array.array("i", [1, 2]),)], "struct<_1:array<int>>", [([1, 2],)]),
+    ]
+    for data, schema, rows in cases:
+        df = session.createDataFrame(data)
+        assert df.schema.simpleString() == schema, data
+        assert [tuple(r) for r in df.collect()] == rows, data
+    for text in ["1.25", "a\x00b", "\uffff"]:
+        # A string stays text, and shares a string column with an integer.
+        df = session.createDataFrame([Row(value=text), Row(value=5)])
+        assert [tuple(r) for r in df.collect()] == [(text,), ("5",)]
+
+
+def test_create_dataframe_raises_merge_errors_as_pyspark(session):
+    from pyspark.errors import PySparkException
+
+    cases = [
+        ([(1,), (2.5,)], "CANNOT_MERGE_TYPE", {"data_type1": "LongType", 
"data_type2": "DoubleType"}),
+        # A merge error within a value names its field.
+        ([([1, 2.5],)], "CANNOT_INFER_TYPE_FOR_FIELD", {"field_name": "_1"}),
+        ([({"a": 1, "b": 2.5},)], "CANNOT_INFER_TYPE_FOR_FIELD", 
{"field_name": "_1"}),
+        # A type that stays null cannot be determined.
+        ([({"k": None},)], "CANNOT_DETERMINE_TYPE", {}),
+        ([([None],)], "CANNOT_DETERMINE_TYPE", {}),
+    ]
+    for data, condition, parameters in cases:
+        with pytest.raises(PySparkException) as error:
+            session.createDataFrame(data)
+        assert error.value.getCondition() == condition, data
+        assert error.value.getMessageParameters() == parameters, data
+
+
+def 
test_create_dataframe_infers_by_the_session_configurations_as_pyspark(session):
+    naive = datetime(2020, 1, 2, 3, 4, 5)
+    cases = [
+        ("spark.sql.pyspark.legacy.inferMapTypeFromFirstPair.enabled", "true",
+         [({"a": 1, "b": 2.5},)], "struct<_1:map<string,bigint>>", [({"a": 1, 
"b": 2},)]),
+        ("spark.sql.pyspark.legacy.inferArrayTypeFromFirstElement.enabled", 
"true",
+         [([1, 2.5],)], "struct<_1:array<bigint>>", [([1, 2],)]),
+        ("spark.sql.pyspark.inferNestedDictAsStruct.enabled", "true",
+         [({"a": {"b": 1}, "c": None},)], 
"struct<_1:struct<a:struct<b:bigint>>>",
+         [(Row(a=Row(b=1)),)]),
+        # A naive datetime is TIMESTAMP_NTZ, but not in a list, whose elements 
PySpark
+        # infers without `prefer_timestamp_ntz`.
+        ("spark.sql.timestampType", "TIMESTAMP_NTZ", [(naive,)], 
"struct<_1:timestamp_ntz>",
+         [(naive,)]),
+        ("spark.sql.timestampType", "TIMESTAMP_NTZ", [([naive],)], 
"struct<_1:array<timestamp>>",
+         None),
+        # PySpark passes the elements of a list `prefer_timestamp_ntz` for
+        # `infer_map_from_first_pair`, so that a dict in a list takes the map 
type of its
+        # first pair.
+        ("spark.sql.timestampType", "TIMESTAMP_NTZ", [([{"a": 1, "b": 2.5}],)],
+         "struct<_1:array<map<string,bigint>>>", [([{"a": 1, "b": 2}],)]),
+    ]
+    for key, value, data, schema, rows in cases:
+        session.conf.set(key, value)
+        try:
+            df = session.createDataFrame(data)
+            assert df.schema.simpleString() == schema, (key, data)
+            if rows is not None:
+                assert [tuple(r) for r in df.collect()] == rows, (key, data)
+        finally:
+            session.conf.unset(key)
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index a4214c6..8422e36 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -6,6 +6,7 @@ Python side has real, measurable coverage in CI without a 
running server. Tests
 that require a live server live in test_dropin_e2e.py and are gated on 
SPARK_REMOTE.
 """
 
+import array
 import datetime
 import decimal
 import json
@@ -3462,6 +3463,71 @@ def 
test_create_dataframe_cannot_type_an_all_null_field_as_pyspark(data):
     assert error.value.getCondition() == "CANNOT_DETERMINE_TYPE"
 
 
[email protected]("data,condition,parameters", [
+    # The types of every row merge, as in PySpark, and types of other classes 
do not.
+    ([(1,), (2.5,)], "CANNOT_MERGE_TYPE", {"data_type1": "LongType", 
"data_type2": "DoubleType"}),
+    ([(True,), (1,)], "CANNOT_MERGE_TYPE", {"data_type1": "BooleanType", 
"data_type2": "LongType"}),
+    ([(decimal.Decimal("1.5"),), (1,)], "CANNOT_MERGE_TYPE",
+     {"data_type1": "DecimalType", "data_type2": "LongType"}),
+    # A row is a Row, a tuple, a list, a dict or an object, which None is not.
+    ([(1,), None], "CANNOT_INFER_SCHEMA_FOR_TYPE", {"data_type": "NoneType"}),
+    # A value of a type that cannot be inferred names its field.
+    ([({1, 2},)], "CANNOT_INFER_TYPE_FOR_FIELD", {"field_name": "_1"}),
+    ([(array.array("q", [1]),)], "CANNOT_INFER_TYPE_FOR_FIELD", {"field_name": 
"_1"}),
+])
+def test_create_dataframe_raises_inference_errors_as_pyspark(data, condition, 
parameters):
+    from pyspark.errors import PySparkTypeError
+
+    with pytest.raises(PySparkTypeError) as error:
+        _local_create_dataframe(data)
+    assert error.value.getCondition() == condition
+    assert error.value.getMessageParameters() == parameters
+
+
+def test_create_dataframe_checks_every_row_against_the_merged_schema():
+    from pyspark.errors import PySparkValueError
+
+    # Rows of different fields merge into one struct, which each Row must then 
fill.
+    with pytest.raises(PySparkValueError) as error:
+        _local_create_dataframe([Row(a=1), Row(b=2)])
+    assert error.value.getCondition() == "AXIS_LENGTH_MISMATCH"
+    assert error.value.getMessageParameters() == {"expected_length": "2", 
"actual_length": "1"}
+
+
[email protected]("schema,arg_type", [
+    (T.ArrayType(T.LongType()), "ArrayType"),
+    (T.MapType(T.StringType(), T.LongType()), "MapType"),
+    ("array<int>", "ArrayType"),
+    (5, "int"),
+])
+def test_create_dataframe_schema_is_names_a_struct_or_an_atomic_type(schema, 
arg_type):
+    from pyspark.errors import PySparkTypeError
+
+    with pytest.raises(PySparkTypeError) as error:
+        _local_create_dataframe([(1,)], schema)
+    assert error.value.getCondition() == "NOT_EXPECTED_TYPE"
+    assert error.value.getMessageParameters() == {
+        "arg_name": "schema", "expected_type": "list, None or StructType", 
"arg_type": arg_type}
+
+
+def test_create_dataframe_of_a_dataframe_raises_as_pyspark():
+    from pyspark.errors import PySparkTypeError
+
+    spark = _offline_session()
+    with pytest.raises(PySparkTypeError) as error:
+        spark.createDataFrame(spark.range(1))
+    assert error.value.getCondition() == "INVALID_TYPE"
+    assert error.value.getMessageParameters() == {"arg_name": "data", 
"arg_type": "DataFrame"}
+
+
+def test_create_dataframe_warns_of_the_arguments_that_connect_ignores():
+    # As PySpark Connect, which ignores both.
+    with pytest.warns(UserWarning, match="'samplingRatio' is ignored"):
+        _local_create_dataframe([(1,)], None, 0.5)
+    with pytest.warns(UserWarning, match="'verifySchema' is ignored"):
+        _local_create_dataframe([(1,)], None, None, True)
+
+
 # --------------------------------------------------------------------------- 
schema cache
 
 
@@ -4802,34 +4868,36 @@ def 
test_reader_schema_is_sent_as_pyspark_sends_it(mock_connect, read):
     "", "1.25", "-42", "NaN", "a\x00b", "é雪😀", "e\u0301", "\uffff",
 ])
 def test_exact_string_converts_as_text(text):
-    from pyspark.errors import PySparkRuntimeError
-
-    # A string, even when it reads as a number: it shares a column with a 
string, not with
-    # an integer.
+    # A string, even when it reads as a number, is text: with a string or an 
integer in
+    # another row, the column is a string column, as PySpark merges the types 
(the live
+    # suite checks the values).
     assert _local_create_dataframe([Row(value=text), Row(value="y")]).isLocal()
-    with pytest.raises(PySparkRuntimeError):
-        _local_create_dataframe([Row(value=text), Row(value=5)])
+    assert _local_create_dataframe([Row(value=text), Row(value=5)]).isLocal()
 
 
 @pytest.mark.parametrize("text", ["\ud800", "\udfff", "a\ud800b"])
 def test_exact_string_surrogate_error_preserves_generic_fallback(text):
-    with pytest.raises(TypeError):
+    # As PySpark, where PyArrow cannot encode a lone surrogate either.
+    with pytest.raises(UnicodeEncodeError, match="surrogates not allowed"):
         _local_create_dataframe([Row(value=text)])
 
 
 def test_exact_string_subclass_uses_generic_conversion():
-    # Pin existing subclass behavior; treating subclasses as text is a separate
-    # behavior change.
-    from pyspark.errors import PySparkRuntimeError
+    from pyspark.errors import PySparkTypeError
 
     class IndexString(str):
         def __index__(self):
             return 17
 
-    # Converted through __index__, it shares a column with an integer, not 
with a string.
-    assert _local_create_dataframe([Row(value=IndexString("x")), 
Row(value=5)]).isLocal()
-    with pytest.raises(PySparkRuntimeError):
-        _local_create_dataframe([Row(value=IndexString("x")), Row(value="y")])
+    # As in PySpark, which takes an instance of a subclass by its `__dict__`: 
an empty
+    # struct, which merges with neither an integer nor a string.
+    assert _local_create_dataframe([Row(value=IndexString("x"))]).isLocal()
+    for other, data_type in [(5, "LongType"), ("y", "StringType")]:
+        with pytest.raises(PySparkTypeError) as error:
+            _local_create_dataframe([Row(value=IndexString("x")), 
Row(value=other)])
+        assert error.value.getCondition() == "CANNOT_MERGE_TYPE"
+        assert error.value.getMessageParameters() == {
+            "data_type1": "StructType", "data_type2": data_type}
 
 
 # --------------------------------------------------------------------------- 
printSchema


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to