This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 42c5b10 [SPARK-60908][CONNECT] Infer and convert local data in
createDataFrame as PySpark does
42c5b10 is described below
commit 42c5b101b3ceb89f054557323b6d836c5d7e351b
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 08:39:24 2026 +0900
[SPARK-60908][CONNECT] Infer and convert local data in createDataFrame as
PySpark does
### What changes were proposed in this pull request?
[SPARK-60908](https://issues.apache.org/jira/browse/SPARK-60908):
`createDataFrame` of local rows now infers and converts its data as PySpark
Connect does.
- A new module, `crates/pyspark-rs/src/infer.rs`, ports PySpark 4.3.0's
`_infer_type`, `_infer_schema`, `_merge_type` and `_has_nulltype`, and the
converters of `LocalDataToArrowConversion`, keeping their order of checks and
their errors. The schema is the merge of every row's type. A string type
absorbs another atomic type, whose values then become strings (`True` as
`'true'`), a null type takes the other type, and types of different classes
raise `CANNOT_MERGE_TYPE`, or `CANNOT_INFE [...]
- The path of local rows follows PySpark Connect's. Dict rows are sorted by
their keys, a row of another value, such as `1`, is a row of that value, and
every row is taken apart and then converted column by column. A list of names
names the inferred fields, and inference extends it with `_2`, `_3`, ...; the
number of names must then equal the number of fields.
- The four configurations that inference depends on
(`spark.sql.pyspark.inferNestedDictAsStruct.enabled`, the two
`spark.sql.pyspark.legacy.infer...` ones and `spark.sql.timestampType`) are
read in one request, and only when a value depends on them: a dict, a list of
more than one element, or a naive datetime. Other data is still inferred
without a request. PySpark's inference of list elements, which passes
`prefer_timestamp_ntz` for `infer_map_from_first_pair` and leaves the former f
[...]
- As in PySpark Connect:
- A schema that is neither a list of names nor a struct or atomic type
raises `NOT_EXPECTED_TYPE`.
- A DataFrame as data raises `INVALID_TYPE`.
- `samplingRatio` and `verifySchema` warn that they are ignored.
- Data that is not a list, such as a tuple or a generator, is accepted.
- A TIMESTAMP value is the exact instant of its datetime, and a
TIMESTAMP_NTZ value its wall-clock time.
- In the Rust core:
- `RuntimeConf::get_many` reads several keys in one request.
- A struct value whose fields are the declared ones, in order, is encoded
by position, so that fields of the same name keep their own values.
- A batch without columns keeps its number of rows, and a struct without
fields its length, as PySpark's `createDataFrame([()])` gives a row without
columns.
- A pandas DataFrame is converted as before.
### Why are the changes needed?
Each field took the type of its first non-null value:
- `createDataFrame([(1,), ("a",)])` and `[([1, "a"],)]` raised an
unclassified error rather than giving a string column of `'1'` and `'a'`, and
`array<string>`.
- `[(1,), (2.5,)]` and `[([1, 2.5],)]` raised an unclassified error rather
than `CANNOT_MERGE_TYPE` and `CANNOT_INFER_TYPE_FOR_FIELD`.
- `[([None],)]` and `[({"k": None},)]` inferred `array<string>` and
`map<string,string>` rather than raising `CANNOT_DETERMINE_TYPE`, and
`[([None],), ([1],)]` raised an error rather than inferring `array<bigint>`.
- A `Decimal` inferred `decimal(38,s)` with the scale of its value, rather
than `decimal(38,18)`.
- Tuples, namedtuples and objects in a field, `array.array`, the inference
configurations, and the errors for an unsupported value or row were not as in
PySpark.
### Does this PR introduce _any_ user-facing change?
Yes, `createDataFrame` of local rows infers, converts and fails as in
PySpark Connect 4.3.0.
The tests that #127 (SPARK-60684) added now check PySpark's results:
- A string and an integer in two rows share a string column rather than
raising.
- A lone surrogate raises `UnicodeEncodeError`, as PyArrow does.
- An instance of a `str` subclass, which PySpark takes by its `__dict__`,
is an empty struct.
Some values still cannot be sent, and raise `TypeError` as before:
`datetime.time`, `timedelta` (whose input
[SPARK-60659](https://issues.apache.org/jira/browse/SPARK-60659) adds in #116,
which can now convert it in `infer::convert`), a value of a user-defined type,
`VariantVal`, and `Geometry` and `Geography`. Their types are inferred as in
PySpark.
### How was this patch tested?
- Against a local Spark 4.3.0 server, a probe of forty inference cases
gives output identical to official PySpark 4.3.0, apart from the `time` and
`timedelta` values above. The probes of #149 (thirty-one `createDataFrame`
cases of names, schemas and rows, nine of dict rows, two of null-first rows and
seven of namedtuples) and the sixteen map cases of #161 are still identical.
- New offline tests check the inference errors, the schema and data
arguments, and the warnings. Their cases also agree with official PySpark 4.3.0
against the server.
- New live tests check the merged types and values, the merge errors, and
inference under each of the four configurations. They also pass with official
PySpark 4.3.0.
- New unit tests check that a struct value of the declared fields is
encoded by position, and that a struct without fields keeps its length, which
encoding one used to panic on.
- The lib tests and every integration test, e2e included, against the local
server (641 passed), the build of all tests with `wasm-udf`, the offline Python
suites (611 passed, 8 live cases skipped), and the live drop-in and Arrow
suites (55 passed) and live cases (10 passed). Clippy reports no diagnostics in
the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #167 from HyukjinKwon/SPARK-60908-infer-merge.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/errors.rs | 8 +
crates/pyspark-rs/src/infer.rs | 1098 +++++++++++++++++++++++++++++++++++
crates/pyspark-rs/src/lib.rs | 1 +
crates/pyspark-rs/src/row.rs | 9 +
crates/pyspark-rs/src/session.rs | 643 ++++++++++----------
crates/spark-connect/src/conf.rs | 6 +
crates/spark-connect/src/session.rs | 78 ++-
python/tests/test_dropin_e2e.py | 91 +++
python/tests/test_dropin_offline.py | 96 ++-
9 files changed, 1703 insertions(+), 327 deletions(-)
diff --git a/crates/pyspark-rs/src/errors.rs b/crates/pyspark-rs/src/errors.rs
index 1da4078..4ea7a0c 100644
--- a/crates/pyspark-rs/src/errors.rs
+++ b/crates/pyspark-rs/src/errors.rs
@@ -193,3 +193,11 @@ impl<T> ResultExt<T> for Result<T, SparkError> {
self.map_err(spark_error_to_py_exception)
}
}
+
+/// For a boxed error, as a closure that runs without the GIL can return one,
which keeps
+/// its result small.
+impl<T> ResultExt<T> for Result<T, Box<SparkError>> {
+ fn to_pyerr(self) -> PyResult<T> {
+ self.map_err(|e| spark_error_to_py_exception(*e))
+ }
+}
diff --git a/crates/pyspark-rs/src/infer.rs b/crates/pyspark-rs/src/infer.rs
new file mode 100644
index 0000000..4d3386b
--- /dev/null
+++ b/crates/pyspark-rs/src/infer.rs
@@ -0,0 +1,1098 @@
+//! Schema inference and conversion of local data for `createDataFrame`, as
PySpark 4.3.0
+//! does them: `_infer_schema`, `_infer_type`, `_merge_type` and
`_has_nulltype` of
+//! `pyspark.sql.types`, and `LocalDataToArrowConversion` of
`pyspark.sql.conversion`, with
+//! which Spark Connect converts local data. The checks run in PySpark's order
and raise its
+//! errors, so that the same data gives the same schema, values and errors.
+
+use std::cell::OnceCell;
+use std::collections::{HashMap, HashSet};
+
+use pyo3::exceptions::{PyAssertionError, PyTypeError, PyValueError};
+use pyo3::intern;
+use pyo3::prelude::*;
+use pyo3::sync::PyOnceLock;
+use pyo3::types::{
+ IntoPyDict, PyBool, PyByteArray, PyBytes, PyDict, PyFloat, PyInt, PyList,
PyString, PyTuple,
+ PyType,
+};
+use spark_connect::conf::RuntimeConf;
+use spark_connect::row::{StructValue, Value};
+use spark_connect::types::{DataType, GeographyType, GeometryType, StructField};
+
+use crate::column::raise_pyspark;
+use crate::errors::ResultExt;
+use crate::row::PyRow;
+use crate::values::{PyGeography, PyGeometry, PyVariantVal};
+
+static DECIMAL: PyOnceLock<Py<PyType>> = PyOnceLock::new();
+static DATE: PyOnceLock<Py<PyType>> = PyOnceLock::new();
+static DATETIME: PyOnceLock<Py<PyType>> = PyOnceLock::new();
+static TIME: PyOnceLock<Py<PyType>> = PyOnceLock::new();
+static TIMEDELTA: PyOnceLock<Py<PyType>> = PyOnceLock::new();
+static ARRAY: PyOnceLock<Py<PyType>> = PyOnceLock::new();
+static SORTED: PyOnceLock<Py<PyAny>> = PyOnceLock::new();
+
+/// The session configurations that inference reads, as PySpark reads them for
+/// `createDataFrame`.
+struct Confs {
+ dict_as_struct: bool,
+ array_from_first_element: bool,
+ map_from_first_pair: bool,
+ prefer_timestamp_ntz: bool,
+}
+
+/// The configurations, read from the server, in one request, the first time
that inference
+/// needs one. Only dicts, lists of more than one element and naive datetimes
depend on them,
+/// so other data is inferred without a request, as without a server.
+pub(crate) struct LazyConfs {
+ conf: RuntimeConf,
+ confs: OnceCell<Confs>,
+}
+
+impl LazyConfs {
+ pub(crate) fn new(conf: RuntimeConf) -> Self {
+ LazyConfs {
+ conf,
+ confs: OnceCell::new(),
+ }
+ }
+
+ fn get(&self, py: Python<'_>) -> PyResult<&Confs> {
+ if let Some(confs) = self.confs.get() {
+ return Ok(confs);
+ }
+ let keys = [
+ "spark.sql.pyspark.inferNestedDictAsStruct.enabled",
+ "spark.sql.pyspark.legacy.inferArrayTypeFromFirstElement.enabled",
+ "spark.sql.pyspark.legacy.inferMapTypeFromFirstPair.enabled",
+ "spark.sql.timestampType",
+ ];
+ let conf = &self.conf;
+ let values = py
+ .detach(|| conf.get_many(&keys).map_err(Box::new))
+ .to_pyerr()?;
+ let is =
+ |i: usize, expected: &str| values.get(i).and_then(|v|
v.as_deref()) == Some(expected);
+ Ok(self.confs.get_or_init(|| Confs {
+ dict_as_struct: is(0, "true"),
+ array_from_first_element: is(1, "true"),
+ map_from_first_pair: is(2, "true"),
+ prefer_timestamp_ntz: is(3, "TIMESTAMP_NTZ"),
+ }))
+ }
+}
+
+/// Where an option of inference takes its value from.
+#[derive(Clone, Copy)]
+enum Source {
+ /// False, where PySpark leaves the option at its default.
+ Off,
+ /// `spark.sql.pyspark.legacy.inferMapTypeFromFirstPair.enabled`.
+ MapFromFirstPair,
+ /// `spark.sql.timestampType` being `TIMESTAMP_NTZ`.
+ PreferTimestampNtz,
+}
+
+/// The options that inference passes down, as PySpark does.
`infer_dict_as_struct` and
+/// `infer_array_from_first_element` always come from the configurations, and
the other two
+/// shift where PySpark passes them on.
+#[derive(Clone, Copy)]
+pub(crate) struct Options<'c> {
+ confs: &'c LazyConfs,
+ map_from_first_pair: Source,
+ prefer_timestamp_ntz: Source,
+}
+
+impl<'c> Options<'c> {
+ pub(crate) fn new(confs: &'c LazyConfs) -> Self {
+ Options {
+ confs,
+ map_from_first_pair: Source::MapFromFirstPair,
+ prefer_timestamp_ntz: Source::PreferTimestampNtz,
+ }
+ }
+
+ /// For the elements of a list. PySpark passes the options to
`_infer_type` by position
+ /// without `infer_map_from_first_pair`, so that it takes the value of
+ /// `prefer_timestamp_ntz`, which is left false.
+ fn for_elements(self) -> Self {
+ Options {
+ map_from_first_pair: self.prefer_timestamp_ntz,
+ prefer_timestamp_ntz: Source::Off,
+ ..self
+ }
+ }
+
+ /// For a row in a value, such as a tuple, a Row or an object, which
PySpark infers
+ /// without `infer_map_from_first_pair`.
+ fn for_nested_rows(self) -> Self {
+ Options {
+ map_from_first_pair: Source::Off,
+ ..self
+ }
+ }
+
+ fn value(&self, py: Python<'_>, source: Source) -> PyResult<bool> {
+ Ok(match source {
+ Source::Off => false,
+ Source::MapFromFirstPair =>
self.confs.get(py)?.map_from_first_pair,
+ Source::PreferTimestampNtz =>
self.confs.get(py)?.prefer_timestamp_ntz,
+ })
+ }
+}
+
+pub(crate) fn string_type() -> DataType {
+ DataType::String {
+ collation: "UTF8_BINARY".to_string(),
+ }
+}
+
+fn nullable_field(name: String, data_type: DataType) -> StructField {
+ StructField {
+ name,
+ data_type,
+ nullable: true,
+ metadata: Default::default(),
+ }
+}
+
+fn map_type(key_type: DataType, value_type: DataType) -> DataType {
+ DataType::Map {
+ key_type: Box::new(key_type),
+ value_type: Box::new(value_type),
+ value_contains_null: true,
+ }
+}
+
+pub(crate) fn type_name(obj: &Bound<'_, PyAny>) -> PyResult<String> {
+ Ok(obj.get_type().name()?.to_string())
+}
+
+/// A nullable field of `name`, which PySpark's `StructField` asserts is a
string.
+fn field_of(name: &Bound<'_, PyAny>, data_type: DataType) ->
PyResult<StructField> {
+ match name.cast::<PyString>() {
+ Ok(name) => Ok(nullable_field(name.to_cow()?.into_owned(), data_type)),
+ Err(_) => Err(PyAssertionError::new_err(format!(
+ "field name {} should be a string",
+ name.str()?
+ ))),
+ }
+}
+
+/// The type of `obj`, as PySpark's `_infer_type`.
+pub(crate) fn infer_type(obj: &Bound<'_, PyAny>, o: Options<'_>) ->
PyResult<DataType> {
+ let py = obj.py();
+ if obj.is_none() {
+ return Ok(DataType::Null);
+ }
+ // The types that PySpark maps by `type(obj)`, before it looks for
`__UDT__`, which none
+ // of them can have: neither these classes nor their instances take new
attributes.
+ if obj.is_exact_instance_of::<PyBool>() {
+ return Ok(DataType::Boolean);
+ }
+ if obj.is_exact_instance_of::<PyInt>() {
+ return Ok(DataType::Long);
+ }
+ if obj.is_exact_instance_of::<PyFloat>() {
+ return Ok(DataType::Double);
+ }
+ if obj.is_exact_instance_of::<PyString>() {
+ return Ok(string_type());
+ }
+ if obj.is_exact_instance_of::<PyBytes>() ||
obj.is_exact_instance_of::<PyByteArray>() {
+ return Ok(DataType::Binary);
+ }
+ let ty = obj.get_type();
+ if ty.is(DECIMAL.import(py, "decimal", "Decimal")?) {
+ // As PySpark: the precision and scale of a value may differ from row
to row.
+ return Ok(DataType::Decimal {
+ precision: 38,
+ scale: 18,
+ });
+ }
+ if ty.is(DATETIME.import(py, "datetime", "datetime")?) {
+ let naive = obj.getattr(intern!(py, "tzinfo"))?.is_none();
+ return Ok(if naive && o.value(py, o.prefer_timestamp_ntz)? {
+ DataType::TimestampNtz
+ } else {
+ DataType::Timestamp
+ });
+ }
+ if ty.is(DATE.import(py, "datetime", "date")?) {
+ return Ok(DataType::Date);
+ }
+ if ty.is(TIME.import(py, "datetime", "time")?) {
+ return Ok(DataType::Time { precision: 6 });
+ }
+ if ty.is(TIMEDELTA.import(py, "datetime", "timedelta")?) {
+ // `DayTimeIntervalType()`, from days to seconds.
+ return Ok(DataType::DayTimeInterval {
+ start_field: 0,
+ end_field: 3,
+ });
+ }
+ if obj.is_exact_instance_of::<PyGeometry>() {
+ let srid = obj.call_method0(intern!(py, "getSrid"))?.extract()?;
+ return Ok(DataType::Geometry { srid });
+ }
+ if obj.is_exact_instance_of::<PyGeography>() {
+ let srid = obj.call_method0(intern!(py, "getSrid"))?.extract()?;
+ return Ok(DataType::Geography { srid });
+ }
+ if obj.hasattr(intern!(py, "__UDT__"))? {
+ return crate::types::py_to_data_type(&obj.getattr(intern!(py,
"__UDT__"))?);
+ }
+ if let Ok(dict) = obj.cast::<PyDict>() {
+ return infer_dict(dict, o);
+ }
+ if let Ok(list) = obj.cast::<PyList>() {
+ return infer_list(list, o);
+ }
+ if obj.is_instance(ARRAY.import(py, "array", "array")?.as_any())? {
+ let typecode: String = obj.getattr(intern!(py,
"typecode"))?.extract()?;
+ return match array_element_type(&typecode) {
+ Some(element_type) => Ok(DataType::Array {
+ element_type: Box::new(element_type),
+ contains_null: false,
+ }),
+ None => Err(raise_pyspark(
+ py,
+ "PySparkTypeError",
+ "UNSUPPORTED_DATA_TYPE",
+ &[("data_type", &format!("array({typecode})"))],
+ )),
+ };
+ }
+ if obj.is_instance_of::<PyVariantVal>() {
+ return Ok(DataType::Variant);
+ }
+ match infer_schema(obj, None, o.for_nested_rows()) {
+ Err(e) if e.is_instance_of::<PyTypeError>(py) => Err(raise_pyspark(
+ py,
+ "PySparkTypeError",
+ "UNSUPPORTED_DATA_TYPE",
+ &[("data_type", &type_name(obj)?)],
+ )),
+ result => result,
+ }
+}
+
+/// A dict: a struct of its keys with non-null values, the map type of its
first pair with
+/// neither null, or, by default, the merged types of its keys and of its
values.
+fn infer_dict(dict: &Bound<'_, PyDict>, o: Options<'_>) -> PyResult<DataType> {
+ let py = dict.py();
+ if o.confs.get(py)?.dict_as_struct {
+ let mut fields = Vec::new();
+ for (key, value) in dict.iter() {
+ if !key.is_none() && !value.is_none() {
+ let data_type = infer_type(&value, o)?;
+ fields.push(field_of(&key, data_type)?);
+ }
+ }
+ return Ok(DataType::Struct { fields });
+ }
+ if o.value(py, o.map_from_first_pair)? {
+ for (key, value) in dict.iter() {
+ if !key.is_none() && !value.is_none() {
+ return Ok(map_type(infer_type(&key, o)?, infer_type(&value,
o)?));
+ }
+ }
+ return Ok(map_type(DataType::Null, DataType::Null));
+ }
+ let (mut key_type, mut value_type) = (DataType::Null, DataType::Null);
+ for (key, value) in dict.iter() {
+ if !key.is_none() {
+ key_type = merge_type(py, key_type, infer_type(&key, o)?)?;
+ }
+ if !value.is_none() {
+ value_type = merge_type(py, value_type, infer_type(&value, o)?)?;
+ }
+ }
+ Ok(map_type(key_type, value_type))
+}
+
+/// A list: an array of the merged types of its elements, or of its first
element's type.
+fn infer_list(list: &Bound<'_, PyList>, o: Options<'_>) -> PyResult<DataType> {
+ let py = list.py();
+ let elements = o.for_elements();
+ let element_type = match list.len() {
+ 0 => DataType::Null,
+ // One element has its own type either way, without reading the
configuration.
+ 1 => infer_type(&list.get_item(0)?, elements)?,
+ _ if o.confs.get(py)?.array_from_first_element =>
infer_type(&list.get_item(0)?, elements)?,
+ _ => {
+ let mut merged = DataType::Null;
+ for (i, value) in list.iter().enumerate() {
+ let data_type = infer_type(&value, elements)?;
+ merged = if i == 0 {
+ data_type
+ } else {
+ merge_type(py, merged, data_type)?
+ };
+ }
+ merged
+ }
+ };
+ Ok(DataType::Array {
+ element_type: Box::new(element_type),
+ contains_null: true,
+ })
+}
+
+/// The element type of an `array.array` of `typecode`, as PySpark maps the
typecodes by the
+/// sizes of their C types; the unsigned ones take one more bit, and none
needs more than 64.
+fn array_element_type(typecode: &str) -> Option<DataType> {
+ use std::ffi::{c_int, c_long, c_schar, c_short, c_uchar, c_uint, c_ulong,
c_ushort};
+ use std::mem::size_of;
+ let integral = |size: usize, unsigned: bool| match size * 8 +
usize::from(unsigned) {
+ 0..=8 => Some(DataType::Byte),
+ 9..=16 => Some(DataType::Short),
+ 17..=32 => Some(DataType::Integer),
+ 33..=64 => Some(DataType::Long),
+ _ => None,
+ };
+ match typecode {
+ "f" => Some(DataType::Float),
+ "d" => Some(DataType::Double),
+ "u" => Some(string_type()),
+ "b" => integral(size_of::<c_schar>(), false),
+ "h" => integral(size_of::<c_short>(), false),
+ "i" => integral(size_of::<c_int>(), false),
+ "l" => integral(size_of::<c_long>(), false),
+ "B" => integral(size_of::<c_uchar>(), true),
+ "H" => integral(size_of::<c_ushort>(), true),
+ "I" => integral(size_of::<c_uint>(), true),
+ "L" => integral(size_of::<c_ulong>(), true),
+ _ => None,
+ }
+}
+
+type Items<'py> = Vec<(Bound<'py, PyAny>, Bound<'py, PyAny>)>;
+
+/// `sorted(mapping.items())`, as PySpark orders the fields of a dict or of an
object.
+fn sorted_items<'py>(mapping: &Bound<'py, PyAny>) -> PyResult<Items<'py>> {
+ let py = mapping.py();
+ let sorted = SORTED.import(py, "builtins", "sorted")?;
+ let items = sorted.call1((mapping.call_method0(intern!(py, "items"))?,))?;
+ items
+ .try_iter()?
+ .map(|item| {
+ let item = item?;
+ Ok((item.get_item(0)?, item.get_item(1)?))
+ })
+ .collect()
+}
+
+/// The values of a tuple or list, named by `names` as `_infer_schema` zips
them. Without
+/// names, they are `_1`, `_2`, and so on; a shorter list of names is extended
so, in place.
+pub(crate) fn positional<'py>(
+ py: Python<'py>,
+ values: Vec<Bound<'py, PyAny>>,
+ names: Option<&mut Vec<String>>,
+) -> Items<'py> {
+ let defaults: Vec<String>;
+ let names: &[String] = match names {
+ Some(names) => {
+ let width = values.len();
+ if names.len() < width {
+ let start = names.len() + 1;
+ names.extend((start..=width).map(|i| format!("_{i}")));
+ }
+ names
+ }
+ None => {
+ defaults = (1..=values.len()).map(|i| format!("_{i}")).collect();
+ &defaults
+ }
+ };
+ names
+ .iter()
+ .map(|name| PyString::new(py, name).into_any())
+ .zip(values)
+ .collect()
+}
+
+/// The (name, value) pairs of a row, as `_infer_schema` takes them: a dict by
its sorted
+/// keys, a Row or a namedtuple by its field names, another tuple or a list by
`names`, and
+/// another object by the sorted names of its `__dict__`.
+fn row_items<'py>(
+ row: &Bound<'py, PyAny>,
+ names: Option<&mut Vec<String>>,
+) -> PyResult<Items<'py>> {
+ let py = row.py();
+ if row.is_instance_of::<PyDict>() {
+ return sorted_items(row);
+ }
+ if let Ok(pyrow) = row.cast::<PyRow>() {
+ let pyrow = pyrow.borrow();
+ let (fields, values) = pyrow.parts(py)?;
+ return Ok(match fields {
+ Some(fields) => fields
+ .iter()
+ .map(|f| PyString::new(py, f).into_any())
+ .zip(values.iter())
+ .collect(),
+ None => positional(py, values.iter().collect(), names),
+ });
+ }
+ if row.is_instance_of::<PyTuple>() || row.is_instance_of::<PyList>() {
+ let fields = if row.hasattr(intern!(py, "__fields__"))? {
+ Some(row.getattr(intern!(py, "__fields__"))?)
+ } else if row.hasattr(intern!(py, "_fields"))? {
+ Some(row.getattr(intern!(py, "_fields"))?)
+ } else {
+ None
+ };
+ let values = row.try_iter()?.collect::<PyResult<Vec<_>>>()?;
+ return match fields {
+ Some(fields) => Ok(fields
+ .try_iter()?
+ .collect::<PyResult<Vec<_>>>()?
+ .into_iter()
+ .zip(values)
+ .collect()),
+ None => Ok(positional(py, values, names)),
+ };
+ }
+ if row.hasattr(intern!(py, "__dict__"))? {
+ return sorted_items(&row.getattr(intern!(py, "__dict__"))?);
+ }
+ Err(raise_pyspark(
+ py,
+ "PySparkTypeError",
+ "CANNOT_INFER_SCHEMA_FOR_TYPE",
+ &[("data_type", &type_name(row)?)],
+ ))
+}
+
+/// The struct type of (name, value) pairs. A `TypeError` in inferring a
value, as from a
+/// type that cannot be inferred or merged, becomes
`CANNOT_INFER_TYPE_FOR_FIELD`.
+pub(crate) fn infer_fields(py: Python<'_>, items: Items<'_>, o: Options<'_>)
-> PyResult<DataType> {
+ let mut fields = Vec::with_capacity(items.len());
+ for (name, value) in items {
+ let data_type = match infer_type(&value, o) {
+ Err(e) if e.is_instance_of::<PyTypeError>(py) => {
+ return Err(raise_pyspark(
+ py,
+ "PySparkTypeError",
+ "CANNOT_INFER_TYPE_FOR_FIELD",
+ &[("field_name", &name.str()?.to_cow()?)],
+ ));
+ }
+ result => result?,
+ };
+ fields.push(field_of(&name, data_type)?);
+ }
+ Ok(DataType::Struct { fields })
+}
+
+/// The struct type of a row, as PySpark's `_infer_schema`.
+pub(crate) fn infer_schema(
+ row: &Bound<'_, PyAny>,
+ names: Option<&mut Vec<String>>,
+ o: Options<'_>,
+) -> PyResult<DataType> {
+ let items = row_items(row, names)?;
+ infer_fields(row.py(), items, o)
+}
+
+/// Whether PySpark's class of `dt` is an `AtomicType`.
+pub(crate) fn is_atomic(dt: &DataType) -> bool {
+ !matches!(
+ dt,
+ DataType::Null
+ | DataType::CalendarInterval
+ | DataType::Array { .. }
+ | DataType::Map { .. }
+ | DataType::Struct { .. }
+ | DataType::Udt { .. }
+ | DataType::Unparsed { .. }
+ )
+}
+
+/// The name of PySpark's class of `dt`, by which its errors name a type.
+pub(crate) fn class_name(dt: &DataType) -> String {
+ let name = match dt {
+ DataType::Null => "NullType",
+ DataType::Boolean => "BooleanType",
+ DataType::Byte => "ByteType",
+ DataType::Short => "ShortType",
+ DataType::Integer => "IntegerType",
+ DataType::Long => "LongType",
+ DataType::Float => "FloatType",
+ DataType::Double => "DoubleType",
+ DataType::Decimal { .. } => "DecimalType",
+ DataType::String { .. } => "StringType",
+ DataType::Char { .. } => "CharType",
+ DataType::Varchar { .. } => "VarcharType",
+ DataType::Binary => "BinaryType",
+ DataType::Date => "DateType",
+ DataType::Timestamp => "TimestampType",
+ DataType::TimestampNtz => "TimestampNTZType",
+ DataType::Time { .. } => "TimeType",
+ DataType::TimestampNtzNanos { .. } => "TimestampNTZNanosType",
+ DataType::TimestampLtzNanos { .. } => "TimestampLTZNanosType",
+ DataType::CalendarInterval => "CalendarIntervalType",
+ DataType::YearMonthInterval { .. } => "YearMonthIntervalType",
+ DataType::DayTimeInterval { .. } => "DayTimeIntervalType",
+ DataType::Array { .. } => "ArrayType",
+ DataType::Map { .. } => "MapType",
+ DataType::Struct { .. } => "StructType",
+ DataType::Variant => "VariantType",
+ DataType::Geometry { .. } => "GeometryType",
+ DataType::Geography { .. } => "GeographyType",
+ DataType::Unparsed { .. } => "UnparsedDataType",
+ DataType::Udt { python_class, .. } => {
+ // A user-defined type is its own class, which `pyClass` names in
full.
+ let class = python_class.as_deref().unwrap_or("UserDefinedType");
+ return class.rsplit('.').next().unwrap_or(class).to_string();
+ }
+ };
+ name.to_string()
+}
+
+/// Whether two types are of the same PySpark class, as `_merge_type` compares
them.
+fn same_class(a: &DataType, b: &DataType) -> bool {
+ match (a, b) {
+ (DataType::Udt { .. }, DataType::Udt { .. }) => class_name(a) ==
class_name(b),
+ _ => std::mem::discriminant(a) == std::mem::discriminant(b),
+ }
+}
+
+/// The merge of two inferred types, as PySpark's `_merge_type`: a null type
takes the
+/// other, a string absorbs another atomic type, and structs, arrays and maps
merge by
+/// their parts. Other types must be of the same class, and the first one is
taken.
+pub(crate) fn merge_type(py: Python<'_>, a: DataType, b: DataType) ->
PyResult<DataType> {
+ use DataType as D;
+ match (&a, &b) {
+ (D::Null, _) => return Ok(b),
+ (_, D::Null) => return Ok(a),
+ (D::Timestamp, D::TimestampNtz) => return Ok(a),
+ (D::TimestampNtz, D::Timestamp) => return Ok(b),
+ (D::Geometry { srid: x }, D::Geometry { srid: y }) if x != y => {
+ return Ok(D::Geometry {
+ srid: GeometryType::MIXED_SRID,
+ });
+ }
+ (D::Geography { srid: x }, D::Geography { srid: y }) if x != y => {
+ return Ok(D::Geography {
+ srid: GeographyType::MIXED_SRID,
+ });
+ }
+ _ => {}
+ }
+ if is_atomic(&a) && matches!(b, D::String { .. }) {
+ return Ok(b);
+ }
+ if matches!(a, D::String { .. }) && is_atomic(&b) {
+ return Ok(a);
+ }
+ if !same_class(&a, &b) {
+ return Err(raise_pyspark(
+ py,
+ "PySparkTypeError",
+ "CANNOT_MERGE_TYPE",
+ &[
+ ("data_type1", &class_name(&a)),
+ ("data_type2", &class_name(&b)),
+ ],
+ ));
+ }
+ Ok(match (a, b) {
+ (D::Struct { fields: a }, D::Struct { fields: b }) => {
+ // `dict((f.name, f.dataType) for f in b.fields)`: each name at
its first place,
+ // with the type of its last field.
+ let mut index: HashMap<String, usize> =
HashMap::with_capacity(b.len());
+ let mut others: Vec<(String, DataType)> =
Vec::with_capacity(b.len());
+ for f in b {
+ match index.get(&f.name) {
+ Some(&i) => others[i].1 = f.data_type,
+ None => {
+ index.insert(f.name.clone(), others.len());
+ others.push((f.name, f.data_type));
+ }
+ }
+ }
+ let mut names: HashSet<String> = HashSet::with_capacity(a.len());
+ let mut fields = Vec::with_capacity(a.len() + others.len());
+ for f in a {
+ let other = match index.get(&f.name) {
+ Some(&i) => others[i].1.clone(),
+ None => D::Null,
+ };
+ names.insert(f.name.clone());
+ fields.push(nullable_field(f.name, merge_type(py, f.data_type,
other)?));
+ }
+ for (name, data_type) in others {
+ if !names.contains(&name) {
+ fields.push(nullable_field(name, data_type));
+ }
+ }
+ D::Struct { fields }
+ }
+ (
+ D::Array {
+ element_type: x, ..
+ },
+ D::Array {
+ element_type: y, ..
+ },
+ ) => D::Array {
+ element_type: Box::new(merge_type(py, *x, *y)?),
+ contains_null: true,
+ },
+ (
+ D::Map {
+ key_type: ka,
+ value_type: va,
+ ..
+ },
+ D::Map {
+ key_type: kb,
+ value_type: vb,
+ ..
+ },
+ ) => map_type(merge_type(py, *ka, *kb)?, merge_type(py, *va, *vb)?),
+ (a, _) => a,
+ })
+}
+
+/// Whether `dt` has a null type in it, which PySpark cannot infer data of.
+pub(crate) fn has_null_type(dt: &DataType) -> bool {
+ match dt {
+ DataType::Struct { fields } => fields.iter().any(|f|
has_null_type(&f.data_type)),
+ DataType::Array { element_type, .. } => has_null_type(element_type),
+ DataType::Map {
+ key_type,
+ value_type,
+ ..
+ } => has_null_type(key_type) || has_null_type(value_type),
+ other => matches!(other, DataType::Null),
+ }
+}
+
+/// A PySpark error of `class` with a message alone, as PySpark raises some of
them.
+fn pyspark_message(py: Python<'_>, class: &str, message: &str) -> PyErr {
+ let build = || -> PyResult<PyErr> {
+ let cls = py.import("pyspark.errors")?.getattr(class)?;
+ Ok(PyErr::from_value(cls.call1((message,))?))
+ };
+ build().unwrap_or_else(|e| e)
+}
+
+fn data_type_str(py: Python<'_>, dt: &DataType) -> PyResult<String> {
+ Ok(crate::types::data_type_to_py(py, dt)?
+ .bind(py)
+ .str()?
+ .to_cow()?
+ .into_owned())
+}
+
+fn must_not_be_none(py: Python<'_>, dt: &DataType) -> PyResult<PyErr> {
+ Ok(pyspark_message(
+ py,
+ "PySparkValueError",
+ &format!("input for {} must not be None", data_type_str(py, dt)?),
+ ))
+}
+
+pub(crate) fn axis_length_mismatch(py: Python<'_>, expected: usize, actual:
usize) -> PyErr {
+ raise_pyspark(
+ py,
+ "PySparkValueError",
+ "AXIS_LENGTH_MISMATCH",
+ &[
+ ("expected_length", &expected.to_string()),
+ ("actual_length", &actual.to_string()),
+ ],
+ )
+}
+
+/// What PySpark's `assert` raises, with no message.
+fn assertion() -> PyErr {
+ PyAssertionError::new_err(())
+}
+
+/// The error of a value that this client cannot send in a column of its type
yet.
+fn unsupported(obj: &Bound<'_, PyAny>) -> PyResult<PyErr> {
+ Ok(PyTypeError::new_err(format!(
+ "unsupported value type for createDataFrame: {}",
+ type_name(obj)?
+ )))
+}
+
+/// The error of a value that PyArrow cannot convert to the type it is named
by.
+fn arrow_invalid(obj: &Bound<'_, PyAny>, arrow_type: &str) -> PyResult<PyErr> {
+ Ok(PyValueError::new_err(format!(
+ "Could not convert {} with type {}: tried to convert to {arrow_type}",
+ obj.repr()?,
+ type_name(obj)?
+ )))
+}
+
+/// Microseconds of a `timedelta`, exactly.
+fn delta_micros(delta: &Bound<'_, PyAny>) -> PyResult<i64> {
+ let py = delta.py();
+ let days: i64 = delta.getattr(intern!(py, "days"))?.extract()?;
+ let seconds: i64 = delta.getattr(intern!(py, "seconds"))?.extract()?;
+ let micros: i64 = delta.getattr(intern!(py, "microseconds"))?.extract()?;
+ Ok(days * 86_400_000_000 + seconds * 1_000_000 + micros)
+}
+
+/// Microseconds since the epoch of the instant of a datetime, which PySpark
takes with
+/// `astimezone(timezone.utc)`: a naive datetime is in the local time zone.
+fn instant_micros(obj: &Bound<'_, PyAny>) -> PyResult<i64> {
+ let py = obj.py();
+ let datetime = py.import(intern!(py, "datetime"))?;
+ let utc = datetime
+ .getattr(intern!(py, "timezone"))?
+ .getattr(intern!(py, "utc"))?;
+ let aware = obj.call_method1(intern!(py, "astimezone"), (&utc,))?;
+ let epoch = DATETIME
+ .import(py, "datetime", "datetime")?
+ .call((1970, 1, 1), Some(&[("tzinfo", &utc)].into_py_dict(py)?))?;
+ delta_micros(&aware.sub(epoch)?)
+}
+
+/// Microseconds since 1970-01-01 00:00 of the wall-clock time of a naive
datetime, as
+/// PyArrow takes a TIMESTAMP_NTZ value.
+fn wall_clock_micros(obj: &Bound<'_, PyAny>) -> PyResult<i64> {
+ let py = obj.py();
+ let epoch = DATETIME
+ .import(py, "datetime", "datetime")?
+ .call1((1970, 1, 1))?;
+ delta_micros(&obj.sub(epoch)?)
+}
+
+/// A field value of `dt` from `obj`, as PySpark's converters and then PyArrow
convert it.
+pub(crate) fn convert(obj: &Bound<'_, PyAny>, dt: &DataType, nullable: bool)
-> PyResult<Value> {
+ let py = obj.py();
+ if obj.is_none() {
+ // A null type takes None either way.
+ if !nullable && !matches!(dt, DataType::Null) {
+ return Err(must_not_be_none(py, dt)?);
+ }
+ return Ok(Value::Null);
+ }
+ match dt {
+ DataType::Null => Err(pyspark_message(
+ py,
+ "PySparkValueError",
+ &format!(
+ "input for {} must be None, but got {}",
+ data_type_str(py, dt)?,
+ obj.str()?
+ ),
+ )),
+ DataType::Struct { fields } => convert_struct(obj, fields),
+ DataType::Array {
+ element_type,
+ contains_null,
+ } => {
+ if !obj.is_instance_of::<PyList>()
+ && !obj.is_instance(ARRAY.import(py, "array",
"array")?.as_any())?
+ {
+ return Err(assertion());
+ }
+ let mut items = Vec::with_capacity(obj.len()?);
+ for item in obj.try_iter()? {
+ items.push(convert(&item?, element_type, *contains_null)?);
+ }
+ Ok(Value::List(items))
+ }
+ DataType::Map {
+ key_type,
+ value_type,
+ value_contains_null,
+ } => {
+ let dict = obj.cast::<PyDict>().map_err(|_| assertion())?;
+ let mut entries = Vec::with_capacity(dict.len());
+ for (key, value) in dict.iter() {
+ entries.push((
+ convert(&key, key_type, false)?,
+ convert(&value, value_type, *value_contains_null)?,
+ ));
+ }
+ Ok(Value::Map(entries))
+ }
+ DataType::Binary => {
+ if let Ok(bytes) = obj.cast::<PyBytes>() {
+ Ok(Value::Binary(bytes.as_bytes().to_vec()))
+ } else if let Ok(bytes) = obj.cast::<PyByteArray>() {
+ Ok(Value::Binary(bytes.to_vec()))
+ } else {
+ Err(assertion())
+ }
+ }
+ DataType::Timestamp | DataType::TimestampLtzNanos { .. } => {
+ if !obj.is_instance(DATETIME.import(py, "datetime",
"datetime")?.as_any())? {
+ return Err(assertion());
+ }
+ Ok(Value::Timestamp(instant_micros(obj)?))
+ }
+ DataType::TimestampNtz | DataType::TimestampNtzNanos { .. } => {
+ if !obj.is_instance(DATETIME.import(py, "datetime",
"datetime")?.as_any())?
+ || !obj.getattr(intern!(py, "tzinfo"))?.is_none()
+ {
+ return Err(assertion());
+ }
+ Ok(Value::Timestamp(wall_clock_micros(obj)?))
+ }
+ DataType::Decimal { precision, scale } => {
+ let decimal = DECIMAL.import(py, "decimal", "Decimal")?;
+ if !obj.is_instance(decimal.as_any())? {
+ return Err(assertion());
+ }
+ if obj.call_method0(intern!(py, "is_nan"))?.is_truthy()? {
+ if !nullable {
+ return Err(must_not_be_none(py, dt)?);
+ }
+ return Ok(Value::Null);
+ }
+ // `value.quantize(Decimal("1E-<scale>"),
context=Context(prec=<precision>,
+ // rounding=ROUND_HALF_EVEN))`, written without an exponent.
+ let module = py.import(intern!(py, "decimal"))?;
+ let exp = decimal.call1((format!("1E-{scale}"),))?;
+ let context = module.getattr(intern!(py, "Context"))?.call(
+ (),
+ Some(
+ &[
+ ("prec", precision.into_pyobject(py)?.into_any()),
+ ("rounding", module.getattr(intern!(py,
"ROUND_HALF_EVEN"))?),
+ ]
+ .into_py_dict(py)?,
+ ),
+ )?;
+ let quantized = obj.call_method(
+ intern!(py, "quantize"),
+ (exp,),
+ Some(&[("context", context)].into_py_dict(py)?),
+ )?;
+ let text =
+ PyString::new(py, "{:f}").call_method1(intern!(py, "format"),
(quantized,))?;
+ Ok(Value::Decimal {
+ value: text.extract()?,
+ precision: Some(*precision),
+ scale: Some(*scale),
+ })
+ }
+ DataType::String { .. } => {
+ let text = if obj.is_exact_instance_of::<PyString>() {
+ obj.cast::<PyString>()?.to_cow()?.into_owned()
+ } else if obj.is_instance_of::<PyBool>() {
+ // As PySpark Classic, which converts a bool to a string in
the JVM.
+ (if obj.is_truthy()? { "true" } else { "false" }).to_string()
+ } else {
+ obj.str()?.to_cow()?.into_owned()
+ };
+ Ok(Value::String(text))
+ }
+ DataType::Char { .. } | DataType::Varchar { .. } => {
+ if let Ok(text) = obj.cast::<PyString>() {
+ Ok(Value::String(text.to_cow()?.into_owned()))
+ } else if let Ok(bytes) = obj.cast::<PyBytes>() {
+ Ok(Value::String(
+ String::from_utf8(bytes.as_bytes().to_vec())
+ .map_err(|e| PyValueError::new_err(e.to_string()))?,
+ ))
+ } else {
+ Err(PyTypeError::new_err(format!(
+ "Expected bytes, got a '{}' object",
+ type_name(obj)?
+ )))
+ }
+ }
+ DataType::Boolean => {
+ if obj.is_instance_of::<PyBool>() {
+ Ok(Value::Bool(obj.is_truthy()?))
+ } else {
+ Err(arrow_invalid(obj, "boolean")?)
+ }
+ }
+ DataType::Byte | DataType::Short | DataType::Integer | DataType::Long
=> {
+ convert_integral(obj, dt)
+ }
+ DataType::Float | DataType::Double => {
+ // PyArrow takes a float, an int or a bool.
+ if obj.is_instance_of::<PyFloat>() ||
obj.is_instance_of::<PyInt>() {
+ let value: f64 = obj.extract()?;
+ Ok(match dt {
+ DataType::Float => Value::Float(value as f32),
+ _ => Value::Double(value),
+ })
+ } else {
+ let arrow_type = if matches!(dt, DataType::Float) {
+ "float"
+ } else {
+ "double"
+ };
+ Err(arrow_invalid(obj, arrow_type)?)
+ }
+ }
+ DataType::Date => {
+ // PyArrow takes a date, or the date of a datetime.
+ if !obj.is_instance(DATE.import(py, "datetime",
"date")?.as_any())? {
+ return Err(arrow_invalid(obj, "date32")?);
+ }
+ let ordinal: i64 = obj.call_method0(intern!(py,
"toordinal"))?.extract()?;
+ Ok(Value::Date((ordinal - 719_163) as i32))
+ }
+ _ => Err(unsupported(obj)?),
+ }
+}
+
+/// An integral value, as PyArrow converts one: an int in range, or a float
truncated.
+fn convert_integral(obj: &Bound<'_, PyAny>, dt: &DataType) -> PyResult<Value> {
+ let (bits, arrow_type) = match dt {
+ DataType::Byte => (8, "int8"),
+ DataType::Short => (16, "int16"),
+ DataType::Integer => (32, "int32"),
+ _ => (64, "int64"),
+ };
+ if obj.is_instance_of::<PyBool>() {
+ return Err(PyTypeError::new_err("Expected integer, got bool"));
+ }
+ let value: i64 = if obj.is_instance_of::<PyInt>() {
+ obj.extract()?
+ } else if obj.is_instance_of::<PyFloat>() {
+ let value: f64 = obj.extract()?;
+ // The range of i64 is [-2^63, 2^63), both bounds exact as f64.
+ if !(i64::MIN as f64..i64::MAX as f64).contains(&value.trunc()) {
+ return Err(arrow_invalid(obj, arrow_type)?);
+ }
+ value.trunc() as i64
+ } else {
+ return Err(arrow_invalid(obj, arrow_type)?);
+ };
+ if bits < 64 && !(-(1i64 << (bits - 1))..(1i64 << (bits -
1))).contains(&value) {
+ return Err(PyValueError::new_err(format!(
+ "Value {value} too large to fit in C integer type"
+ )));
+ }
+ Ok(match dt {
+ DataType::Byte => Value::Byte(value as i8),
+ DataType::Short => Value::Short(value as i16),
+ DataType::Integer => Value::Integer(value as i32),
+ _ => Value::Long(value),
+ })
+}
+
+/// The values of `fields` from a mapping, by their names, as
`mapping.get(name)`.
+fn values_by_name<'py>(
+ mapping: &Bound<'py, PyAny>,
+ fields: &[StructField],
+) -> PyResult<Vec<Bound<'py, PyAny>>> {
+ let py = mapping.py();
+ if let Ok(dict) = mapping.cast::<PyDict>() {
+ return fields
+ .iter()
+ .map(|f| {
+ Ok(dict
+ .get_item(&f.name)?
+ .unwrap_or_else(|| py.None().into_bound(py)))
+ })
+ .collect();
+ }
+ fields
+ .iter()
+ .map(|f| mapping.call_method1(intern!(py, "get"), (&f.name,)))
+ .collect()
+}
+
+/// The values of a tuple or a Row, which must be as many as `fields`.
+fn values_by_position<'py>(
+ py: Python<'py>,
+ values: Vec<Bound<'py, PyAny>>,
+ fields: &[StructField],
+) -> PyResult<Vec<Bound<'py, PyAny>>> {
+ if values.len() == fields.len() {
+ Ok(values)
+ } else {
+ Err(axis_length_mismatch(py, fields.len(), values.len()))
+ }
+}
+
+/// A struct value, as PySpark converts one: a tuple or a Row by position, a
dict by name,
+/// and another object by the names of its `__dict__`.
+fn convert_struct(obj: &Bound<'_, PyAny>, fields: &[StructField]) ->
PyResult<Value> {
+ let py = obj.py();
+ let values = if let Ok(row) = obj.cast::<PyRow>() {
+ let (_, values) = row.borrow().parts(py)?;
+ values_by_position(py, values.iter().collect(), fields)?
+ } else if let Ok(tuple) = obj.cast::<PyTuple>() {
+ values_by_position(py, tuple.iter().collect(), fields)?
+ } else if obj.is_instance_of::<PyDict>() {
+ values_by_name(obj, fields)?
+ } else if let Ok(attributes) = obj.getattr(intern!(py, "__dict__")) {
+ values_by_name(&attributes, fields)?
+ } else {
+ // PySpark asserts that another value has a `__dict__`.
+ return Err(PyAssertionError::new_err(format!(
+ "{} {}",
+ obj.get_type().str()?,
+ obj.str()?
+ )));
+ };
+ struct_of(&values, fields)
+}
+
+fn struct_of(values: &[Bound<'_, PyAny>], fields: &[StructField]) ->
PyResult<Value> {
+ let names: std::sync::Arc<[String]> = fields.iter().map(|f|
f.name.clone()).collect();
+ let values = values
+ .iter()
+ .zip(fields)
+ .map(|(value, f)| convert(value, &f.data_type, f.nullable))
+ .collect::<PyResult<Vec<_>>>()?;
+ Ok(Value::Struct(StructValue::new(names, values)))
+}
+
+/// `dict(sorted(row.items()))`, as PySpark sorts dict rows to respect their
inferred schema.
+pub(crate) fn sorted_dict<'py>(row: &Bound<'py, PyAny>) -> PyResult<Bound<'py,
PyAny>> {
+ let py = row.py();
+ let sorted = SORTED.import(py, "builtins", "sorted")?;
+ let items = sorted.call1((row.call_method0(intern!(py, "items"))?,))?;
+ Ok(py.get_type::<PyDict>().call1((items,))?.into_any())
+}
+
+/// The field of the single column that data of an atomic schema has, as
PySpark names it.
+pub(crate) fn value_field(data_type: DataType) -> StructField {
+ nullable_field("value".to_string(), data_type)
+}
+
+/// The values of a row in the order of `fields`, as PySpark's `to_row` takes
them: nulls for
+/// None, a tuple or a Row by position, a dict by name, another object by its
`__dict__`, and
+/// another value, such as a list, by position.
+pub(crate) fn row_values<'py>(
+ item: &Bound<'py, PyAny>,
+ fields: &[StructField],
+) -> PyResult<Vec<Bound<'py, PyAny>>> {
+ let py = item.py();
+ if item.is_none() {
+ return Ok(vec![py.None().into_bound(py); fields.len()]);
+ }
+ if let Ok(row) = item.cast::<PyRow>() {
+ let (_, values) = row.borrow().parts(py)?;
+ return values_by_position(py, values.iter().collect(), fields);
+ }
+ if let Ok(tuple) = item.cast::<PyTuple>() {
+ return values_by_position(py, tuple.iter().collect(), fields);
+ }
+ if item.is_instance_of::<PyDict>() {
+ return values_by_name(item, fields);
+ }
+ if item.is_instance_of::<PyVariantVal>() {
+ return Err(pyspark_message(
+ py,
+ "PySparkValueError",
+ "Rows cannot be of type VariantVal",
+ ));
+ }
+ if item.hasattr(intern!(py, "__dict__"))? {
+ return values_by_name(&item.getattr(intern!(py, "__dict__"))?, fields);
+ }
+ let length = item.len()?;
+ if length != fields.len() {
+ return Err(axis_length_mismatch(py, fields.len(), length));
+ }
+ item.try_iter()?.collect()
+}
diff --git a/crates/pyspark-rs/src/lib.rs b/crates/pyspark-rs/src/lib.rs
index 8f2ab6b..e5b3c6f 100644
--- a/crates/pyspark-rs/src/lib.rs
+++ b/crates/pyspark-rs/src/lib.rs
@@ -33,6 +33,7 @@ mod errors;
mod eval_type;
mod functions;
mod group;
+mod infer;
mod ml;
mod observation;
mod pipelines;
diff --git a/crates/pyspark-rs/src/row.rs b/crates/pyspark-rs/src/row.rs
index fe43303..53d8b21 100644
--- a/crates/pyspark-rs/src/row.rs
+++ b/crates/pyspark-rs/src/row.rs
@@ -90,6 +90,15 @@ impl PyRow {
}
}
+ /// The field names, if the row has them (`__fields__`), and the values,
as the tuple
+ /// that a PySpark row is, for reading the row as PySpark's schema
inference does.
+ pub(crate) fn parts<'py>(
+ &self,
+ py: Python<'py>,
+ ) -> PyResult<(Option<&[String]>, Bound<'py, PyTuple>)> {
+ Ok((self.named.then(|| self.fields()), self.values(py)?))
+ }
+
/// The row with Spark values, for the rest of the client, which converts
the values of a
/// row made in Python as it converts those of a tuple.
pub(crate) fn to_row(&self, py: Python<'_>) -> PyResult<Row> {
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 8807b8c..1432ff3 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -2,7 +2,7 @@
use pyo3::prelude::*;
use pyo3::types::{PyDict, PyList};
-use spark_connect::row::{Row, Value};
+use spark_connect::row::{Row, StructValue, Value};
use spark_connect::session::SparkSession;
use spark_connect::types::{DataType, StructField};
use spark_connect::udf::{CommonInlineUserDefinedFunctionExpression,
PythonUDFPayload};
@@ -278,6 +278,268 @@ impl PySparkSession {
pub fn new(session: SparkSession) -> Self {
PySparkSession { session }
}
+
+ /// A DataFrame of local rows, as PySpark Connect creates one: the schema
is given, or
+ /// inferred by merging the types of every row, and each value is
converted to the type
+ /// of its field. A list of names names the inferred fields and is applied
with `toDF`.
+ fn create_from_local(
+ &self,
+ py: Python<'_>,
+ data: &Bound<'_, PyAny>,
+ spec: Spec,
+ ) -> PyResult<PyDataFrame> {
+ use crate::infer;
+
+ let (given, mut cols) = match spec {
+ Spec::None => (None, None),
+ Spec::Names(names) => (None, Some(names)),
+ Spec::Struct(fields) => (Some(fields), None),
+ Spec::Atomic(atomic) => (Some(vec![infer::value_field(atomic)]),
None),
+ };
+ // Empty data with a length needs a schema to type its relation, and
gets no toDF.
+ if data.len().is_ok_and(|n| n == 0) {
+ let Some(fields) = given else {
+ return Err(crate::column::raise_pyspark(
+ py,
+ "PySparkValueError",
+ "CANNOT_INFER_EMPTY_SCHEMA",
+ &[],
+ ));
+ };
+ let schema = DataType::Struct { fields };
+ let df = py
+ .detach(|| {
+ self.session
+ .create_dataframe(Vec::new(), schema)
+ .map_err(Box::new)
+ })
+ .to_pyerr()?;
+ return Ok(PyDataFrame::new(df));
+ }
+ let mut items: Vec<Bound<'_, PyAny>> =
data.try_iter()?.collect::<PyResult<_>>()?;
+ let Some(first) = items.first().cloned() else {
+ // As PySpark takes the first row of data without a length.
+ return Err(pyo3::exceptions::PyIndexError::new_err(
+ "list index out of range",
+ ));
+ };
+ // As PySpark: dict rows are sorted by their keys, and rows that are
none of a Row, a
+ // tuple, a list, a dict or an object, such as 1, are rows of that
single value.
+ let single = if first.is_instance_of::<PyDict>() {
+ for item in items.iter_mut() {
+ if !item.is_none() {
+ *item = infer::sorted_dict(item)?;
+ }
+ }
+ false
+ } else {
+ !(first.is_instance_of::<crate::row::PyRow>()
+ || first.is_instance_of::<pyo3::types::PyTuple>()
+ || first.is_instance_of::<PyList>()
+ || first.hasattr("__dict__")?)
+ };
+
+ let fields = match given {
+ Some(fields) => fields,
+ None => {
+ // The configurations are read only when a value depends on
them.
+ let confs = infer::LazyConfs::new(self.session.conf());
+ let options = infer::Options::new(&confs);
+ let mut merged: Option<DataType> = None;
+ for item in &items {
+ let row_type = if single {
+ let items = infer::positional(py, vec![item.clone()],
cols.as_mut());
+ infer::infer_fields(py, items, options)?
+ } else {
+ infer::infer_schema(item, cols.as_mut(), options)?
+ };
+ merged = Some(match merged {
+ None => row_type,
+ Some(merged) => infer::merge_type(py, merged,
row_type)?,
+ });
+ }
+ match merged {
+ Some(schema) if infer::has_null_type(&schema) => {
+ return Err(crate::column::raise_pyspark(
+ py,
+ "PySparkValueError",
+ "CANNOT_DETERMINE_TYPE",
+ &[],
+ ));
+ }
+ Some(DataType::Struct { fields }) => fields,
+ _ => unreachable!("rows are inferred as structs"),
+ }
+ }
+ };
+
+ // As PySpark: every row is taken apart first, then each column is
converted.
+ let mut parts: Vec<Vec<Bound<'_, PyAny>>> =
Vec::with_capacity(items.len());
+ for item in &items {
+ parts.push(if single {
+ if fields.len() != 1 {
+ return Err(infer::axis_length_mismatch(py, fields.len(),
1));
+ }
+ vec![item.clone()]
+ } else {
+ infer::row_values(item, &fields)?
+ });
+ }
+ let mut values: Vec<Vec<Value>> = (0..parts.len())
+ .map(|_| Vec::with_capacity(fields.len()))
+ .collect();
+ for (i, field) in fields.iter().enumerate() {
+ for (row, out) in parts.iter().zip(values.iter_mut()) {
+ out.push(infer::convert(&row[i], &field.data_type,
field.nullable)?);
+ }
+ }
+ // The names, with those that inference added, must name every column.
+ if let Some(cols) = &cols {
+ if cols.len() != fields.len() {
+ return Err(infer::axis_length_mismatch(py, cols.len(),
fields.len()));
+ }
+ }
+ let names: std::sync::Arc<[String]> = fields.iter().map(|f|
f.name.clone()).collect();
+ let rows: Vec<Row> = values
+ .into_iter()
+ .map(|values| StructValue::new(names.clone(), values).into())
+ .collect();
+ let schema = DataType::Struct { fields };
+ let df = py
+ .detach(|| {
+ self.session
+ .create_dataframe(rows, schema)
+ .map_err(Box::new)
+ })
+ .to_pyerr()?;
+ // As pyspark, column names are applied with toDF. The local relation
already carries
+ // them, so this only changes the plan, which the server resolves into
a Project that
+ // is not local; it keeps isLocal() as in pyspark and is not redundant.
+ Ok(PyDataFrame::new(match cols.take() {
+ Some(cols) if !cols.is_empty() =>
df.to_df(cols.iter().map(String::as_str).collect()),
+ _ => df,
+ }))
+ }
+
+ /// A DataFrame of a pandas DataFrame: its rows as lists with NaN and NaT
as None, typed
+ /// by the first non-null value of each column, and its column labels, as
strings, or the
+ /// given names, applied with `toDF`.
+ fn create_from_pandas(
+ &self,
+ py: Python<'_>,
+ data: &Bound<'_, PyAny>,
+ spec: Spec,
+ ) -> PyResult<PyDataFrame> {
+ let mut columns = Vec::new();
+ for label in data.getattr("columns")?.try_iter()? {
+ columns.push(label?.str()?.extract::<String>()?);
+ }
+ // df.astype(object).where(df.notna(), None).values.tolist() -> rows
with None for NA.
+ let notna = data.call_method0("notna")?;
+ let obj = data.call_method1("astype", ("object",))?;
+ let replaced = obj.call_method1("where", (notna, py.None()))?;
+ let values = replaced.getattr("values")?.call_method0("tolist")?;
+ let data = values.cast_into::<PyList>()?;
+ let empty = data.is_empty();
+ let cannot_infer_empty = || {
+ crate::column::raise_pyspark(py, "PySparkValueError",
"CANNOT_INFER_EMPTY_SCHEMA", &[])
+ };
+ // The given fields, and the names that toDF applies, which empty data
does not get.
+ let (given, names) = match spec {
+ Spec::Atomic(atomic) if !empty => {
+ // pyspark reports str() of the parsed type, which is its repr.
+ let data_type = crate::types::data_type_to_py(py, &atomic)?
+ .bind(py)
+ .repr()?
+ .extract::<String>()?;
+ return Err(crate::column::raise_pyspark(
+ py,
+ "PySparkTypeError",
+ "UNSUPPORTED_DATA_TYPE_FOR_ARROW",
+ &[("data_type", &data_type)],
+ ));
+ }
+ Spec::Atomic(atomic) =>
(Some(vec![crate::infer::value_field(atomic)]), None),
+ Spec::Struct(fields) => {
+ let names = fields.iter().map(|f| f.name.clone()).collect();
+ (Some(fields), (!empty).then_some(names))
+ }
+ _ if empty => return Err(cannot_infer_empty()),
+ Spec::None if columns.is_empty() => return
Err(cannot_infer_empty()),
+ Spec::None => (None, Some(columns)),
+ Spec::Names(mut names) => {
+ // Extended with `_2`, `_3`, ... for more columns, as pyspark
does.
+ let width = columns.len();
+ let start = names.len() + 1;
+ names.extend((start..=width).map(|i| format!("_{i}")));
+ (None, Some(names))
+ }
+ };
+ let row_names: Vec<String> = match (&given, &names) {
+ (Some(fields), _) => fields.iter().map(|f|
f.name.clone()).collect(),
+ (None, Some(names)) => names.clone(),
+ (None, None) => Vec::new(),
+ };
+ let mut rows: Vec<Row> = Vec::with_capacity(data.len());
+ for row in data.iter() {
+ let values = row
+ .try_iter()?
+ .map(|v| py_to_value(&v?))
+ .collect::<PyResult<Vec<_>>>()?;
+ if values.len() != row_names.len() {
+ // As pyspark, which checks every row against the width of the
schema.
+ return Err(crate::infer::axis_length_mismatch(
+ py,
+ row_names.len(),
+ values.len(),
+ ));
+ }
+ rows.push(Row::new(row_names.clone(), values));
+ }
+ // A given struct wins; otherwise each field takes the type of its
first non-null
+ // value (empty input has a struct schema, checked above).
+ let fields = match given {
+ Some(fields) => fields,
+ None => {
+ let mut fields: Vec<StructField> =
Vec::with_capacity(row_names.len());
+ for (i, name) in row_names.iter().enumerate() {
+ let Some(value) = rows
+ .iter()
+ .filter_map(|row| row.get(i))
+ .find(|value| !matches!(value, Value::Null))
+ else {
+ return Err(crate::column::raise_pyspark(
+ py,
+ "PySparkValueError",
+ "CANNOT_DETERMINE_TYPE",
+ &[],
+ ));
+ };
+ fields.push(StructField {
+ name: name.clone(),
+ data_type: value_to_datatype(value),
+ nullable: true,
+ metadata: Default::default(),
+ });
+ }
+ fields
+ }
+ };
+ let schema = DataType::Struct { fields };
+ let df = py
+ .detach(|| {
+ self.session
+ .create_dataframe(rows, schema)
+ .map_err(Box::new)
+ })
+ .to_pyerr()?;
+ Ok(PyDataFrame::new(match names {
+ Some(names) if !names.is_empty() => {
+ df.to_df(names.iter().map(String::as_str).collect())
+ }
+ _ => df,
+ }))
+ }
}
/// Whether `obj` is an instance of `module.class`, without importing `module`
when it is not
@@ -296,6 +558,59 @@ fn is_loaded_instance(
}
}
+/// A `createDataFrame` schema, as PySpark Connect reads one.
+enum Spec {
+ None,
+ /// Column names, which name the inferred fields and are applied with
`toDF`.
+ Names(Vec<String>),
+ /// The fields of a struct type, or of a DDL string of one.
+ Struct(Vec<StructField>),
+ /// An atomic type, of the single column `value`.
+ Atomic(DataType),
+}
+
+/// The schema argument of `createDataFrame`: None, a list or tuple of names,
or a struct or
+/// atomic type, given as a type or a DDL string. Other types raise
`NOT_EXPECTED_TYPE`.
+fn schema_spec(py: Python<'_>, schema: Option<&Bound<'_, PyAny>>) ->
PyResult<Spec> {
+ let Some(schema) = schema.filter(|s| !s.is_none()) else {
+ return Ok(Spec::None);
+ };
+ let not_expected = |arg_type: &str| {
+ crate::column::raise_pyspark(
+ py,
+ "PySparkTypeError",
+ "NOT_EXPECTED_TYPE",
+ &[
+ ("arg_name", "schema"),
+ ("expected_type", "list, None or StructType"),
+ ("arg_type", arg_type),
+ ],
+ )
+ };
+ if schema.is_instance_of::<PyList>() ||
schema.is_instance_of::<pyo3::types::PyTuple>() {
+ let mut names = Vec::new();
+ for name in schema.try_iter()? {
+ let name = name?;
+ names.push(match name.cast::<pyo3::types::PyString>() {
+ Ok(name) => name.to_cow()?.into_owned(),
+ // PySpark encodes a name that is not a string, which most
values cannot do.
+ Err(_) => name.call_method1("encode",
("utf-8",))?.str()?.to_string(),
+ });
+ }
+ return Ok(Spec::Names(names));
+ }
+ if !schema.is_instance_of::<pyo3::types::PyString>()
+ && !schema.is_instance_of::<crate::types::PyDataType>()
+ {
+ return Err(not_expected(&crate::infer::type_name(schema)?));
+ }
+ Ok(match crate::types::py_to_data_type(schema)? {
+ DataType::Struct { fields } => Spec::Struct(fields),
+ atomic if crate::infer::is_atomic(&atomic) => Spec::Atomic(atomic),
+ other => return Err(not_expected(&crate::infer::class_name(&other))),
+ })
+}
+
#[pymethods]
impl PySparkSession {
/// The session builder, reached as `SparkSession.builder` (class
attribute,
@@ -381,319 +696,53 @@ impl PySparkSession {
/// Create a DataFrame from local data (a list of rows, a pandas
DataFrame, or a pyarrow
/// Table) and an optional schema, as in pyspark.
///
- /// Column names follow pyspark: without a schema, plain rows are named
`_1`, `_2`, ...
- /// and Rows and dicts keep their own names. A list of names, extended
with `_2`, `_3`,
- /// ... for wider plain rows, and the columns of a pandas DataFrame (or
the field names
- /// of its schema) are applied with `toDF`. Without a struct schema, the
field types are
- /// inferred from the first row.
+ /// Local rows are inferred, converted and named as PySpark Connect does
it, in
+ /// `crate::infer`. The columns of a pandas DataFrame, or the field names
of its schema,
+ /// are applied with `toDF`, and its fields take the type of their first
non-null value.
#[pyo3(signature = (data, schema=None, samplingRatio=None,
verifySchema=None))]
fn createDataFrame(
&self,
py: Python<'_>,
data: &Bound<'_, PyAny>,
schema: Option<&Bound<'_, PyAny>>,
- samplingRatio: Option<f64>,
- verifySchema: Option<bool>,
+ samplingRatio: Option<&Bound<'_, PyAny>>,
+ verifySchema: Option<&Bound<'_, PyAny>>,
) -> PyResult<PyDataFrame> {
- use crate::row::PyRow;
-
- // `samplingRatio` only affects RDD-based schema inference in
reference PySpark and is
- // unused for local data over Spark Connect (reference ignores it here
too).
- // `verifySchema` toggles client-side row/type verification; this path
builds the plan
- // directly and does not perform that extra verification, so both are
accepted for
- // signature parity. See PARITY_GAP_ANALYSIS.md.
- let _ = (samplingRatio, verifySchema);
-
- // A caller holding a Table has already imported PyArrow.
- if is_loaded_instance(py, data, "pyarrow", "Table")? {
- return crate::pyarrow_import::create_dataframe(py, &self.session,
data, schema);
- }
-
- // A pandas DataFrame (as pyspark does): its rows as lists with
NaN/NaT mapped to None,
- // and its column labels, as strings, which are applied below.
- let mut pandas_columns: Option<Vec<String>> = None;
- let is_pandas = py
- .import("pandas")
- .and_then(|m| m.getattr("DataFrame"))
- .and_then(|c| data.is_instance(&c))
- .unwrap_or(false);
- let data_owned: Bound<'_, PyList>;
- let data: &Bound<'_, PyList> = if is_pandas {
- let mut labels = Vec::new();
- for label in data.getattr("columns")?.try_iter()? {
- labels.push(label?.str()?.extract::<String>()?);
- }
- pandas_columns = Some(labels);
- // df.astype(object).where(df.notna(), None).values.tolist() ->
rows with None for NA.
- let notna = data.call_method0("notna")?;
- let obj = data.call_method1("astype", ("object",))?;
- let replaced = obj.call_method1("where", (notna, py.None()))?;
- let values = replaced.getattr("values")?.call_method0("tolist")?;
- data_owned = values.cast_into::<PyList>()?;
- &data_owned
- } else {
- data.cast::<PyList>()?
- };
-
- // Resolve the schema: a list of column names (types inferred), a DDL
string or
- // a StructType (names + types explicit), or None (names from Row /
default).
- enum Spec {
- None,
- Names(Vec<String>),
- Struct(Vec<StructField>),
+ if data.is_none() {
+ return Err(pyo3::exceptions::PyAssertionError::new_err(()));
}
- let mut atomic_schema: Option<DataType> = None;
- let spec = match schema {
- None => Spec::None,
- Some(s) => {
- if let Ok(names) = s.extract::<Vec<String>>() {
- Spec::Names(names)
- } else {
- // Any DataType object (StructType, PyDataType) or a DDL
string,
- // resolved through the shared converter. A struct becomes
the
- // schema directly; a scalar AtomicType becomes a single
"value"
- // column (matching pyspark createDataFrame(data,
AtomicType())).
- match crate::types::py_to_data_type(s)? {
- DataType::Struct { fields } => Spec::Struct(fields),
- atomic => {
- atomic_schema = Some(atomic.clone());
-
Spec::Struct(vec![spark_connect::types::StructField {
- name: "value".to_string(),
- data_type: atomic,
- nullable: true,
- metadata: std::collections::BTreeMap::new(),
- }])
- }
- }
- }
- }
- };
-
- // As pyspark, empty input needs a schema to type its relation, and
gets no toDF.
- let empty = data.is_empty();
- if empty && !matches!(spec, Spec::Struct(_)) {
+ if data.is_instance_of::<PyDataFrame>() {
return Err(crate::column::raise_pyspark(
py,
- "PySparkValueError",
- "CANNOT_INFER_EMPTY_SCHEMA",
- &[],
+ "PySparkTypeError",
+ "INVALID_TYPE",
+ &[("arg_name", "data"), ("arg_type", "DataFrame")],
));
}
- if !empty {
- if let Some(columns) = &pandas_columns {
- if let Some(atomic) = &atomic_schema {
- // pyspark reports str() of the parsed type, which is its
repr.
- let data_type = crate::types::data_type_to_py(py, atomic)?
- .bind(py)
- .repr()?
- .extract::<String>()?;
- return Err(crate::column::raise_pyspark(
- py,
- "PySparkTypeError",
- "UNSUPPORTED_DATA_TYPE_FOR_ARROW",
- &[("data_type", &data_type)],
- ));
- }
- if matches!(spec, Spec::None) && columns.is_empty() {
- return Err(crate::column::raise_pyspark(
- py,
- "PySparkValueError",
- "CANNOT_INFER_EMPTY_SCHEMA",
- &[],
- ));
- }
- }
- }
-
- // How a row carries its values, as in pyspark: positionally (lists,
tuples and
- // scalars), positionally with names of its own (Rows, and namedtuples
by their
- // `_fields`), or by name (dicts, and other objects through their
`__dict__`, whose
- // keys are sorted as `_infer_schema` does).
- enum Item {
- Plain(Vec<Value>),
- Named(Vec<String>, Vec<Value>),
- Record(Vec<(String, Value)>),
- }
- let record = |dict: &Bound<'_, PyDict>| -> PyResult<Item> {
- let mut pairs: Vec<(String, Value)> =
Vec::with_capacity(dict.len());
- for (k, v) in dict.iter() {
- pairs.push((k.str()?.to_string(), py_to_value(&v)?));
+ // As PySpark Connect, which ignores both.
+ for (name, value) in [
+ ("samplingRatio", samplingRatio),
+ ("verifySchema", verifySchema),
+ ] {
+ if value.is_some_and(|v| !v.is_none()) {
+ py.import("warnings")?.call_method1(
+ "warn",
+ (format!(
+ "'{name}' is ignored. It is not supported with Spark
Connect."
+ ),),
+ )?;
}
- pairs.sort_by(|a, b| a.0.cmp(&b.0));
- Ok(Item::Record(pairs))
- };
- let mut items: Vec<Item> = Vec::with_capacity(data.len());
- for item in data.iter() {
- items.push(if let Ok(dict) = item.cast::<PyDict>() {
- record(dict)?
- } else if let Ok(pyrow) = item.extract::<PyRef<PyRow>>() {
- let row = pyrow.to_row(py)?;
- Item::Named(row.fields().to_vec(), row.values().to_vec())
- } else if let Ok(lst) = item.cast::<PyList>() {
- Item::Plain(
- lst.iter()
- .map(|v| py_to_value(&v))
- .collect::<PyResult<_>>()?,
- )
- } else if let Ok(tuple) = item.cast::<pyo3::types::PyTuple>() {
- let values = tuple
- .iter()
- .map(|v| py_to_value(&v))
- .collect::<PyResult<_>>()?;
- match tuple
- .getattr("_fields")
- .and_then(|fields| fields.extract::<Vec<String>>())
- {
- Ok(fields) => Item::Named(fields, values),
- Err(_) => Item::Plain(values),
- }
- } else if let Some(attributes) = item
- .getattr("__dict__")
- .ok()
- .and_then(|attributes| attributes.cast_into::<PyDict>().ok())
- {
- record(&attributes)?
- } else {
- // A scalar row (int/str/float/...): a single-field row,
matching
- // `createDataFrame([1, 2, 3], IntegerType())`. The field name
comes from
- // the schema when given, else defaults below.
- Item::Plain(vec![py_to_value(&item)?])
- });
}
- // Plain rows are as wide as the widest of them: pyspark's schema
inference names
- // them `_1`, `_2`, ..., and extends a shorter list of names the same
way.
- let plain_width = items
- .iter()
- .filter_map(|item| match item {
- Item::Plain(values) => Some(values.len()),
- _ => None,
- })
- .max()
- .unwrap_or(0);
- // The keys of the dict rows, in the order pyspark merges their
schemas: each row's
- // sorted keys that no earlier row had.
- let mut record_names: Vec<String> = Vec::new();
- let mut seen = std::collections::HashSet::new();
- for item in &items {
- if let Item::Record(pairs) = item {
- for (key, _) in pairs {
- if seen.insert(key.as_str()) {
- record_names.push(key.clone());
- }
- }
- }
+ // A caller holding a Table, or a pandas DataFrame, has already
imported its module.
+ if is_loaded_instance(py, data, "pyarrow", "Table")? {
+ return crate::pyarrow_import::create_dataframe(py, &self.session,
data, schema);
}
- let extend = |names: &[String], width: usize| -> Vec<String> {
- let mut names = names.to_vec();
- names.extend((names.len()..width).map(|i| format!("_{}", i + 1)));
- names
- };
- // The names pyspark applies with toDF: a list of names, and the
columns of a pandas
- // DataFrame or the field names of its schema.
- let to_df_names: Option<Vec<String>> = match (&spec, &pandas_columns) {
- _ if empty => None,
- (Spec::Names(names), Some(columns)) => Some(extend(names,
columns.len())),
- (Spec::Names(names), None) => Some(extend(names, plain_width)),
- (Spec::None, Some(columns)) => Some(columns.clone()),
- (Spec::Struct(fields), Some(_)) => {
- Some(fields.iter().map(|f| f.name.clone()).collect())
- }
- (_, None) => None,
- };
- let default_names = extend(&[], plain_width);
- // A dict row's value for `name`, as pyspark looks it up: null when
the key is missing.
- // The pairs are sorted by key, and the first of equal keys is taken.
- let lookup = |pairs: &[(String, Value)], name: &String| {
- let index = pairs.partition_point(|(key, _)| key < name);
- match pairs.get(index) {
- Some((key, value)) if key == name => value.clone(),
- _ => Value::Null,
- }
- };
-
- let mut rows: Vec<Row> = Vec::with_capacity(items.len());
- for item in items {
- let (names, values) = match (&spec, &to_df_names, item) {
- // A struct schema takes dict values by its field names, so
that missing
- // keys are null and other keys are ignored, and other rows by
position.
- (Spec::Struct(fields), _, item) => {
- let names: Vec<String> = fields.iter().map(|f|
f.name.clone()).collect();
- let values = match item {
- Item::Record(pairs) => names.iter().map(|n|
lookup(&pairs, n)).collect(),
- Item::Plain(values) | Item::Named(_, values) => values,
- };
- (names, values)
- }
- (_, names, Item::Record(pairs)) => (
- names.clone().unwrap_or_else(|| record_names.clone()),
- record_names.iter().map(|n| lookup(&pairs, n)).collect(),
- ),
- (_, Some(names), Item::Plain(values) | Item::Named(_, values))
=> {
- (names.clone(), values)
- }
- (_, None, Item::Named(names, values)) => (names, values),
- (_, None, Item::Plain(values)) => (default_names.clone(),
values),
- };
- if names.len() != values.len() {
- // As pyspark, which checks every row against the width of the
schema.
- return Err(crate::column::raise_pyspark(
- py,
- "PySparkValueError",
- "AXIS_LENGTH_MISMATCH",
- &[
- ("expected_length", &names.len().to_string()),
- ("actual_length", &values.len().to_string()),
- ],
- ));
- }
- rows.push(Row::new(names, values));
+ let spec = schema_spec(py, schema)?;
+ if is_loaded_instance(py, data, "pandas", "DataFrame")? {
+ return self.create_from_pandas(py, data, spec);
}
-
- // An explicit struct schema wins; otherwise each field takes the type
of its first
- // non-null value (empty input has a struct schema, checked above).
pyspark merges
- // the types of all rows, which gives the same type when they agree,
and cannot
- // type a field that is null in every row.
- let schema_dtype = match spec {
- Spec::Struct(fields) => DataType::Struct { fields },
- _ => {
- let mut fields: Vec<StructField> =
Vec::with_capacity(rows[0].len());
- for (i, name) in rows[0].fields().iter().enumerate() {
- let Some(value) = rows
- .iter()
- .filter_map(|row| row.get(i))
- .find(|value| !matches!(value, Value::Null))
- else {
- return Err(crate::column::raise_pyspark(
- py,
- "PySparkValueError",
- "CANNOT_DETERMINE_TYPE",
- &[],
- ));
- };
- fields.push(StructField {
- name: name.clone(),
- data_type: value_to_datatype(value),
- nullable: true,
- metadata: Default::default(),
- });
- }
- DataType::Struct { fields }
- }
- };
-
- let df = py
- .detach(|| self.session.create_dataframe(rows, schema_dtype))
- .to_pyerr()?;
- // As pyspark, column names are applied with toDF. The local relation
already carries
- // them, so this only changes the plan, which the server resolves into
a Project that
- // is not local; it keeps isLocal() as in pyspark and is not redundant.
- Ok(PyDataFrame::new(match to_df_names {
- Some(names) if !names.is_empty() => {
- df.to_df(names.iter().map(String::as_str).collect())
- }
- _ => df,
- }))
+ self.create_from_local(py, data, spec)
}
/// Get the catalog API (`spark.catalog`).
diff --git a/crates/spark-connect/src/conf.rs b/crates/spark-connect/src/conf.rs
index d43d611..3227f11 100644
--- a/crates/spark-connect/src/conf.rs
+++ b/crates/spark-connect/src/conf.rs
@@ -62,6 +62,12 @@ impl RuntimeConf {
Ok(results.into_iter().next().flatten())
}
+ /// Get the values of several configuration keys in one request, in the
order of `keys`,
+ /// with `None` for a key that does not exist on the server.
+ pub fn get_many(&self, keys: &[&str]) -> Result<Vec<Option<String>>> {
+ block_on(self.client.get_configs(keys))
+ }
+
/// Get a configuration value, returning `default` when the key is unset —
instead of
/// erroring as `get` does for an unknown key. Uses the server's
`GetWithDefault`
/// operation, mirroring `RuntimeConfig.get(key, default)`.
diff --git a/crates/spark-connect/src/session.rs
b/crates/spark-connect/src/session.rs
index a66366f..a791e1f 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -724,8 +724,9 @@ fn rows_to_arrow_ipc(rows: &[Row], schema: &DataType) ->
Result<Vec<u8>> {
columns.push(values_to_arrow(&column, &field.data_type)?);
}
- // Create RecordBatch
- let batch = RecordBatch::try_new(Arc::new(arrow_schema), columns)
+ // Create RecordBatch, with the number of rows, which a batch without
columns needs.
+ let options =
arrow::record_batch::RecordBatchOptions::new().with_row_count(Some(rows.len()));
+ let batch = RecordBatch::try_new_with_options(Arc::new(arrow_schema),
columns, &options)
.map_err(|e| SparkError::connect_msg(format!("Failed to create Arrow
batch: {}", e)))?;
// Write to IPC stream
@@ -1031,14 +1032,19 @@ fn values_to_arrow(values: &[Value], dt: &DataType) ->
Result<Arc<dyn arrow::arr
}
Value::Struct(sf) => {
valid.append(true);
+ // A struct of the declared fields in order, as the
Python client
+ // converts one, is taken by position, so that fields
of the same
+ // name keep their own values. Otherwise match by
field name, else
+ // by position.
+ let in_order = sf.len() == fields.len()
+ && sf.fields().iter().zip(fields).all(|(n, f)| *n
== f.name);
for (i, f) in fields.iter().enumerate() {
- // Prefer match by field name, else positional.
- let val = sf
- .get_by_name(&f.name)
- .or_else(|| sf.get(i))
- .cloned()
- .unwrap_or(Value::Null);
- cols[i].push(val);
+ let val = if in_order {
+ sf.get(i)
+ } else {
+ sf.get_by_name(&f.name).or_else(|| sf.get(i))
+ };
+ cols[i].push(val.cloned().unwrap_or(Value::Null));
}
}
// A Python dict supplied for a struct field (e.g.
array<struct> data)
@@ -1072,11 +1078,15 @@ fn values_to_arrow(values: &[Value], dt: &DataType) ->
Result<Arc<dyn arrow::arr
})
.collect::<Result<Vec<_>>>()?
.into();
- Ok(Arc::new(StructArray::new(
+ // A struct without fields takes its length from the values.
+ StructArray::try_new_with_length(
afields,
arrays,
Some(NullBuffer::new(valid.finish())),
- )))
+ values.len(),
+ )
+ .map(|array| Arc::new(array) as Arc<dyn Array>)
+ .map_err(|e| SparkError::connect_msg(format!("struct build: {e}")))
}
DataType::Map {
key_type,
@@ -1733,12 +1743,48 @@ mod tests {
}
#[test]
- fn input_ipc_preserves_empty_input_and_zero_column_error() {
+ fn struct_values_of_the_declared_fields_are_taken_by_position() {
+ // Fields of the same name keep their own values.
+ let fields = vec![
+ struct_field("a", DataType::Long),
+ struct_field("a", DataType::Long),
+ ];
+ let names = vec!["a".to_string(), "a".to_string()];
+ let value = Value::Struct(Row::new(names, vec![Value::Long(1),
Value::Long(2)]).into());
+ let array = values_to_arrow(&[value], &DataType::Struct { fields
}).unwrap();
+ let array = array.as_any().downcast_ref::<StructArray>().unwrap();
+ let column = |i: usize| {
+ let column = array
+ .column(i)
+ .as_any()
+ .downcast_ref::<arrow::array::Int64Array>();
+ column.unwrap().value(0)
+ };
+ assert_eq!((column(0), column(1)), (1, 2));
+ }
+
+ #[test]
+ fn struct_values_without_fields_keep_their_length() {
+ let empty = Value::Struct(Row::empty().into());
+ let array = values_to_arrow(&[empty, Value::Null], &DataType::Struct {
fields: vec![] });
+ let array = array.unwrap();
+ assert_eq!(array.len(), 2);
+ assert!(array.is_valid(0));
+ assert!(array.is_null(1));
+ }
+
+ #[test]
+ fn input_ipc_preserves_empty_input_and_rows_without_columns() {
assert!(rows_to_arrow_ipc(&[], &DataType::Long).unwrap().is_empty());
- let error = rows_to_arrow_ipc(&[Row::empty()], &schema(vec![]))
- .unwrap_err()
- .to_string();
- assert!(error.contains("Failed to create Arrow batch"), "{error}");
+ // Rows without columns keep their number, as PySpark's table without
columns does.
+ let data = rows_to_arrow_ipc(&[Row::empty(), Row::empty()],
&schema(vec![])).unwrap();
+ let batches = StreamReader::try_new(Cursor::new(data), None)
+ .unwrap()
+ .collect::<std::result::Result<Vec<_>, _>>()
+ .unwrap();
+ assert_eq!(batches.len(), 1);
+ assert_eq!(batches[0].num_columns(), 0);
+ assert_eq!(batches[0].num_rows(), 2);
}
#[test]
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index ceb4569..d6ec5e0 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -381,3 +381,94 @@ def
test_maps_keep_their_entries_in_order_with_keys_of_their_type(session):
df = session.createDataFrame([({2: "b", 1: "a"},)])
assert df.schema.simpleString() == "struct<_1:map<bigint,string>>"
assert list(df.first()[0].items()) == [(2, "b"), (1, "a")]
+
+
+def test_create_dataframe_merges_the_types_of_every_row_as_pyspark(session):
+ import array
+ import decimal
+
+ Point = namedtuple("Point", ["x", "y"])
+
+ class Attributes:
+ def __init__(self):
+ self.b = 1
+ self.a = "x"
+
+ cases = [
+ # A string absorbs another atomic type, whose values become strings,
and a null
+ # takes the type of another row.
+ ([(1, None), ("a", 2.5), (True, None)], "struct<_1:string,_2:double>",
+ [("1", None), ("a", 2.5), ("true", None)]),
+ ([([1, "a"],)], "struct<_1:array<string>>", [(["1", "a"],)]),
+ ([{"a": 1}, {"a": "x"}], "struct<a:string>", [("1",), ("x",)]),
+ ([({"a": [1]},), ({"a": [None]},)],
"struct<_1:map<string,array<bigint>>>",
+ [({"a": [1]},), ({"a": [None]},)]),
+ # A decimal is decimal(38,18), a tuple, a namedtuple or an object a
struct, and an
+ # array.array an array of its typecode.
+ ([(decimal.Decimal("1.5"),)], "struct<_1:decimal(38,18)>",
+ [(decimal.Decimal("1.500000000000000000"),)]),
+ ([((1, 2),)], "struct<_1:struct<_1:bigint,_2:bigint>>", [(Row(_1=1,
_2=2),)]),
+ ([(Point(1, 2),)], "struct<_1:struct<x:bigint,y:bigint>>", [(Row(x=1,
y=2),)]),
+ ([(Attributes(),)], "struct<_1:struct<a:string,b:bigint>>",
[(Row(a="x", b=1),)]),
+ ([(array.array("i", [1, 2]),)], "struct<_1:array<int>>", [([1, 2],)]),
+ ]
+ for data, schema, rows in cases:
+ df = session.createDataFrame(data)
+ assert df.schema.simpleString() == schema, data
+ assert [tuple(r) for r in df.collect()] == rows, data
+ for text in ["1.25", "a\x00b", "\uffff"]:
+ # A string stays text, and shares a string column with an integer.
+ df = session.createDataFrame([Row(value=text), Row(value=5)])
+ assert [tuple(r) for r in df.collect()] == [(text,), ("5",)]
+
+
+def test_create_dataframe_raises_merge_errors_as_pyspark(session):
+ from pyspark.errors import PySparkException
+
+ cases = [
+ ([(1,), (2.5,)], "CANNOT_MERGE_TYPE", {"data_type1": "LongType",
"data_type2": "DoubleType"}),
+ # A merge error within a value names its field.
+ ([([1, 2.5],)], "CANNOT_INFER_TYPE_FOR_FIELD", {"field_name": "_1"}),
+ ([({"a": 1, "b": 2.5},)], "CANNOT_INFER_TYPE_FOR_FIELD",
{"field_name": "_1"}),
+ # A type that stays null cannot be determined.
+ ([({"k": None},)], "CANNOT_DETERMINE_TYPE", {}),
+ ([([None],)], "CANNOT_DETERMINE_TYPE", {}),
+ ]
+ for data, condition, parameters in cases:
+ with pytest.raises(PySparkException) as error:
+ session.createDataFrame(data)
+ assert error.value.getCondition() == condition, data
+ assert error.value.getMessageParameters() == parameters, data
+
+
+def
test_create_dataframe_infers_by_the_session_configurations_as_pyspark(session):
+ naive = datetime(2020, 1, 2, 3, 4, 5)
+ cases = [
+ ("spark.sql.pyspark.legacy.inferMapTypeFromFirstPair.enabled", "true",
+ [({"a": 1, "b": 2.5},)], "struct<_1:map<string,bigint>>", [({"a": 1,
"b": 2},)]),
+ ("spark.sql.pyspark.legacy.inferArrayTypeFromFirstElement.enabled",
"true",
+ [([1, 2.5],)], "struct<_1:array<bigint>>", [([1, 2],)]),
+ ("spark.sql.pyspark.inferNestedDictAsStruct.enabled", "true",
+ [({"a": {"b": 1}, "c": None},)],
"struct<_1:struct<a:struct<b:bigint>>>",
+ [(Row(a=Row(b=1)),)]),
+ # A naive datetime is TIMESTAMP_NTZ, but not in a list, whose elements
PySpark
+ # infers without `prefer_timestamp_ntz`.
+ ("spark.sql.timestampType", "TIMESTAMP_NTZ", [(naive,)],
"struct<_1:timestamp_ntz>",
+ [(naive,)]),
+ ("spark.sql.timestampType", "TIMESTAMP_NTZ", [([naive],)],
"struct<_1:array<timestamp>>",
+ None),
+ # PySpark passes the elements of a list `prefer_timestamp_ntz` for
+ # `infer_map_from_first_pair`, so that a dict in a list takes the map
type of its
+ # first pair.
+ ("spark.sql.timestampType", "TIMESTAMP_NTZ", [([{"a": 1, "b": 2.5}],)],
+ "struct<_1:array<map<string,bigint>>>", [([{"a": 1, "b": 2}],)]),
+ ]
+ for key, value, data, schema, rows in cases:
+ session.conf.set(key, value)
+ try:
+ df = session.createDataFrame(data)
+ assert df.schema.simpleString() == schema, (key, data)
+ if rows is not None:
+ assert [tuple(r) for r in df.collect()] == rows, (key, data)
+ finally:
+ session.conf.unset(key)
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index a4214c6..8422e36 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -6,6 +6,7 @@ Python side has real, measurable coverage in CI without a
running server. Tests
that require a live server live in test_dropin_e2e.py and are gated on
SPARK_REMOTE.
"""
+import array
import datetime
import decimal
import json
@@ -3462,6 +3463,71 @@ def
test_create_dataframe_cannot_type_an_all_null_field_as_pyspark(data):
assert error.value.getCondition() == "CANNOT_DETERMINE_TYPE"
[email protected]("data,condition,parameters", [
+ # The types of every row merge, as in PySpark, and types of other classes
do not.
+ ([(1,), (2.5,)], "CANNOT_MERGE_TYPE", {"data_type1": "LongType",
"data_type2": "DoubleType"}),
+ ([(True,), (1,)], "CANNOT_MERGE_TYPE", {"data_type1": "BooleanType",
"data_type2": "LongType"}),
+ ([(decimal.Decimal("1.5"),), (1,)], "CANNOT_MERGE_TYPE",
+ {"data_type1": "DecimalType", "data_type2": "LongType"}),
+ # A row is a Row, a tuple, a list, a dict or an object, which None is not.
+ ([(1,), None], "CANNOT_INFER_SCHEMA_FOR_TYPE", {"data_type": "NoneType"}),
+ # A value of a type that cannot be inferred names its field.
+ ([({1, 2},)], "CANNOT_INFER_TYPE_FOR_FIELD", {"field_name": "_1"}),
+ ([(array.array("q", [1]),)], "CANNOT_INFER_TYPE_FOR_FIELD", {"field_name":
"_1"}),
+])
+def test_create_dataframe_raises_inference_errors_as_pyspark(data, condition,
parameters):
+ from pyspark.errors import PySparkTypeError
+
+ with pytest.raises(PySparkTypeError) as error:
+ _local_create_dataframe(data)
+ assert error.value.getCondition() == condition
+ assert error.value.getMessageParameters() == parameters
+
+
+def test_create_dataframe_checks_every_row_against_the_merged_schema():
+ from pyspark.errors import PySparkValueError
+
+ # Rows of different fields merge into one struct, which each Row must then
fill.
+ with pytest.raises(PySparkValueError) as error:
+ _local_create_dataframe([Row(a=1), Row(b=2)])
+ assert error.value.getCondition() == "AXIS_LENGTH_MISMATCH"
+ assert error.value.getMessageParameters() == {"expected_length": "2",
"actual_length": "1"}
+
+
[email protected]("schema,arg_type", [
+ (T.ArrayType(T.LongType()), "ArrayType"),
+ (T.MapType(T.StringType(), T.LongType()), "MapType"),
+ ("array<int>", "ArrayType"),
+ (5, "int"),
+])
+def test_create_dataframe_schema_is_names_a_struct_or_an_atomic_type(schema,
arg_type):
+ from pyspark.errors import PySparkTypeError
+
+ with pytest.raises(PySparkTypeError) as error:
+ _local_create_dataframe([(1,)], schema)
+ assert error.value.getCondition() == "NOT_EXPECTED_TYPE"
+ assert error.value.getMessageParameters() == {
+ "arg_name": "schema", "expected_type": "list, None or StructType",
"arg_type": arg_type}
+
+
+def test_create_dataframe_of_a_dataframe_raises_as_pyspark():
+ from pyspark.errors import PySparkTypeError
+
+ spark = _offline_session()
+ with pytest.raises(PySparkTypeError) as error:
+ spark.createDataFrame(spark.range(1))
+ assert error.value.getCondition() == "INVALID_TYPE"
+ assert error.value.getMessageParameters() == {"arg_name": "data",
"arg_type": "DataFrame"}
+
+
+def test_create_dataframe_warns_of_the_arguments_that_connect_ignores():
+ # As PySpark Connect, which ignores both.
+ with pytest.warns(UserWarning, match="'samplingRatio' is ignored"):
+ _local_create_dataframe([(1,)], None, 0.5)
+ with pytest.warns(UserWarning, match="'verifySchema' is ignored"):
+ _local_create_dataframe([(1,)], None, None, True)
+
+
# ---------------------------------------------------------------------------
schema cache
@@ -4802,34 +4868,36 @@ def
test_reader_schema_is_sent_as_pyspark_sends_it(mock_connect, read):
"", "1.25", "-42", "NaN", "a\x00b", "é雪😀", "e\u0301", "\uffff",
])
def test_exact_string_converts_as_text(text):
- from pyspark.errors import PySparkRuntimeError
-
- # A string, even when it reads as a number: it shares a column with a
string, not with
- # an integer.
+ # A string, even when it reads as a number, is text: with a string or an
integer in
+ # another row, the column is a string column, as PySpark merges the types
(the live
+ # suite checks the values).
assert _local_create_dataframe([Row(value=text), Row(value="y")]).isLocal()
- with pytest.raises(PySparkRuntimeError):
- _local_create_dataframe([Row(value=text), Row(value=5)])
+ assert _local_create_dataframe([Row(value=text), Row(value=5)]).isLocal()
@pytest.mark.parametrize("text", ["\ud800", "\udfff", "a\ud800b"])
def test_exact_string_surrogate_error_preserves_generic_fallback(text):
- with pytest.raises(TypeError):
+ # As PySpark, where PyArrow cannot encode a lone surrogate either.
+ with pytest.raises(UnicodeEncodeError, match="surrogates not allowed"):
_local_create_dataframe([Row(value=text)])
def test_exact_string_subclass_uses_generic_conversion():
- # Pin existing subclass behavior; treating subclasses as text is a separate
- # behavior change.
- from pyspark.errors import PySparkRuntimeError
+ from pyspark.errors import PySparkTypeError
class IndexString(str):
def __index__(self):
return 17
- # Converted through __index__, it shares a column with an integer, not
with a string.
- assert _local_create_dataframe([Row(value=IndexString("x")),
Row(value=5)]).isLocal()
- with pytest.raises(PySparkRuntimeError):
- _local_create_dataframe([Row(value=IndexString("x")), Row(value="y")])
+ # As in PySpark, which takes an instance of a subclass by its `__dict__`:
an empty
+ # struct, which merges with neither an integer nor a string.
+ assert _local_create_dataframe([Row(value=IndexString("x"))]).isLocal()
+ for other, data_type in [(5, "LongType"), ("y", "StringType")]:
+ with pytest.raises(PySparkTypeError) as error:
+ _local_create_dataframe([Row(value=IndexString("x")),
Row(value=other)])
+ assert error.value.getCondition() == "CANNOT_MERGE_TYPE"
+ assert error.value.getMessageParameters() == {
+ "data_type1": "StructType", "data_type2": data_type}
# ---------------------------------------------------------------------------
printSchema
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]