This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 3191f4b [SPARK-61031][CONNECT][PYTHON] Take the arguments of the
readers, the writer and the catalog, and create the tables of createTable on
the call, in the Python client as PySpark does
3191f4b is described below
commit 3191f4ba2a1eacb79f248258b092122c850173c3
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 11:49:48 2026 +0900
[SPARK-61031][CONNECT][PYTHON] Take the arguments of the readers, the
writer and the catalog, and create the tables of createTable on the call, in
the Python client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61031](https://issues.apache.org/jira/browse/SPARK-61031): the
Python client now takes the arguments of the readers, `df.write` and
`spark.catalog` as PySpark does, and `createTable` creates the table on the
call.
Readers:
- `spark.read.load(path)` and the format readers, `csv`, `json`, `orc`,
`text` and `xml`, take a path or a list of paths, and `parquet(*paths)` takes
any number of them. As PySpark's `DataSource` asserts them, any other path
fails an assertion.
- `csv`, `json` and `xml` also take a DataFrame of one string column, whose
strings they parse with a `Parse` relation, of the schema and the options of
the reader. The server parses a DDL schema, as PySpark's `StructType.fromDDL`
does. `json` and `xml` raise `NOT_EXPECTED_TYPE` for any other `path`. A parse
keeps the schema and the options on the reader, but not the format, which only
`load` sets.
Writer:
- `bucketBy(numBuckets, col, *cols)` and `sortBy(col, *cols)` take column
names, or a list or tuple of them alone. They raise `NOT_EXPECTED_TYPE` for a
`numBuckets` that is not an int and for columns that are not strings, and
`CANNOT_SET_TOGETHER` for a list together with more columns.
- `clusterBy` takes a list or tuple of columns too, and asserts that there
is at least one.
- `mode(None)` keeps the default mode.
Catalog:
- `createTable` creates the table on the call, as PySpark's
`df._to_table()` does, of the schema and the string options given. It raises
`NOT_EXPECTED_TYPE` for a schema that is not a StructType.
- `createExternalTable` warns that it is deprecated and creates the table
as `createTable` does.
- `tableExists`, `functionExists` and `listColumns` warn that `dbName` is
deprecated.
The core adds `DataFrameReader::load_paths` and `parse`, a `Parse` plan,
and `Catalog::create_table_with` and `create_external_table_with`, of a schema
and options. It now creates these tables on the call.
### Why are the changes needed?
The client:
- raised a TypeError for a list of paths, as in `spark.read.csv([path])`,
and for a DataFrame of strings, and a ValueError for `parquet` of more than one
path
- raised TypeErrors for the lists of columns of `bucketBy`, `sortBy` and
`clusterBy`, and for `mode(None)`, and TypeErrors rather than PySpark's errors
for the other arguments of `bucketBy`
- ignored the schema and the options of `createTable` and
`createExternalTable`, and created the table only when the returned DataFrame
ran
- did not warn of `createExternalTable` and of `dbName`
### Does this PR introduce _any_ user-facing change?
Yes. These methods take the arguments and raise the errors of PySpark
4.3.0, and `createTable` creates the table on the call.
### How was this patch tested?
- A new live test parses DataFrames of strings with `csv`, `json` and
`xml`, loads lists of paths, and creates tables with `createTable` and
`createExternalTable`. It passes with official PySpark 4.3.0.
- New offline tests check, against a mock Connect server, the `Parse`
relations, the paths that the readers load, and the `CreateTable` that
`createTable` and `createExternalTable` send on the call. They also check the
errors and the warnings.
- A new lib test checks the `Parse` plan, and new e2e tests check `Parse`
and the tables created against the server.
- On a local server, 52 cases of these methods behave as in official
PySpark 4.3.0: their results, errors, warnings and reader state. So do 19
`Parse` schemas.
- The lib tests of both crates (572 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (1826 passed, 8 skipped), and the
live drop-in and Arrow suites (110 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #236 from HyukjinKwon/SPARK-61031-reader-writer.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/catalog.rs | 128 +++++++++++++++---
crates/pyspark-rs/src/dataframe.rs | 46 +++++--
crates/pyspark-rs/src/readwriter.rs | 171 ++++++++++++++++++++----
crates/spark-connect/src/catalog.rs | 76 ++++++++---
crates/spark-connect/src/plan.rs | 40 ++++++
crates/spark-connect/src/readwriter.rs | 82 +++++++++++-
crates/spark-connect/tests/e2e_coverage.rs | 96 ++++++++++++-
python/tests/test_dropin_e2e.py | 46 +++++++
python/tests/test_dropin_offline.py | 208 +++++++++++++++++++++++++++++
9 files changed, 813 insertions(+), 80 deletions(-)
diff --git a/crates/pyspark-rs/src/catalog.rs b/crates/pyspark-rs/src/catalog.rs
index 62c5cef..18e13e7 100644
--- a/crates/pyspark-rs/src/catalog.rs
+++ b/crates/pyspark-rs/src/catalog.rs
@@ -8,6 +8,70 @@ use spark_connect::catalog::{
use crate::dataframe::PyDataFrame;
use crate::errors::ResultExt;
+/// PySpark's FutureWarning of the deprecated `dbName` of the catalog method
`method`.
+fn warn_db_name(py: Python<'_>, method: &str, db_name: Option<&str>) ->
PyResult<()> {
+ if db_name.is_some() {
+ py.import("warnings")?.call_method1(
+ "warn",
+ (
+ format!(
+ "`dbName` has been deprecated since Spark 3.4 and might be
removed in a \
+ future version. Use {method}(`dbName.tableName`) instead."
+ ),
+ py.get_type::<pyo3::exceptions::PyFutureWarning>(),
+ ),
+ )?;
+ }
+ Ok(())
+}
+
+/// The schema of `createTable`, a StructType, as PySpark's, else
`NOT_EXPECTED_TYPE`.
+fn table_schema(
+ schema: Option<&Bound<'_, PyAny>>,
+) -> PyResult<Option<spark_connect::types::DataType>> {
+ let Some(schema) = schema.filter(|schema| !schema.is_none()) else {
+ return Ok(None);
+ };
+ if let Ok(st) = schema.extract::<PyRef<'_, crate::types::PyStructType>>() {
+ return Ok(Some(spark_connect::types::DataType::Struct {
+ fields: st.fields.clone(),
+ }));
+ }
+ let type_name = schema.get_type().name()?.to_string();
+ Err(crate::column::raise_pyspark(
+ schema.py(),
+ "PySparkTypeError",
+ "NOT_EXPECTED_TYPE",
+ &[
+ ("arg_name", "schema"),
+ ("expected_type", "struct type"),
+ ("arg_type", &type_name),
+ ],
+ ))
+}
+
+/// The `**options` of `createTable`, as PySpark's: strings, where a `None` is
left out.
+fn table_options(
+ options: Option<&Bound<'_, pyo3::types::PyDict>>,
+) -> PyResult<std::collections::HashMap<String, String>> {
+ let mut map = std::collections::HashMap::new();
+ for (key, value) in options.into_iter().flat_map(|options| options.iter())
{
+ if value.is_none() {
+ continue;
+ }
+ if !value.is_instance_of::<pyo3::types::PyString>() {
+ // As the protobuf map of strings that PySpark sets the options in.
+ return Err(pyo3::exceptions::PyTypeError::new_err(format!(
+ "{} has type {}, but expected one of: (<class 'bytes'>, <class
'str'>)",
+ value.repr()?,
+ value.get_type().str()?
+ )));
+ }
+ map.insert(key.extract()?, value.extract()?);
+ }
+ Ok(map)
+}
+
/// `pyspark.sql.catalog.CatalogMetadata`.
#[pyclass(name = "CatalogMetadata", module = "pyspark.sql.catalog")]
pub struct PyCatalogMetadata {
@@ -290,7 +354,8 @@ impl PyCatalog {
/// Check if table exists. Mirrors `Catalog.tableExists(tableName,
dbName=None) -> bool`.
#[pyo3(signature = (tableName, dbName=None))]
- fn tableExists(&self, tableName: &str, dbName: Option<&str>) ->
PyResult<bool> {
+ fn tableExists(&self, py: Python<'_>, tableName: &str, dbName:
Option<&str>) -> PyResult<bool> {
+ warn_db_name(py, "tableExists", dbName)?;
self.catalog
.table_exists_with_database(tableName, dbName)
.to_pyerr()
@@ -322,7 +387,13 @@ impl PyCatalog {
/// Check if function exists. Mirrors
`Catalog.functionExists(functionName, dbName=None) -> bool`.
#[pyo3(signature = (functionName, dbName=None))]
- fn functionExists(&self, functionName: &str, dbName: Option<&str>) ->
PyResult<bool> {
+ fn functionExists(
+ &self,
+ py: Python<'_>,
+ functionName: &str,
+ dbName: Option<&str>,
+ ) -> PyResult<bool> {
+ warn_db_name(py, "functionExists", dbName)?;
self.catalog
.function_exists_with_database(functionName, dbName)
.to_pyerr()
@@ -338,7 +409,13 @@ impl PyCatalog {
/// List columns of a table. Mirrors `Catalog.listColumns(tableName,
dbName=None) -> List[Column]`.
#[pyo3(signature = (tableName, dbName=None))]
#[allow(non_snake_case)]
- fn listColumns(&self, tableName: &str, dbName: Option<&str>) ->
PyResult<Vec<PyCatalogColumn>> {
+ fn listColumns(
+ &self,
+ py: Python<'_>,
+ tableName: &str,
+ dbName: Option<&str>,
+ ) -> PyResult<Vec<PyCatalogColumn>> {
+ warn_db_name(py, "listColumns", dbName)?;
Ok(self
.catalog
.list_columns_typed(tableName, dbName)
@@ -403,11 +480,13 @@ impl PyCatalog {
self.catalog.recover_partitions(tableName).to_pyerr()
}
- /// Create a table from a data source.
+ /// Create a table from a data source, as PySpark's: of a StructType
`schema` and string
+ /// `**options`, created on return.
#[pyo3(signature = (tableName, path=None, source=None, schema=None,
description=None, **options))]
- #[allow(non_snake_case, unused_variables)]
+ #[allow(non_snake_case, clippy::too_many_arguments)]
fn createTable(
&self,
+ py: Python<'_>,
tableName: &str,
path: Option<&str>,
source: Option<&str>,
@@ -415,27 +494,46 @@ impl PyCatalog {
description: Option<&str>,
options: Option<&Bound<'_, pyo3::types::PyDict>>,
) -> PyResult<PyDataFrame> {
- self.catalog
- .create_table(tableName, path, source, description)
- .map(PyDataFrame::new)
- .to_pyerr()
+ let schema = table_schema(schema)?;
+ let options = table_options(options)?;
+ py.detach(|| {
+ self.catalog
+ .create_table_with(
+ tableName,
+ path,
+ source,
+ schema.as_ref(),
+ description,
+ options,
+ )
+ .map_err(Box::new)
+ })
+ .map(PyDataFrame::new)
+ .to_pyerr()
}
- /// Create an external table from a data source.
+ /// Create an external table from a data source: deprecated, as PySpark's,
which warns and
+ /// creates the table as `createTable` does.
#[pyo3(signature = (tableName, path=None, source=None, schema=None,
**options))]
- #[allow(non_snake_case, unused_variables)]
+ #[allow(non_snake_case)]
fn createExternalTable(
&self,
+ py: Python<'_>,
tableName: &str,
path: Option<&str>,
source: Option<&str>,
schema: Option<&Bound<'_, PyAny>>,
options: Option<&Bound<'_, pyo3::types::PyDict>>,
) -> PyResult<PyDataFrame> {
- self.catalog
- .create_external_table(tableName, path, source)
- .map(PyDataFrame::new)
- .to_pyerr()
+ py.import("warnings")?.call_method1(
+ "warn",
+ (
+ "createExternalTable is deprecated since Spark 4.0, please use
createTable \
+ instead.",
+ py.get_type::<pyo3::exceptions::PyFutureWarning>(),
+ ),
+ )?;
+ self.createTable(py, tableName, path, source, schema, None, options)
}
/// Register a Python function as a UDF, returning the registered UDF.
Deprecated
diff --git a/crates/pyspark-rs/src/dataframe.rs
b/crates/pyspark-rs/src/dataframe.rs
index cfffa05..af7993f 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -2763,8 +2763,14 @@ fn apply_write_common(
#[pymethods]
impl PyDataFrameWriter {
- fn mode<'py>(mut slf: PyRefMut<'py, Self>, saveMode: &str) ->
PyResult<PyRefMut<'py, Self>> {
- slf.inner = slf.inner.clone().mode(saveMode);
+ /// The save mode, as PySpark's: `None` keeps the default, "error".
+ fn mode<'py>(
+ mut slf: PyRefMut<'py, Self>,
+ saveMode: Option<&str>,
+ ) -> PyResult<PyRefMut<'py, Self>> {
+ if let Some(save_mode) = saveMode {
+ slf.inner = slf.inner.clone().mode(save_mode);
+ }
Ok(slf)
}
@@ -2819,33 +2825,45 @@ impl PyDataFrameWriter {
#[pyo3(name = "bucketBy", signature = (numBuckets, col, *cols))]
fn bucket_by<'py>(
mut slf: PyRefMut<'py, Self>,
- numBuckets: i32,
- col: String,
- cols: Vec<String>,
+ numBuckets: &Bound<'_, PyAny>,
+ col: &Bound<'_, PyAny>,
+ cols: Vec<Bound<'_, PyAny>>,
) -> PyResult<PyRefMut<'py, Self>> {
- let mut all = vec![col];
- all.extend(cols);
- slf.inner = slf.inner.clone().bucket_by(numBuckets, all);
+ let num_buckets = crate::readwriter::bucket_count(numBuckets)?;
+ let columns = crate::readwriter::writer_columns(col, &cols)?;
+ slf.inner = slf.inner.clone().bucket_by(num_buckets, columns);
Ok(slf)
}
#[pyo3(name = "sortBy", signature = (col, *cols))]
fn sort_by<'py>(
mut slf: PyRefMut<'py, Self>,
- col: String,
- cols: Vec<String>,
+ col: &Bound<'_, PyAny>,
+ cols: Vec<Bound<'_, PyAny>>,
) -> PyResult<PyRefMut<'py, Self>> {
- let mut all = vec![col];
- all.extend(cols);
- slf.inner = slf.inner.clone().sort_by(all);
+ let columns = crate::readwriter::writer_columns(col, &cols)?;
+ slf.inner = slf.inner.clone().sort_by(columns);
Ok(slf)
}
+ /// The clustering columns, as PySpark's: column names, or a list or tuple
of them, at
+ /// least one.
#[pyo3(name = "clusterBy", signature = (*cols))]
fn cluster_by<'py>(
mut slf: PyRefMut<'py, Self>,
- cols: Vec<String>,
+ cols: Vec<Bound<'_, PyAny>>,
) -> PyResult<PyRefMut<'py, Self>> {
+ let cols: Vec<String> = match cols.as_slice() {
+ [only] if only.is_instance_of::<PyList>() ||
only.is_instance_of::<PyTuple>() => {
+ only.extract()?
+ }
+ _ => cols.iter().map(|c| c.extract()).collect::<PyResult<_>>()?,
+ };
+ if cols.is_empty() {
+ return Err(pyo3::exceptions::PyAssertionError::new_err(
+ "clusterBy needs one or more clustering columns.",
+ ));
+ }
slf.inner = slf.inner.clone().cluster_by(cols);
Ok(slf)
}
diff --git a/crates/pyspark-rs/src/readwriter.rs
b/crates/pyspark-rs/src/readwriter.rs
index 217034a..998b028 100644
--- a/crates/pyspark-rs/src/readwriter.rs
+++ b/crates/pyspark-rs/src/readwriter.rs
@@ -7,6 +7,7 @@ use pyo3::types::PyDict;
use spark_connect::readwriter::DataFrameReader;
use crate::dataframe::PyDataFrame;
+use crate::errors::ResultExt;
/// Python wrapper for the batch DataFrameReader.
///
@@ -45,6 +46,133 @@ impl PyDataFrameReader {
fn commit(&mut self, r: &DataFrameReader, format: &str) {
self.inner = r.clone().format(format);
}
+
+ /// The DataFrame of `r`, this reader with the options of a call of
`json`, `csv` or `xml`
+ /// of `path`, as PySpark's: a DataFrame of strings, which it parses as
`parse_format`,
+ /// keeping the schema and the options on this reader but not the format,
or else a path
+ /// or a list of them, which it loads as `format`. `csv`, unlike the
others, passes any
+ /// other `path` to `load`, which asserts it.
+ fn read_or_parse(
+ &mut self,
+ r: DataFrameReader,
+ format: &str,
+ parse_format: spark_connect::plan::ParseFormat,
+ path: &Bound<'_, PyAny>,
+ ) -> PyResult<PyDataFrame> {
+ if let Ok(df) = path.extract::<PyRef<'_, PyDataFrame>>() {
+ let parsed = r
+ .clone()
+ .parse(parse_format, &df.dataframe)
+ .map(PyDataFrame::new)
+ .to_pyerr()?;
+ self.inner = r;
+ return Ok(parsed);
+ }
+ if path.is_instance_of::<pyo3::types::PyString>()
+ || path.is_instance_of::<pyo3::types::PyList>()
+ || parse_format == spark_connect::plan::ParseFormat::Csv
+ {
+ let paths = paths_of(path)?;
+ self.commit(&r, format);
+ return Ok(PyDataFrame::new(r.format(format).load_paths(paths)));
+ }
+ Err(not_expected_type(path, "path", "str, list, or DataFrame"))
+ }
+}
+
+/// The paths of `path`, a path or a list of them, as PySpark's `load` takes
them: anything
+/// else fails its assertion.
+fn paths_of(path: &Bound<'_, PyAny>) -> PyResult<Vec<String>> {
+ if path.is_instance_of::<pyo3::types::PyString>() {
+ return Ok(vec![path.extract()?]);
+ }
+ if !path.is_instance_of::<pyo3::types::PyList>() {
+ return Err(pyo3::exceptions::PyAssertionError::new_err(()));
+ }
+ path_list(&path.try_iter()?.collect::<PyResult<Vec<_>>>()?)
+}
+
+/// The paths of `paths`, strings, as PySpark's `load` asserts them.
+fn path_list(paths: &[Bound<'_, PyAny>]) -> PyResult<Vec<String>> {
+ paths
+ .iter()
+ .map(|path| {
+ if path.is_instance_of::<pyo3::types::PyString>() {
+ path.extract()
+ } else {
+ Err(pyo3::exceptions::PyAssertionError::new_err(()))
+ }
+ })
+ .collect()
+}
+
+/// PySpark's `NOT_EXPECTED_TYPE` error of the argument `name`, `value`, which
should be of
+/// the type `expected`.
+fn not_expected_type(value: &Bound<'_, PyAny>, name: &str, expected: &str) ->
PyErr {
+ let type_name = value
+ .get_type()
+ .name()
+ .map(|name| name.to_string())
+ .unwrap_or_default();
+ crate::column::raise_pyspark(
+ value.py(),
+ "PySparkTypeError",
+ "NOT_EXPECTED_TYPE",
+ &[
+ ("arg_name", name),
+ ("expected_type", expected),
+ ("arg_type", &type_name),
+ ],
+ )
+}
+
+/// The columns of `df.write.bucketBy(numBuckets, col, *cols)` and
`sortBy(col, *cols)`, as
+/// PySpark's: column names, or a list or tuple of them alone.
+pub(crate) fn writer_columns(
+ col: &Bound<'_, PyAny>,
+ cols: &[Bound<'_, PyAny>],
+) -> PyResult<Vec<String>> {
+ let sequence =
+ col.is_instance_of::<pyo3::types::PyList>() ||
col.is_instance_of::<pyo3::types::PyTuple>();
+ let (col, cols) = if sequence {
+ if !cols.is_empty() {
+ let type_name = col.get_type().name()?.to_string();
+ return Err(crate::column::raise_pyspark(
+ col.py(),
+ "PySparkValueError",
+ "CANNOT_SET_TOGETHER",
+ &[("arg_list", &format!("`col` of type {type_name} and
`cols`"))],
+ ));
+ }
+ // An empty list or tuple raises an IndexError, as PySpark's `col[0]`.
+ let first = col.get_item(0)?;
+ let rest: Vec<Bound<'_, PyAny>> =
col.try_iter()?.skip(1).collect::<PyResult<_>>()?;
+ (first, rest)
+ } else {
+ (col.clone(), cols.to_vec())
+ };
+ if let Some(other) = cols
+ .iter()
+ .find(|c| !c.is_instance_of::<pyo3::types::PyString>())
+ {
+ return Err(not_expected_type(other, "cols", "list[str]"));
+ }
+ if !col.is_instance_of::<pyo3::types::PyString>() {
+ return Err(not_expected_type(&col, "col", "list[str]"));
+ }
+ std::iter::once(&col)
+ .chain(cols.iter())
+ .map(|c| c.extract())
+ .collect()
+}
+
+/// The number of buckets of `df.write.bucketBy`, an int, as PySpark's.
+pub(crate) fn bucket_count(num_buckets: &Bound<'_, PyAny>) -> PyResult<i32> {
+ if num_buckets.is_instance_of::<pyo3::types::PyInt>() {
+ num_buckets.extract()
+ } else {
+ Err(not_expected_type(num_buckets, "numBuckets", "int"))
+ }
}
/// Apply a single reader option (skipping None; bool->"true"/"false").
@@ -168,7 +296,7 @@ impl PyDataFrameReader {
#[pyo3(signature = (path=None, format=None, schema=None, **options))]
fn load(
&mut self,
- path: Option<&str>,
+ path: Option<&Bound<'_, PyAny>>,
format: Option<&str>,
schema: Option<&Bound<'_, PyAny>>,
options: Option<&Bound<'_, PyDict>>,
@@ -180,9 +308,10 @@ impl PyDataFrameReader {
if let Some(s) = schema {
r = apply_reader_schema(r, s)?;
}
+ let paths = path.map(paths_of).transpose()?.unwrap_or_default();
// PySpark's one-shot load sets format/schema/options on the reader
itself.
self.inner = r.clone();
- Ok(PyDataFrame::new(r.load(path)))
+ Ok(PyDataFrame::new(r.load_paths(paths)))
}
/// Read a table by name.
@@ -202,7 +331,7 @@ impl PyDataFrameReader {
#[allow(non_snake_case, clippy::too_many_arguments)]
fn json(
&mut self,
- path: &str,
+ path: &Bound<'_, PyAny>,
schema: Option<&Bound<'_, PyAny>>,
primitivesAsString: Option<&Bound<'_, PyAny>>,
prefersDecimal: Option<&Bound<'_, PyAny>>,
@@ -261,27 +390,21 @@ impl PyDataFrameReader {
r = set_opt(r, "modifiedAfter", modifiedAfter)?;
r = set_opt(r, "allowNonNumericNumbers", allowNonNumericNumbers)?;
r = set_opt(r, "useUnsafeRow", useUnsafeRow)?;
- self.commit(&r, "json");
- Ok(PyDataFrame::new(r.json(path)))
+ self.read_or_parse(r, "json", spark_connect::plan::ParseFormat::Json,
path)
}
/// Read Parquet file(s).
#[pyo3(signature = (*paths, **options))]
fn parquet(
&mut self,
- paths: Vec<String>,
+ paths: Vec<Bound<'_, PyAny>>,
options: Option<&Bound<'_, PyDict>>,
) -> PyResult<PyDataFrame> {
- // Reference `parquet(*paths)` accepts multiple paths; the connect
core reader
- // currently loads a single path, so we support exactly one here.
- if paths.len() != 1 {
- return Err(PyErr::new::<pyo3::exceptions::PyValueError, _>(
- "parquet() currently supports exactly one path",
- ));
- }
+ // As PySpark's `parquet(*paths)`, of every path.
let r = self.with_opts(options)?;
+ let paths = path_list(&paths)?;
self.commit(&r, "parquet");
- Ok(PyDataFrame::new(r.parquet(&paths[0])))
+ Ok(PyDataFrame::new(r.format("parquet").load_paths(paths)))
}
/// Read CSV file(s) - full pyspark signature; each named option is
@@ -290,7 +413,7 @@ impl PyDataFrameReader {
#[allow(non_snake_case, clippy::too_many_arguments)]
fn csv(
&mut self,
- path: &str,
+ path: &Bound<'_, PyAny>,
schema: Option<&Bound<'_, PyAny>>,
sep: Option<&Bound<'_, PyAny>>,
encoding: Option<&Bound<'_, PyAny>>,
@@ -365,8 +488,7 @@ impl PyDataFrameReader {
r = set_opt(r, "modifiedBefore", modifiedBefore)?;
r = set_opt(r, "modifiedAfter", modifiedAfter)?;
r = set_opt(r, "unescapedQuoteHandling", unescapedQuoteHandling)?;
- self.commit(&r, "csv");
- Ok(PyDataFrame::new(r.csv(path)))
+ self.read_or_parse(r, "csv", spark_connect::plan::ParseFormat::Csv,
path)
}
/// Read ORC file(s) - full pyspark signature; each named option is
@@ -375,7 +497,7 @@ impl PyDataFrameReader {
#[allow(non_snake_case, clippy::too_many_arguments)]
fn orc(
&mut self,
- path: &str,
+ path: &Bound<'_, PyAny>,
mergeSchema: Option<&Bound<'_, PyAny>>,
pathGlobFilter: Option<&Bound<'_, PyAny>>,
recursiveFileLookup: Option<&Bound<'_, PyAny>>,
@@ -388,8 +510,9 @@ impl PyDataFrameReader {
r = set_opt(r, "recursiveFileLookup", recursiveFileLookup)?;
r = set_opt(r, "modifiedBefore", modifiedBefore)?;
r = set_opt(r, "modifiedAfter", modifiedAfter)?;
+ let paths = paths_of(path)?;
self.commit(&r, "orc");
- Ok(PyDataFrame::new(r.orc(path)))
+ Ok(PyDataFrame::new(r.format("orc").load_paths(paths)))
}
/// Read TEXT file(s) - full pyspark signature; each named option is
@@ -398,7 +521,7 @@ impl PyDataFrameReader {
#[allow(non_snake_case, clippy::too_many_arguments)]
fn text(
&mut self,
- paths: &str,
+ paths: &Bound<'_, PyAny>,
wholetext: Option<&Bound<'_, PyAny>>,
lineSep: Option<&Bound<'_, PyAny>>,
pathGlobFilter: Option<&Bound<'_, PyAny>>,
@@ -413,8 +536,9 @@ impl PyDataFrameReader {
r = set_opt(r, "recursiveFileLookup", recursiveFileLookup)?;
r = set_opt(r, "modifiedBefore", modifiedBefore)?;
r = set_opt(r, "modifiedAfter", modifiedAfter)?;
+ let paths = paths_of(paths)?;
self.commit(&r, "text");
- Ok(PyDataFrame::new(r.text(paths)))
+ Ok(PyDataFrame::new(r.format("text").load_paths(paths)))
}
/// Read XML file(s) - full pyspark signature; each named option is
@@ -423,7 +547,7 @@ impl PyDataFrameReader {
#[allow(non_snake_case, clippy::too_many_arguments)]
fn xml(
&mut self,
- path: &str,
+ path: &Bound<'_, PyAny>,
rowTag: Option<&Bound<'_, PyAny>>,
schema: Option<&Bound<'_, PyAny>>,
excludeAttribute: Option<&Bound<'_, PyAny>>,
@@ -466,8 +590,7 @@ impl PyDataFrameReader {
r = set_opt(r, "multiLine", multiLine)?;
r = set_opt(r, "samplingRatio", samplingRatio)?;
r = set_opt(r, "locale", locale)?;
- self.commit(&r, "xml");
- Ok(PyDataFrame::new(r.xml(path)))
+ self.read_or_parse(r, "xml", spark_connect::plan::ParseFormat::Xml,
path)
}
/// Read from a JDBC source. Mirrors `DataFrameReader.jdbc(url, table,
diff --git a/crates/spark-connect/src/catalog.rs
b/crates/spark-connect/src/catalog.rs
index 8915923..c211558 100644
--- a/crates/spark-connect/src/catalog.rs
+++ b/crates/spark-connect/src/catalog.rs
@@ -2,6 +2,8 @@
//!
//! Provides access to database and table metadata, and catalog operations.
+use std::collections::HashMap;
+
use spark_connect_core::error::{Result, SparkError};
use spark_connect_core::runtime::block_on;
use spark_connect_proto as proto;
@@ -523,19 +525,31 @@ impl Catalog {
path: Option<&str>,
source: Option<&str>,
) -> Result<DataFrame> {
- let mut create_ext = proto::CreateExternalTable::default();
- create_ext.table_name = table_name.to_string();
- if let Some(p) = path {
- create_ext.path = Some(p.to_string());
- }
- if let Some(s) = source {
- create_ext.source = Some(s.to_string());
- }
+ self.create_external_table_with(table_name, path, source, None,
HashMap::new())
+ }
+
+ /// Create an external table of a schema and options, mirroring
+ /// `Catalog.createExternalTable(tableName, path, source, schema,
**options)`.
+ pub fn create_external_table_with(
+ &self,
+ table_name: &str,
+ path: Option<&str>,
+ source: Option<&str>,
+ schema: Option<&crate::types::DataType>,
+ options: HashMap<String, String>,
+ ) -> Result<DataFrame> {
+ let create_ext = proto::CreateExternalTable {
+ table_name: table_name.to_string(),
+ path: path.map(str::to_string),
+ source: source.map(str::to_string),
+ schema: schema.map(|s| s.to_proto()),
+ options,
+ };
let mut catalog_msg = proto::Catalog::default();
catalog_msg.cat_type =
Some(proto::catalog::CatType::CreateExternalTable(create_ext));
- self.execute_catalog_as_dataframe(&catalog_msg)
+ self.execute_catalog_eagerly(&catalog_msg)
}
/// Create a table.
@@ -548,22 +562,33 @@ impl Catalog {
source: Option<&str>,
description: Option<&str>,
) -> Result<DataFrame> {
- let mut create_tbl = proto::CreateTable::default();
- create_tbl.table_name = table_name.to_string();
- if let Some(p) = path {
- create_tbl.path = Some(p.to_string());
- }
- if let Some(s) = source {
- create_tbl.source = Some(s.to_string());
- }
- if let Some(d) = description {
- create_tbl.description = Some(d.to_string());
- }
+ self.create_table_with(table_name, path, source, None, description,
HashMap::new())
+ }
+
+ /// Create a table of a schema and options, mirroring
+ /// `Catalog.createTable(tableName, path, source, schema, description,
**options)`.
+ pub fn create_table_with(
+ &self,
+ table_name: &str,
+ path: Option<&str>,
+ source: Option<&str>,
+ schema: Option<&crate::types::DataType>,
+ description: Option<&str>,
+ options: HashMap<String, String>,
+ ) -> Result<DataFrame> {
+ let create_tbl = proto::CreateTable {
+ table_name: table_name.to_string(),
+ path: path.map(str::to_string),
+ source: source.map(str::to_string),
+ description: description.map(str::to_string),
+ schema: schema.map(|s| s.to_proto()),
+ options,
+ };
let mut catalog_msg = proto::Catalog::default();
catalog_msg.cat_type =
Some(proto::catalog::CatType::CreateTable(create_tbl));
- self.execute_catalog_as_dataframe(&catalog_msg)
+ self.execute_catalog_eagerly(&catalog_msg)
}
/// Drop a temporary view.
@@ -1056,6 +1081,15 @@ impl Catalog {
Ok(DataFrame::new(self.session.clone(), plan))
}
+ /// The DataFrame of a catalog operation, executed once first, as PySpark's
+ /// `createTable` executes it eagerly (`df._to_table()`), so that the
table exists on
+ /// return.
+ fn execute_catalog_eagerly(&self, catalog: &proto::Catalog) ->
Result<DataFrame> {
+ let df = self.execute_catalog_as_dataframe(catalog)?;
+ df.to_arrow()?;
+ Ok(df)
+ }
+
/// Build an ExecutePlanRequest for a catalog operation.
fn build_execute_catalog_request(
&self,
diff --git a/crates/spark-connect/src/plan.rs b/crates/spark-connect/src/plan.rs
index 8f4ba30..a8341cf 100644
--- a/crates/spark-connect/src/plan.rs
+++ b/crates/spark-connect/src/plan.rs
@@ -182,6 +182,14 @@ pub enum LogicalPlan {
/// The parameters, literal expressions or columns, as PySpark's `lit`
of them.
parameters: Vec<Expression>,
},
+ /// Parse: a DataFrame of one string column parsed into a structured
DataFrame, as
+ /// PySpark's `Parse` of `spark.read.json`, `csv` and `xml` of a DataFrame.
+ Parse {
+ input: Arc<LogicalPlan>,
+ format: ParseFormat,
+ schema: Option<DataType>,
+ options: HashMap<String, String>,
+ },
/// Unpivot: `df.unpivot(...)`.
Unpivot {
input: Arc<LogicalPlan>,
@@ -1049,6 +1057,20 @@ impl LogicalPlan {
relation.rel_type =
Some(proto::relation::RelType::Hint(Box::new(hint)));
}
+ LogicalPlan::Parse {
+ input,
+ format,
+ schema,
+ options,
+ } => {
+ relation.rel_type =
Some(proto::relation::RelType::Parse(Box::new(proto::Parse {
+ input: Some(Box::new(relation_of(input, plan_ids))),
+ format: format.to_proto() as i32,
+ schema: schema.as_ref().map(|s| s.to_proto()),
+ options: options.clone(),
+ })));
+ }
+
LogicalPlan::Unpivot {
input,
ids,
@@ -1897,6 +1919,24 @@ pub fn to_schema(input: impl Into<Arc<LogicalPlan>>,
schema: DataType) -> Logica
}
}
+/// The format of a `Parse`: PySpark's `ParseFormat` of `spark.read.csv`,
`json` and `xml`.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum ParseFormat {
+ Csv,
+ Json,
+ Xml,
+}
+
+impl ParseFormat {
+ fn to_proto(self) -> proto::parse::ParseFormat {
+ match self {
+ ParseFormat::Csv => proto::parse::ParseFormat::Csv,
+ ParseFormat::Json => proto::parse::ParseFormat::Json,
+ ParseFormat::Xml => proto::parse::ParseFormat::Xml,
+ }
+ }
+}
+
/// Create a Hint plan, of parameters given as strings: an integer-looking
parameter is an
/// Integer (int32) literal, matching PySpark's `lit(int)`, otherwise a String
literal. (A
/// Long literal is rejected as a partitionNum by e.g. the REBALANCE hint,
which expects an
diff --git a/crates/spark-connect/src/readwriter.rs
b/crates/spark-connect/src/readwriter.rs
index 9292288..fd08538 100644
--- a/crates/spark-connect/src/readwriter.rs
+++ b/crates/spark-connect/src/readwriter.rs
@@ -76,7 +76,13 @@ impl DataFrameReader {
/// Load data from the specified path(s) with the configured format,
schema, and options.
pub fn load(self, path: Option<&str>) -> DataFrame {
- let paths = path.map(|p| vec![p.to_string()]);
+ self.load_paths(path.map(|p| vec![p.to_string()]).unwrap_or_default())
+ }
+
+ /// Load data from `paths`, mirroring `DataFrameReader.load` of a list of
paths, with the
+ /// configured format, schema, and options.
+ pub fn load_paths(self, paths: Vec<String>) -> DataFrame {
+ let paths = Some(paths);
let plan = LogicalPlan::Read {
read_type: ReadType::DataSource {
format: self.format,
@@ -105,6 +111,27 @@ impl DataFrameReader {
DataFrame::new(self.session, plan)
}
+ /// Parse the strings of `input`, a DataFrame of one string column, as
`format`, with the
+ /// schema and the options of this reader, mirroring
`DataFrameReader.json`, `csv` and `xml`
+ /// of a DataFrame: a schema of JSON, or else of DDL, which the server
parses, as PySpark's
+ /// `StructType.fromDDL`.
+ pub fn parse(self, format: crate::plan::ParseFormat, input: &DataFrame) ->
Result<DataFrame> {
+ let schema = if self.schema.is_empty() {
+ None
+ } else if self.schema.starts_with('{') {
+ Some(crate::types::DataType::from_json_str(&self.schema)?)
+ } else {
+ Some(self.session.parse_ddl(&self.schema)?)
+ };
+ let plan = LogicalPlan::Parse {
+ input: Arc::clone(&input.plan),
+ format,
+ schema,
+ options: self.options,
+ };
+ Ok(DataFrame::new(self.session, plan))
+ }
+
/// Read from a named table.
pub fn table(self, table_name: &str) -> DataFrame {
let plan = LogicalPlan::Read {
@@ -876,4 +903,57 @@ mod tests {
_ => panic!("expected Read plan"),
}
}
+
+ #[test]
+ fn parse_sends_the_strings_with_the_schema_and_options_of_the_reader() {
+ use crate::plan::ParseFormat;
+ use crate::types::DataType;
+ use proto::relation::RelType;
+
+ let spark = session();
+ // A lazy plan, as `spark.sql` runs on the call.
+ let strings = spark.range(1).unwrap();
+ let parse_of = |reader: DataFrameReader, format| match reader
+ .parse(format, &strings)
+ .unwrap()
+ .relation()
+ .rel_type
+ {
+ Some(RelType::Parse(parse)) => parse,
+ other => panic!("expected Parse, got {other:?}"),
+ };
+ // A schema of JSON, as PySpark's `StructType.fromJson`; one of DDL
the server parses
+ // (the e2e tests check it).
+ let json = DataType::from_ddl("a INT, b STRING").unwrap().json();
+ let parse = parse_of(
+ spark.read().schema(json).option("sep", ";"),
+ ParseFormat::Csv,
+ );
+ assert_eq!(parse.format, proto::parse::ParseFormat::Csv as i32);
+ assert_eq!(
+ parse.schema,
+ Some(DataType::from_ddl("a INT, b STRING").unwrap().to_proto())
+ );
+ assert_eq!(parse.options.get("sep").map(String::as_str), Some(";"));
+ assert!(matches!(
+ parse.input.and_then(|input| input.rel_type),
+ Some(RelType::Range(_))
+ ));
+ let json = DataType::from_ddl("c BIGINT").unwrap().json();
+ let parse = parse_of(spark.read().schema(json), ParseFormat::Json);
+ assert_eq!(parse.format, proto::parse::ParseFormat::Json as i32);
+ assert_eq!(
+ parse.schema,
+ Some(DataType::from_ddl("c BIGINT").unwrap().to_proto())
+ );
+ // Without one, the server infers it.
+ let parse = parse_of(spark.read(), ParseFormat::Xml);
+ assert_eq!(parse.format, proto::parse::ParseFormat::Xml as i32);
+ assert_eq!(parse.schema, None);
+ assert!(spark
+ .read()
+ .schema("{not json".to_string())
+ .parse(ParseFormat::Json, &strings)
+ .is_err());
+ }
}
diff --git a/crates/spark-connect/tests/e2e_coverage.rs
b/crates/spark-connect/tests/e2e_coverage.rs
index 8e13236..ed61e12 100644
--- a/crates/spark-connect/tests/e2e_coverage.rs
+++ b/crates/spark-connect/tests/e2e_coverage.rs
@@ -213,11 +213,97 @@ fn catalog_extras_surface() {
let _ = c.refresh_table("cov_view2");
let _ = c.clear_cache();
- // create_table / create_external_table: the client builds and submits the
- // catalog command; with a bogus source the server rejects it, but the
client
- // serialization path runs either way. Accept success or a server error.
- let _ = c.create_table("cov_created_tbl", None, Some("parquet"),
Some("desc"));
- let _ = c.create_external_table("cov_ext_tbl", None, Some("parquet"));
+ // create_table and create_external_table create the table on the call, as
PySpark's,
+ // of the schema, the description and the options given.
+ let drop = |name: &str| {
+ let _ = s
+ .sql(&format!("DROP TABLE IF EXISTS {name}"))
+ .and_then(|d| d.collect());
+ };
+ drop("cov_eager_tbl");
+ let schema = spark_connect::types::DataType::from_ddl("a BIGINT, b
STRING").unwrap();
+ let options = HashMap::from([("compression".to_string(),
"snappy".to_string())]);
+ let created = c
+ .create_table_with(
+ "cov_eager_tbl",
+ None,
+ Some("parquet"),
+ Some(&schema),
+ Some("desc"),
+ options,
+ )
+ .unwrap();
+ assert!(c.table_exists("cov_eager_tbl").unwrap());
+ assert_eq!(created.columns().unwrap(), vec!["a", "b"]);
+ assert_eq!(
+ c.get_table_typed("cov_eager_tbl").unwrap().description,
+ Some("desc".to_string())
+ );
+ drop("cov_eager_tbl");
+ let dir = std::env::temp_dir().join(format!("cov_eager_ext_{}",
std::process::id()));
+ let path = dir.to_string_lossy().into_owned();
+ s.range(3)
+ .unwrap()
+ .write()
+ .mode("overwrite")
+ .parquet(&path)
+ .unwrap();
+ drop("cov_eager_ext");
+ let external = c
+ .create_external_table_with(
+ "cov_eager_ext",
+ Some(&path),
+ Some("parquet"),
+ None,
+ HashMap::new(),
+ )
+ .unwrap();
+ assert!(c.table_exists("cov_eager_ext").unwrap());
+ assert_eq!(external.count().unwrap(), 3);
+ drop("cov_eager_ext");
+ let _ = std::fs::remove_dir_all(&dir);
+}
+
+/// `spark.read.csv`, `json` and `xml` of a DataFrame of strings parse them,
as PySpark's
+/// `Parse`, with the schema and the options of the reader.
+#[test]
+fn reader_parses_a_dataframe_of_strings() {
+ if !should_run() {
+ return;
+ }
+ use spark_connect::plan::ParseFormat;
+ let s = session();
+ let csv = s.sql("SELECT '1;x' AS value").unwrap();
+ let parsed = s
+ .read()
+ .schema("a INT, b STRING".to_string())
+ .option("sep", ";")
+ .parse(ParseFormat::Csv, &csv)
+ .unwrap();
+ assert_eq!(parsed.columns().unwrap(), vec!["a", "b"]);
+ let rows = parsed.collect().unwrap();
+ assert_eq!(rows.len(), 1);
+ assert_eq!(rows[0].get(0).and_then(|v| v.as_i64()), Some(1));
+ let json = s.sql(r#"SELECT '{"a": 2}' AS value"#).unwrap();
+ let schema =
+
r#"{"type":"struct","fields":[{"name":"a","type":"long","nullable":true,"metadata":{}}]}"#;
+ let parsed = s
+ .read()
+ .schema(schema.to_string())
+ .parse(ParseFormat::Json, &json)
+ .unwrap();
+ assert_eq!(
+ parsed.collect().unwrap()[0].get(0).and_then(|v| v.as_i64()),
+ Some(2)
+ );
+ // Without a schema, the server infers one.
+ let xml = s.sql("SELECT '<row><a>3</a></row>' AS value").unwrap();
+ let parsed = s
+ .read()
+ .option("rowTag", "row")
+ .parse(ParseFormat::Xml, &xml)
+ .unwrap();
+ assert_eq!(parsed.columns().unwrap(), vec!["a"]);
}
/// DDL catalog operations added for v4.2.0 parity: create/drop database,
drop/analyze/
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index a2efb5e..e588adc 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -791,6 +791,52 @@ def test_reader_schema_keeps_field_metadata(session,
tmp_path, ddl):
assert stream.schema.fields[0].metadata == {"comment": "the key"}
+def test_reader_writer_and_catalog_return_the_results_of_pyspark(session,
tmp_path):
+ # As PySpark's: csv, json and xml of a DataFrame of strings parse them,
the readers load
+ # every path, and createTable creates the table on the call, of its schema
and options.
+ import warnings
+
+ strings = session.createDataFrame([("1;x",), ("2;y",)], "value string")
+ assert session.read.csv(strings, sep=";", schema="a INT, b
STRING").collect() == [
+ Row(a=1, b="x"), Row(a=2, b="y")]
+ assert session.read.csv(strings, sep=";").columns == ["_c0", "_c1"]
+ documents = session.createDataFrame([('{"a": 1, "b": "x"}',)], "value
string")
+ assert session.read.json(documents).collect() == [Row(a=1, b="x")]
+ assert session.read.json(documents, schema=StructType([StructField("a",
LongType())])) \
+ .collect() == [Row(a=1)]
+ elements = session.createDataFrame([("<row><a>1</a></row>",)], "value
string")
+ assert session.read.xml(elements, rowTag="row").collect() == [Row(a=1)]
+ for name in ("p1", "p2"):
+ session.range(2).write.mode("overwrite").parquet(str(tmp_path / name))
+ paths = [str(tmp_path / "p1"), str(tmp_path / "p2")]
+ assert session.read.parquet(*paths).count() == 4
+ assert session.read.format("parquet").load(paths).count() == 4
+ session.range(3).write.mode(None).json(str(tmp_path / "j"))
+ assert session.read.json([str(tmp_path / "j")]).count() == 3
+ session.sql("DROP TABLE IF EXISTS e2e_created_eagerly")
+ try:
+ schema = StructType([StructField("a", LongType()), StructField("b",
StringType())])
+ created = session.catalog.createTable(
+ "e2e_created_eagerly", source="parquet", schema=schema,
description="d",
+ compression="snappy")
+ assert session.catalog.tableExists("e2e_created_eagerly")
+ assert created.columns == ["a", "b"]
+ assert session.catalog.getTable("e2e_created_eagerly").description ==
"d"
+ finally:
+ session.sql("DROP TABLE IF EXISTS e2e_created_eagerly")
+ session.sql("DROP TABLE IF EXISTS e2e_external_eagerly")
+ try:
+ with warnings.catch_warnings(record=True) as record:
+ warnings.simplefilter("always")
+ external = session.catalog.createExternalTable(
+ "e2e_external_eagerly", path=paths[0], source="parquet")
+ assert session.catalog.tableExists("e2e_external_eagerly",
"default")
+ assert external.count() == 2
+ assert [w.category for w in record] == [FutureWarning, FutureWarning]
+ finally:
+ session.sql("DROP TABLE IF EXISTS e2e_external_eagerly")
+
+
def test_subqueries_and_table_arguments_return_the_rows_of_pyspark(session):
# As PySpark's DataFrame.scalar(), DataFrame.exists(), Column.isin(df) and
asTable():
# scalar, correlated and IN subqueries, and the rows of a DataFrame as a
UDTF argument.
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 9d4bd24..b0bcb74 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -9297,6 +9297,214 @@ def
test_reader_schema_is_sent_as_pyspark_sends_it(mock_connect, read):
"arg_name": "schema", "expected_type": "str or struct type",
"arg_type": "LongType"}
+def _sent_relation(request, field):
+ """The relation `field` of the root of a captured AnalyzePlanRequest or
ExecutePlanRequest:
+ AnalyzePlanRequest.schema=4 -> plan=1, or ExecutePlanRequest.plan=3; then
Plan.root=1."""
+ message = parse_message(request)
+ plan = parse_message(message[4][0])[1][0] if 4 in message else
message[3][0]
+ root = parse_message(parse_message(plan)[1][0])
+ assert field in root, sorted(root)
+ return parse_message(root[field][0])
+
+
+def _struct_field_names(data_type):
+ # DataType.struct=21 -> Struct.fields=1 -> StructField.name=1.
+ struct = parse_message(parse_message(data_type)[21][0])
+ return [parse_message(field)[1][0].decode() for field in struct[1]]
+
+
[email protected]("read, format, option", [
+ (lambda spark, df: spark.read.csv(df, schema="a INT, b STRING", sep=";"),
1, b"sep"),
+ (lambda spark, df: spark.read.json(df, schema="a INT, b STRING",
lineSep=";"), 2,
+ b"lineSep"),
+ (lambda spark, df: spark.read.xml(df, schema="a INT, b STRING",
rowTag=";"), 3, b"rowTag"),
+], ids=["csv", "json", "xml"])
+def test_reader_parses_a_dataframe_of_strings_as_pyspark(mock_connect, read,
format, option):
+ # As PySpark's, csv, json and xml of a DataFrame parse its strings: a
Parse (Relation.parse
+ # =30) of the DataFrame (input=1) in the format (format=2), with the
schema (schema=3) of
+ # the reader and its options (options=4).
+ spark, service = mock_connect
+ strings = T.StructType([T.StructField("value", T.StringType())])
+ read(spark, spark.createDataFrame([("1;x",)], strings)).columns
+ parse = _sent_relation(service.analyze_requests[-1], 30)
+ assert parse[2] == [format]
+ # The server parses a DDL schema, as PySpark's `StructType.fromDDL`:
+ # AnalyzePlanRequest.ddl_parse=11 -> ddl_string=1.
+ assert [parse_message(parse_message(request)[11][0])[1][0]
+ for request in service.analyze_requests if 11 in
parse_message(request)] == [
+ b"a INT, b STRING"]
+ assert _struct_field_names(parse[3][0]) == ["a", "b"]
+ assert parse_message(parse[4][0]) == {1: [option], 2: [b";"]}
+ assert 1 in parse
+ # A schema of JSON, as a StructType sends it, or none.
+ struct = T.StructType([T.StructField("c", T.LongType())])
+ spark.read.json(spark.createDataFrame([("{}",)], strings),
schema=struct).columns
+ assert _struct_field_names(_sent_relation(service.analyze_requests[-1],
30)[3][0]) == ["c"]
+ spark.read.csv(spark.createDataFrame([("1",)], strings)).columns
+ assert 3 not in _sent_relation(service.analyze_requests[-1], 30)
+ # The reader keeps the schema and the options of a parse, but not its
format, which
+ # only `load` sets.
+ reader = spark.read.format("parquet")
+ reader.csv(spark.createDataFrame([("1",)], strings), schema="a INT",
sep=";")
+ assert (reader._format, reader._schema, reader._options) == ("parquet", "a
INT", {"sep": ";"})
+ reader.csv("p", sep=",")
+ assert (reader._format, reader._schema, reader._options) == ("csv", "a
INT", {"sep": ","})
+
+
+def test_reader_paths_are_checked_as_pyspark(offline_spark):
+ from pyspark.errors import PySparkTypeError
+
+ # json and xml take a path, a list of them or a DataFrame; csv loads any
other path,
+ # which load then asserts, as PySpark's.
+ for read in (offline_spark.read.json, offline_spark.read.xml):
+ for path in (1, ("p",)):
+ with pytest.raises(PySparkTypeError) as error:
+ read(path)
+ assert error.value.getCondition() == "NOT_EXPECTED_TYPE"
+ assert error.value.getMessageParameters() == {
+ "arg_name": "path", "expected_type": "str, list, or DataFrame",
+ "arg_type": type(path).__name__}
+ for read in (
+ lambda path: offline_spark.read.csv(path),
+ lambda path: offline_spark.read.load(path),
+ lambda path: offline_spark.read.orc(path),
+ lambda path: offline_spark.read.text(path),
+ lambda path: offline_spark.read.json([path]),
+ ):
+ for path in (("p",), 1):
+ with pytest.raises(AssertionError):
+ read(path)
+ with pytest.raises(AssertionError):
+ offline_spark.read.parquet("p", 1)
+
+
+def _sent_data_source_paths(request):
+ # Relation.read=2 -> Read.data_source=2 -> DataSource.paths=4.
+ read = _sent_relation(request, 2)
+ return [path.decode() for path in parse_message(read[2][0]).get(4, [])]
+
+
[email protected]("read", [
+ lambda spark: spark.read.json(["a", "b"]),
+ lambda spark: spark.read.csv(["a", "b"]),
+ lambda spark: spark.read.xml(["a", "b"], rowTag="row"),
+ lambda spark: spark.read.parquet("a", "b"),
+ lambda spark: spark.read.orc(["a", "b"]),
+ lambda spark: spark.read.text(["a", "b"]),
+ lambda spark: spark.read.load(["a", "b"], format="json"),
+], ids=["json", "csv", "xml", "parquet", "orc", "text", "load"])
+def test_reader_loads_every_path_as_pyspark(mock_connect, read):
+ spark, service = mock_connect
+ read(spark).columns
+ assert _sent_data_source_paths(service.analyze_requests[-1]) == ["a", "b"]
+
+
+def
test_writer_bucket_sort_and_cluster_columns_are_checked_as_pyspark(offline_spark):
+ from pyspark.errors import PySparkTypeError, PySparkValueError
+
+ w = offline_spark.range(3).write
+ assert w.mode(None) is w
+ assert w.bucketBy(True, "id") is w
+ assert w.bucketBy(2, ["id"]) is w and w.bucketBy(2, ("id", "x")) is w
+ assert w.sortBy(["id"]) is w and w.sortBy("id", "x") is w
+ assert w.clusterBy(["id"]) is w and w.clusterBy(("id", "x")) is w
+ for call, condition, parameters in [
+ (lambda: w.bucketBy("2", "id"), "NOT_EXPECTED_TYPE",
+ {"arg_name": "numBuckets", "expected_type": "int", "arg_type":
"str"}),
+ (lambda: w.bucketBy(2.0, "id"), "NOT_EXPECTED_TYPE",
+ {"arg_name": "numBuckets", "expected_type": "int", "arg_type":
"float"}),
+ (lambda: w.bucketBy(2, ["id"], "x"), "CANNOT_SET_TOGETHER",
+ {"arg_list": "`col` of type list and `cols`"}),
+ (lambda: w.sortBy(("id",), "x"), "CANNOT_SET_TOGETHER",
+ {"arg_list": "`col` of type tuple and `cols`"}),
+ (lambda: w.bucketBy(2, 1), "NOT_EXPECTED_TYPE",
+ {"arg_name": "col", "expected_type": "list[str]", "arg_type": "int"}),
+ (lambda: w.bucketBy(2, "id", 1), "NOT_EXPECTED_TYPE",
+ {"arg_name": "cols", "expected_type": "list[str]", "arg_type":
"int"}),
+ (lambda: w.sortBy(["id", 1]), "NOT_EXPECTED_TYPE",
+ {"arg_name": "cols", "expected_type": "list[str]", "arg_type":
"int"}),
+ (lambda: w.sortBy(1), "NOT_EXPECTED_TYPE",
+ {"arg_name": "col", "expected_type": "list[str]", "arg_type": "int"}),
+ ]:
+ with pytest.raises((PySparkTypeError, PySparkValueError)) as error:
+ call()
+ assert error.value.getCondition() == condition
+ assert error.value.getMessageParameters() == parameters
+ # As PySpark's `col[0]` of an empty list, and its assertion of the
clustering columns.
+ with pytest.raises(IndexError):
+ w.bucketBy(2, [])
+ for call in (w.clusterBy, lambda: w.clusterBy([])):
+ with pytest.raises(AssertionError, match="clusterBy needs one or
more"):
+ call()
+
+
+def _sent_catalog(request, field):
+ # Relation.catalog=200 -> the Catalog operation `field`.
+ return parse_message(_sent_relation(request, 200)[field][0])
+
+
+def test_catalog_creates_tables_eagerly_as_pyspark(mock_connect):
+ import warnings
+
+ from pyspark.errors import PySparkTypeError
+
+ spark, service = mock_connect
+ struct = T.StructType([T.StructField("a", T.LongType()),
T.StructField("b", T.StringType())])
+ # As PySpark's, createTable runs on the call, a CreateTable
(Catalog.create_table=14) of
+ # the table (table_name=1), path=2, source=3, description=4, schema=5 and
options=6, of
+ # strings, where a None is left out.
+ spark.catalog.createTable(
+ "t", path="/p", source="parquet", schema=struct, description="d",
k="v", skipped=None)
+ create = _sent_catalog(service.execute_requests[-1], 14)
+ assert (create[1], create[2], create[3], create[4]) == ([b"t"], [b"/p"],
[b"parquet"], [b"d"])
+ assert _struct_field_names(create[5][0]) == ["a", "b"]
+ assert [parse_message(entry) for entry in create[6]] == [{1: [b"k"], 2:
[b"v"]}]
+ with pytest.raises(TypeError, match="1 has type <class 'int'>"):
+ spark.catalog.createTable("t", source="parquet", k=1)
+ for schema in ("a INT", T.LongType()):
+ with pytest.raises(PySparkTypeError) as error:
+ spark.catalog.createTable("t", source="parquet", schema=schema)
+ assert error.value.getCondition() == "NOT_EXPECTED_TYPE"
+ assert error.value.getMessageParameters() == {
+ "arg_name": "schema", "expected_type": "struct type",
+ "arg_type": type(schema).__name__}
+ # createExternalTable warns that it is deprecated and creates the table as
createTable.
+ calls = len(service.execute_requests)
+ with pytest.warns(FutureWarning, match="createExternalTable is deprecated
since Spark 4.0"):
+ spark.catalog.createExternalTable("e", path="/e", source="csv",
schema=struct, sep=";")
+ assert len(service.execute_requests) == calls + 1
+ create = _sent_catalog(service.execute_requests[-1], 14)
+ assert (create[1], create[2], create[3]) == ([b"e"], [b"/e"], [b"csv"])
+ assert 4 not in create and _struct_field_names(create[5][0]) == ["a", "b"]
+ assert [parse_message(entry) for entry in create[6]] == [{1: [b"sep"], 2:
[b";"]}]
+ with warnings.catch_warnings():
+ warnings.simplefilter("ignore", FutureWarning)
+ with pytest.raises(PySparkTypeError):
+ spark.catalog.createExternalTable("e", schema=1)
+
+
[email protected]("call, method", [
+ (lambda catalog, *db: catalog.tableExists("t", *db), "tableExists"),
+ (lambda catalog, *db: catalog.functionExists("f", *db), "functionExists"),
+ (lambda catalog, *db: catalog.listColumns("t", *db), "listColumns"),
+], ids=["tableExists", "functionExists", "listColumns"])
+def test_catalog_warns_of_the_deprecated_db_name_as_pyspark(mock_connect,
call, method):
+ import contextlib
+ import warnings
+
+ spark, _ = mock_connect
+ # As PySpark's, only a call of `dbName` warns, before its request, which
the mock answers
+ # with no rows.
+ with warnings.catch_warnings(record=True) as record:
+ warnings.simplefilter("always")
+ for db in ((), ("db",)):
+ with contextlib.suppress(Exception):
+ call(spark.catalog, *db)
+ assert [str(w.message) for w in record if issubclass(w.category,
FutureWarning)] == [
+ "`dbName` has been deprecated since Spark 3.4 and might be removed in
a future version. "
+ f"Use {method}(`dbName.tableName`) instead."]
+
+
# Exact built-in string conversion guards. A Row made in Python keeps its
values as they
# are, as PySpark's does, so these check the conversion that createDataFrame
makes.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]