This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 6db70f0 [SPARK-61101][CONNECT][PYTHON] Validate the save mode of
DataFrameWriter, and the arguments of DataFrameReader, in the Python client as
PySpark does
6db70f0 is described below
commit 6db70f09cd6e9a8a2893b68425c381b022f04180
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 17:49:45 2026 +0900
[SPARK-61101][CONNECT][PYTHON] Validate the save mode of DataFrameWriter,
and the arguments of DataFrameReader, in the Python client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61101](https://issues.apache.org/jira/browse/SPARK-61101): the batch
reader and writer now check their arguments as PySpark Connect's do.
As PySpark Connect's:
- A write raises `UNSUPPORTED_OPERATION` when the last save mode set is not
`append`, `overwrite`, `error` or `ignore`, in any case. This covers `save`,
`saveAsTable`, `insertInto`, the format methods and `jdbc`. The core's
`DataFrameWriter` fails the write; it used to ignore the mode.
- `format(None)` of `DataFrameReader`, `DataFrameWriter` and
`DataStreamReader` unsets the format, so the server uses its default source.
The core's readers and writer gain `clear_format`.
- `DataFrameReader._jreader` raises `JVM_ATTRIBUTE_NOT_SUPPORTED`.
- `DataFrameReader.jdbc`:
- sets the format `jdbc` on the reader first
- with a `column`, asserts `lowerBound`, `upperBound` and `numPartitions`
(for example "lowerBound can not be None when ``column`` is specified") and
ignores `predicates`
- without a `column`, ignores the bounds and `numPartitions`
- takes the column, the bounds, `numPartitions` and the values of
`properties` as options of any type, like `DataFrameWriter.jdbc`'s `properties`
- `DataFrameReader.changes` of a reader with a schema raises the
`AnalysisException` `_LEGACY_ERROR_TEMP_1189`, as `DataStreamReader.changes`
already does. The two share the code now.
- `csv`, `text` and `orc` of None read without a path.
The client keeps `errorifexists` as the save mode `error`. PySpark
documents it, and pandas-on-Spark's `validate_mode` passes it to the writer.
PySpark Connect 4.3.0 raises `UNSUPPORTED_OPERATION` for it, against its own
documentation.
### Why are the changes needed?
The client:
- accepted any save mode: `df.write.mode("bad").save(path)` saved the data
with the default mode, and a typo like `mode("overwite")` failed later with
`PATH_ALREADY_EXISTS`
- raised a `TypeError` for `format(None)`
- sent a JDBC read of a column without bounds to the server
- dropped the schema of a reader in `changes`
- raised an `AttributeError` for `_jreader`
- raised an `AssertionError` for `csv(None)`, `text(None)` and `orc(None)`
### Does this PR introduce _any_ user-facing change?
Yes. The reader and the writer take their arguments, and raise their
errors, as in PySpark 4.3.0.
### How was this patch tested?
- A new live test checks an unsupported save mode, which writes nothing;
`format(None)`, which writes and reads Parquet; `changes` of a schema; `jdbc`
of a column without bounds; `_jreader`; and `text(None)`. It passes with
official PySpark 4.3.0 and fails without this change.
- A new offline test checks these errors and the requests sent for each
mode and format, and for the options and predicates of `jdbc`.
- A new core test checks the save modes of a write.
- On a local server, 160 cases of the reader and the writer with good and
bad arguments give the same results as official PySpark 4.3.0. These are left:
- the protobuf and PyO3 messages of a `TypeError` of an argument of a
wrong type
- `errorifexists`, kept as above
- `bucketBy(0, ...)`: PySpark Connect drops the bucketing (it sends one
only for a positive count), while the client sends it and the server raises
`INVALID_BUCKET_COUNT`, as classic PySpark does
- The lib tests of both crates (589 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2051 passed, 12 skipped), and the
live drop-in and Arrow suites (155 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #302 from HyukjinKwon/SPARK-61101-readwriter-args.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/dataframe.rs | 23 ++++--
crates/pyspark-rs/src/readwriter.rs | 128 +++++++++++++++++++++++----------
crates/pyspark-rs/src/streaming.rs | 27 +++----
crates/spark-connect/src/readwriter.rs | 76 +++++++++++++++++++-
crates/spark-connect/src/streaming.rs | 7 ++
python/tests/test_dropin_e2e.py | 27 +++++++
python/tests/test_dropin_offline.py | 76 ++++++++++++++++++++
7 files changed, 304 insertions(+), 60 deletions(-)
diff --git a/crates/pyspark-rs/src/dataframe.rs
b/crates/pyspark-rs/src/dataframe.rs
index 1d47c1a..16e6fd1 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -3002,8 +3002,16 @@ impl PyDataFrameWriter {
Ok(slf)
}
- fn format<'py>(mut slf: PyRefMut<'py, Self>, source: &str) ->
PyResult<PyRefMut<'py, Self>> {
- slf.inner = slf.inner.clone().format(source);
+ /// The format of the write. As PySpark's, None unsets it, so that the
server writes its
+ /// default source.
+ fn format<'py>(
+ mut slf: PyRefMut<'py, Self>,
+ source: Option<&str>,
+ ) -> PyResult<PyRefMut<'py, Self>> {
+ slf.inner = match source {
+ Some(source) => slf.inner.clone().format(source),
+ None => slf.inner.clone().clear_format(),
+ };
Ok(slf)
}
@@ -3096,7 +3104,8 @@ impl PyDataFrameWriter {
Ok(slf)
}
- /// Write to an external JDBC table. Connection details go through
`properties`.
+ /// Write to an external JDBC table. Connection details go through
`properties`, options of
+ /// any type (None skipped, bools lowercased), as PySpark's.
#[pyo3(signature = (url, table, mode=None, properties=None))]
fn jdbc(
&mut self,
@@ -3104,7 +3113,7 @@ impl PyDataFrameWriter {
url: &str,
table: &str,
mode: Option<&str>,
- properties: Option<std::collections::HashMap<String, String>>,
+ properties: Option<&Bound<'_, PyDict>>,
) -> PyResult<()> {
// Same order as PySpark: format, mode, properties, then url/dbtable
(which
// win over same-named properties), all kept on the writer, then save.
@@ -3113,7 +3122,11 @@ impl PyDataFrameWriter {
w = w.mode(m);
}
if let Some(props) = properties {
- w = w.options(props);
+ for (key, value) in props.iter() {
+ if let Some(value) = crate::coerce_option_value(&value)? {
+ w = w.option(&key.str()?.to_cow()?, &value);
+ }
+ }
}
w = w.option("url", url).option("dbtable", table);
self.inner = w.clone();
diff --git a/crates/pyspark-rs/src/readwriter.rs
b/crates/pyspark-rs/src/readwriter.rs
index 998b028..2c22f04 100644
--- a/crates/pyspark-rs/src/readwriter.rs
+++ b/crates/pyspark-rs/src/readwriter.rs
@@ -80,9 +80,12 @@ impl PyDataFrameReader {
}
}
-/// The paths of `path`, a path or a list of them, as PySpark's `load` takes
them: anything
-/// else fails its assertion.
+/// The paths of `path`, a path or a list of them, or None for none, as
PySpark's `load` takes
+/// them: anything else fails its assertion.
fn paths_of(path: &Bound<'_, PyAny>) -> PyResult<Vec<String>> {
+ if path.is_none() {
+ return Ok(vec![]);
+ }
if path.is_instance_of::<pyo3::types::PyString>() {
return Ok(vec![path.extract()?]);
}
@@ -216,6 +219,29 @@ pub(crate) fn reader_schema(schema: &Bound<'_, PyAny>) ->
PyResult<String> {
))
}
+/// The `AnalysisException` of `_LEGACY_ERROR_TEMP_1189` that PySpark's
`DataFrameReader.changes`
+/// and `DataStreamReader.changes` raise for a reader of a schema, of the
message of the JVM
+/// error class, which PySpark's error classes lack.
+pub(crate) fn changes_of_schema_error(py: Python<'_>) -> PyErr {
+ let exception = (|| {
+ let kwargs = PyDict::new(py);
+ kwargs.set_item("errorClass", "_LEGACY_ERROR_TEMP_1189")?;
+ let parameters = PyDict::new(py);
+ parameters.set_item("operation", "changes")?;
+ kwargs.set_item("messageParameters", parameters)?;
+ py.import("pyspark.errors")?
+ .getattr("AnalysisException")?
+ .call(
+ ("User specified schema not supported with `changes`.",),
+ Some(&kwargs),
+ )
+ })();
+ match exception {
+ Ok(exception) => PyErr::from_value(exception),
+ Err(err) => err,
+ }
+}
+
/// Apply the `schema` argument of a format method, as pyspark does through
`schema()`.
fn apply_reader_schema(r: DataFrameReader, v: &Bound<'_, PyAny>) ->
PyResult<DataFrameReader> {
Ok(r.schema(reader_schema(v)?))
@@ -223,9 +249,13 @@ fn apply_reader_schema(r: DataFrameReader, v: &Bound<'_,
PyAny>) -> PyResult<Dat
#[pymethods]
impl PyDataFrameReader {
- /// Set the source format (e.g. "parquet", "json", "csv").
- fn format<'py>(mut slf: PyRefMut<'py, Self>, source: &str) ->
PyRefMut<'py, Self> {
- slf.inner = slf.inner.clone().format(source);
+ /// Set the source format (e.g. "parquet", "json", "csv"). As PySpark's,
None unsets it, so
+ /// that the server reads its default source.
+ fn format<'py>(mut slf: PyRefMut<'py, Self>, source: Option<&str>) ->
PyRefMut<'py, Self> {
+ slf.inner = match source {
+ Some(source) => slf.inner.clone().format(source),
+ None => slf.inner.clone().clear_format(),
+ };
slf
}
@@ -289,6 +319,18 @@ impl PyDataFrameReader {
self.inner.get_options().clone()
}
+ /// The JVM reader of classic PySpark, which PySpark Connect's raises
+ /// `JVM_ATTRIBUTE_NOT_SUPPORTED` for.
+ #[getter]
+ fn _jreader(&self, py: Python<'_>) -> PyResult<()> {
+ Err(crate::column::raise_pyspark(
+ py,
+ "PySparkAttributeError",
+ "JVM_ATTRIBUTE_NOT_SUPPORTED",
+ &[("attr_name", "_jreader")],
+ ))
+ }
+
/// Load data from the (optional) path. Mirrors the reference
/// `DataFrameReader.load(path=None, format=None, schema=None,
**options)`: the
/// one-shot form applies format/schema/options here before reading,
equivalent to
@@ -319,9 +361,13 @@ impl PyDataFrameReader {
Ok(PyDataFrame::new(self.inner.clone().table(tableName)))
}
- /// Read the CDC changes of a named table. Mirrors
`DataFrameReader.changes`.
+ /// Read the CDC changes of a named table. Mirrors
`DataFrameReader.changes`, which raises an
+ /// `AnalysisException` of `_LEGACY_ERROR_TEMP_1189` for a reader of a
schema.
#[allow(non_snake_case)]
- fn changes(&self, tableName: &str) -> PyResult<PyDataFrame> {
+ fn changes(&self, py: Python<'_>, tableName: &str) ->
PyResult<PyDataFrame> {
+ if !self.inner.get_schema().is_empty() {
+ return Err(changes_of_schema_error(py));
+ }
Ok(PyDataFrame::new(self.inner.clone().changes(tableName)))
}
@@ -593,48 +639,58 @@ impl PyDataFrameReader {
self.read_or_parse(r, "xml", spark_connect::plan::ParseFormat::Xml,
path)
}
- /// Read from a JDBC source. Mirrors `DataFrameReader.jdbc(url, table,
- /// column=None, lowerBound=None, upperBound=None, numPartitions=None,
- /// predicates=None, properties=None)`: the column/bound/partition args
and the
- /// connection `properties` are threaded as reader options (connect
represents
- /// them that way); `predicates` stays the partitioning predicate list.
+ /// Read from a JDBC source. Mirrors `DataFrameReader.jdbc(url, table,
column=None,
+ /// lowerBound=None, upperBound=None, numPartitions=None, predicates=None,
properties=None)`,
+ /// which keeps the format, then the options, on the reader:
+ /// - of a `column`, it asserts the bounds and `numPartitions`, and reads
the partitions of
+ /// the column, ignoring `predicates`;
+ /// - else, it reads the partitions of `predicates`, ignoring the bounds
and `numPartitions`.
+ ///
+ /// The column, the bounds, `numPartitions` and the values of `properties`
are options, of any
+ /// type (None skipped, bools lowercased).
#[pyo3(signature = (url, table, column=None, lowerBound=None,
upperBound=None, numPartitions=None, predicates=None, properties=None))]
#[allow(non_snake_case, clippy::too_many_arguments)]
fn jdbc(
&mut self,
url: &str,
table: &str,
- column: Option<String>,
+ column: Option<&Bound<'_, PyAny>>,
lowerBound: Option<&Bound<'_, PyAny>>,
upperBound: Option<&Bound<'_, PyAny>>,
- numPartitions: Option<i32>,
+ numPartitions: Option<&Bound<'_, PyAny>>,
predicates: Option<Vec<String>>,
- properties: Option<HashMap<String, String>>,
+ properties: Option<&Bound<'_, PyDict>>,
) -> PyResult<PyDataFrame> {
+ self.inner = self.inner.clone().format("jdbc");
let mut r = self.inner.clone();
- if let Some(c) = column {
- r = r.option("partitionColumn", &c);
- }
- if let Some(lb) = lowerBound {
- r = r.option("lowerBound", &lb.str()?.to_string());
- }
- if let Some(ub) = upperBound {
- r = r.option("upperBound", &ub.str()?.to_string());
- }
- if let Some(n) = numPartitions {
- r = r.option("numPartitions", &n.to_string());
- }
- if let Some(props) = properties {
- for (k, v) in props {
- r = r.option(&k, &v);
+ let predicates = match column {
+ Some(column) => {
+ for (bound, name) in [
+ (lowerBound, "lowerBound"),
+ (upperBound, "upperBound"),
+ (numPartitions, "numPartitions"),
+ ] {
+ if bound.is_none() {
+ return
Err(pyo3::exceptions::PyAssertionError::new_err(format!(
+ "{name} can not be None when ``column`` is
specified"
+ )));
+ }
+ }
+ r = set_opt(r, "partitionColumn", Some(column))?;
+ r = set_opt(r, "lowerBound", lowerBound)?;
+ r = set_opt(r, "upperBound", upperBound)?;
+ r = set_opt(r, "numPartitions", numPartitions)?;
+ None
+ }
+ None => predicates,
+ };
+ if let Some(properties) = properties {
+ for (key, value) in properties.iter() {
+ r = set_opt(r, &key.str()?.to_cow()?, Some(&value))?;
}
}
- // PySpark's jdbc also leaves format/url/dbtable on the reader.
- self.inner = r
- .clone()
- .format("jdbc")
- .option("url", url)
- .option("dbtable", table);
+ // PySpark's jdbc also leaves url/dbtable on the reader.
+ self.inner = r.clone().option("url", url).option("dbtable", table);
Ok(PyDataFrame::new(r.jdbc(url, table, predicates)))
}
}
diff --git a/crates/pyspark-rs/src/streaming.rs
b/crates/pyspark-rs/src/streaming.rs
index 583d15e..5de4b14 100644
--- a/crates/pyspark-rs/src/streaming.rs
+++ b/crates/pyspark-rs/src/streaming.rs
@@ -148,8 +148,16 @@ impl PyDataStreamReader {
#[pymethods]
impl PyDataStreamReader {
- fn format<'py>(mut slf: PyRefMut<'py, Self>, source: &str) ->
PyResult<PyRefMut<'py, Self>> {
- slf.inner = slf.inner.clone().format(source);
+ /// Set the source format. As PySpark's, None unsets it, so that the
server reads its
+ /// default source.
+ fn format<'py>(
+ mut slf: PyRefMut<'py, Self>,
+ source: Option<&str>,
+ ) -> PyResult<PyRefMut<'py, Self>> {
+ slf.inner = match source {
+ Some(source) => slf.inner.clone().format(source),
+ None => slf.inner.clone().clear_format(),
+ };
Ok(slf)
}
@@ -278,20 +286,7 @@ impl PyDataStreamReader {
/// schema.
fn changes(&self, py: Python<'_>, tableName: &str) ->
PyResult<PyDataFrame> {
if !self.inner.get_schema().is_empty() {
- // Of the message of the JVM error class, which PySpark's error
classes lack.
- let kwargs = PyDict::new(py);
- kwargs.set_item("errorClass", "_LEGACY_ERROR_TEMP_1189")?;
- let parameters = PyDict::new(py);
- parameters.set_item("operation", "changes")?;
- kwargs.set_item("messageParameters", parameters)?;
- let exception = py
- .import("pyspark.errors")?
- .getattr("AnalysisException")?
- .call(
- ("User specified schema not supported with `changes`.",),
- Some(&kwargs),
- )?;
- return Err(PyErr::from_value(exception));
+ return Err(crate::readwriter::changes_of_schema_error(py));
}
Ok(PyDataFrame::new(self.inner.clone().changes(tableName)))
}
diff --git a/crates/spark-connect/src/readwriter.rs
b/crates/spark-connect/src/readwriter.rs
index 3ed5044..c1d8ea9 100644
--- a/crates/spark-connect/src/readwriter.rs
+++ b/crates/spark-connect/src/readwriter.rs
@@ -41,6 +41,13 @@ impl DataFrameReader {
self
}
+ /// Unset the format, so that the server reads its default source
+ /// (`spark.sql.sources.default`), as PySpark's `format(None)`.
+ pub fn clear_format(mut self) -> Self {
+ self.format = None;
+ self
+ }
+
/// Set the schema from a DDL string or JSON string.
pub fn schema(mut self, schema: String) -> Self {
self.schema = schema;
@@ -277,6 +284,8 @@ pub struct DataFrameWriter {
input_plan: Arc<LogicalPlan>,
format: Option<String>,
mode: SaveMode,
+ /// The last mode set when it is not a save mode, which fails the write.
+ unsupported_mode: Option<String>,
options: HashMap<String, String>,
partition_cols: Vec<String>,
cluster_cols: Vec<String>,
@@ -299,6 +308,7 @@ impl DataFrameWriter {
input_plan,
format: None,
mode: SaveMode::ErrorIfExists,
+ unsupported_mode: None,
options: HashMap::new(),
partition_cols: vec![],
cluster_cols: vec![],
@@ -331,10 +341,16 @@ impl DataFrameWriter {
self
}
- /// Set the save mode.
+ /// Set the save mode: `append`, `overwrite`, `error` (or `errorifexists`)
or `ignore`, in
+ /// any case. As PySpark's, the write fails with `UNSUPPORTED_OPERATION`
when the last mode
+ /// set is any other string.
pub fn mode(mut self, mode: &str) -> Self {
- if let Some(m) = SaveMode::from_str(mode) {
- self.mode = m;
+ match SaveMode::from_str(mode) {
+ Some(m) => {
+ self.mode = m;
+ self.unsupported_mode = None;
+ }
+ None => self.unsupported_mode = Some(mode.to_string()),
}
self
}
@@ -345,6 +361,13 @@ impl DataFrameWriter {
self
}
+ /// Unset the format, so that the server writes its default source
+ /// (`spark.sql.sources.default`), as PySpark's `format(None)`.
+ pub fn clear_format(mut self) -> Self {
+ self.format = None;
+ self
+ }
+
/// Set a single option.
pub fn option(mut self, key: &str, value: &str) -> Self {
self.options.insert(key.to_string(), value.to_string());
@@ -385,6 +408,12 @@ impl DataFrameWriter {
&self,
save_type: Option<proto::write_operation::SaveType>,
) -> Result<proto::WriteOperation> {
+ if let Some(mode) = &self.unsupported_mode {
+ return Err(spark_connect_core::error::SparkError::value(
+ "UNSUPPORTED_OPERATION",
+ &[("operation", mode)],
+ ));
+ }
let mut op = proto::WriteOperation::default();
op.input = Some(build_input_relation(&self.input_plan, &self.session));
op.source = self.format.clone();
@@ -669,6 +698,47 @@ mod tests {
}
}
+ #[test]
+ fn v1_write_operation_of_an_unsupported_mode_fails() {
+ // As PySpark's: the last mode set decides, in any case, and any other
string fails the
+ // write with UNSUPPORTED_OPERATION.
+ let spark = session();
+ let df = spark.range(3).unwrap();
+ let path = || {
+ Some(proto::write_operation::SaveType::Path(
+ "/tmp/out".to_string(),
+ ))
+ };
+ for mode in ["bad", "default", "", " append"] {
+ let err = df
+ .write()
+ .mode("append")
+ .mode(mode)
+ .build_write_operation(path())
+ .unwrap_err();
+ assert_eq!(
+ err.get_condition().as_deref(),
+ Some("UNSUPPORTED_OPERATION")
+ );
+ assert_eq!(err.params.get("operation").map(String::as_str),
Some(mode));
+ }
+ for (mode, expected) in [
+ ("APPEND", SaveMode::Append),
+ ("Overwrite", SaveMode::Overwrite),
+ ("error", SaveMode::ErrorIfExists),
+ ("errorifexists", SaveMode::ErrorIfExists),
+ ("ignore", SaveMode::Ignore),
+ ] {
+ let op = df
+ .write()
+ .mode("bad")
+ .mode(mode)
+ .build_write_operation(path())
+ .unwrap();
+ assert_eq!(op.mode, expected.to_proto());
+ }
+ }
+
#[test]
fn v1_write_operation_save_as_table() {
let spark = session();
diff --git a/crates/spark-connect/src/streaming.rs
b/crates/spark-connect/src/streaming.rs
index bd864c0..9e17be0 100644
--- a/crates/spark-connect/src/streaming.rs
+++ b/crates/spark-connect/src/streaming.rs
@@ -47,6 +47,13 @@ impl DataStreamReader {
self
}
+ /// Unset the format, so that the server reads its default source
+ /// (`spark.sql.sources.default`), as PySpark's `format(None)`.
+ pub fn clear_format(mut self) -> Self {
+ self.format = None;
+ self
+ }
+
/// Set the schema from a DDL string or JSON string.
pub fn schema(mut self, schema: impl Into<String>) -> Self {
self.schema = schema.into();
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 3f09677..13cd21f 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1302,6 +1302,33 @@ def
test_dataframe_columns_resolve_against_their_dataframe_as_pyspark(session):
assert error.value.getCondition() == "AMBIGUOUS_COLUMN_REFERENCE"
+def test_reader_and_writer_check_their_arguments_as_pyspark(session, tmp_path):
+ # As PySpark Connect's: a write of an unsupported save mode raises
UNSUPPORTED_OPERATION
+ # and writes nothing; format(None) writes and reads the default source,
Parquet; the reader
+ # refuses a schema in changes and asserts the bounds of a JDBC column;
text of None reads
+ # no rows.
+ from pyspark.errors import PySparkAttributeError
+
+ df = session.createDataFrame([(1, "a"), (2, "b")], "id int, s string")
+ path = str(tmp_path / "out")
+ with pytest.raises(PySparkValueError) as e:
+ df.write.mode("bad").save(path)
+ assert e.value.getCondition() == "UNSUPPORTED_OPERATION"
+ assert e.value.getMessageParameters() == {"operation": "bad"}
+ assert not os.path.exists(path)
+ df.write.format("json").format(None).save(path)
+ assert sorted(session.read.parquet(path).collect()) == [Row(id=1, s="a"),
Row(id=2, s="b")]
+ assert session.read.format("json").format(None).load(path).count() == 2
+ with pytest.raises(AnalysisException) as e:
+ session.read.schema("a int").changes("t")
+ assert e.value.getCondition() == "_LEGACY_ERROR_TEMP_1189"
+ with pytest.raises(AssertionError, match="lowerBound can not be None"):
+ session.read.jdbc("jdbc:x", "t", column="c")
+ with pytest.raises(PySparkAttributeError) as e:
+ session.read._jreader
+ assert e.value.getCondition() == "JVM_ATTRIBUTE_NOT_SUPPORTED"
+ assert session.read.text(None).count() == 0
+
def test_writer_v2_takes_its_arguments_as_pyspark(session):
# As PySpark's DataFrameWriterV2: tableProperty(property, None) leaves the
property unset,
# partitionedBy raises NOT_EXPECTED_TYPE of col for a list, and overwrite
takes a column
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index dd533eb..b4c5b63 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -11029,6 +11029,82 @@ def
test_reader_parses_a_dataframe_of_strings_as_pyspark(mock_connect, read, for
assert (reader._format, reader._schema, reader._options) == ("csv", "a
INT", {"sep": ","})
+def test_reader_and_writer_check_their_arguments_as_pyspark(mock_connect):
+ # As PySpark Connect's: a write of a mode other than append, overwrite,
error (or
+ # errorifexists, which PySpark documents) and ignore raises
UNSUPPORTED_OPERATION, of the
+ # last mode set; format(None) unsets the format; the reader raises
JVM_ATTRIBUTE_NOT_SUPPORTED
+ # of _jreader, asserts the bounds of a JDBC column and refuses a schema in
changes; and csv,
+ # text and orc of None read without a path.
+ from pyspark.errors import AnalysisException, PySparkAttributeError,
PySparkValueError
+
+ spark, service = mock_connect
+ df = spark.range(3)
+ for write, mode in [
+ (lambda: df.write.mode("bad").save("p"), "bad"),
+ (lambda: df.write.mode("append").mode("default").saveAsTable("t"),
"default"),
+ (lambda: df.write.mode(" append").insertInto("t"), " append"),
+ (lambda: df.write.parquet("p", mode=""), ""),
+ (lambda: df.write.jdbc("jdbc:x", "t", mode="bad"), "bad"),
+ ]:
+ with pytest.raises(PySparkValueError) as e:
+ write()
+ assert e.value.getCondition() == "UNSUPPORTED_OPERATION"
+ assert e.value.getMessageParameters() == {"operation": mode}
+
+ def sent_write():
+ # ExecutePlanRequest.plan=3, Plan.command=2, Command.write_operation=2.
+ plan = parse_message(parse_message(service.execute_requests[-1])[3][0])
+ return parse_message(parse_message(plan[2][0])[2][0])
+
+ # WriteOperation.source=2 and mode=5, of SAVE_MODE_APPEND=1 and
SAVE_MODE_ERROR_IF_EXISTS=3.
+ df.write.mode("bad").mode("APPEND").format("json").format(None).save("p")
+ assert 2 not in sent_write() and sent_write()[5] == [1]
+ df.write.mode("errorifexists").insertInto("t")
+ assert sent_write()[5] == [3]
+ df.write.jdbc("jdbc:x", "t", properties={"user": 1, "flag": True, "skip":
None})
+ # WriteOperation.options=9, of entries of key=1 and value=2.
+ options = {parse_message(entry)[1][0]: parse_message(entry)[2][0] for
entry in sent_write()[9]}
+ assert options == {b"user": b"1", b"flag": b"true", b"url": b"jdbc:x",
b"dbtable": b"t"}
+
+ reader = spark.read.format("csv")
+ assert reader.format(None) is reader and reader._format is None
+ stream = spark.readStream.format("rate")
+ assert stream.format(None) is stream
+ with pytest.raises(PySparkAttributeError) as e:
+ spark.read._jreader
+ assert e.value.getCondition() == "JVM_ATTRIBUTE_NOT_SUPPORTED"
+ assert e.value.getMessageParameters() == {"attr_name": "_jreader"}
+ with pytest.raises(AnalysisException) as e:
+ spark.read.schema("a int").changes("t")
+ assert e.value.getCondition() == "_LEGACY_ERROR_TEMP_1189"
+ assert e.value.getMessageParameters() == {"operation": "changes"}
+ assert "User specified schema not supported with `changes`." in
str(e.value)
+ for bounds, missing in [({}, "lowerBound"), ({"lowerBound": 1},
"upperBound"),
+ ({"lowerBound": 1, "upperBound": 2},
"numPartitions")]:
+ reader = spark.read
+ with pytest.raises(AssertionError, match=f"{missing} can not be None
when ``column``"):
+ reader.jdbc("jdbc:x", "t", column="c", **bounds)
+ assert reader._format == "jdbc" and reader._options == {}
+
+ def sent_data_source():
+ # Relation.read=2 -> Read.data_source=2: options=3, of entries of
key=1 and value=2,
+ # and predicates=5.
+ source = parse_message(_sent_relation(service.analyze_requests[-1],
2)[2][0])
+ options = {parse_message(e)[1][0].decode():
parse_message(e)[2][0].decode()
+ for e in source.get(3, [])}
+ return options, [p.decode() for p in source.get(5, [])]
+
+ spark.read.jdbc("jdbc:x", "t", column="c", lowerBound=1, upperBound=2,
numPartitions="3",
+ predicates=["p"], properties={"user": 1, "flag":
True}).columns
+ assert sent_data_source() == ({
+ "partitionColumn": "c", "lowerBound": "1", "upperBound": "2",
"numPartitions": "3",
+ "user": "1", "flag": "true", "url": "jdbc:x", "dbtable": "t"}, [])
+ spark.read.jdbc("jdbc:x", "t", lowerBound=1, predicates=["p"]).columns
+ assert sent_data_source() == ({"url": "jdbc:x", "dbtable": "t"}, ["p"])
+ for read in (spark.read.csv, spark.read.text, spark.read.orc):
+ read(None).columns
+ assert _sent_data_source_paths(service.analyze_requests[-1]) == []
+
def test_reader_paths_are_checked_as_pyspark(offline_spark):
from pyspark.errors import PySparkTypeError
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]