This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 6db70f0  [SPARK-61101][CONNECT][PYTHON] Validate the save mode of 
DataFrameWriter, and the arguments of DataFrameReader, in the Python client as 
PySpark does
6db70f0 is described below

commit 6db70f09cd6e9a8a2893b68425c381b022f04180
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 17:49:45 2026 +0900

    [SPARK-61101][CONNECT][PYTHON] Validate the save mode of DataFrameWriter, 
and the arguments of DataFrameReader, in the Python client as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61101](https://issues.apache.org/jira/browse/SPARK-61101): the batch 
reader and writer now check their arguments as PySpark Connect's do.
    
    As PySpark Connect's:
    - A write raises `UNSUPPORTED_OPERATION` when the last save mode set is not 
`append`, `overwrite`, `error` or `ignore`, in any case. This covers `save`, 
`saveAsTable`, `insertInto`, the format methods and `jdbc`. The core's 
`DataFrameWriter` fails the write; it used to ignore the mode.
    - `format(None)` of `DataFrameReader`, `DataFrameWriter` and 
`DataStreamReader` unsets the format, so the server uses its default source. 
The core's readers and writer gain `clear_format`.
    - `DataFrameReader._jreader` raises `JVM_ATTRIBUTE_NOT_SUPPORTED`.
    - `DataFrameReader.jdbc`:
      - sets the format `jdbc` on the reader first
      - with a `column`, asserts `lowerBound`, `upperBound` and `numPartitions` 
(for example "lowerBound can not be None when ``column`` is specified") and 
ignores `predicates`
      - without a `column`, ignores the bounds and `numPartitions`
      - takes the column, the bounds, `numPartitions` and the values of 
`properties` as options of any type, like `DataFrameWriter.jdbc`'s `properties`
    - `DataFrameReader.changes` of a reader with a schema raises the 
`AnalysisException` `_LEGACY_ERROR_TEMP_1189`, as `DataStreamReader.changes` 
already does. The two share the code now.
    - `csv`, `text` and `orc` of None read without a path.
    
    The client keeps `errorifexists` as the save mode `error`. PySpark 
documents it, and pandas-on-Spark's `validate_mode` passes it to the writer. 
PySpark Connect 4.3.0 raises `UNSUPPORTED_OPERATION` for it, against its own 
documentation.
    
    ### Why are the changes needed?
    
    The client:
    - accepted any save mode: `df.write.mode("bad").save(path)` saved the data 
with the default mode, and a typo like `mode("overwite")` failed later with 
`PATH_ALREADY_EXISTS`
    - raised a `TypeError` for `format(None)`
    - sent a JDBC read of a column without bounds to the server
    - dropped the schema of a reader in `changes`
    - raised an `AttributeError` for `_jreader`
    - raised an `AssertionError` for `csv(None)`, `text(None)` and `orc(None)`
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. The reader and the writer take their arguments, and raise their 
errors, as in PySpark 4.3.0.
    
    ### How was this patch tested?
    
    - A new live test checks an unsupported save mode, which writes nothing; 
`format(None)`, which writes and reads Parquet; `changes` of a schema; `jdbc` 
of a column without bounds; `_jreader`; and `text(None)`. It passes with 
official PySpark 4.3.0 and fails without this change.
    - A new offline test checks these errors and the requests sent for each 
mode and format, and for the options and predicates of `jdbc`.
    - A new core test checks the save modes of a write.
    - On a local server, 160 cases of the reader and the writer with good and 
bad arguments give the same results as official PySpark 4.3.0. These are left:
      - the protobuf and PyO3 messages of a `TypeError` of an argument of a 
wrong type
      - `errorifexists`, kept as above
      - `bucketBy(0, ...)`: PySpark Connect drops the bucketing (it sends one 
only for a positive count), while the client sends it and the server raises 
`INVALID_BUCKET_COUNT`, as classic PySpark does
    - The lib tests of both crates (589 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2051 passed, 12 skipped), and the 
live drop-in and Arrow suites (155 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #302 from HyukjinKwon/SPARK-61101-readwriter-args.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/dataframe.rs     |  23 ++++--
 crates/pyspark-rs/src/readwriter.rs    | 128 +++++++++++++++++++++++----------
 crates/pyspark-rs/src/streaming.rs     |  27 +++----
 crates/spark-connect/src/readwriter.rs |  76 +++++++++++++++++++-
 crates/spark-connect/src/streaming.rs  |   7 ++
 python/tests/test_dropin_e2e.py        |  27 +++++++
 python/tests/test_dropin_offline.py    |  76 ++++++++++++++++++++
 7 files changed, 304 insertions(+), 60 deletions(-)

diff --git a/crates/pyspark-rs/src/dataframe.rs 
b/crates/pyspark-rs/src/dataframe.rs
index 1d47c1a..16e6fd1 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -3002,8 +3002,16 @@ impl PyDataFrameWriter {
         Ok(slf)
     }
 
-    fn format<'py>(mut slf: PyRefMut<'py, Self>, source: &str) -> 
PyResult<PyRefMut<'py, Self>> {
-        slf.inner = slf.inner.clone().format(source);
+    /// The format of the write. As PySpark's, None unsets it, so that the 
server writes its
+    /// default source.
+    fn format<'py>(
+        mut slf: PyRefMut<'py, Self>,
+        source: Option<&str>,
+    ) -> PyResult<PyRefMut<'py, Self>> {
+        slf.inner = match source {
+            Some(source) => slf.inner.clone().format(source),
+            None => slf.inner.clone().clear_format(),
+        };
         Ok(slf)
     }
 
@@ -3096,7 +3104,8 @@ impl PyDataFrameWriter {
         Ok(slf)
     }
 
-    /// Write to an external JDBC table. Connection details go through 
`properties`.
+    /// Write to an external JDBC table. Connection details go through 
`properties`, options of
+    /// any type (None skipped, bools lowercased), as PySpark's.
     #[pyo3(signature = (url, table, mode=None, properties=None))]
     fn jdbc(
         &mut self,
@@ -3104,7 +3113,7 @@ impl PyDataFrameWriter {
         url: &str,
         table: &str,
         mode: Option<&str>,
-        properties: Option<std::collections::HashMap<String, String>>,
+        properties: Option<&Bound<'_, PyDict>>,
     ) -> PyResult<()> {
         // Same order as PySpark: format, mode, properties, then url/dbtable 
(which
         // win over same-named properties), all kept on the writer, then save.
@@ -3113,7 +3122,11 @@ impl PyDataFrameWriter {
             w = w.mode(m);
         }
         if let Some(props) = properties {
-            w = w.options(props);
+            for (key, value) in props.iter() {
+                if let Some(value) = crate::coerce_option_value(&value)? {
+                    w = w.option(&key.str()?.to_cow()?, &value);
+                }
+            }
         }
         w = w.option("url", url).option("dbtable", table);
         self.inner = w.clone();
diff --git a/crates/pyspark-rs/src/readwriter.rs 
b/crates/pyspark-rs/src/readwriter.rs
index 998b028..2c22f04 100644
--- a/crates/pyspark-rs/src/readwriter.rs
+++ b/crates/pyspark-rs/src/readwriter.rs
@@ -80,9 +80,12 @@ impl PyDataFrameReader {
     }
 }
 
-/// The paths of `path`, a path or a list of them, as PySpark's `load` takes 
them: anything
-/// else fails its assertion.
+/// The paths of `path`, a path or a list of them, or None for none, as 
PySpark's `load` takes
+/// them: anything else fails its assertion.
 fn paths_of(path: &Bound<'_, PyAny>) -> PyResult<Vec<String>> {
+    if path.is_none() {
+        return Ok(vec![]);
+    }
     if path.is_instance_of::<pyo3::types::PyString>() {
         return Ok(vec![path.extract()?]);
     }
@@ -216,6 +219,29 @@ pub(crate) fn reader_schema(schema: &Bound<'_, PyAny>) -> 
PyResult<String> {
     ))
 }
 
+/// The `AnalysisException` of `_LEGACY_ERROR_TEMP_1189` that PySpark's 
`DataFrameReader.changes`
+/// and `DataStreamReader.changes` raise for a reader of a schema, of the 
message of the JVM
+/// error class, which PySpark's error classes lack.
+pub(crate) fn changes_of_schema_error(py: Python<'_>) -> PyErr {
+    let exception = (|| {
+        let kwargs = PyDict::new(py);
+        kwargs.set_item("errorClass", "_LEGACY_ERROR_TEMP_1189")?;
+        let parameters = PyDict::new(py);
+        parameters.set_item("operation", "changes")?;
+        kwargs.set_item("messageParameters", parameters)?;
+        py.import("pyspark.errors")?
+            .getattr("AnalysisException")?
+            .call(
+                ("User specified schema not supported with `changes`.",),
+                Some(&kwargs),
+            )
+    })();
+    match exception {
+        Ok(exception) => PyErr::from_value(exception),
+        Err(err) => err,
+    }
+}
+
 /// Apply the `schema` argument of a format method, as pyspark does through 
`schema()`.
 fn apply_reader_schema(r: DataFrameReader, v: &Bound<'_, PyAny>) -> 
PyResult<DataFrameReader> {
     Ok(r.schema(reader_schema(v)?))
@@ -223,9 +249,13 @@ fn apply_reader_schema(r: DataFrameReader, v: &Bound<'_, 
PyAny>) -> PyResult<Dat
 
 #[pymethods]
 impl PyDataFrameReader {
-    /// Set the source format (e.g. "parquet", "json", "csv").
-    fn format<'py>(mut slf: PyRefMut<'py, Self>, source: &str) -> 
PyRefMut<'py, Self> {
-        slf.inner = slf.inner.clone().format(source);
+    /// Set the source format (e.g. "parquet", "json", "csv"). As PySpark's, 
None unsets it, so
+    /// that the server reads its default source.
+    fn format<'py>(mut slf: PyRefMut<'py, Self>, source: Option<&str>) -> 
PyRefMut<'py, Self> {
+        slf.inner = match source {
+            Some(source) => slf.inner.clone().format(source),
+            None => slf.inner.clone().clear_format(),
+        };
         slf
     }
 
@@ -289,6 +319,18 @@ impl PyDataFrameReader {
         self.inner.get_options().clone()
     }
 
+    /// The JVM reader of classic PySpark, which PySpark Connect's raises
+    /// `JVM_ATTRIBUTE_NOT_SUPPORTED` for.
+    #[getter]
+    fn _jreader(&self, py: Python<'_>) -> PyResult<()> {
+        Err(crate::column::raise_pyspark(
+            py,
+            "PySparkAttributeError",
+            "JVM_ATTRIBUTE_NOT_SUPPORTED",
+            &[("attr_name", "_jreader")],
+        ))
+    }
+
     /// Load data from the (optional) path. Mirrors the reference
     /// `DataFrameReader.load(path=None, format=None, schema=None, 
**options)`: the
     /// one-shot form applies format/schema/options here before reading, 
equivalent to
@@ -319,9 +361,13 @@ impl PyDataFrameReader {
         Ok(PyDataFrame::new(self.inner.clone().table(tableName)))
     }
 
-    /// Read the CDC changes of a named table. Mirrors 
`DataFrameReader.changes`.
+    /// Read the CDC changes of a named table. Mirrors 
`DataFrameReader.changes`, which raises an
+    /// `AnalysisException` of `_LEGACY_ERROR_TEMP_1189` for a reader of a 
schema.
     #[allow(non_snake_case)]
-    fn changes(&self, tableName: &str) -> PyResult<PyDataFrame> {
+    fn changes(&self, py: Python<'_>, tableName: &str) -> 
PyResult<PyDataFrame> {
+        if !self.inner.get_schema().is_empty() {
+            return Err(changes_of_schema_error(py));
+        }
         Ok(PyDataFrame::new(self.inner.clone().changes(tableName)))
     }
 
@@ -593,48 +639,58 @@ impl PyDataFrameReader {
         self.read_or_parse(r, "xml", spark_connect::plan::ParseFormat::Xml, 
path)
     }
 
-    /// Read from a JDBC source. Mirrors `DataFrameReader.jdbc(url, table,
-    /// column=None, lowerBound=None, upperBound=None, numPartitions=None,
-    /// predicates=None, properties=None)`: the column/bound/partition args 
and the
-    /// connection `properties` are threaded as reader options (connect 
represents
-    /// them that way); `predicates` stays the partitioning predicate list.
+    /// Read from a JDBC source. Mirrors `DataFrameReader.jdbc(url, table, 
column=None,
+    /// lowerBound=None, upperBound=None, numPartitions=None, predicates=None, 
properties=None)`,
+    /// which keeps the format, then the options, on the reader:
+    /// - of a `column`, it asserts the bounds and `numPartitions`, and reads 
the partitions of
+    ///   the column, ignoring `predicates`;
+    /// - else, it reads the partitions of `predicates`, ignoring the bounds 
and `numPartitions`.
+    ///
+    /// The column, the bounds, `numPartitions` and the values of `properties` 
are options, of any
+    /// type (None skipped, bools lowercased).
     #[pyo3(signature = (url, table, column=None, lowerBound=None, 
upperBound=None, numPartitions=None, predicates=None, properties=None))]
     #[allow(non_snake_case, clippy::too_many_arguments)]
     fn jdbc(
         &mut self,
         url: &str,
         table: &str,
-        column: Option<String>,
+        column: Option<&Bound<'_, PyAny>>,
         lowerBound: Option<&Bound<'_, PyAny>>,
         upperBound: Option<&Bound<'_, PyAny>>,
-        numPartitions: Option<i32>,
+        numPartitions: Option<&Bound<'_, PyAny>>,
         predicates: Option<Vec<String>>,
-        properties: Option<HashMap<String, String>>,
+        properties: Option<&Bound<'_, PyDict>>,
     ) -> PyResult<PyDataFrame> {
+        self.inner = self.inner.clone().format("jdbc");
         let mut r = self.inner.clone();
-        if let Some(c) = column {
-            r = r.option("partitionColumn", &c);
-        }
-        if let Some(lb) = lowerBound {
-            r = r.option("lowerBound", &lb.str()?.to_string());
-        }
-        if let Some(ub) = upperBound {
-            r = r.option("upperBound", &ub.str()?.to_string());
-        }
-        if let Some(n) = numPartitions {
-            r = r.option("numPartitions", &n.to_string());
-        }
-        if let Some(props) = properties {
-            for (k, v) in props {
-                r = r.option(&k, &v);
+        let predicates = match column {
+            Some(column) => {
+                for (bound, name) in [
+                    (lowerBound, "lowerBound"),
+                    (upperBound, "upperBound"),
+                    (numPartitions, "numPartitions"),
+                ] {
+                    if bound.is_none() {
+                        return 
Err(pyo3::exceptions::PyAssertionError::new_err(format!(
+                            "{name} can not be None when ``column`` is 
specified"
+                        )));
+                    }
+                }
+                r = set_opt(r, "partitionColumn", Some(column))?;
+                r = set_opt(r, "lowerBound", lowerBound)?;
+                r = set_opt(r, "upperBound", upperBound)?;
+                r = set_opt(r, "numPartitions", numPartitions)?;
+                None
+            }
+            None => predicates,
+        };
+        if let Some(properties) = properties {
+            for (key, value) in properties.iter() {
+                r = set_opt(r, &key.str()?.to_cow()?, Some(&value))?;
             }
         }
-        // PySpark's jdbc also leaves format/url/dbtable on the reader.
-        self.inner = r
-            .clone()
-            .format("jdbc")
-            .option("url", url)
-            .option("dbtable", table);
+        // PySpark's jdbc also leaves url/dbtable on the reader.
+        self.inner = r.clone().option("url", url).option("dbtable", table);
         Ok(PyDataFrame::new(r.jdbc(url, table, predicates)))
     }
 }
diff --git a/crates/pyspark-rs/src/streaming.rs 
b/crates/pyspark-rs/src/streaming.rs
index 583d15e..5de4b14 100644
--- a/crates/pyspark-rs/src/streaming.rs
+++ b/crates/pyspark-rs/src/streaming.rs
@@ -148,8 +148,16 @@ impl PyDataStreamReader {
 
 #[pymethods]
 impl PyDataStreamReader {
-    fn format<'py>(mut slf: PyRefMut<'py, Self>, source: &str) -> 
PyResult<PyRefMut<'py, Self>> {
-        slf.inner = slf.inner.clone().format(source);
+    /// Set the source format. As PySpark's, None unsets it, so that the 
server reads its
+    /// default source.
+    fn format<'py>(
+        mut slf: PyRefMut<'py, Self>,
+        source: Option<&str>,
+    ) -> PyResult<PyRefMut<'py, Self>> {
+        slf.inner = match source {
+            Some(source) => slf.inner.clone().format(source),
+            None => slf.inner.clone().clear_format(),
+        };
         Ok(slf)
     }
 
@@ -278,20 +286,7 @@ impl PyDataStreamReader {
     /// schema.
     fn changes(&self, py: Python<'_>, tableName: &str) -> 
PyResult<PyDataFrame> {
         if !self.inner.get_schema().is_empty() {
-            // Of the message of the JVM error class, which PySpark's error 
classes lack.
-            let kwargs = PyDict::new(py);
-            kwargs.set_item("errorClass", "_LEGACY_ERROR_TEMP_1189")?;
-            let parameters = PyDict::new(py);
-            parameters.set_item("operation", "changes")?;
-            kwargs.set_item("messageParameters", parameters)?;
-            let exception = py
-                .import("pyspark.errors")?
-                .getattr("AnalysisException")?
-                .call(
-                    ("User specified schema not supported with `changes`.",),
-                    Some(&kwargs),
-                )?;
-            return Err(PyErr::from_value(exception));
+            return Err(crate::readwriter::changes_of_schema_error(py));
         }
         Ok(PyDataFrame::new(self.inner.clone().changes(tableName)))
     }
diff --git a/crates/spark-connect/src/readwriter.rs 
b/crates/spark-connect/src/readwriter.rs
index 3ed5044..c1d8ea9 100644
--- a/crates/spark-connect/src/readwriter.rs
+++ b/crates/spark-connect/src/readwriter.rs
@@ -41,6 +41,13 @@ impl DataFrameReader {
         self
     }
 
+    /// Unset the format, so that the server reads its default source
+    /// (`spark.sql.sources.default`), as PySpark's `format(None)`.
+    pub fn clear_format(mut self) -> Self {
+        self.format = None;
+        self
+    }
+
     /// Set the schema from a DDL string or JSON string.
     pub fn schema(mut self, schema: String) -> Self {
         self.schema = schema;
@@ -277,6 +284,8 @@ pub struct DataFrameWriter {
     input_plan: Arc<LogicalPlan>,
     format: Option<String>,
     mode: SaveMode,
+    /// The last mode set when it is not a save mode, which fails the write.
+    unsupported_mode: Option<String>,
     options: HashMap<String, String>,
     partition_cols: Vec<String>,
     cluster_cols: Vec<String>,
@@ -299,6 +308,7 @@ impl DataFrameWriter {
             input_plan,
             format: None,
             mode: SaveMode::ErrorIfExists,
+            unsupported_mode: None,
             options: HashMap::new(),
             partition_cols: vec![],
             cluster_cols: vec![],
@@ -331,10 +341,16 @@ impl DataFrameWriter {
         self
     }
 
-    /// Set the save mode.
+    /// Set the save mode: `append`, `overwrite`, `error` (or `errorifexists`) 
or `ignore`, in
+    /// any case. As PySpark's, the write fails with `UNSUPPORTED_OPERATION` 
when the last mode
+    /// set is any other string.
     pub fn mode(mut self, mode: &str) -> Self {
-        if let Some(m) = SaveMode::from_str(mode) {
-            self.mode = m;
+        match SaveMode::from_str(mode) {
+            Some(m) => {
+                self.mode = m;
+                self.unsupported_mode = None;
+            }
+            None => self.unsupported_mode = Some(mode.to_string()),
         }
         self
     }
@@ -345,6 +361,13 @@ impl DataFrameWriter {
         self
     }
 
+    /// Unset the format, so that the server writes its default source
+    /// (`spark.sql.sources.default`), as PySpark's `format(None)`.
+    pub fn clear_format(mut self) -> Self {
+        self.format = None;
+        self
+    }
+
     /// Set a single option.
     pub fn option(mut self, key: &str, value: &str) -> Self {
         self.options.insert(key.to_string(), value.to_string());
@@ -385,6 +408,12 @@ impl DataFrameWriter {
         &self,
         save_type: Option<proto::write_operation::SaveType>,
     ) -> Result<proto::WriteOperation> {
+        if let Some(mode) = &self.unsupported_mode {
+            return Err(spark_connect_core::error::SparkError::value(
+                "UNSUPPORTED_OPERATION",
+                &[("operation", mode)],
+            ));
+        }
         let mut op = proto::WriteOperation::default();
         op.input = Some(build_input_relation(&self.input_plan, &self.session));
         op.source = self.format.clone();
@@ -669,6 +698,47 @@ mod tests {
         }
     }
 
+    #[test]
+    fn v1_write_operation_of_an_unsupported_mode_fails() {
+        // As PySpark's: the last mode set decides, in any case, and any other 
string fails the
+        // write with UNSUPPORTED_OPERATION.
+        let spark = session();
+        let df = spark.range(3).unwrap();
+        let path = || {
+            Some(proto::write_operation::SaveType::Path(
+                "/tmp/out".to_string(),
+            ))
+        };
+        for mode in ["bad", "default", "", " append"] {
+            let err = df
+                .write()
+                .mode("append")
+                .mode(mode)
+                .build_write_operation(path())
+                .unwrap_err();
+            assert_eq!(
+                err.get_condition().as_deref(),
+                Some("UNSUPPORTED_OPERATION")
+            );
+            assert_eq!(err.params.get("operation").map(String::as_str), 
Some(mode));
+        }
+        for (mode, expected) in [
+            ("APPEND", SaveMode::Append),
+            ("Overwrite", SaveMode::Overwrite),
+            ("error", SaveMode::ErrorIfExists),
+            ("errorifexists", SaveMode::ErrorIfExists),
+            ("ignore", SaveMode::Ignore),
+        ] {
+            let op = df
+                .write()
+                .mode("bad")
+                .mode(mode)
+                .build_write_operation(path())
+                .unwrap();
+            assert_eq!(op.mode, expected.to_proto());
+        }
+    }
+
     #[test]
     fn v1_write_operation_save_as_table() {
         let spark = session();
diff --git a/crates/spark-connect/src/streaming.rs 
b/crates/spark-connect/src/streaming.rs
index bd864c0..9e17be0 100644
--- a/crates/spark-connect/src/streaming.rs
+++ b/crates/spark-connect/src/streaming.rs
@@ -47,6 +47,13 @@ impl DataStreamReader {
         self
     }
 
+    /// Unset the format, so that the server reads its default source
+    /// (`spark.sql.sources.default`), as PySpark's `format(None)`.
+    pub fn clear_format(mut self) -> Self {
+        self.format = None;
+        self
+    }
+
     /// Set the schema from a DDL string or JSON string.
     pub fn schema(mut self, schema: impl Into<String>) -> Self {
         self.schema = schema.into();
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 3f09677..13cd21f 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1302,6 +1302,33 @@ def 
test_dataframe_columns_resolve_against_their_dataframe_as_pyspark(session):
     assert error.value.getCondition() == "AMBIGUOUS_COLUMN_REFERENCE"
 
 
+def test_reader_and_writer_check_their_arguments_as_pyspark(session, tmp_path):
+    # As PySpark Connect's: a write of an unsupported save mode raises 
UNSUPPORTED_OPERATION
+    # and writes nothing; format(None) writes and reads the default source, 
Parquet; the reader
+    # refuses a schema in changes and asserts the bounds of a JDBC column; 
text of None reads
+    # no rows.
+    from pyspark.errors import PySparkAttributeError
+
+    df = session.createDataFrame([(1, "a"), (2, "b")], "id int, s string")
+    path = str(tmp_path / "out")
+    with pytest.raises(PySparkValueError) as e:
+        df.write.mode("bad").save(path)
+    assert e.value.getCondition() == "UNSUPPORTED_OPERATION"
+    assert e.value.getMessageParameters() == {"operation": "bad"}
+    assert not os.path.exists(path)
+    df.write.format("json").format(None).save(path)
+    assert sorted(session.read.parquet(path).collect()) == [Row(id=1, s="a"), 
Row(id=2, s="b")]
+    assert session.read.format("json").format(None).load(path).count() == 2
+    with pytest.raises(AnalysisException) as e:
+        session.read.schema("a int").changes("t")
+    assert e.value.getCondition() == "_LEGACY_ERROR_TEMP_1189"
+    with pytest.raises(AssertionError, match="lowerBound can not be None"):
+        session.read.jdbc("jdbc:x", "t", column="c")
+    with pytest.raises(PySparkAttributeError) as e:
+        session.read._jreader
+    assert e.value.getCondition() == "JVM_ATTRIBUTE_NOT_SUPPORTED"
+    assert session.read.text(None).count() == 0
+
 def test_writer_v2_takes_its_arguments_as_pyspark(session):
     # As PySpark's DataFrameWriterV2: tableProperty(property, None) leaves the 
property unset,
     # partitionedBy raises NOT_EXPECTED_TYPE of col for a list, and overwrite 
takes a column
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index dd533eb..b4c5b63 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -11029,6 +11029,82 @@ def 
test_reader_parses_a_dataframe_of_strings_as_pyspark(mock_connect, read, for
     assert (reader._format, reader._schema, reader._options) == ("csv", "a 
INT", {"sep": ","})
 
 
+def test_reader_and_writer_check_their_arguments_as_pyspark(mock_connect):
+    # As PySpark Connect's: a write of a mode other than append, overwrite, 
error (or
+    # errorifexists, which PySpark documents) and ignore raises 
UNSUPPORTED_OPERATION, of the
+    # last mode set; format(None) unsets the format; the reader raises 
JVM_ATTRIBUTE_NOT_SUPPORTED
+    # of _jreader, asserts the bounds of a JDBC column and refuses a schema in 
changes; and csv,
+    # text and orc of None read without a path.
+    from pyspark.errors import AnalysisException, PySparkAttributeError, 
PySparkValueError
+
+    spark, service = mock_connect
+    df = spark.range(3)
+    for write, mode in [
+        (lambda: df.write.mode("bad").save("p"), "bad"),
+        (lambda: df.write.mode("append").mode("default").saveAsTable("t"), 
"default"),
+        (lambda: df.write.mode(" append").insertInto("t"), " append"),
+        (lambda: df.write.parquet("p", mode=""), ""),
+        (lambda: df.write.jdbc("jdbc:x", "t", mode="bad"), "bad"),
+    ]:
+        with pytest.raises(PySparkValueError) as e:
+            write()
+        assert e.value.getCondition() == "UNSUPPORTED_OPERATION"
+        assert e.value.getMessageParameters() == {"operation": mode}
+
+    def sent_write():
+        # ExecutePlanRequest.plan=3, Plan.command=2, Command.write_operation=2.
+        plan = parse_message(parse_message(service.execute_requests[-1])[3][0])
+        return parse_message(parse_message(plan[2][0])[2][0])
+
+    # WriteOperation.source=2 and mode=5, of SAVE_MODE_APPEND=1 and 
SAVE_MODE_ERROR_IF_EXISTS=3.
+    df.write.mode("bad").mode("APPEND").format("json").format(None).save("p")
+    assert 2 not in sent_write() and sent_write()[5] == [1]
+    df.write.mode("errorifexists").insertInto("t")
+    assert sent_write()[5] == [3]
+    df.write.jdbc("jdbc:x", "t", properties={"user": 1, "flag": True, "skip": 
None})
+    # WriteOperation.options=9, of entries of key=1 and value=2.
+    options = {parse_message(entry)[1][0]: parse_message(entry)[2][0] for 
entry in sent_write()[9]}
+    assert options == {b"user": b"1", b"flag": b"true", b"url": b"jdbc:x", 
b"dbtable": b"t"}
+
+    reader = spark.read.format("csv")
+    assert reader.format(None) is reader and reader._format is None
+    stream = spark.readStream.format("rate")
+    assert stream.format(None) is stream
+    with pytest.raises(PySparkAttributeError) as e:
+        spark.read._jreader
+    assert e.value.getCondition() == "JVM_ATTRIBUTE_NOT_SUPPORTED"
+    assert e.value.getMessageParameters() == {"attr_name": "_jreader"}
+    with pytest.raises(AnalysisException) as e:
+        spark.read.schema("a int").changes("t")
+    assert e.value.getCondition() == "_LEGACY_ERROR_TEMP_1189"
+    assert e.value.getMessageParameters() == {"operation": "changes"}
+    assert "User specified schema not supported with `changes`." in 
str(e.value)
+    for bounds, missing in [({}, "lowerBound"), ({"lowerBound": 1}, 
"upperBound"),
+                            ({"lowerBound": 1, "upperBound": 2}, 
"numPartitions")]:
+        reader = spark.read
+        with pytest.raises(AssertionError, match=f"{missing} can not be None 
when ``column``"):
+            reader.jdbc("jdbc:x", "t", column="c", **bounds)
+        assert reader._format == "jdbc" and reader._options == {}
+
+    def sent_data_source():
+        # Relation.read=2 -> Read.data_source=2: options=3, of entries of 
key=1 and value=2,
+        # and predicates=5.
+        source = parse_message(_sent_relation(service.analyze_requests[-1], 
2)[2][0])
+        options = {parse_message(e)[1][0].decode(): 
parse_message(e)[2][0].decode()
+                   for e in source.get(3, [])}
+        return options, [p.decode() for p in source.get(5, [])]
+
+    spark.read.jdbc("jdbc:x", "t", column="c", lowerBound=1, upperBound=2, 
numPartitions="3",
+                    predicates=["p"], properties={"user": 1, "flag": 
True}).columns
+    assert sent_data_source() == ({
+        "partitionColumn": "c", "lowerBound": "1", "upperBound": "2", 
"numPartitions": "3",
+        "user": "1", "flag": "true", "url": "jdbc:x", "dbtable": "t"}, [])
+    spark.read.jdbc("jdbc:x", "t", lowerBound=1, predicates=["p"]).columns
+    assert sent_data_source() == ({"url": "jdbc:x", "dbtable": "t"}, ["p"])
+    for read in (spark.read.csv, spark.read.text, spark.read.orc):
+        read(None).columns
+        assert _sent_data_source_paths(service.analyze_requests[-1]) == []
+
 def test_reader_paths_are_checked_as_pyspark(offline_spark):
     from pyspark.errors import PySparkTypeError
 


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to