This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new abc14ce8 [SPARK-61179][CONNECT][PYTHON] Unset a DataFrameReader option 
of None, and take the description of Catalog.createExternalTable, in the Python 
client as PySpark does
abc14ce8 is described below

commit abc14ce897803d5618afa912910f4d79ddc7aff5
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Nov 9 06:21:24 2026 +0900

    [SPARK-61179][CONNECT][PYTHON] Unset a DataFrameReader option of None, and 
take the description of Catalog.createExternalTable, in the Python client as 
PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61179](https://issues.apache.org/jira/browse/SPARK-61179): PySpark's 
`DataFrameReader.option(key, None)`, `DataFrameReader.options(key=None)` and 
`DataFrameReader.load(..., key=None)` unset the option, and 
`Catalog.createExternalTable(..., description=...)` passes the description to 
`createTable`, which stores it as the table comment.
    
    - `DataFrameReader.option`, `options` and `load` unset an option given 
`None`, through a new core `DataFrameReader::unset_option`. The format methods 
(`csv(path, header=None)` and the like) still skip `None` and keep the earlier 
value, as PySpark does, and so do the `DataStreamReader` and writer option 
methods, which PySpark also keeps.
    - `Catalog.createExternalTable` takes `description` as a named argument and 
passes it to `createTable`, instead of sending it as a table option.
    
    ### Why are the changes needed?
    
    A reader option set to `None` kept its earlier value, so the reader read 
with the header that PySpark drops:
    
    ```python
    spark.read.option("header", "true").option("header", 
None).format("csv").load(path).columns
    # PySpark 4.3.0: ['_c0', '_c1']
    # before this change: ['a', 'b']
    ```
    
    `createExternalTable` with a description did not set the table comment:
    
    ```python
    spark.catalog.createExternalTable("t", path=p, source="parquet", 
description="the table")
    spark.catalog.getTable("t").description
    # PySpark 4.3.0: 'the table'
    # before this change: None (and SHOW TBLPROPERTIES listed 
option.description)
    ```
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `DataFrameReader.option`, `options` and `load` with a `None` value now 
unset the option, as in PySpark. `Catalog.createExternalTable(description=...)` 
sets the table comment, as in PySpark, and no longer adds a table option.
    
    One difference remains: PySpark's `DataFrameReader._options` lists an unset 
key with the value `None`, and the Rust-backed reader drops the key.
    
    ### How was this patch tested?
    
    - 
`test_reader_none_unsets_an_option_and_external_description_is_the_comment` in 
`python/tests/test_dropin_e2e.py`. It checks the reader columns after `option`, 
`options` and `load` with `None` after an earlier `header` value, that the 
`csv(path, header=None)` kwarg keeps the earlier value, and that a reused 
reader keeps its option, and it checks `getTable(...).description`, the 
`Comment` row of `DESCRIBE TABLE EXTENDED`, and that `SHOW TBLPROPERTIES` lists 
no description. It fails wi [...]
    - The same catalog and `DESCRIBE` results were checked with official 
PySpark 4.3.0, this change and master by a probe script.
    - `option(key, None)` keeps the earlier value in official PySpark 4.3.0 for 
`DataFrameWriter`, `DataFrameWriterV2` and `DataStreamReader` (probed through 
`_write` and `_options`), and on the build for `DataFrameWriter` (by reading 
back the written header) and `DataStreamReader` (through `_options`). The 
build's `DataFrameWriterV2` path is unchanged and was not probed.
    - The lib tests of both crates (608 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2227 passed, 8 skipped), and the 
live drop-in and Arrow suites (211 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #377 from HyukjinKwon/SPARK-61179-reader-none-external-description.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/catalog.rs       |  7 +++--
 crates/pyspark-rs/src/readwriter.rs    | 56 +++++++++++++++++++---------------
 crates/spark-connect/src/readwriter.rs |  6 ++++
 python/tests/test_dropin_e2e.py        | 36 ++++++++++++++++++++++
 4 files changed, 77 insertions(+), 28 deletions(-)

diff --git a/crates/pyspark-rs/src/catalog.rs b/crates/pyspark-rs/src/catalog.rs
index 80680ba5..d0ca8da4 100644
--- a/crates/pyspark-rs/src/catalog.rs
+++ b/crates/pyspark-rs/src/catalog.rs
@@ -453,8 +453,8 @@ impl PyCatalog {
 
     /// Create an external table from a data source: deprecated, as PySpark's, 
which warns and
     /// creates the table as `createTable` does.
-    #[pyo3(signature = (tableName, path=None, source=None, schema=None, 
**options))]
-    #[allow(non_snake_case)]
+    #[pyo3(signature = (tableName, path=None, source=None, schema=None, 
description=None, **options))]
+    #[allow(non_snake_case, clippy::too_many_arguments)]
     fn createExternalTable(
         &self,
         py: Python<'_>,
@@ -462,6 +462,7 @@ impl PyCatalog {
         path: Option<&str>,
         source: Option<&str>,
         schema: Option<&Bound<'_, PyAny>>,
+        description: Option<&str>,
         options: Option<&Bound<'_, pyo3::types::PyDict>>,
     ) -> PyResult<PyDataFrame> {
         py.import("warnings")?.call_method1(
@@ -472,7 +473,7 @@ impl PyCatalog {
                 py.get_type::<pyo3::exceptions::PyFutureWarning>(),
             ),
         )?;
-        self.createTable(py, tableName, path, source, schema, None, options)
+        self.createTable(py, tableName, path, source, schema, description, 
options)
     }
 
     /// Register a Python function as a UDF, returning the registered UDF. 
Deprecated
diff --git a/crates/pyspark-rs/src/readwriter.rs 
b/crates/pyspark-rs/src/readwriter.rs
index df9515d7..8ffb1c3d 100644
--- a/crates/pyspark-rs/src/readwriter.rs
+++ b/crates/pyspark-rs/src/readwriter.rs
@@ -33,15 +33,7 @@ impl PyDataFrameReader {
     /// A copy of the current reader with per-call `**options` applied 
(skipping
     /// None, lowercasing bools), for a format-specific read.
     fn with_opts(&self, options: Option<&Bound<'_, PyDict>>) -> 
PyResult<DataFrameReader> {
-        let mut r = self.inner.clone();
-        if let Some(opts) = options {
-            for (k, v) in opts.iter() {
-                if let Some(val) = crate::coerce_option_value(&v)? {
-                    r = r.option(&k.str()?.to_string(), &val);
-                }
-            }
-        }
-        Ok(r)
+        apply_options(self.inner.clone(), options, false)
     }
 
     /// Keep the options of a format-specific read on this reader, as PySpark 
does
@@ -183,6 +175,27 @@ pub(crate) fn bucket_count(num_buckets: &Bound<'_, PyAny>) 
-> PyResult<i32> {
     }
 }
 
+/// The reader `r` with the `options` of a call, lowercasing bools. A None 
value unsets its
+/// option when `unset_none`, as PySpark's `DataFrameReader.option`, `options` 
and `load` do,
+/// and is skipped otherwise, as the `_set_opts` of its format methods does.
+fn apply_options(
+    mut r: DataFrameReader,
+    options: Option<&Bound<'_, PyDict>>,
+    unset_none: bool,
+) -> PyResult<DataFrameReader> {
+    if let Some(opts) = options {
+        for (k, v) in opts.iter() {
+            let key = k.str()?.to_string();
+            match crate::coerce_option_value(&v)? {
+                Some(val) => r = r.option(&key, &val),
+                None if unset_none => r = r.unset_option(&key),
+                None => {}
+            }
+        }
+    }
+    Ok(r)
+}
+
 /// Apply a single reader option (skipping None; bool->"true"/"false").
 fn set_opt(
     r: DataFrameReader,
@@ -306,35 +319,28 @@ impl PyDataFrameReader {
         Ok(slf)
     }
 
-    /// Set a single read option. `None` leaves the option unset (not the 
string
-    /// "None"); booleans lowercase to "true"/"false" (reference `to_str` 
semantics).
+    /// Set a single read option. `None` unsets the option, as PySpark's does 
(not the
+    /// string "None"); booleans lowercase to "true"/"false" (reference 
`to_str` semantics).
     fn option<'py>(
         mut slf: PyRefMut<'py, Self>,
         key: &str,
         value: &Bound<'_, PyAny>,
     ) -> PyResult<PyRefMut<'py, Self>> {
-        if let Some(v) = crate::coerce_option_value(value)? {
-            slf.inner = slf.inner.clone().option(key, &v);
-        }
+        slf.inner = match crate::coerce_option_value(value)? {
+            Some(v) => slf.inner.clone().option(key, &v),
+            None => slf.inner.clone().unset_option(key),
+        };
         Ok(slf)
     }
 
     /// Set multiple read options. Mirrors reference 
`DataFrameReader.options(**options)`
-    /// - keyword args; `None` values are skipped and booleans lowercased.
+    /// - keyword args; a `None` value unsets the option and booleans 
lowercase.
     #[pyo3(signature = (**options))]
     fn options<'py>(
         mut slf: PyRefMut<'py, Self>,
         options: Option<&Bound<'_, PyDict>>,
     ) -> PyResult<PyRefMut<'py, Self>> {
-        let mut map = HashMap::new();
-        if let Some(options) = options {
-            for (k, v) in options.iter() {
-                if let Some(val) = crate::coerce_option_value(&v)? {
-                    map.insert(k.str()?.to_string(), val);
-                }
-            }
-        }
-        slf.inner = slf.inner.clone().options(map);
+        slf.inner = apply_options(slf.inner.clone(), options, true)?;
         Ok(slf)
     }
 
@@ -380,7 +386,7 @@ impl PyDataFrameReader {
         schema: Option<&Bound<'_, PyAny>>,
         options: Option<&Bound<'_, PyDict>>,
     ) -> PyResult<PyDataFrame> {
-        let mut r = self.with_opts(options)?;
+        let mut r = apply_options(self.inner.clone(), options, true)?;
         if let Some(f) = format {
             r = r.format(f);
         }
diff --git a/crates/spark-connect/src/readwriter.rs 
b/crates/spark-connect/src/readwriter.rs
index c1d8ea98..5680f360 100644
--- a/crates/spark-connect/src/readwriter.rs
+++ b/crates/spark-connect/src/readwriter.rs
@@ -60,6 +60,12 @@ impl DataFrameReader {
         self
     }
 
+    /// Unset a read option, as PySpark's `DataFrameReader.option(key, None)` 
does.
+    pub fn unset_option(mut self, key: &str) -> Self {
+        self.options.remove(key);
+        self
+    }
+
     /// Set multiple options.
     pub fn options(mut self, options: HashMap<String, String>) -> Self {
         self.options.extend(options);
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 0c3a8894..efddc6bc 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1501,6 +1501,42 @@ def test_reader_schema_keeps_field_metadata(session, 
tmp_path, ddl):
     assert stream.schema.fields[0].metadata == {"comment": "the key"}
 
 
+def 
test_reader_none_unsets_an_option_and_external_description_is_the_comment(session,
 tmp_path):
+    # As PySpark's: DataFrameReader.option, options and load unset an option 
given None, while
+    # the format methods skip None and keep it. createExternalTable's 
description is the
+    # comment of createTable, not a property.
+    import warnings
+
+    (tmp_path / "t.csv").write_text("a,b\n1,2\n")
+    path = str(tmp_path / "t.csv")
+    assert session.read.option("header", "true").csv(path).columns == ["a", 
"b"]
+    unset = session.read.option("header", "true").option("header", None)
+    assert unset.format("csv").load(path).columns == ["_c0", "_c1"]
+    unset = session.read.option("header", "true").options(header=None)
+    assert unset.format("csv").load(path).columns == ["_c0", "_c1"]
+    unset = session.read.option("header", "true").format("csv")
+    assert unset.load(path, header=None).columns == ["_c0", "_c1"]
+    kept = session.read.option("header", "true")
+    assert kept.csv(path, header=None).columns == ["a", "b"]
+    assert kept.format("csv").load(path).columns == ["a", "b"]
+    session.range(2).write.mode("overwrite").parquet(str(tmp_path / "ext"))
+    session.sql("DROP TABLE IF EXISTS i3_described_external")
+    try:
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", FutureWarning)
+            session.catalog.createExternalTable(
+                "i3_described_external", path=str(tmp_path / "ext"), 
source="parquet",
+                description="the table")
+        assert session.catalog.getTable("i3_described_external").description 
== "the table"
+        comments = session.sql("DESCRIBE TABLE EXTENDED 
i3_described_external").filter(
+            "col_name = 'Comment'").collect()
+        assert [tuple(r)[:2] for r in comments] == [("Comment", "the table")]
+        keys = [r.key for r in session.sql("SHOW TBLPROPERTIES 
i3_described_external").collect()]
+        assert not any("description" in key for key in keys)
+    finally:
+        session.sql("DROP TABLE IF EXISTS i3_described_external")
+
+
 def test_reader_writer_and_catalog_return_the_results_of_pyspark(session, 
tmp_path):
     # As PySpark's: csv, json and xml of a DataFrame of strings parse them, 
the readers load
     # every path, and createTable creates the table on the call, of its schema 
and options.


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to