This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new abc14ce8 [SPARK-61179][CONNECT][PYTHON] Unset a DataFrameReader option
of None, and take the description of Catalog.createExternalTable, in the Python
client as PySpark does
abc14ce8 is described below
commit abc14ce897803d5618afa912910f4d79ddc7aff5
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Nov 9 06:21:24 2026 +0900
[SPARK-61179][CONNECT][PYTHON] Unset a DataFrameReader option of None, and
take the description of Catalog.createExternalTable, in the Python client as
PySpark does
### What changes were proposed in this pull request?
[SPARK-61179](https://issues.apache.org/jira/browse/SPARK-61179): PySpark's
`DataFrameReader.option(key, None)`, `DataFrameReader.options(key=None)` and
`DataFrameReader.load(..., key=None)` unset the option, and
`Catalog.createExternalTable(..., description=...)` passes the description to
`createTable`, which stores it as the table comment.
- `DataFrameReader.option`, `options` and `load` unset an option given
`None`, through a new core `DataFrameReader::unset_option`. The format methods
(`csv(path, header=None)` and the like) still skip `None` and keep the earlier
value, as PySpark does, and so do the `DataStreamReader` and writer option
methods, which PySpark also keeps.
- `Catalog.createExternalTable` takes `description` as a named argument and
passes it to `createTable`, instead of sending it as a table option.
### Why are the changes needed?
A reader option set to `None` kept its earlier value, so the reader read
with the header that PySpark drops:
```python
spark.read.option("header", "true").option("header",
None).format("csv").load(path).columns
# PySpark 4.3.0: ['_c0', '_c1']
# before this change: ['a', 'b']
```
`createExternalTable` with a description did not set the table comment:
```python
spark.catalog.createExternalTable("t", path=p, source="parquet",
description="the table")
spark.catalog.getTable("t").description
# PySpark 4.3.0: 'the table'
# before this change: None (and SHOW TBLPROPERTIES listed
option.description)
```
### Does this PR introduce _any_ user-facing change?
Yes. `DataFrameReader.option`, `options` and `load` with a `None` value now
unset the option, as in PySpark. `Catalog.createExternalTable(description=...)`
sets the table comment, as in PySpark, and no longer adds a table option.
One difference remains: PySpark's `DataFrameReader._options` lists an unset
key with the value `None`, and the Rust-backed reader drops the key.
### How was this patch tested?
-
`test_reader_none_unsets_an_option_and_external_description_is_the_comment` in
`python/tests/test_dropin_e2e.py`. It checks the reader columns after `option`,
`options` and `load` with `None` after an earlier `header` value, that the
`csv(path, header=None)` kwarg keeps the earlier value, and that a reused
reader keeps its option, and it checks `getTable(...).description`, the
`Comment` row of `DESCRIBE TABLE EXTENDED`, and that `SHOW TBLPROPERTIES` lists
no description. It fails wi [...]
- The same catalog and `DESCRIBE` results were checked with official
PySpark 4.3.0, this change and master by a probe script.
- `option(key, None)` keeps the earlier value in official PySpark 4.3.0 for
`DataFrameWriter`, `DataFrameWriterV2` and `DataStreamReader` (probed through
`_write` and `_options`), and on the build for `DataFrameWriter` (by reading
back the written header) and `DataStreamReader` (through `_options`). The
build's `DataFrameWriterV2` path is unchanged and was not probed.
- The lib tests of both crates (608 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2227 passed, 8 skipped), and the
live drop-in and Arrow suites (211 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #377 from HyukjinKwon/SPARK-61179-reader-none-external-description.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/catalog.rs | 7 +++--
crates/pyspark-rs/src/readwriter.rs | 56 +++++++++++++++++++---------------
crates/spark-connect/src/readwriter.rs | 6 ++++
python/tests/test_dropin_e2e.py | 36 ++++++++++++++++++++++
4 files changed, 77 insertions(+), 28 deletions(-)
diff --git a/crates/pyspark-rs/src/catalog.rs b/crates/pyspark-rs/src/catalog.rs
index 80680ba5..d0ca8da4 100644
--- a/crates/pyspark-rs/src/catalog.rs
+++ b/crates/pyspark-rs/src/catalog.rs
@@ -453,8 +453,8 @@ impl PyCatalog {
/// Create an external table from a data source: deprecated, as PySpark's,
which warns and
/// creates the table as `createTable` does.
- #[pyo3(signature = (tableName, path=None, source=None, schema=None,
**options))]
- #[allow(non_snake_case)]
+ #[pyo3(signature = (tableName, path=None, source=None, schema=None,
description=None, **options))]
+ #[allow(non_snake_case, clippy::too_many_arguments)]
fn createExternalTable(
&self,
py: Python<'_>,
@@ -462,6 +462,7 @@ impl PyCatalog {
path: Option<&str>,
source: Option<&str>,
schema: Option<&Bound<'_, PyAny>>,
+ description: Option<&str>,
options: Option<&Bound<'_, pyo3::types::PyDict>>,
) -> PyResult<PyDataFrame> {
py.import("warnings")?.call_method1(
@@ -472,7 +473,7 @@ impl PyCatalog {
py.get_type::<pyo3::exceptions::PyFutureWarning>(),
),
)?;
- self.createTable(py, tableName, path, source, schema, None, options)
+ self.createTable(py, tableName, path, source, schema, description,
options)
}
/// Register a Python function as a UDF, returning the registered UDF.
Deprecated
diff --git a/crates/pyspark-rs/src/readwriter.rs
b/crates/pyspark-rs/src/readwriter.rs
index df9515d7..8ffb1c3d 100644
--- a/crates/pyspark-rs/src/readwriter.rs
+++ b/crates/pyspark-rs/src/readwriter.rs
@@ -33,15 +33,7 @@ impl PyDataFrameReader {
/// A copy of the current reader with per-call `**options` applied
(skipping
/// None, lowercasing bools), for a format-specific read.
fn with_opts(&self, options: Option<&Bound<'_, PyDict>>) ->
PyResult<DataFrameReader> {
- let mut r = self.inner.clone();
- if let Some(opts) = options {
- for (k, v) in opts.iter() {
- if let Some(val) = crate::coerce_option_value(&v)? {
- r = r.option(&k.str()?.to_string(), &val);
- }
- }
- }
- Ok(r)
+ apply_options(self.inner.clone(), options, false)
}
/// Keep the options of a format-specific read on this reader, as PySpark
does
@@ -183,6 +175,27 @@ pub(crate) fn bucket_count(num_buckets: &Bound<'_, PyAny>)
-> PyResult<i32> {
}
}
+/// The reader `r` with the `options` of a call, lowercasing bools. A None
value unsets its
+/// option when `unset_none`, as PySpark's `DataFrameReader.option`, `options`
and `load` do,
+/// and is skipped otherwise, as the `_set_opts` of its format methods does.
+fn apply_options(
+ mut r: DataFrameReader,
+ options: Option<&Bound<'_, PyDict>>,
+ unset_none: bool,
+) -> PyResult<DataFrameReader> {
+ if let Some(opts) = options {
+ for (k, v) in opts.iter() {
+ let key = k.str()?.to_string();
+ match crate::coerce_option_value(&v)? {
+ Some(val) => r = r.option(&key, &val),
+ None if unset_none => r = r.unset_option(&key),
+ None => {}
+ }
+ }
+ }
+ Ok(r)
+}
+
/// Apply a single reader option (skipping None; bool->"true"/"false").
fn set_opt(
r: DataFrameReader,
@@ -306,35 +319,28 @@ impl PyDataFrameReader {
Ok(slf)
}
- /// Set a single read option. `None` leaves the option unset (not the
string
- /// "None"); booleans lowercase to "true"/"false" (reference `to_str`
semantics).
+ /// Set a single read option. `None` unsets the option, as PySpark's does
(not the
+ /// string "None"); booleans lowercase to "true"/"false" (reference
`to_str` semantics).
fn option<'py>(
mut slf: PyRefMut<'py, Self>,
key: &str,
value: &Bound<'_, PyAny>,
) -> PyResult<PyRefMut<'py, Self>> {
- if let Some(v) = crate::coerce_option_value(value)? {
- slf.inner = slf.inner.clone().option(key, &v);
- }
+ slf.inner = match crate::coerce_option_value(value)? {
+ Some(v) => slf.inner.clone().option(key, &v),
+ None => slf.inner.clone().unset_option(key),
+ };
Ok(slf)
}
/// Set multiple read options. Mirrors reference
`DataFrameReader.options(**options)`
- /// - keyword args; `None` values are skipped and booleans lowercased.
+ /// - keyword args; a `None` value unsets the option and booleans
lowercase.
#[pyo3(signature = (**options))]
fn options<'py>(
mut slf: PyRefMut<'py, Self>,
options: Option<&Bound<'_, PyDict>>,
) -> PyResult<PyRefMut<'py, Self>> {
- let mut map = HashMap::new();
- if let Some(options) = options {
- for (k, v) in options.iter() {
- if let Some(val) = crate::coerce_option_value(&v)? {
- map.insert(k.str()?.to_string(), val);
- }
- }
- }
- slf.inner = slf.inner.clone().options(map);
+ slf.inner = apply_options(slf.inner.clone(), options, true)?;
Ok(slf)
}
@@ -380,7 +386,7 @@ impl PyDataFrameReader {
schema: Option<&Bound<'_, PyAny>>,
options: Option<&Bound<'_, PyDict>>,
) -> PyResult<PyDataFrame> {
- let mut r = self.with_opts(options)?;
+ let mut r = apply_options(self.inner.clone(), options, true)?;
if let Some(f) = format {
r = r.format(f);
}
diff --git a/crates/spark-connect/src/readwriter.rs
b/crates/spark-connect/src/readwriter.rs
index c1d8ea98..5680f360 100644
--- a/crates/spark-connect/src/readwriter.rs
+++ b/crates/spark-connect/src/readwriter.rs
@@ -60,6 +60,12 @@ impl DataFrameReader {
self
}
+ /// Unset a read option, as PySpark's `DataFrameReader.option(key, None)`
does.
+ pub fn unset_option(mut self, key: &str) -> Self {
+ self.options.remove(key);
+ self
+ }
+
/// Set multiple options.
pub fn options(mut self, options: HashMap<String, String>) -> Self {
self.options.extend(options);
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 0c3a8894..efddc6bc 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1501,6 +1501,42 @@ def test_reader_schema_keeps_field_metadata(session,
tmp_path, ddl):
assert stream.schema.fields[0].metadata == {"comment": "the key"}
+def
test_reader_none_unsets_an_option_and_external_description_is_the_comment(session,
tmp_path):
+ # As PySpark's: DataFrameReader.option, options and load unset an option
given None, while
+ # the format methods skip None and keep it. createExternalTable's
description is the
+ # comment of createTable, not a property.
+ import warnings
+
+ (tmp_path / "t.csv").write_text("a,b\n1,2\n")
+ path = str(tmp_path / "t.csv")
+ assert session.read.option("header", "true").csv(path).columns == ["a",
"b"]
+ unset = session.read.option("header", "true").option("header", None)
+ assert unset.format("csv").load(path).columns == ["_c0", "_c1"]
+ unset = session.read.option("header", "true").options(header=None)
+ assert unset.format("csv").load(path).columns == ["_c0", "_c1"]
+ unset = session.read.option("header", "true").format("csv")
+ assert unset.load(path, header=None).columns == ["_c0", "_c1"]
+ kept = session.read.option("header", "true")
+ assert kept.csv(path, header=None).columns == ["a", "b"]
+ assert kept.format("csv").load(path).columns == ["a", "b"]
+ session.range(2).write.mode("overwrite").parquet(str(tmp_path / "ext"))
+ session.sql("DROP TABLE IF EXISTS i3_described_external")
+ try:
+ with warnings.catch_warnings():
+ warnings.simplefilter("ignore", FutureWarning)
+ session.catalog.createExternalTable(
+ "i3_described_external", path=str(tmp_path / "ext"),
source="parquet",
+ description="the table")
+ assert session.catalog.getTable("i3_described_external").description
== "the table"
+ comments = session.sql("DESCRIBE TABLE EXTENDED
i3_described_external").filter(
+ "col_name = 'Comment'").collect()
+ assert [tuple(r)[:2] for r in comments] == [("Comment", "the table")]
+ keys = [r.key for r in session.sql("SHOW TBLPROPERTIES
i3_described_external").collect()]
+ assert not any("description" in key for key in keys)
+ finally:
+ session.sql("DROP TABLE IF EXISTS i3_described_external")
+
+
def test_reader_writer_and_catalog_return_the_results_of_pyspark(session,
tmp_path):
# As PySpark's: csv, json and xml of a DataFrame of strings parse them,
the readers load
# every path, and createTable creates the table on the call, of its schema
and options.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]