This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new ba535db [SPARK-60920][CONNECT][PYTHON] Make reader/writer setters
mutate in place like PySpark in the Rust drop-in
ba535db is described below
commit ba535dbbcb8768d266d053a4372b6288c1ba2846
Author: Liang-Chi Hsieh <[email protected]>
AuthorDate: Thu Nov 5 18:58:09 2026 +0900
[SPARK-60920][CONNECT][PYTHON] Make reader/writer setters mutate in place
like PySpark in the Rust drop-in
### What changes were proposed in this pull request?
JIRA: [SPARK-60920](https://issues.apache.org/jira/browse/SPARK-60920)
The drop-in's `DataFrameReader`, `DataStreamReader`, `DataFrameWriter`,
`DataFrameWriterV2`, `DataStreamWriter` and `MergeIntoWriter`
(`crates/pyspark-rs/src/{readwriter,streaming,dataframe}.rs`) now change
themselves in place, as PySpark's do.
- Each wrapper holds its core builder directly, not in an `Option` that
every setter emptied. A setter replaces the builder in place and returns the
same Python object (`PyRefMut<Self>`).
- Terminal methods (`load`, `table`, `save`, `saveAsTable`, `insertInto`,
`start`, `toTable`, `create`, `merge`, ...) build from a clone, so the object
can be used again afterwards.
- As in PySpark, methods that take inline settings keep them on the object:
the format methods (`json`, `csv`, `parquet`, ..., `jdbc`), and the extra
arguments of `load`, `save`, `saveAsTable`, `insertInto`, `start` and
`toTable`. A call that fails argument validation leaves the object unchanged.
- `DataFrameWriter.jdbc` now applies `url` and `dbtable` after
`properties`, so they win over same-named properties, as in PySpark.
- A merge clause (`whenMatched()`, `whenNotMatched()`,
`whenNotMatchedBySource()`) holds its parent writer, as in PySpark Connect.
Each action appends to that writer and returns it, and a clause can be used
more than once. `withSchemaEvolution()` returns `self`.
- The core builders get `#[derive(Clone)]`; their consuming Rust API does
not change. The two core readers also get read-only `get_format` / `get_schema`
/ `get_options`. On the Python side these back PySpark Connect's private
`_format` / `_schema` / `_options` attributes.
### Why are the changes needed?
Each setter took the inner builder and returned a new wrapper, which left
the original Python object empty. So the usual PySpark pattern
```python
r = spark.read
r.option("a", "b")
r.format("jdbc").load()
```
failed with `RuntimeError: DataFrameReader already consumed`. The vendored
pandas-on-Spark `read_sql_table` uses this pattern: it calls
`reader.option(...)`, `reader.schema(...)` and `reader.options(...)` and
ignores the return values. The same error came up when reusing a reader or
writer after a read or write, which PySpark allows, and after a format method
failed argument validation, because the builder was taken before the arguments
were checked. A merge clause object could also b [...]
### Does this PR introduce _any_ user-facing change?
Yes. Reader, writer and merge-writer setters now return the same object and
change it in place, and these objects can be reused after a read, write or
merge, matching PySpark. Code that only chained calls behaves as before.
### How was this patch tested?
New offline tests in `python/tests/test_dropin_offline.py`:
- calling setters and ignoring their return values, then loading (the
`read_sql_table` call sequence)
- the vendored `read_sql_table` itself, run up to the read, with
`default_session` pointed at an offline session
- reuse of a reader after successful reads, and after reads that fail
argument validation, checking the format, schema and options left on it
- the in-place setters on `DataStreamReader`, `DataFrameWriter`,
`DataFrameWriterV2` and `DataStreamWriter`
- merge clause actions returning the same writer, and a clause used more
than once
This touches the same code in `readwriter.rs` / `streaming.rs` as #151
(reader schema handling). Whichever merges second will need a rebase.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Anthropic)
Closes #158 from viirya/SPARK-60920.
Authored-by: Liang-Chi Hsieh <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/dataframe.rs | 419 +++++++++++++++++----------------
crates/pyspark-rs/src/readwriter.rs | 128 ++++++----
crates/pyspark-rs/src/streaming.rs | 243 +++++++++++--------
crates/spark-connect/src/merge.rs | 1 +
crates/spark-connect/src/readwriter.rs | 18 ++
crates/spark-connect/src/streaming.rs | 17 ++
python/tests/test_dropin_offline.py | 191 +++++++++++++++
7 files changed, 667 insertions(+), 350 deletions(-)
diff --git a/crates/pyspark-rs/src/dataframe.rs
b/crates/pyspark-rs/src/dataframe.rs
index 1dfe89e..8df1b83 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -1436,7 +1436,7 @@ impl PyDataFrame {
#[getter]
fn write(&self) -> PyDataFrameWriter {
PyDataFrameWriter {
- inner: Some(self.dataframe.write()),
+ inner: self.dataframe.write(),
}
}
@@ -1444,7 +1444,7 @@ impl PyDataFrame {
#[pyo3(name = "writeTo")]
fn write_to(&self, table: &str) -> PyDataFrameWriterV2 {
PyDataFrameWriterV2 {
- inner: Some(self.dataframe.write_to(table)),
+ inner: self.dataframe.write_to(table),
}
}
@@ -1469,7 +1469,7 @@ impl PyDataFrame {
#[pyo3(name = "mergeInto")]
fn merge_into(&self, table: &str, condition: &PyColumn) ->
PyMergeIntoWriter {
PyMergeIntoWriter {
- inner: Some(self.dataframe.merge_into(table,
condition.column.clone())),
+ inner: self.dataframe.merge_into(table, condition.column.clone()),
}
}
@@ -1702,17 +1702,28 @@ fn build_replacement_pairs(
}
/// Python wrapper for `MergeIntoWriter` (`df.mergeInto(...)`).
+///
+/// Mirrors PySpark's mutable writer: `withSchemaEvolution` and every clause
action
+/// (`whenMatched().updateAll()`, ...) change this object and return it, and
`merge`
+/// leaves it usable. The core writer is a consuming builder, so the wrapped
value is
+/// replaced in place and `merge` runs on a clone.
#[pyclass(name = "MergeIntoWriter", module = "pyspark.sql.merge")]
pub struct PyMergeIntoWriter {
- inner: Option<spark_connect::MergeIntoWriter>,
+ inner: spark_connect::MergeIntoWriter,
}
-impl PyMergeIntoWriter {
- fn take(&mut self) -> PyResult<spark_connect::MergeIntoWriter> {
- self.inner.take().ok_or_else(|| {
- pyo3::exceptions::PyRuntimeError::new_err("MergeIntoWriter already
consumed")
- })
- }
+/// Apply a clause action to the parent writer in place and return that same
writer
+/// (PySpark's clause methods append to `self.writer` and return it).
+fn merge_clause_action(
+ py: Python<'_>,
+ writer: &Py<PyMergeIntoWriter>,
+ action: impl FnOnce(spark_connect::MergeIntoWriter) ->
spark_connect::MergeIntoWriter,
+) -> PyResult<Py<PyMergeIntoWriter>> {
+ {
+ let mut w = writer.bind(py).try_borrow_mut()?;
+ w.inner = action(w.inner.clone());
+ }
+ Ok(writer.clone_ref(py))
}
#[pymethods]
@@ -1737,41 +1748,40 @@ impl PyMergeIntoWriter {
}
#[pyo3(name = "whenMatched", signature = (condition=None))]
- fn when_matched(&mut self, condition: Option<&PyColumn>) ->
PyResult<PyWhenMatched> {
- let cond = condition.map(|c| c.column.clone());
- Ok(PyWhenMatched {
- inner: Some(self.take()?.when_matched(cond)),
- })
+ fn when_matched(slf: &Bound<'_, Self>, condition: Option<&PyColumn>) ->
PyWhenMatched {
+ PyWhenMatched {
+ writer: slf.clone().unbind(),
+ condition: condition.map(|c| c.column.clone()),
+ }
}
#[pyo3(name = "whenNotMatched", signature = (condition=None))]
- fn when_not_matched(&mut self, condition: Option<&PyColumn>) ->
PyResult<PyWhenNotMatched> {
- let cond = condition.map(|c| c.column.clone());
- Ok(PyWhenNotMatched {
- inner: Some(self.take()?.when_not_matched(cond)),
- })
+ fn when_not_matched(slf: &Bound<'_, Self>, condition: Option<&PyColumn>)
-> PyWhenNotMatched {
+ PyWhenNotMatched {
+ writer: slf.clone().unbind(),
+ condition: condition.map(|c| c.column.clone()),
+ }
}
#[pyo3(name = "whenNotMatchedBySource", signature = (condition=None))]
fn when_not_matched_by_source(
- &mut self,
+ slf: &Bound<'_, Self>,
condition: Option<&PyColumn>,
- ) -> PyResult<PyWhenNotMatchedBySource> {
- let cond = condition.map(|c| c.column.clone());
- Ok(PyWhenNotMatchedBySource {
- inner: Some(self.take()?.when_not_matched_by_source(cond)),
- })
+ ) -> PyWhenNotMatchedBySource {
+ PyWhenNotMatchedBySource {
+ writer: slf.clone().unbind(),
+ condition: condition.map(|c| c.column.clone()),
+ }
}
#[pyo3(name = "withSchemaEvolution")]
- fn with_schema_evolution(&mut self) -> PyResult<PyMergeIntoWriter> {
- Ok(PyMergeIntoWriter {
- inner: Some(self.take()?.with_schema_evolution()),
- })
+ fn with_schema_evolution(mut slf: PyRefMut<'_, Self>) -> PyRefMut<'_,
Self> {
+ slf.inner = slf.inner.clone().with_schema_evolution();
+ slf
}
- fn merge(&mut self) -> PyResult<()> {
- self.take()?.merge().to_pyerr()
+ fn merge(&self) -> PyResult<()> {
+ self.inner.clone().merge().to_pyerr()
}
}
@@ -1785,124 +1795,99 @@ fn assignments_to_map(
.collect()
}
-/// `whenMatched(...)` clause builder.
+/// `whenMatched(...)` clause builder. Holds its parent writer, like PySpark's
+/// `MergeIntoWriter.WhenMatched`; each action may be called more than once.
#[pyclass(name = "WhenMatched", module = "pyspark.sql.merge")]
pub struct PyWhenMatched {
- inner: Option<spark_connect::merge::WhenMatched>,
+ writer: Py<PyMergeIntoWriter>,
+ condition: Option<spark_connect::column::Column>,
}
#[pymethods]
impl PyWhenMatched {
#[pyo3(name = "updateAll")]
- fn update_all(&mut self) -> PyResult<PyMergeIntoWriter> {
- let w = self
- .inner
- .take()
- .ok_or_else(|| pyo3::exceptions::PyRuntimeError::new_err("clause
already consumed"))?;
- Ok(PyMergeIntoWriter {
- inner: Some(w.update_all()),
- })
+ fn update_all(&self, py: Python<'_>) -> PyResult<Py<PyMergeIntoWriter>> {
+ let cond = self.condition.clone();
+ merge_clause_action(py, &self.writer, |w|
w.when_matched(cond).update_all())
}
fn update(
- &mut self,
+ &self,
py: Python<'_>,
assignments: std::collections::HashMap<String, Py<PyColumn>>,
- ) -> PyResult<PyMergeIntoWriter> {
- let w = self
- .inner
- .take()
- .ok_or_else(|| pyo3::exceptions::PyRuntimeError::new_err("clause
already consumed"))?;
- Ok(PyMergeIntoWriter {
- inner: Some(w.update(assignments_to_map(py, assignments))),
- })
+ ) -> PyResult<Py<PyMergeIntoWriter>> {
+ let cond = self.condition.clone();
+ let map = assignments_to_map(py, assignments);
+ merge_clause_action(py, &self.writer, |w|
w.when_matched(cond).update(map))
}
- fn delete(&mut self) -> PyResult<PyMergeIntoWriter> {
- let w = self
- .inner
- .take()
- .ok_or_else(|| pyo3::exceptions::PyRuntimeError::new_err("clause
already consumed"))?;
- Ok(PyMergeIntoWriter {
- inner: Some(w.delete()),
- })
+ fn delete(&self, py: Python<'_>) -> PyResult<Py<PyMergeIntoWriter>> {
+ let cond = self.condition.clone();
+ merge_clause_action(py, &self.writer, |w|
w.when_matched(cond).delete())
}
}
-/// `whenNotMatched(...)` clause builder.
+/// `whenNotMatched(...)` clause builder. Holds its parent writer, like
PySpark's
+/// `MergeIntoWriter.WhenNotMatched`.
#[pyclass(name = "WhenNotMatched", module = "pyspark.sql.merge")]
pub struct PyWhenNotMatched {
- inner: Option<spark_connect::merge::WhenNotMatched>,
+ writer: Py<PyMergeIntoWriter>,
+ condition: Option<spark_connect::column::Column>,
}
#[pymethods]
impl PyWhenNotMatched {
#[pyo3(name = "insertAll")]
- fn insert_all(&mut self) -> PyResult<PyMergeIntoWriter> {
- let w = self
- .inner
- .take()
- .ok_or_else(|| pyo3::exceptions::PyRuntimeError::new_err("clause
already consumed"))?;
- Ok(PyMergeIntoWriter {
- inner: Some(w.insert_all()),
- })
+ fn insert_all(&self, py: Python<'_>) -> PyResult<Py<PyMergeIntoWriter>> {
+ let cond = self.condition.clone();
+ merge_clause_action(py, &self.writer, |w|
w.when_not_matched(cond).insert_all())
}
fn insert(
- &mut self,
+ &self,
py: Python<'_>,
assignments: std::collections::HashMap<String, Py<PyColumn>>,
- ) -> PyResult<PyMergeIntoWriter> {
- let w = self
- .inner
- .take()
- .ok_or_else(|| pyo3::exceptions::PyRuntimeError::new_err("clause
already consumed"))?;
- Ok(PyMergeIntoWriter {
- inner: Some(w.insert(assignments_to_map(py, assignments))),
- })
+ ) -> PyResult<Py<PyMergeIntoWriter>> {
+ let cond = self.condition.clone();
+ let map = assignments_to_map(py, assignments);
+ merge_clause_action(py, &self.writer, |w|
w.when_not_matched(cond).insert(map))
}
}
-/// `whenNotMatchedBySource(...)` clause builder.
+/// `whenNotMatchedBySource(...)` clause builder. Holds its parent writer, like
+/// PySpark's `MergeIntoWriter.WhenNotMatchedBySource`.
#[pyclass(name = "WhenNotMatchedBySource", module = "pyspark.sql.merge")]
pub struct PyWhenNotMatchedBySource {
- inner: Option<spark_connect::merge::WhenNotMatchedBySource>,
+ writer: Py<PyMergeIntoWriter>,
+ condition: Option<spark_connect::column::Column>,
}
#[pymethods]
impl PyWhenNotMatchedBySource {
#[pyo3(name = "updateAll")]
- fn update_all(&mut self) -> PyResult<PyMergeIntoWriter> {
- let w = self
- .inner
- .take()
- .ok_or_else(|| pyo3::exceptions::PyRuntimeError::new_err("clause
already consumed"))?;
- Ok(PyMergeIntoWriter {
- inner: Some(w.update_all()),
+ fn update_all(&self, py: Python<'_>) -> PyResult<Py<PyMergeIntoWriter>> {
+ let cond = self.condition.clone();
+ merge_clause_action(py, &self.writer, |w| {
+ w.when_not_matched_by_source(cond).update_all()
})
}
fn update(
- &mut self,
+ &self,
py: Python<'_>,
assignments: std::collections::HashMap<String, Py<PyColumn>>,
- ) -> PyResult<PyMergeIntoWriter> {
- let w = self
- .inner
- .take()
- .ok_or_else(|| pyo3::exceptions::PyRuntimeError::new_err("clause
already consumed"))?;
- Ok(PyMergeIntoWriter {
- inner: Some(w.update(assignments_to_map(py, assignments))),
+ ) -> PyResult<Py<PyMergeIntoWriter>> {
+ let cond = self.condition.clone();
+ let map = assignments_to_map(py, assignments);
+ merge_clause_action(py, &self.writer, |w| {
+ w.when_not_matched_by_source(cond).update(map)
})
}
- fn delete(&mut self) -> PyResult<PyMergeIntoWriter> {
- let w = self
- .inner
- .take()
- .ok_or_else(|| pyo3::exceptions::PyRuntimeError::new_err("clause
already consumed"))?;
- Ok(PyMergeIntoWriter {
- inner: Some(w.delete()),
+ fn delete(&self, py: Python<'_>) -> PyResult<Py<PyMergeIntoWriter>> {
+ let cond = self.condition.clone();
+ merge_clause_action(py, &self.writer, |w| {
+ w.when_not_matched_by_source(cond).delete()
})
}
}
@@ -1910,16 +1895,10 @@ impl PyWhenNotMatchedBySource {
/// Python wrapper for the V1 `DataFrameWriter` (`df.write`).
#[pyclass(name = "DataFrameWriter", module = "pyspark.sql.readwriter")]
pub struct PyDataFrameWriter {
- inner: Option<spark_connect::readwriter::DataFrameWriter>,
+ inner: spark_connect::readwriter::DataFrameWriter,
}
-impl PyDataFrameWriter {
- fn take(&mut self) -> PyResult<spark_connect::readwriter::DataFrameWriter>
{
- self.inner.take().ok_or_else(|| {
- PyErr::new::<pyo3::exceptions::PyRuntimeError, _>("DataFrameWriter
already consumed")
- })
- }
-}
+impl PyDataFrameWriter {}
/// Apply the shared `format` / `mode` / `partitionBy` / `**options` arguments
that
/// reference `DataFrameWriter.save` and `.saveAsTable` accept inline,
mirroring
@@ -1959,34 +1938,38 @@ fn apply_write_common(
#[pymethods]
impl PyDataFrameWriter {
- fn mode(&mut self, saveMode: &str) -> PyResult<PyDataFrameWriter> {
- Ok(PyDataFrameWriter {
- inner: Some(self.take()?.mode(saveMode)),
- })
+ fn mode<'py>(mut slf: PyRefMut<'py, Self>, saveMode: &str) ->
PyResult<PyRefMut<'py, Self>> {
+ slf.inner = slf.inner.clone().mode(saveMode);
+ Ok(slf)
}
- fn format(&mut self, source: &str) -> PyResult<PyDataFrameWriter> {
- Ok(PyDataFrameWriter {
- inner: Some(self.take()?.format(source)),
- })
+ fn format<'py>(mut slf: PyRefMut<'py, Self>, source: &str) ->
PyResult<PyRefMut<'py, Self>> {
+ slf.inner = slf.inner.clone().format(source);
+ Ok(slf)
}
- fn option(&mut self, key: &str, value: &Bound<'_, PyAny>) ->
PyResult<PyDataFrameWriter> {
+ fn option<'py>(
+ mut slf: PyRefMut<'py, Self>,
+ key: &str,
+ value: &Bound<'_, PyAny>,
+ ) -> PyResult<PyRefMut<'py, Self>> {
// None -> option left unset; bools -> "true"/"false" (reference
`to_str`).
match crate::coerce_option_value(value)? {
- Some(v) => Ok(PyDataFrameWriter {
- inner: Some(self.take()?.option(key, &v)),
- }),
- None => Ok(PyDataFrameWriter {
- inner: Some(self.take()?),
- }),
+ Some(v) => {
+ slf.inner = slf.inner.clone().option(key, &v);
+ Ok(slf)
+ }
+ None => Ok(slf),
}
}
// Mirrors reference `DataFrameWriter.options(**options)`: keyword args;
None values
// skipped, booleans lowercased.
#[pyo3(signature = (**options))]
- fn options(&mut self, options: Option<&Bound<'_, PyDict>>) ->
PyResult<PyDataFrameWriter> {
+ fn options<'py>(
+ mut slf: PyRefMut<'py, Self>,
+ options: Option<&Bound<'_, PyDict>>,
+ ) -> PyResult<PyRefMut<'py, Self>> {
let mut map = std::collections::HashMap::new();
if let Some(options) = options {
for (k, v) in options.iter() {
@@ -1995,46 +1978,51 @@ impl PyDataFrameWriter {
}
}
}
- Ok(PyDataFrameWriter {
- inner: Some(self.take()?.options(map)),
- })
+ slf.inner = slf.inner.clone().options(map);
+ Ok(slf)
}
#[pyo3(name = "partitionBy", signature = (*cols))]
- fn partition_by(&mut self, cols: Vec<Bound<'_, PyAny>>) ->
PyResult<PyDataFrameWriter> {
- Ok(PyDataFrameWriter {
- inner: Some(self.take()?.partition_by(flatten_str_cols(cols)?)),
- })
+ fn partition_by<'py>(
+ mut slf: PyRefMut<'py, Self>,
+ cols: Vec<Bound<'_, PyAny>>,
+ ) -> PyResult<PyRefMut<'py, Self>> {
+ slf.inner = slf.inner.clone().partition_by(flatten_str_cols(cols)?);
+ Ok(slf)
}
#[pyo3(name = "bucketBy", signature = (numBuckets, col, *cols))]
- fn bucket_by(
- &mut self,
+ fn bucket_by<'py>(
+ mut slf: PyRefMut<'py, Self>,
numBuckets: i32,
col: String,
cols: Vec<String>,
- ) -> PyResult<PyDataFrameWriter> {
+ ) -> PyResult<PyRefMut<'py, Self>> {
let mut all = vec![col];
all.extend(cols);
- Ok(PyDataFrameWriter {
- inner: Some(self.take()?.bucket_by(numBuckets, all)),
- })
+ slf.inner = slf.inner.clone().bucket_by(numBuckets, all);
+ Ok(slf)
}
#[pyo3(name = "sortBy", signature = (col, *cols))]
- fn sort_by(&mut self, col: String, cols: Vec<String>) ->
PyResult<PyDataFrameWriter> {
+ fn sort_by<'py>(
+ mut slf: PyRefMut<'py, Self>,
+ col: String,
+ cols: Vec<String>,
+ ) -> PyResult<PyRefMut<'py, Self>> {
let mut all = vec![col];
all.extend(cols);
- Ok(PyDataFrameWriter {
- inner: Some(self.take()?.sort_by(all)),
- })
+ slf.inner = slf.inner.clone().sort_by(all);
+ Ok(slf)
}
#[pyo3(name = "clusterBy", signature = (*cols))]
- fn cluster_by(&mut self, cols: Vec<String>) -> PyResult<PyDataFrameWriter>
{
- Ok(PyDataFrameWriter {
- inner: Some(self.take()?.cluster_by(cols)),
- })
+ fn cluster_by<'py>(
+ mut slf: PyRefMut<'py, Self>,
+ cols: Vec<String>,
+ ) -> PyResult<PyRefMut<'py, Self>> {
+ slf.inner = slf.inner.clone().cluster_by(cols);
+ Ok(slf)
}
/// Write to an external JDBC table. Connection details go through
`properties`.
@@ -2046,17 +2034,17 @@ impl PyDataFrameWriter {
mode: Option<&str>,
properties: Option<std::collections::HashMap<String, String>>,
) -> PyResult<()> {
- let mut w = self
- .take()?
- .format("jdbc")
- .option("url", url)
- .option("dbtable", table);
+ // Same order as PySpark: format, mode, properties, then url/dbtable
(which
+ // win over same-named properties), all kept on the writer, then save.
+ let mut w = self.inner.clone().format("jdbc");
if let Some(m) = mode {
w = w.mode(m);
}
if let Some(props) = properties {
w = w.options(props);
}
+ w = w.option("url", url).option("dbtable", table);
+ self.inner = w.clone();
w.save(None).to_pyerr()
}
@@ -2070,7 +2058,9 @@ impl PyDataFrameWriter {
partitionBy: Option<&Bound<'_, PyAny>>,
options: Option<&Bound<'_, PyDict>>,
) -> PyResult<()> {
- let w = apply_write_common(self.take()?, format, mode, partitionBy,
options)?;
+ let w = apply_write_common(self.inner.clone(), format, mode,
partitionBy, options)?;
+ // PySpark sets the inline format/mode/partitionBy/options on the
writer itself.
+ self.inner = w.clone();
w.save(path.as_deref()).to_pyerr()
}
@@ -2084,7 +2074,8 @@ impl PyDataFrameWriter {
partitionBy: Option<&Bound<'_, PyAny>>,
options: Option<&Bound<'_, PyDict>>,
) -> PyResult<()> {
- let w = apply_write_common(self.take()?, format, mode, partitionBy,
options)?;
+ let w = apply_write_common(self.inner.clone(), format, mode,
partitionBy, options)?;
+ self.inner = w.clone();
w.save_as_table(name).to_pyerr()
}
@@ -2092,10 +2083,12 @@ impl PyDataFrameWriter {
/// mirroring reference `DataFrameWriter.insertInto(tableName,
overwrite=None)`.
#[pyo3(name = "insertInto", signature = (tableName, overwrite=None))]
fn insert_into(&mut self, tableName: &str, overwrite: Option<bool>) ->
PyResult<()> {
- let mut w = self.take()?;
+ let mut w = self.inner.clone();
if let Some(ow) = overwrite {
w = w.mode(if ow { "overwrite" } else { "append" });
}
+ // PySpark sets the overwrite/append mode on the writer itself.
+ self.inner = w.clone();
w.insert_into(tableName).to_pyerr()
}
@@ -2110,7 +2103,7 @@ impl PyDataFrameWriter {
partitionBy: Option<&Bound<'_, PyAny>>,
compression: Option<&Bound<'_, PyAny>>,
) -> PyResult<()> {
- let mut w = self.take()?;
+ let mut w = self.inner.clone();
if let Some(m) = mode {
w = w.mode(&m.str()?.to_string());
}
@@ -2123,6 +2116,8 @@ impl PyDataFrameWriter {
w = w.partition_by(cols);
}
w = wset_opt(w, "compression", compression)?;
+ // PySpark sets the mode/options/format on the writer itself, then
saves.
+ self.inner = w.clone().format("parquet");
w.parquet(path).to_pyerr()
}
@@ -2141,7 +2136,7 @@ impl PyDataFrameWriter {
encoding: Option<&Bound<'_, PyAny>>,
ignoreNullFields: Option<&Bound<'_, PyAny>>,
) -> PyResult<()> {
- let mut w = self.take()?;
+ let mut w = self.inner.clone();
if let Some(m) = mode {
w = w.mode(&m.str()?.to_string());
}
@@ -2151,6 +2146,8 @@ impl PyDataFrameWriter {
w = wset_opt(w, "lineSep", lineSep)?;
w = wset_opt(w, "encoding", encoding)?;
w = wset_opt(w, "ignoreNullFields", ignoreNullFields)?;
+ // PySpark sets the mode/options/format on the writer itself, then
saves.
+ self.inner = w.clone().format("json");
w.json(path).to_pyerr()
}
@@ -2179,7 +2176,7 @@ impl PyDataFrameWriter {
emptyValue: Option<&Bound<'_, PyAny>>,
lineSep: Option<&Bound<'_, PyAny>>,
) -> PyResult<()> {
- let mut w = self.take()?;
+ let mut w = self.inner.clone();
if let Some(m) = mode {
w = w.mode(&m.str()?.to_string());
}
@@ -2199,6 +2196,8 @@ impl PyDataFrameWriter {
w = wset_opt(w, "encoding", encoding)?;
w = wset_opt(w, "emptyValue", emptyValue)?;
w = wset_opt(w, "lineSep", lineSep)?;
+ // PySpark sets the mode/options/format on the writer itself, then
saves.
+ self.inner = w.clone().format("csv");
w.csv(path).to_pyerr()
}
@@ -2213,7 +2212,7 @@ impl PyDataFrameWriter {
partitionBy: Option<&Bound<'_, PyAny>>,
compression: Option<&Bound<'_, PyAny>>,
) -> PyResult<()> {
- let mut w = self.take()?;
+ let mut w = self.inner.clone();
if let Some(m) = mode {
w = w.mode(&m.str()?.to_string());
}
@@ -2226,6 +2225,8 @@ impl PyDataFrameWriter {
w = w.partition_by(cols);
}
w = wset_opt(w, "compression", compression)?;
+ // PySpark sets the mode/options/format on the writer itself, then
saves.
+ self.inner = w.clone().format("orc");
w.orc(path).to_pyerr()
}
@@ -2239,9 +2240,11 @@ impl PyDataFrameWriter {
compression: Option<&Bound<'_, PyAny>>,
lineSep: Option<&Bound<'_, PyAny>>,
) -> PyResult<()> {
- let mut w = self.take()?;
+ let mut w = self.inner.clone();
w = wset_opt(w, "compression", compression)?;
w = wset_opt(w, "lineSep", lineSep)?;
+ // PySpark sets the mode/options/format on the writer itself, then
saves.
+ self.inner = w.clone().format("text");
w.text(path).to_pyerr()
}
@@ -2266,7 +2269,7 @@ impl PyDataFrameWriter {
encoding: Option<&Bound<'_, PyAny>>,
validateName: Option<&Bound<'_, PyAny>>,
) -> PyResult<()> {
- let mut w = self.take()?;
+ let mut w = self.inner.clone();
w = wset_opt(w, "rowTag", rowTag)?;
if let Some(m) = mode {
w = w.mode(&m.str()?.to_string());
@@ -2282,6 +2285,8 @@ impl PyDataFrameWriter {
w = wset_opt(w, "compression", compression)?;
w = wset_opt(w, "encoding", encoding)?;
w = wset_opt(w, "validateName", validateName)?;
+ // PySpark sets the mode/options/format on the writer itself, then
saves.
+ self.inner = w.clone().format("xml");
w.xml(path).to_pyerr()
}
}
@@ -2289,41 +2294,40 @@ impl PyDataFrameWriter {
/// Python wrapper for the V2 `DataFrameWriter` (`df.writeTo`).
#[pyclass(name = "DataFrameWriterV2", module = "pyspark.sql.readwriter")]
pub struct PyDataFrameWriterV2 {
- inner: Option<spark_connect::readwriter::DataFrameWriterV2>,
+ inner: spark_connect::readwriter::DataFrameWriterV2,
}
-impl PyDataFrameWriterV2 {
- fn take(&mut self) ->
PyResult<spark_connect::readwriter::DataFrameWriterV2> {
- self.inner.take().ok_or_else(|| {
- PyErr::new::<pyo3::exceptions::PyRuntimeError,
_>("DataFrameWriterV2 already consumed")
- })
- }
-}
+impl PyDataFrameWriterV2 {}
#[pymethods]
impl PyDataFrameWriterV2 {
- fn using(&mut self, provider: &str) -> PyResult<PyDataFrameWriterV2> {
- Ok(PyDataFrameWriterV2 {
- inner: Some(self.take()?.using(provider)),
- })
+ fn using<'py>(mut slf: PyRefMut<'py, Self>, provider: &str) ->
PyResult<PyRefMut<'py, Self>> {
+ slf.inner = slf.inner.clone().using(provider);
+ Ok(slf)
}
// None -> option left unset; bools -> "true"/"false" (reference
`to_str`), same as
// the other reader/writer option bindings.
- fn option(&mut self, key: &str, value: &Bound<'_, PyAny>) ->
PyResult<PyDataFrameWriterV2> {
+ fn option<'py>(
+ mut slf: PyRefMut<'py, Self>,
+ key: &str,
+ value: &Bound<'_, PyAny>,
+ ) -> PyResult<PyRefMut<'py, Self>> {
match crate::coerce_option_value(value)? {
- Some(v) => Ok(PyDataFrameWriterV2 {
- inner: Some(self.take()?.option(key, &v)),
- }),
- None => Ok(PyDataFrameWriterV2 {
- inner: Some(self.take()?),
- }),
+ Some(v) => {
+ slf.inner = slf.inner.clone().option(key, &v);
+ Ok(slf)
+ }
+ None => Ok(slf),
}
}
// Mirrors reference `DataFrameWriterV2.options(**options)`; None skipped,
bools lowercased.
#[pyo3(signature = (**options))]
- fn options(&mut self, options: Option<&Bound<'_, PyDict>>) ->
PyResult<PyDataFrameWriterV2> {
+ fn options<'py>(
+ mut slf: PyRefMut<'py, Self>,
+ options: Option<&Bound<'_, PyDict>>,
+ ) -> PyResult<PyRefMut<'py, Self>> {
let mut map = std::collections::HashMap::new();
if let Some(options) = options {
for (k, v) in options.iter() {
@@ -2332,65 +2336,72 @@ impl PyDataFrameWriterV2 {
}
}
}
- Ok(PyDataFrameWriterV2 {
- inner: Some(self.take()?.options(map)),
- })
+ slf.inner = slf.inner.clone().options(map);
+ Ok(slf)
}
#[pyo3(name = "tableProperty")]
- fn table_property(&mut self, property: &str, value: &str) ->
PyResult<PyDataFrameWriterV2> {
- Ok(PyDataFrameWriterV2 {
- inner: Some(self.take()?.table_property(property, value)),
- })
+ fn table_property<'py>(
+ mut slf: PyRefMut<'py, Self>,
+ property: &str,
+ value: &str,
+ ) -> PyResult<PyRefMut<'py, Self>> {
+ slf.inner = slf.inner.clone().table_property(property, value);
+ Ok(slf)
}
#[pyo3(name = "partitionedBy", signature = (col, *cols))]
- fn partitioned_by(
- &mut self,
+ fn partitioned_by<'py>(
+ mut slf: PyRefMut<'py, Self>,
col: Bound<'_, PyAny>,
cols: Vec<Bound<'_, PyAny>>,
- ) -> PyResult<PyDataFrameWriterV2> {
+ ) -> PyResult<PyRefMut<'py, Self>> {
let mut all = vec![col];
all.extend(cols);
let columns = to_column_list(all)?;
- Ok(PyDataFrameWriterV2 {
- inner: Some(self.take()?.partition_by(columns)),
- })
+ slf.inner = slf.inner.clone().partition_by(columns);
+ Ok(slf)
}
#[pyo3(name = "clusterBy", signature = (col, *cols))]
- fn cluster_by(&mut self, col: String, cols: Vec<String>) ->
PyResult<PyDataFrameWriterV2> {
+ fn cluster_by<'py>(
+ mut slf: PyRefMut<'py, Self>,
+ col: String,
+ cols: Vec<String>,
+ ) -> PyResult<PyRefMut<'py, Self>> {
let mut all = vec![col];
all.extend(cols);
- Ok(PyDataFrameWriterV2 {
- inner: Some(self.take()?.cluster_by(all)),
- })
+ slf.inner = slf.inner.clone().cluster_by(all);
+ Ok(slf)
}
fn create(&mut self) -> PyResult<()> {
- self.take()?.create().to_pyerr()
+ self.inner.clone().create().to_pyerr()
}
fn replace(&mut self) -> PyResult<()> {
- self.take()?.replace().to_pyerr()
+ self.inner.clone().replace().to_pyerr()
}
#[pyo3(name = "createOrReplace")]
fn create_or_replace(&mut self) -> PyResult<()> {
- self.take()?.create_or_replace().to_pyerr()
+ self.inner.clone().create_or_replace().to_pyerr()
}
fn append(&mut self) -> PyResult<()> {
- self.take()?.append().to_pyerr()
+ self.inner.clone().append().to_pyerr()
}
fn overwrite(&mut self, condition: &PyColumn) -> PyResult<()> {
- self.take()?.overwrite(condition.column.clone()).to_pyerr()
+ self.inner
+ .clone()
+ .overwrite(condition.column.clone())
+ .to_pyerr()
}
#[pyo3(name = "overwritePartitions")]
fn overwrite_partitions(&mut self) -> PyResult<()> {
- self.take()?.overwrite_partitions().to_pyerr()
+ self.inner.clone().overwrite_partitions().to_pyerr()
}
}
diff --git a/crates/pyspark-rs/src/readwriter.rs
b/crates/pyspark-rs/src/readwriter.rs
index 3ef57a5..3c0db1b 100644
--- a/crates/pyspark-rs/src/readwriter.rs
+++ b/crates/pyspark-rs/src/readwriter.rs
@@ -8,31 +8,26 @@ use spark_connect::readwriter::DataFrameReader;
use crate::dataframe::PyDataFrame;
-/// Python wrapper for the batch DataFrameReader. The core reader is a
consuming
-/// builder, so each step takes the inner value and returns a fresh wrapper
(mirrors
-/// the existing PyDataStreamReader).
+/// Python wrapper for the batch DataFrameReader.
+///
+/// PySpark's reader is mutable: `format`/`schema`/`option`/`options` change
`self`
+/// and return it, and a reader can be used again after a read. The core
reader is a
+/// consuming builder, so the setters replace the wrapped value in place and
return
+/// the same Python object, and the terminal reads build their DataFrame from
a clone.
#[pyclass(name = "DataFrameReader", module = "pyspark.sql.readwriter")]
pub struct PyDataFrameReader {
- inner: Option<DataFrameReader>,
+ inner: DataFrameReader,
}
impl PyDataFrameReader {
pub fn new(reader: DataFrameReader) -> Self {
- PyDataFrameReader {
- inner: Some(reader),
- }
- }
-
- fn take(&mut self) -> PyResult<DataFrameReader> {
- self.inner.take().ok_or_else(|| {
- PyErr::new::<pyo3::exceptions::PyRuntimeError, _>("DataFrameReader
already consumed")
- })
+ PyDataFrameReader { inner: reader }
}
- /// Take the inner reader and apply per-call `**options` (skipping None,
- /// lowercasing bools) before a format-specific read.
- fn take_with_opts(&mut self, options: Option<&Bound<'_, PyDict>>) ->
PyResult<DataFrameReader> {
- let mut r = self.take()?;
+ /// A copy of the current reader with per-call `**options` applied
(skipping
+ /// None, lowercasing bools), for a format-specific read.
+ fn with_opts(&self, options: Option<&Bound<'_, PyDict>>) ->
PyResult<DataFrameReader> {
+ let mut r = self.inner.clone();
if let Some(opts) = options {
for (k, v) in opts.iter() {
if let Some(val) = crate::coerce_option_value(&v)? {
@@ -42,6 +37,14 @@ impl PyDataFrameReader {
}
Ok(r)
}
+
+ /// Keep the options of a format-specific read on this reader, as PySpark
does
+ /// (its `json`/`csv`/... set the options and the format on `self`, then
`load`).
+ /// Called only after every argument was validated, so a failed call
leaves the
+ /// reader unchanged.
+ fn commit(&mut self, r: &DataFrameReader, format: &str) {
+ self.inner = r.clone().format(format);
+ }
}
/// Apply a single reader option (skipping None; bool->"true"/"false").
@@ -69,30 +72,37 @@ fn apply_reader_schema(r: DataFrameReader, v: &Bound<'_,
PyAny>) -> PyResult<Dat
#[pymethods]
impl PyDataFrameReader {
/// Set the source format (e.g. "parquet", "json", "csv").
- fn format(&mut self, source: &str) -> PyResult<PyDataFrameReader> {
- Ok(PyDataFrameReader::new(self.take()?.format(source)))
+ fn format<'py>(mut slf: PyRefMut<'py, Self>, source: &str) ->
PyRefMut<'py, Self> {
+ slf.inner = slf.inner.clone().format(source);
+ slf
}
/// Set the schema (a DDL string).
- fn schema(&mut self, schema: &str) -> PyResult<PyDataFrameReader> {
- Ok(PyDataFrameReader::new(
- self.take()?.schema(schema.to_string()),
- ))
+ fn schema<'py>(mut slf: PyRefMut<'py, Self>, schema: &str) ->
PyRefMut<'py, Self> {
+ slf.inner = slf.inner.clone().schema(schema.to_string());
+ slf
}
/// Set a single read option. `None` leaves the option unset (not the
string
/// "None"); booleans lowercase to "true"/"false" (reference `to_str`
semantics).
- fn option(&mut self, key: &str, value: &Bound<'_, PyAny>) ->
PyResult<PyDataFrameReader> {
- match crate::coerce_option_value(value)? {
- Some(v) => Ok(PyDataFrameReader::new(self.take()?.option(key,
&v))),
- None => Ok(PyDataFrameReader::new(self.take()?)),
+ fn option<'py>(
+ mut slf: PyRefMut<'py, Self>,
+ key: &str,
+ value: &Bound<'_, PyAny>,
+ ) -> PyResult<PyRefMut<'py, Self>> {
+ if let Some(v) = crate::coerce_option_value(value)? {
+ slf.inner = slf.inner.clone().option(key, &v);
}
+ Ok(slf)
}
/// Set multiple read options. Mirrors reference
`DataFrameReader.options(**options)`
/// - keyword args; `None` values are skipped and booleans lowercased.
#[pyo3(signature = (**options))]
- fn options(&mut self, options: Option<&Bound<'_, PyDict>>) ->
PyResult<PyDataFrameReader> {
+ fn options<'py>(
+ mut slf: PyRefMut<'py, Self>,
+ options: Option<&Bound<'_, PyDict>>,
+ ) -> PyResult<PyRefMut<'py, Self>> {
let mut map = HashMap::new();
if let Some(options) = options {
for (k, v) in options.iter() {
@@ -101,7 +111,26 @@ impl PyDataFrameReader {
}
}
}
- Ok(PyDataFrameReader::new(self.take()?.options(map)))
+ slf.inner = slf.inner.clone().options(map);
+ Ok(slf)
+ }
+
+ /// The configured source format (PySpark Connect's
`DataFrameReader._format`).
+ #[getter]
+ fn _format(&self) -> Option<String> {
+ self.inner.get_format().map(str::to_string)
+ }
+
+ /// The configured schema string, empty when unset
(`DataFrameReader._schema`).
+ #[getter]
+ fn _schema(&self) -> String {
+ self.inner.get_schema().to_string()
+ }
+
+ /// The configured read options (`DataFrameReader._options`); a copy.
+ #[getter]
+ fn _options(&self) -> HashMap<String, String> {
+ self.inner.get_options().clone()
}
/// Load data from the (optional) path. Mirrors the reference
@@ -116,25 +145,27 @@ impl PyDataFrameReader {
schema: Option<&Bound<'_, PyAny>>,
options: Option<&Bound<'_, PyDict>>,
) -> PyResult<PyDataFrame> {
- let mut r = self.take_with_opts(options)?;
+ let mut r = self.with_opts(options)?;
if let Some(f) = format {
r = r.format(f);
}
if let Some(s) = schema {
r = apply_reader_schema(r, s)?;
}
+ // PySpark's one-shot load sets format/schema/options on the reader
itself.
+ self.inner = r.clone();
Ok(PyDataFrame::new(r.load(path)))
}
/// Read a table by name.
- fn table(&mut self, tableName: &str) -> PyResult<PyDataFrame> {
- Ok(PyDataFrame::new(self.take()?.table(tableName)))
+ fn table(&self, tableName: &str) -> PyResult<PyDataFrame> {
+ Ok(PyDataFrame::new(self.inner.clone().table(tableName)))
}
/// Read the CDC changes of a named table. Mirrors
`DataFrameReader.changes`.
#[allow(non_snake_case)]
- fn changes(&mut self, tableName: &str) -> PyResult<PyDataFrame> {
- Ok(PyDataFrame::new(self.take()?.changes(tableName)))
+ fn changes(&self, tableName: &str) -> PyResult<PyDataFrame> {
+ Ok(PyDataFrame::new(self.inner.clone().changes(tableName)))
}
/// Read JSON file(s) - full pyspark signature; each named option is
@@ -170,7 +201,7 @@ impl PyDataFrameReader {
allowNonNumericNumbers: Option<&Bound<'_, PyAny>>,
useUnsafeRow: Option<&Bound<'_, PyAny>>,
) -> PyResult<PyDataFrame> {
- let mut r = self.take()?;
+ let mut r = self.inner.clone();
if let Some(v) = schema {
r = apply_reader_schema(r, v)?;
}
@@ -202,6 +233,7 @@ impl PyDataFrameReader {
r = set_opt(r, "modifiedAfter", modifiedAfter)?;
r = set_opt(r, "allowNonNumericNumbers", allowNonNumericNumbers)?;
r = set_opt(r, "useUnsafeRow", useUnsafeRow)?;
+ self.commit(&r, "json");
Ok(PyDataFrame::new(r.json(path)))
}
@@ -219,9 +251,9 @@ impl PyDataFrameReader {
"parquet() currently supports exactly one path",
));
}
- Ok(PyDataFrame::new(
- self.take_with_opts(options)?.parquet(&paths[0]),
- ))
+ let r = self.with_opts(options)?;
+ self.commit(&r, "parquet");
+ Ok(PyDataFrame::new(r.parquet(&paths[0])))
}
/// Read CSV file(s) - full pyspark signature; each named option is
@@ -265,7 +297,7 @@ impl PyDataFrameReader {
modifiedAfter: Option<&Bound<'_, PyAny>>,
unescapedQuoteHandling: Option<&Bound<'_, PyAny>>,
) -> PyResult<PyDataFrame> {
- let mut r = self.take()?;
+ let mut r = self.inner.clone();
if let Some(v) = schema {
r = apply_reader_schema(r, v)?;
}
@@ -305,6 +337,7 @@ impl PyDataFrameReader {
r = set_opt(r, "modifiedBefore", modifiedBefore)?;
r = set_opt(r, "modifiedAfter", modifiedAfter)?;
r = set_opt(r, "unescapedQuoteHandling", unescapedQuoteHandling)?;
+ self.commit(&r, "csv");
Ok(PyDataFrame::new(r.csv(path)))
}
@@ -321,12 +354,13 @@ impl PyDataFrameReader {
modifiedBefore: Option<&Bound<'_, PyAny>>,
modifiedAfter: Option<&Bound<'_, PyAny>>,
) -> PyResult<PyDataFrame> {
- let mut r = self.take()?;
+ let mut r = self.inner.clone();
r = set_opt(r, "mergeSchema", mergeSchema)?;
r = set_opt(r, "pathGlobFilter", pathGlobFilter)?;
r = set_opt(r, "recursiveFileLookup", recursiveFileLookup)?;
r = set_opt(r, "modifiedBefore", modifiedBefore)?;
r = set_opt(r, "modifiedAfter", modifiedAfter)?;
+ self.commit(&r, "orc");
Ok(PyDataFrame::new(r.orc(path)))
}
@@ -344,13 +378,14 @@ impl PyDataFrameReader {
modifiedBefore: Option<&Bound<'_, PyAny>>,
modifiedAfter: Option<&Bound<'_, PyAny>>,
) -> PyResult<PyDataFrame> {
- let mut r = self.take()?;
+ let mut r = self.inner.clone();
r = set_opt(r, "wholetext", wholetext)?;
r = set_opt(r, "lineSep", lineSep)?;
r = set_opt(r, "pathGlobFilter", pathGlobFilter)?;
r = set_opt(r, "recursiveFileLookup", recursiveFileLookup)?;
r = set_opt(r, "modifiedBefore", modifiedBefore)?;
r = set_opt(r, "modifiedAfter", modifiedAfter)?;
+ self.commit(&r, "text");
Ok(PyDataFrame::new(r.text(paths)))
}
@@ -381,7 +416,7 @@ impl PyDataFrameReader {
samplingRatio: Option<&Bound<'_, PyAny>>,
locale: Option<&Bound<'_, PyAny>>,
) -> PyResult<PyDataFrame> {
- let mut r = self.take()?;
+ let mut r = self.inner.clone();
r = set_opt(r, "rowTag", rowTag)?;
if let Some(v) = schema {
r = apply_reader_schema(r, v)?;
@@ -403,6 +438,7 @@ impl PyDataFrameReader {
r = set_opt(r, "multiLine", multiLine)?;
r = set_opt(r, "samplingRatio", samplingRatio)?;
r = set_opt(r, "locale", locale)?;
+ self.commit(&r, "xml");
Ok(PyDataFrame::new(r.xml(path)))
}
@@ -424,7 +460,7 @@ impl PyDataFrameReader {
predicates: Option<Vec<String>>,
properties: Option<HashMap<String, String>>,
) -> PyResult<PyDataFrame> {
- let mut r = self.take()?;
+ let mut r = self.inner.clone();
if let Some(c) = column {
r = r.option("partitionColumn", &c);
}
@@ -442,6 +478,12 @@ impl PyDataFrameReader {
r = r.option(&k, &v);
}
}
+ // PySpark's jdbc also leaves format/url/dbtable on the reader.
+ self.inner = r
+ .clone()
+ .format("jdbc")
+ .option("url", url)
+ .option("dbtable", table);
Ok(PyDataFrame::new(r.jdbc(url, table, predicates)))
}
}
diff --git a/crates/pyspark-rs/src/streaming.rs
b/crates/pyspark-rs/src/streaming.rs
index 09977bc..88d3baf 100644
--- a/crates/pyspark-rs/src/streaming.rs
+++ b/crates/pyspark-rs/src/streaming.rs
@@ -40,7 +40,6 @@ impl StreamingQueryListener for PyListenerAdapter {
}
}
-/// Python wrapper for DataStreamReader.
/// Apply a named read option to a streaming reader when supplied (mirrors the
batch
/// `set_opt` in readwriter.rs): `None` leaves the option unset (not the
string "None");
/// booleans lowercase to "true"/"false" via `coerce_option_value`.
@@ -74,55 +73,56 @@ fn extract_partition_cols(v: &Bound<'_, PyAny>) ->
PyResult<Vec<String>> {
v.extract::<Vec<String>>()
}
+/// Python wrapper for DataStreamReader.
+///
+/// Like PySpark's reader, the setters change this object and return it, and
the
+/// reader can be used again after a read: the core reader is a consuming
builder, so
+/// the setters replace the wrapped value in place and the reads work on a
clone.
#[pyclass(name = "DataStreamReader", module =
"pyspark.sql.streaming.readwriter")]
pub struct PyDataStreamReader {
- inner: Option<DataStreamReader>,
+ inner: DataStreamReader,
}
impl PyDataStreamReader {
pub fn new(reader: DataStreamReader) -> Self {
- PyDataStreamReader {
- inner: Some(reader),
- }
- }
-
- fn take(&mut self) -> PyResult<DataStreamReader> {
- self.inner.take().ok_or_else(|| {
- PyErr::new::<pyo3::exceptions::PyRuntimeError,
_>("DataStreamReader already consumed")
- })
+ PyDataStreamReader { inner: reader }
}
}
#[pymethods]
impl PyDataStreamReader {
- fn format(&mut self, source: &str) -> PyResult<PyDataStreamReader> {
- Ok(PyDataStreamReader {
- inner: Some(self.take()?.format(source)),
- })
+ fn format<'py>(mut slf: PyRefMut<'py, Self>, source: &str) ->
PyResult<PyRefMut<'py, Self>> {
+ slf.inner = slf.inner.clone().format(source);
+ Ok(slf)
}
- fn schema(&mut self, schema: &str) -> PyResult<PyDataStreamReader> {
- Ok(PyDataStreamReader {
- inner: Some(self.take()?.schema(schema)),
- })
+ fn schema<'py>(mut slf: PyRefMut<'py, Self>, schema: &str) ->
PyResult<PyRefMut<'py, Self>> {
+ slf.inner = slf.inner.clone().schema(schema);
+ Ok(slf)
}
- fn option(&mut self, key: &str, value: &Bound<'_, PyAny>) ->
PyResult<PyDataStreamReader> {
+ fn option<'py>(
+ mut slf: PyRefMut<'py, Self>,
+ key: &str,
+ value: &Bound<'_, PyAny>,
+ ) -> PyResult<PyRefMut<'py, Self>> {
// None -> option left unset; bools -> "true"/"false" (reference
`to_str`).
match crate::coerce_option_value(value)? {
- Some(v) => Ok(PyDataStreamReader {
- inner: Some(self.take()?.option(key, &v)),
- }),
- None => Ok(PyDataStreamReader {
- inner: Some(self.take()?),
- }),
+ Some(v) => {
+ slf.inner = slf.inner.clone().option(key, &v);
+ Ok(slf)
+ }
+ None => Ok(slf),
}
}
// Mirrors reference `DataStreamReader.options(**options)`: keyword args;
None values
// skipped, booleans lowercased.
#[pyo3(signature = (**options))]
- fn options(&mut self, options: Option<&Bound<'_, PyDict>>) ->
PyResult<PyDataStreamReader> {
+ fn options<'py>(
+ mut slf: PyRefMut<'py, Self>,
+ options: Option<&Bound<'_, PyDict>>,
+ ) -> PyResult<PyRefMut<'py, Self>> {
let mut opts = HashMap::new();
if let Some(options) = options {
for (k, v) in options.iter() {
@@ -131,9 +131,26 @@ impl PyDataStreamReader {
}
}
}
- Ok(PyDataStreamReader {
- inner: Some(self.take()?.options(opts)),
- })
+ slf.inner = slf.inner.clone().options(opts);
+ Ok(slf)
+ }
+
+ /// The configured source format (PySpark Connect's
`DataStreamReader._format`).
+ #[getter]
+ fn _format(&self) -> Option<String> {
+ self.inner.get_format().map(str::to_string)
+ }
+
+ /// The configured schema string, empty when unset
(`DataStreamReader._schema`).
+ #[getter]
+ fn _schema(&self) -> String {
+ self.inner.get_schema().to_string()
+ }
+
+ /// The configured read options (`DataStreamReader._options`); a copy.
+ #[getter]
+ fn _options(&self) -> HashMap<String, String> {
+ self.inner.get_options().clone()
}
#[pyo3(signature = (path=None, format=None, schema=None, **options))]
@@ -144,7 +161,7 @@ impl PyDataStreamReader {
schema: Option<&Bound<'_, PyAny>>,
options: Option<&Bound<'_, PyDict>>,
) -> PyResult<PyDataFrame> {
- let mut r = self.take()?;
+ let mut r = self.inner.clone();
if let Some(fmt) = format {
r = r.format(fmt);
}
@@ -160,12 +177,14 @@ impl PyDataStreamReader {
}
r = r.options(opts);
}
+ // PySpark's one-shot load sets format/schema/options on the reader
itself.
+ self.inner = r.clone();
Ok(PyDataFrame::new(r.load(path)))
}
/// Set the source name (for checkpoint stability). Mirrors
`DataStreamReader.name`,
/// which validates the name is a non-empty `[A-Za-z0-9_]+` string.
- fn name(&mut self, source_name: &str) -> PyResult<PyDataStreamReader> {
+ fn name<'py>(mut slf: PyRefMut<'py, Self>, source_name: &str) ->
PyResult<PyRefMut<'py, Self>> {
if source_name.is_empty()
|| !source_name
.chars()
@@ -175,14 +194,13 @@ impl PyDataStreamReader {
"Invalid streaming source name: {source_name:?}; only ASCII
letters, digits, and underscores are allowed"
)));
}
- Ok(PyDataStreamReader {
- inner: Some(self.take()?.name(source_name)),
- })
+ slf.inner = slf.inner.clone().name(source_name);
+ Ok(slf)
}
/// Read the streaming CDC changes of a named table. Mirrors
`DataStreamReader.changes`.
- fn changes(&mut self, tableName: &str) -> PyResult<PyDataFrame> {
- Ok(PyDataFrame::new(self.take()?.changes(tableName)))
+ fn changes(&self, tableName: &str) -> PyResult<PyDataFrame> {
+ Ok(PyDataFrame::new(self.inner.clone().changes(tableName)))
}
/// Load an XML streaming source. Mirrors `DataStreamReader.xml(...)`: set
each named
@@ -212,7 +230,7 @@ impl PyDataStreamReader {
samplingRatio: Option<&Bound<'_, PyAny>>,
locale: Option<&Bound<'_, PyAny>>,
) -> PyResult<PyDataFrame> {
- let mut r = self.take()?.format("xml");
+ let mut r = self.inner.clone().format("xml");
r = set_ropt(r, "rowTag", rowTag)?;
if let Some(v) = schema {
r = apply_stream_schema(r, v)?;
@@ -234,11 +252,13 @@ impl PyDataStreamReader {
r = set_ropt(r, "multiLine", multiLine)?;
r = set_ropt(r, "samplingRatio", samplingRatio)?;
r = set_ropt(r, "locale", locale)?;
+ // Like PySpark, the options and the xml format stay on the reader.
+ self.inner = r.clone();
Ok(PyDataFrame::new(r.load(Some(path))))
}
- fn table(&mut self, tableName: &str) -> PyResult<PyDataFrame> {
- let df = self.take()?.table(tableName);
+ fn table(&self, tableName: &str) -> PyResult<PyDataFrame> {
+ let df = self.inner.clone().table(tableName);
Ok(PyDataFrame::new(df))
}
@@ -270,7 +290,7 @@ impl PyDataStreamReader {
allowNonNumericNumbers: Option<&Bound<'_, PyAny>>,
useUnsafeRow: Option<&Bound<'_, PyAny>>,
) -> PyResult<PyDataFrame> {
- let mut r = self.take()?;
+ let mut r = self.inner.clone();
if let Some(v) = schema {
r = apply_stream_schema(r, v)?;
}
@@ -299,6 +319,8 @@ impl PyDataStreamReader {
r = set_ropt(r, "recursiveFileLookup", recursiveFileLookup)?;
r = set_ropt(r, "allowNonNumericNumbers", allowNonNumericNumbers)?;
r = set_ropt(r, "useUnsafeRow", useUnsafeRow)?;
+ // Like PySpark, the options and the format stay on the reader.
+ self.inner = r.clone().format("json");
Ok(PyDataFrame::new(r.json(path)))
}
@@ -313,12 +335,14 @@ impl PyDataStreamReader {
datetimeRebaseMode: Option<&Bound<'_, PyAny>>,
int96RebaseMode: Option<&Bound<'_, PyAny>>,
) -> PyResult<PyDataFrame> {
- let mut r = self.take()?;
+ let mut r = self.inner.clone();
r = set_ropt(r, "mergeSchema", mergeSchema)?;
r = set_ropt(r, "pathGlobFilter", pathGlobFilter)?;
r = set_ropt(r, "recursiveFileLookup", recursiveFileLookup)?;
r = set_ropt(r, "datetimeRebaseMode", datetimeRebaseMode)?;
r = set_ropt(r, "int96RebaseMode", int96RebaseMode)?;
+ // Like PySpark, the options and the format stay on the reader.
+ self.inner = r.clone().format("parquet");
Ok(PyDataFrame::new(r.parquet(path)))
}
@@ -358,7 +382,7 @@ impl PyDataStreamReader {
recursiveFileLookup: Option<&Bound<'_, PyAny>>,
unescapedQuoteHandling: Option<&Bound<'_, PyAny>>,
) -> PyResult<PyDataFrame> {
- let mut r = self.take()?;
+ let mut r = self.inner.clone();
if let Some(v) = schema {
r = apply_stream_schema(r, v)?;
}
@@ -395,6 +419,8 @@ impl PyDataStreamReader {
r = set_ropt(r, "pathGlobFilter", pathGlobFilter)?;
r = set_ropt(r, "recursiveFileLookup", recursiveFileLookup)?;
r = set_ropt(r, "unescapedQuoteHandling", unescapedQuoteHandling)?;
+ // Like PySpark, the options and the format stay on the reader.
+ self.inner = r.clone().format("csv");
Ok(PyDataFrame::new(r.csv(path)))
}
@@ -407,10 +433,12 @@ impl PyDataStreamReader {
pathGlobFilter: Option<&Bound<'_, PyAny>>,
recursiveFileLookup: Option<&Bound<'_, PyAny>>,
) -> PyResult<PyDataFrame> {
- let mut r = self.take()?;
+ let mut r = self.inner.clone();
r = set_ropt(r, "mergeSchema", mergeSchema)?;
r = set_ropt(r, "pathGlobFilter", pathGlobFilter)?;
r = set_ropt(r, "recursiveFileLookup", recursiveFileLookup)?;
+ // Like PySpark, the options and the format stay on the reader.
+ self.inner = r.clone().format("orc");
Ok(PyDataFrame::new(r.orc(path)))
}
@@ -424,11 +452,13 @@ impl PyDataStreamReader {
pathGlobFilter: Option<&Bound<'_, PyAny>>,
recursiveFileLookup: Option<&Bound<'_, PyAny>>,
) -> PyResult<PyDataFrame> {
- let mut r = self.take()?;
+ let mut r = self.inner.clone();
r = set_ropt(r, "wholetext", wholetext)?;
r = set_ropt(r, "lineSep", lineSep)?;
r = set_ropt(r, "pathGlobFilter", pathGlobFilter)?;
r = set_ropt(r, "recursiveFileLookup", recursiveFileLookup)?;
+ // Like PySpark, the options and the format stay on the reader.
+ self.inner = r.clone().format("text");
Ok(PyDataFrame::new(r.text(path)))
}
}
@@ -480,20 +510,12 @@ impl PyTrigger {
/// Python wrapper for DataStreamWriter.
#[pyclass(name = "DataStreamWriter", module =
"pyspark.sql.streaming.readwriter")]
pub struct PyDataStreamWriter {
- inner: Option<DataStreamWriter>,
+ inner: DataStreamWriter,
}
impl PyDataStreamWriter {
pub fn new(writer: DataStreamWriter) -> Self {
- PyDataStreamWriter {
- inner: Some(writer),
- }
- }
-
- fn take(&mut self) -> PyResult<DataStreamWriter> {
- self.inner.take().ok_or_else(|| {
- PyErr::new::<pyo3::exceptions::PyRuntimeError,
_>("DataStreamWriter already consumed")
- })
+ PyDataStreamWriter { inner: writer }
}
/// Apply the one-shot writer options accepted by `start`/`toTable`
@@ -507,7 +529,7 @@ impl PyDataStreamWriter {
query_name: Option<&str>,
options: Option<&Bound<'_, PyDict>>,
) -> PyResult<DataStreamWriter> {
- let mut w = self.take()?;
+ let mut w = self.inner.clone();
if let Some(fmt) = format {
w = w.format(fmt);
}
@@ -531,40 +553,49 @@ impl PyDataStreamWriter {
}
w = w.options(opts);
}
+ // PySpark's start/toTable set these on the writer itself before
starting.
+ self.inner = w.clone();
Ok(w)
}
}
#[pymethods]
impl PyDataStreamWriter {
- fn outputMode(&mut self, outputMode: &str) -> PyResult<PyDataStreamWriter>
{
- Ok(PyDataStreamWriter {
- inner: Some(self.take()?.output_mode(outputMode)),
- })
+ fn outputMode<'py>(
+ mut slf: PyRefMut<'py, Self>,
+ outputMode: &str,
+ ) -> PyResult<PyRefMut<'py, Self>> {
+ slf.inner = slf.inner.clone().output_mode(outputMode);
+ Ok(slf)
}
- fn format(&mut self, source: &str) -> PyResult<PyDataStreamWriter> {
- Ok(PyDataStreamWriter {
- inner: Some(self.take()?.format(source)),
- })
+ fn format<'py>(mut slf: PyRefMut<'py, Self>, source: &str) ->
PyResult<PyRefMut<'py, Self>> {
+ slf.inner = slf.inner.clone().format(source);
+ Ok(slf)
}
- fn option(&mut self, key: &str, value: &Bound<'_, PyAny>) ->
PyResult<PyDataStreamWriter> {
+ fn option<'py>(
+ mut slf: PyRefMut<'py, Self>,
+ key: &str,
+ value: &Bound<'_, PyAny>,
+ ) -> PyResult<PyRefMut<'py, Self>> {
// None -> option left unset; bools -> "true"/"false" (reference
`to_str`).
match crate::coerce_option_value(value)? {
- Some(v) => Ok(PyDataStreamWriter {
- inner: Some(self.take()?.option(key, &v)),
- }),
- None => Ok(PyDataStreamWriter {
- inner: Some(self.take()?),
- }),
+ Some(v) => {
+ slf.inner = slf.inner.clone().option(key, &v);
+ Ok(slf)
+ }
+ None => Ok(slf),
}
}
// Mirrors reference `DataStreamWriter.options(**options)`: keyword args;
None values
// skipped, booleans lowercased.
#[pyo3(signature = (**options))]
- fn options(&mut self, options: Option<&Bound<'_, PyDict>>) ->
PyResult<PyDataStreamWriter> {
+ fn options<'py>(
+ mut slf: PyRefMut<'py, Self>,
+ options: Option<&Bound<'_, PyDict>>,
+ ) -> PyResult<PyRefMut<'py, Self>> {
let mut opts = HashMap::new();
if let Some(options) = options {
for (k, v) in options.iter() {
@@ -573,44 +604,49 @@ impl PyDataStreamWriter {
}
}
}
- Ok(PyDataStreamWriter {
- inner: Some(self.take()?.options(opts)),
- })
+ slf.inner = slf.inner.clone().options(opts);
+ Ok(slf)
}
#[pyo3(signature = (*cols))]
- fn partitionBy(&mut self, cols: Vec<String>) ->
PyResult<PyDataStreamWriter> {
+ fn partitionBy<'py>(
+ mut slf: PyRefMut<'py, Self>,
+ cols: Vec<String>,
+ ) -> PyResult<PyRefMut<'py, Self>> {
let col_refs: Vec<&str> = cols.iter().map(|s| s.as_str()).collect();
- Ok(PyDataStreamWriter {
- inner: Some(self.take()?.partition_by(col_refs)),
- })
+ slf.inner = slf.inner.clone().partition_by(col_refs);
+ Ok(slf)
}
#[pyo3(signature = (*cols))]
- fn clusterBy(&mut self, cols: Vec<String>) -> PyResult<PyDataStreamWriter>
{
+ fn clusterBy<'py>(
+ mut slf: PyRefMut<'py, Self>,
+ cols: Vec<String>,
+ ) -> PyResult<PyRefMut<'py, Self>> {
let col_refs: Vec<&str> = cols.iter().map(|s| s.as_str()).collect();
- Ok(PyDataStreamWriter {
- inner: Some(self.take()?.cluster_by(col_refs)),
- })
+ slf.inner = slf.inner.clone().cluster_by(col_refs);
+ Ok(slf)
}
- fn queryName(&mut self, queryName: &str) -> PyResult<PyDataStreamWriter> {
- Ok(PyDataStreamWriter {
- inner: Some(self.take()?.query_name(queryName)),
- })
+ fn queryName<'py>(
+ mut slf: PyRefMut<'py, Self>,
+ queryName: &str,
+ ) -> PyResult<PyRefMut<'py, Self>> {
+ slf.inner = slf.inner.clone().query_name(queryName);
+ Ok(slf)
}
/// `DataStreamWriter.trigger(...)`: mirrors the reference keyword API —
exactly one
/// of `processingTime` / `once` / `availableNow` / `continuous` is given.
#[pyo3(signature = (*, processingTime=None, once=None, continuous=None,
availableNow=None, realTime=None))]
- fn trigger(
- &mut self,
+ fn trigger<'py>(
+ mut slf: PyRefMut<'py, Self>,
processingTime: Option<&str>,
once: Option<bool>,
continuous: Option<&str>,
availableNow: Option<bool>,
realTime: Option<&str>,
- ) -> PyResult<PyDataStreamWriter> {
+ ) -> PyResult<PyRefMut<'py, Self>> {
let trigger = if let Some(interval) = processingTime {
Trigger::ProcessingTime(interval.to_string())
} else if once == Some(true) {
@@ -630,9 +666,8 @@ impl PyDataStreamWriter {
"trigger() requires exactly one of processingTime, once,
continuous, availableNow, realTime",
));
};
- Ok(PyDataStreamWriter {
- inner: Some(self.take()?.trigger(trigger)),
- })
+ slf.inner = slf.inner.clone().trigger(trigger);
+ Ok(slf)
}
#[pyo3(signature = (path=None, format=None, outputMode=None,
partitionBy=None, queryName=None, **options))]
@@ -672,11 +707,11 @@ impl PyDataStreamWriter {
/// function (via the bundled `pyspark.cloudpickle`) and attach it as the
/// foreach-batch sink. Pickling happens here so the Python skin can
re-export this
/// class directly rather than subclassing a (non-subclassable) PyO3 type.
- fn foreachBatch(
- &mut self,
+ fn foreachBatch<'py>(
+ mut slf: PyRefMut<'py, Self>,
py: Python<'_>,
func: &Bound<'_, PyAny>,
- ) -> PyResult<PyDataStreamWriter> {
+ ) -> PyResult<PyRefMut<'py, Self>> {
let command = crate::dataframe::py_cloudpickle(py, func)?;
let payload = PythonUDFPayload::new(
spark_connect::types::DataType::Struct { fields: vec![] },
@@ -684,16 +719,19 @@ impl PyDataStreamWriter {
command,
crate::dataframe::py_version(py),
);
- Ok(PyDataStreamWriter {
- inner: Some(self.take()?.foreach_batch(payload)),
- })
+ slf.inner = slf.inner.clone().foreach_batch(payload);
+ Ok(slf)
}
/// `DataStreamWriter.foreach(f)`: wrap the row handler as the reference
client does
/// — `(f, None, serializer, serializer)` with
`AutoBatchedSerializer(CPickleSerializer())`
/// — and cloudpickle it so the worker deserializes it against its own
/// `pyspark.serializers`. Built here to avoid a Python subclass of the
PyO3 class.
- fn foreach(&mut self, py: Python<'_>, f: &Bound<'_, PyAny>) ->
PyResult<PyDataStreamWriter> {
+ fn foreach<'py>(
+ mut slf: PyRefMut<'py, Self>,
+ py: Python<'_>,
+ f: &Bound<'_, PyAny>,
+ ) -> PyResult<PyRefMut<'py, Self>> {
let serializers = py.import("pyspark.serializers")?;
let cpickle = serializers.getattr("CPickleSerializer")?.call0()?;
let serializer = serializers
@@ -717,9 +755,8 @@ impl PyDataStreamWriter {
command,
crate::dataframe::py_version(py),
);
- Ok(PyDataStreamWriter {
- inner: Some(self.take()?.foreach(payload)),
- })
+ slf.inner = slf.inner.clone().foreach(payload);
+ Ok(slf)
}
}
diff --git a/crates/spark-connect/src/merge.rs
b/crates/spark-connect/src/merge.rs
index 1c723db..fc57ad5 100644
--- a/crates/spark-connect/src/merge.rs
+++ b/crates/spark-connect/src/merge.rs
@@ -38,6 +38,7 @@ fn merge_action(
}
/// Fluent builder for a MERGE INTO command. Create it via
[`crate::dataframe::DataFrame::merge_into`].
+#[derive(Clone)]
pub struct MergeIntoWriter {
session: SparkSession,
source_plan: Arc<LogicalPlan>,
diff --git a/crates/spark-connect/src/readwriter.rs
b/crates/spark-connect/src/readwriter.rs
index 54fafeb..f103bb1 100644
--- a/crates/spark-connect/src/readwriter.rs
+++ b/crates/spark-connect/src/readwriter.rs
@@ -16,6 +16,7 @@ use crate::session::SparkSession;
/// DataFrameReader for reading data from various sources.
///
/// Mirrors `pyspark.sql.connect.readwriter.DataFrameReader`.
+#[derive(Clone)]
pub struct DataFrameReader {
session: SparkSession,
format: Option<String>,
@@ -58,6 +59,21 @@ impl DataFrameReader {
self
}
+ /// The configured source format, if any.
+ pub fn get_format(&self) -> Option<&str> {
+ self.format.as_deref()
+ }
+
+ /// The configured schema (a DDL/JSON string; empty when unset).
+ pub fn get_schema(&self) -> &str {
+ &self.schema
+ }
+
+ /// The configured read options.
+ pub fn get_options(&self) -> &HashMap<String, String> {
+ &self.options
+ }
+
/// Load data from the specified path(s) with the configured format,
schema, and options.
pub fn load(self, path: Option<&str>) -> DataFrame {
let paths = path.map(|p| vec![p.to_string()]);
@@ -330,6 +346,7 @@ impl TableSaveMethod {
/// DataFrameWriter for writing DataFrames to various destinations.
///
/// Mirrors `pyspark.sql.connect.readwriter.DataFrameWriter`.
+#[derive(Clone)]
pub struct DataFrameWriter {
session: SparkSession,
input_plan: Arc<LogicalPlan>,
@@ -515,6 +532,7 @@ impl DataFrameWriter {
/// DataFrameWriterV2 for the v2 write API (`DataFrame.writeTo`).
///
/// Mirrors `pyspark.sql.connect.readwriter.DataFrameWriterV2`.
+#[derive(Clone)]
pub struct DataFrameWriterV2 {
session: SparkSession,
input_plan: Arc<LogicalPlan>,
diff --git a/crates/spark-connect/src/streaming.rs
b/crates/spark-connect/src/streaming.rs
index 600ae28..9a03323 100644
--- a/crates/spark-connect/src/streaming.rs
+++ b/crates/spark-connect/src/streaming.rs
@@ -21,6 +21,7 @@ use crate::udf::PythonUDFPayload;
/// DataStreamReader for reading streaming data from various sources.
///
/// Mirrors `pyspark.sql.connect.streaming.DataStreamReader`.
+#[derive(Clone)]
pub struct DataStreamReader {
session: SparkSession,
format: Option<String>,
@@ -71,6 +72,21 @@ impl DataStreamReader {
self
}
+ /// The configured source format, if any.
+ pub fn get_format(&self) -> Option<&str> {
+ self.format.as_deref()
+ }
+
+ /// The configured schema (a DDL/JSON string; empty when unset).
+ pub fn get_schema(&self) -> &str {
+ &self.schema
+ }
+
+ /// The configured read options.
+ pub fn get_options(&self) -> &HashMap<String, String> {
+ &self.options
+ }
+
/// Load streaming data from the specified path(s).
pub fn load(self, path: Option<&str>) -> DataFrame {
let paths = path.map(|p| vec![p.to_string()]);
@@ -242,6 +258,7 @@ pub enum Trigger {
/// DataStreamWriter for writing streaming data to various sinks.
///
/// Mirrors `pyspark.sql.connect.streaming.DataStreamWriter`.
+#[derive(Clone)]
pub struct DataStreamWriter {
session: SparkSession,
plan: Arc<LogicalPlan>,
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 0fb6533..01c2c44 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -4418,3 +4418,194 @@ def
test_connect_registry_sends_auto_cdc_define_flow(mock_connect):
assert 4 not in plain
assert 10 not in details and 11 not in details and 12 not in details
+
+# ------------------------------------------------- reader/writer in-place
setters
+
[email protected]
+def offline_spark():
+ # The gRPC channel connects lazily, so building readers/writers and lazy
+ # DataFrames needs no server.
+ from pyspark.sql import SparkSession
+
+ return SparkSession.builder.remote("sc://localhost:1").getOrCreate()
+
+
+def test_reader_setters_mutate_and_return_self(offline_spark):
+ # PySpark: the setters change the reader and return it; the return value
may be
+ # ignored (the pattern pandas-on-Spark read_sql_table uses).
+ r = offline_spark.read
+ assert r.option("a", "b") is r
+ assert r.options(c="d", skipped=None, flag=True) is r
+ assert r.schema("x INT") is r
+ assert r.format("jdbc") is r
+ assert r._format == "jdbc"
+ assert r._schema == "x INT"
+ assert r._options == {"a": "b", "c": "d", "flag": "true"}
+ assert type(r.load()).__name__ == "DataFrame"
+
+
+def test_reader_read_sql_table_pattern(offline_spark):
+ # Mirrors pyspark/pandas/namespace.py read_sql_table, which discards every
+ # setter's return value before loading.
+ reader = offline_spark.read
+ reader.option("dbtable", "t")
+ reader.option("url", "jdbc:postgresql:db")
+ reader.schema("id INT")
+ reader.options(fetchsize=10)
+ sdf = reader.format("jdbc").load()
+ assert type(sdf).__name__ == "DataFrame"
+ assert reader._options == {
+ "dbtable": "t", "url": "jdbc:postgresql:db", "fetchsize": "10",
+ }
+
+
+def test_pandas_read_sql_table_reader_stage(offline_spark, monkeypatch):
+ # Drive the vendored read_sql_table itself up to the point it hands the
loaded
+ # frame on. default_session() asks the server for a config, so point it at
the
+ # offline session, and stop at _get_index_map (the first step after the
read).
+ # pandas-on-Spark does not import under pandas >= 3.0 (pyspark.pandas.frame
+ # needs pandas.tseries.frequencies.DateOffset), so skip on any ImportError.
+ ns = pytest.importorskip("pyspark.pandas.namespace", exc_type=ImportError)
+
+ class _Loaded(Exception):
+ pass
+
+ seen = {}
+
+ def fake_index_map(sdf, index_col):
+ seen["sdf"] = sdf
+ raise _Loaded
+
+ monkeypatch.setattr(ns, "default_session", lambda: offline_spark)
+ monkeypatch.setattr(ns, "_get_index_map", fake_index_map)
+ with pytest.raises(_Loaded):
+ ns.read_sql_table(
+ "t", "jdbc:postgresql:db", schema="id INT", options={"fetchsize":
"1"}
+ )
+ assert type(seen["sdf"]).__name__ == "DataFrame"
+
+
+def test_reader_reusable_after_load(offline_spark):
+ r = offline_spark.read.format("parquet")
+ r.load("/a")
+ r.load("/b")
+ r.table("t")
+ r.csv("/c", header=True)
+ # Like PySpark, a format method keeps its options and format on the reader.
+ assert r._format == "csv"
+ assert r._options == {"header": "true"}
+ r.json("/d")
+ r.parquet("/e", mergeSchema=False)
+ r.jdbc("jdbc:x", "tbl", properties={"user": "u"})
+ assert r._format == "jdbc"
+ assert r._options["dbtable"] == "tbl" and r._options["user"] == "u"
+ assert r.option("k", "v") is r
+ assert type(r.load()).__name__ == "DataFrame"
+
+
+def test_reader_reusable_after_failed_read(offline_spark):
+ r = offline_spark.read.option("a", "b")
+ with pytest.raises(Exception):
+ r.csv("/p", schema=object())
+ with pytest.raises(Exception):
+ r.json("/p", schema=12)
+ # A failed call leaves the reader as it was.
+ assert r._options == {"a": "b"}
+ assert r._format is None
+ assert type(r.format("csv").load("/p")).__name__ == "DataFrame"
+
+
+def test_stream_reader_setters_mutate_and_return_self(offline_spark):
+ r = offline_spark.readStream
+ assert r.format("rate") is r
+ assert r.option("rowsPerSecond", 5) is r
+ assert r.options(numPartitions=2, skipped=None) is r
+ assert r.schema("v LONG") is r
+ assert r.name("src_1") is r
+ with pytest.raises(ValueError):
+ r.name("bad name")
+ assert r._format == "rate"
+ assert r._schema == "v LONG"
+ assert r._options == {"rowsPerSecond": "5", "numPartitions": "2"}
+ r.load()
+ r.json("/j")
+ assert r._format == "json"
+ r.xml("/x", rowTag="row")
+ assert r._format == "xml" and r._options["rowTag"] == "row"
+ r.table("t")
+ assert type(r.load()).__name__ == "DataFrame"
+
+
+def test_writer_setters_mutate_and_return_self(offline_spark):
+ w = offline_spark.range(3).write
+ assert w.mode("overwrite") is w
+ assert w.format("parquet") is w
+ assert w.option("a", "b") is w
+ assert w.option("skipped", None) is w
+ assert w.options(c="d") is w
+ assert w.partitionBy("id") is w
+ assert w.bucketBy(2, "id") is w
+ assert w.sortBy("id") is w
+ assert w.clusterBy("id") is w
+ # A failed format call (bad partitionBy) leaves the writer usable.
+ with pytest.raises(Exception):
+ w.parquet("/p", partitionBy=123)
+ assert w.mode("append") is w
+
+
+def test_writer_v2_setters_mutate_and_return_self(offline_spark):
+ from pyspark.sql import functions as F
+
+ w = offline_spark.range(3).writeTo("t")
+ assert w.using("parquet") is w
+ assert w.option("a", "b") is w
+ assert w.options(c="d") is w
+ assert w.tableProperty("p", "v") is w
+ assert w.partitionedBy(F.col("id")) is w
+ assert w.clusterBy("id") is w
+
+
+def test_stream_writer_setters_mutate_and_return_self(offline_spark):
+ w = offline_spark.readStream.format("rate").load().writeStream
+ assert w.outputMode("append") is w
+ assert w.format("memory") is w
+ assert w.option("a", "b") is w
+ assert w.options(c="d") is w
+ assert w.partitionBy("value") is w
+ assert w.clusterBy("value") is w
+ assert w.queryName("q") is w
+ with pytest.raises(ValueError):
+ w.trigger()
+ assert w.trigger(processingTime="1 second") is w
+ assert w.foreachBatch(lambda df, i: None) is w
+ assert w.foreach(lambda row: None) is w
+
+
+def test_merge_into_writer_mutates_and_returns_self(offline_spark):
+ # PySpark: withSchemaEvolution and every clause action change the writer
and
+ # return that same writer; a clause object can be used more than once.
+ from pyspark.sql import functions as F
+
+ src = offline_spark.range(3)
+ m = src.mergeInto("target", F.col("id") == F.col("target.id"))
+ assert m.withSchemaEvolution() is m
+
+ matched = m.whenMatched(F.col("id") > 1)
+ assert matched.updateAll() is m
+ assert matched.update({"id": F.col("id")}) is m
+ assert matched.delete() is m
+ assert m.whenMatched().delete() is m
+
+ not_matched = m.whenNotMatched()
+ assert not_matched.insertAll() is m
+ assert not_matched.insert({"id": F.col("id")}) is m
+
+ by_source = m.whenNotMatchedBySource(F.col("target.id") < 0)
+ assert by_source.updateAll() is m
+ assert by_source.update({"id": F.lit(0)}) is m
+ assert by_source.delete() is m
+
+ # The ignore-the-return-value form keeps working on the original object.
+ m.whenNotMatched().insertAll()
+ m.withSchemaEvolution()
+ assert m.whenMatched().updateAll() is m
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]