This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new ba535db  [SPARK-60920][CONNECT][PYTHON] Make reader/writer setters 
mutate in place like PySpark in the Rust drop-in
ba535db is described below

commit ba535dbbcb8768d266d053a4372b6288c1ba2846
Author: Liang-Chi Hsieh <[email protected]>
AuthorDate: Thu Nov 5 18:58:09 2026 +0900

    [SPARK-60920][CONNECT][PYTHON] Make reader/writer setters mutate in place 
like PySpark in the Rust drop-in
    
    ### What changes were proposed in this pull request?
    
    JIRA: [SPARK-60920](https://issues.apache.org/jira/browse/SPARK-60920)
    
    The drop-in's `DataFrameReader`, `DataStreamReader`, `DataFrameWriter`, 
`DataFrameWriterV2`, `DataStreamWriter` and `MergeIntoWriter` 
(`crates/pyspark-rs/src/{readwriter,streaming,dataframe}.rs`) now change 
themselves in place, as PySpark's do.
    
    - Each wrapper holds its core builder directly, not in an `Option` that 
every setter emptied. A setter replaces the builder in place and returns the 
same Python object (`PyRefMut<Self>`).
    - Terminal methods (`load`, `table`, `save`, `saveAsTable`, `insertInto`, 
`start`, `toTable`, `create`, `merge`, ...) build from a clone, so the object 
can be used again afterwards.
    - As in PySpark, methods that take inline settings keep them on the object: 
the format methods (`json`, `csv`, `parquet`, ..., `jdbc`), and the extra 
arguments of `load`, `save`, `saveAsTable`, `insertInto`, `start` and 
`toTable`. A call that fails argument validation leaves the object unchanged.
    - `DataFrameWriter.jdbc` now applies `url` and `dbtable` after 
`properties`, so they win over same-named properties, as in PySpark.
    - A merge clause (`whenMatched()`, `whenNotMatched()`, 
`whenNotMatchedBySource()`) holds its parent writer, as in PySpark Connect. 
Each action appends to that writer and returns it, and a clause can be used 
more than once. `withSchemaEvolution()` returns `self`.
    - The core builders get `#[derive(Clone)]`; their consuming Rust API does 
not change. The two core readers also get read-only `get_format` / `get_schema` 
/ `get_options`. On the Python side these back PySpark Connect's private 
`_format` / `_schema` / `_options` attributes.
    
    ### Why are the changes needed?
    
    Each setter took the inner builder and returned a new wrapper, which left 
the original Python object empty. So the usual PySpark pattern
    
    ```python
    r = spark.read
    r.option("a", "b")
    r.format("jdbc").load()
    ```
    
    failed with `RuntimeError: DataFrameReader already consumed`. The vendored 
pandas-on-Spark `read_sql_table` uses this pattern: it calls 
`reader.option(...)`, `reader.schema(...)` and `reader.options(...)` and 
ignores the return values. The same error came up when reusing a reader or 
writer after a read or write, which PySpark allows, and after a format method 
failed argument validation, because the builder was taken before the arguments 
were checked. A merge clause object could also b [...]
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Reader, writer and merge-writer setters now return the same object and 
change it in place, and these objects can be reused after a read, write or 
merge, matching PySpark. Code that only chained calls behaves as before.
    
    ### How was this patch tested?
    
    New offline tests in `python/tests/test_dropin_offline.py`:
    - calling setters and ignoring their return values, then loading (the 
`read_sql_table` call sequence)
    - the vendored `read_sql_table` itself, run up to the read, with 
`default_session` pointed at an offline session
    - reuse of a reader after successful reads, and after reads that fail 
argument validation, checking the format, schema and options left on it
    - the in-place setters on `DataStreamReader`, `DataFrameWriter`, 
`DataFrameWriterV2` and `DataStreamWriter`
    - merge clause actions returning the same writer, and a clause used more 
than once
    
    This touches the same code in `readwriter.rs` / `streaming.rs` as #151 
(reader schema handling). Whichever merges second will need a rebase.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Anthropic)
    
    Closes #158 from viirya/SPARK-60920.
    
    Authored-by: Liang-Chi Hsieh <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/dataframe.rs     | 419 +++++++++++++++++----------------
 crates/pyspark-rs/src/readwriter.rs    | 128 ++++++----
 crates/pyspark-rs/src/streaming.rs     | 243 +++++++++++--------
 crates/spark-connect/src/merge.rs      |   1 +
 crates/spark-connect/src/readwriter.rs |  18 ++
 crates/spark-connect/src/streaming.rs  |  17 ++
 python/tests/test_dropin_offline.py    | 191 +++++++++++++++
 7 files changed, 667 insertions(+), 350 deletions(-)

diff --git a/crates/pyspark-rs/src/dataframe.rs 
b/crates/pyspark-rs/src/dataframe.rs
index 1dfe89e..8df1b83 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -1436,7 +1436,7 @@ impl PyDataFrame {
     #[getter]
     fn write(&self) -> PyDataFrameWriter {
         PyDataFrameWriter {
-            inner: Some(self.dataframe.write()),
+            inner: self.dataframe.write(),
         }
     }
 
@@ -1444,7 +1444,7 @@ impl PyDataFrame {
     #[pyo3(name = "writeTo")]
     fn write_to(&self, table: &str) -> PyDataFrameWriterV2 {
         PyDataFrameWriterV2 {
-            inner: Some(self.dataframe.write_to(table)),
+            inner: self.dataframe.write_to(table),
         }
     }
 
@@ -1469,7 +1469,7 @@ impl PyDataFrame {
     #[pyo3(name = "mergeInto")]
     fn merge_into(&self, table: &str, condition: &PyColumn) -> 
PyMergeIntoWriter {
         PyMergeIntoWriter {
-            inner: Some(self.dataframe.merge_into(table, 
condition.column.clone())),
+            inner: self.dataframe.merge_into(table, condition.column.clone()),
         }
     }
 
@@ -1702,17 +1702,28 @@ fn build_replacement_pairs(
 }
 
 /// Python wrapper for `MergeIntoWriter` (`df.mergeInto(...)`).
+///
+/// Mirrors PySpark's mutable writer: `withSchemaEvolution` and every clause 
action
+/// (`whenMatched().updateAll()`, ...) change this object and return it, and 
`merge`
+/// leaves it usable. The core writer is a consuming builder, so the wrapped 
value is
+/// replaced in place and `merge` runs on a clone.
 #[pyclass(name = "MergeIntoWriter", module = "pyspark.sql.merge")]
 pub struct PyMergeIntoWriter {
-    inner: Option<spark_connect::MergeIntoWriter>,
+    inner: spark_connect::MergeIntoWriter,
 }
 
-impl PyMergeIntoWriter {
-    fn take(&mut self) -> PyResult<spark_connect::MergeIntoWriter> {
-        self.inner.take().ok_or_else(|| {
-            pyo3::exceptions::PyRuntimeError::new_err("MergeIntoWriter already 
consumed")
-        })
-    }
+/// Apply a clause action to the parent writer in place and return that same 
writer
+/// (PySpark's clause methods append to `self.writer` and return it).
+fn merge_clause_action(
+    py: Python<'_>,
+    writer: &Py<PyMergeIntoWriter>,
+    action: impl FnOnce(spark_connect::MergeIntoWriter) -> 
spark_connect::MergeIntoWriter,
+) -> PyResult<Py<PyMergeIntoWriter>> {
+    {
+        let mut w = writer.bind(py).try_borrow_mut()?;
+        w.inner = action(w.inner.clone());
+    }
+    Ok(writer.clone_ref(py))
 }
 
 #[pymethods]
@@ -1737,41 +1748,40 @@ impl PyMergeIntoWriter {
     }
 
     #[pyo3(name = "whenMatched", signature = (condition=None))]
-    fn when_matched(&mut self, condition: Option<&PyColumn>) -> 
PyResult<PyWhenMatched> {
-        let cond = condition.map(|c| c.column.clone());
-        Ok(PyWhenMatched {
-            inner: Some(self.take()?.when_matched(cond)),
-        })
+    fn when_matched(slf: &Bound<'_, Self>, condition: Option<&PyColumn>) -> 
PyWhenMatched {
+        PyWhenMatched {
+            writer: slf.clone().unbind(),
+            condition: condition.map(|c| c.column.clone()),
+        }
     }
 
     #[pyo3(name = "whenNotMatched", signature = (condition=None))]
-    fn when_not_matched(&mut self, condition: Option<&PyColumn>) -> 
PyResult<PyWhenNotMatched> {
-        let cond = condition.map(|c| c.column.clone());
-        Ok(PyWhenNotMatched {
-            inner: Some(self.take()?.when_not_matched(cond)),
-        })
+    fn when_not_matched(slf: &Bound<'_, Self>, condition: Option<&PyColumn>) 
-> PyWhenNotMatched {
+        PyWhenNotMatched {
+            writer: slf.clone().unbind(),
+            condition: condition.map(|c| c.column.clone()),
+        }
     }
 
     #[pyo3(name = "whenNotMatchedBySource", signature = (condition=None))]
     fn when_not_matched_by_source(
-        &mut self,
+        slf: &Bound<'_, Self>,
         condition: Option<&PyColumn>,
-    ) -> PyResult<PyWhenNotMatchedBySource> {
-        let cond = condition.map(|c| c.column.clone());
-        Ok(PyWhenNotMatchedBySource {
-            inner: Some(self.take()?.when_not_matched_by_source(cond)),
-        })
+    ) -> PyWhenNotMatchedBySource {
+        PyWhenNotMatchedBySource {
+            writer: slf.clone().unbind(),
+            condition: condition.map(|c| c.column.clone()),
+        }
     }
 
     #[pyo3(name = "withSchemaEvolution")]
-    fn with_schema_evolution(&mut self) -> PyResult<PyMergeIntoWriter> {
-        Ok(PyMergeIntoWriter {
-            inner: Some(self.take()?.with_schema_evolution()),
-        })
+    fn with_schema_evolution(mut slf: PyRefMut<'_, Self>) -> PyRefMut<'_, 
Self> {
+        slf.inner = slf.inner.clone().with_schema_evolution();
+        slf
     }
 
-    fn merge(&mut self) -> PyResult<()> {
-        self.take()?.merge().to_pyerr()
+    fn merge(&self) -> PyResult<()> {
+        self.inner.clone().merge().to_pyerr()
     }
 }
 
@@ -1785,124 +1795,99 @@ fn assignments_to_map(
         .collect()
 }
 
-/// `whenMatched(...)` clause builder.
+/// `whenMatched(...)` clause builder. Holds its parent writer, like PySpark's
+/// `MergeIntoWriter.WhenMatched`; each action may be called more than once.
 #[pyclass(name = "WhenMatched", module = "pyspark.sql.merge")]
 pub struct PyWhenMatched {
-    inner: Option<spark_connect::merge::WhenMatched>,
+    writer: Py<PyMergeIntoWriter>,
+    condition: Option<spark_connect::column::Column>,
 }
 
 #[pymethods]
 impl PyWhenMatched {
     #[pyo3(name = "updateAll")]
-    fn update_all(&mut self) -> PyResult<PyMergeIntoWriter> {
-        let w = self
-            .inner
-            .take()
-            .ok_or_else(|| pyo3::exceptions::PyRuntimeError::new_err("clause 
already consumed"))?;
-        Ok(PyMergeIntoWriter {
-            inner: Some(w.update_all()),
-        })
+    fn update_all(&self, py: Python<'_>) -> PyResult<Py<PyMergeIntoWriter>> {
+        let cond = self.condition.clone();
+        merge_clause_action(py, &self.writer, |w| 
w.when_matched(cond).update_all())
     }
 
     fn update(
-        &mut self,
+        &self,
         py: Python<'_>,
         assignments: std::collections::HashMap<String, Py<PyColumn>>,
-    ) -> PyResult<PyMergeIntoWriter> {
-        let w = self
-            .inner
-            .take()
-            .ok_or_else(|| pyo3::exceptions::PyRuntimeError::new_err("clause 
already consumed"))?;
-        Ok(PyMergeIntoWriter {
-            inner: Some(w.update(assignments_to_map(py, assignments))),
-        })
+    ) -> PyResult<Py<PyMergeIntoWriter>> {
+        let cond = self.condition.clone();
+        let map = assignments_to_map(py, assignments);
+        merge_clause_action(py, &self.writer, |w| 
w.when_matched(cond).update(map))
     }
 
-    fn delete(&mut self) -> PyResult<PyMergeIntoWriter> {
-        let w = self
-            .inner
-            .take()
-            .ok_or_else(|| pyo3::exceptions::PyRuntimeError::new_err("clause 
already consumed"))?;
-        Ok(PyMergeIntoWriter {
-            inner: Some(w.delete()),
-        })
+    fn delete(&self, py: Python<'_>) -> PyResult<Py<PyMergeIntoWriter>> {
+        let cond = self.condition.clone();
+        merge_clause_action(py, &self.writer, |w| 
w.when_matched(cond).delete())
     }
 }
 
-/// `whenNotMatched(...)` clause builder.
+/// `whenNotMatched(...)` clause builder. Holds its parent writer, like 
PySpark's
+/// `MergeIntoWriter.WhenNotMatched`.
 #[pyclass(name = "WhenNotMatched", module = "pyspark.sql.merge")]
 pub struct PyWhenNotMatched {
-    inner: Option<spark_connect::merge::WhenNotMatched>,
+    writer: Py<PyMergeIntoWriter>,
+    condition: Option<spark_connect::column::Column>,
 }
 
 #[pymethods]
 impl PyWhenNotMatched {
     #[pyo3(name = "insertAll")]
-    fn insert_all(&mut self) -> PyResult<PyMergeIntoWriter> {
-        let w = self
-            .inner
-            .take()
-            .ok_or_else(|| pyo3::exceptions::PyRuntimeError::new_err("clause 
already consumed"))?;
-        Ok(PyMergeIntoWriter {
-            inner: Some(w.insert_all()),
-        })
+    fn insert_all(&self, py: Python<'_>) -> PyResult<Py<PyMergeIntoWriter>> {
+        let cond = self.condition.clone();
+        merge_clause_action(py, &self.writer, |w| 
w.when_not_matched(cond).insert_all())
     }
 
     fn insert(
-        &mut self,
+        &self,
         py: Python<'_>,
         assignments: std::collections::HashMap<String, Py<PyColumn>>,
-    ) -> PyResult<PyMergeIntoWriter> {
-        let w = self
-            .inner
-            .take()
-            .ok_or_else(|| pyo3::exceptions::PyRuntimeError::new_err("clause 
already consumed"))?;
-        Ok(PyMergeIntoWriter {
-            inner: Some(w.insert(assignments_to_map(py, assignments))),
-        })
+    ) -> PyResult<Py<PyMergeIntoWriter>> {
+        let cond = self.condition.clone();
+        let map = assignments_to_map(py, assignments);
+        merge_clause_action(py, &self.writer, |w| 
w.when_not_matched(cond).insert(map))
     }
 }
 
-/// `whenNotMatchedBySource(...)` clause builder.
+/// `whenNotMatchedBySource(...)` clause builder. Holds its parent writer, like
+/// PySpark's `MergeIntoWriter.WhenNotMatchedBySource`.
 #[pyclass(name = "WhenNotMatchedBySource", module = "pyspark.sql.merge")]
 pub struct PyWhenNotMatchedBySource {
-    inner: Option<spark_connect::merge::WhenNotMatchedBySource>,
+    writer: Py<PyMergeIntoWriter>,
+    condition: Option<spark_connect::column::Column>,
 }
 
 #[pymethods]
 impl PyWhenNotMatchedBySource {
     #[pyo3(name = "updateAll")]
-    fn update_all(&mut self) -> PyResult<PyMergeIntoWriter> {
-        let w = self
-            .inner
-            .take()
-            .ok_or_else(|| pyo3::exceptions::PyRuntimeError::new_err("clause 
already consumed"))?;
-        Ok(PyMergeIntoWriter {
-            inner: Some(w.update_all()),
+    fn update_all(&self, py: Python<'_>) -> PyResult<Py<PyMergeIntoWriter>> {
+        let cond = self.condition.clone();
+        merge_clause_action(py, &self.writer, |w| {
+            w.when_not_matched_by_source(cond).update_all()
         })
     }
 
     fn update(
-        &mut self,
+        &self,
         py: Python<'_>,
         assignments: std::collections::HashMap<String, Py<PyColumn>>,
-    ) -> PyResult<PyMergeIntoWriter> {
-        let w = self
-            .inner
-            .take()
-            .ok_or_else(|| pyo3::exceptions::PyRuntimeError::new_err("clause 
already consumed"))?;
-        Ok(PyMergeIntoWriter {
-            inner: Some(w.update(assignments_to_map(py, assignments))),
+    ) -> PyResult<Py<PyMergeIntoWriter>> {
+        let cond = self.condition.clone();
+        let map = assignments_to_map(py, assignments);
+        merge_clause_action(py, &self.writer, |w| {
+            w.when_not_matched_by_source(cond).update(map)
         })
     }
 
-    fn delete(&mut self) -> PyResult<PyMergeIntoWriter> {
-        let w = self
-            .inner
-            .take()
-            .ok_or_else(|| pyo3::exceptions::PyRuntimeError::new_err("clause 
already consumed"))?;
-        Ok(PyMergeIntoWriter {
-            inner: Some(w.delete()),
+    fn delete(&self, py: Python<'_>) -> PyResult<Py<PyMergeIntoWriter>> {
+        let cond = self.condition.clone();
+        merge_clause_action(py, &self.writer, |w| {
+            w.when_not_matched_by_source(cond).delete()
         })
     }
 }
@@ -1910,16 +1895,10 @@ impl PyWhenNotMatchedBySource {
 /// Python wrapper for the V1 `DataFrameWriter` (`df.write`).
 #[pyclass(name = "DataFrameWriter", module = "pyspark.sql.readwriter")]
 pub struct PyDataFrameWriter {
-    inner: Option<spark_connect::readwriter::DataFrameWriter>,
+    inner: spark_connect::readwriter::DataFrameWriter,
 }
 
-impl PyDataFrameWriter {
-    fn take(&mut self) -> PyResult<spark_connect::readwriter::DataFrameWriter> 
{
-        self.inner.take().ok_or_else(|| {
-            PyErr::new::<pyo3::exceptions::PyRuntimeError, _>("DataFrameWriter 
already consumed")
-        })
-    }
-}
+impl PyDataFrameWriter {}
 
 /// Apply the shared `format` / `mode` / `partitionBy` / `**options` arguments 
that
 /// reference `DataFrameWriter.save` and `.saveAsTable` accept inline, 
mirroring
@@ -1959,34 +1938,38 @@ fn apply_write_common(
 
 #[pymethods]
 impl PyDataFrameWriter {
-    fn mode(&mut self, saveMode: &str) -> PyResult<PyDataFrameWriter> {
-        Ok(PyDataFrameWriter {
-            inner: Some(self.take()?.mode(saveMode)),
-        })
+    fn mode<'py>(mut slf: PyRefMut<'py, Self>, saveMode: &str) -> 
PyResult<PyRefMut<'py, Self>> {
+        slf.inner = slf.inner.clone().mode(saveMode);
+        Ok(slf)
     }
 
-    fn format(&mut self, source: &str) -> PyResult<PyDataFrameWriter> {
-        Ok(PyDataFrameWriter {
-            inner: Some(self.take()?.format(source)),
-        })
+    fn format<'py>(mut slf: PyRefMut<'py, Self>, source: &str) -> 
PyResult<PyRefMut<'py, Self>> {
+        slf.inner = slf.inner.clone().format(source);
+        Ok(slf)
     }
 
-    fn option(&mut self, key: &str, value: &Bound<'_, PyAny>) -> 
PyResult<PyDataFrameWriter> {
+    fn option<'py>(
+        mut slf: PyRefMut<'py, Self>,
+        key: &str,
+        value: &Bound<'_, PyAny>,
+    ) -> PyResult<PyRefMut<'py, Self>> {
         // None -> option left unset; bools -> "true"/"false" (reference 
`to_str`).
         match crate::coerce_option_value(value)? {
-            Some(v) => Ok(PyDataFrameWriter {
-                inner: Some(self.take()?.option(key, &v)),
-            }),
-            None => Ok(PyDataFrameWriter {
-                inner: Some(self.take()?),
-            }),
+            Some(v) => {
+                slf.inner = slf.inner.clone().option(key, &v);
+                Ok(slf)
+            }
+            None => Ok(slf),
         }
     }
 
     // Mirrors reference `DataFrameWriter.options(**options)`: keyword args; 
None values
     // skipped, booleans lowercased.
     #[pyo3(signature = (**options))]
-    fn options(&mut self, options: Option<&Bound<'_, PyDict>>) -> 
PyResult<PyDataFrameWriter> {
+    fn options<'py>(
+        mut slf: PyRefMut<'py, Self>,
+        options: Option<&Bound<'_, PyDict>>,
+    ) -> PyResult<PyRefMut<'py, Self>> {
         let mut map = std::collections::HashMap::new();
         if let Some(options) = options {
             for (k, v) in options.iter() {
@@ -1995,46 +1978,51 @@ impl PyDataFrameWriter {
                 }
             }
         }
-        Ok(PyDataFrameWriter {
-            inner: Some(self.take()?.options(map)),
-        })
+        slf.inner = slf.inner.clone().options(map);
+        Ok(slf)
     }
 
     #[pyo3(name = "partitionBy", signature = (*cols))]
-    fn partition_by(&mut self, cols: Vec<Bound<'_, PyAny>>) -> 
PyResult<PyDataFrameWriter> {
-        Ok(PyDataFrameWriter {
-            inner: Some(self.take()?.partition_by(flatten_str_cols(cols)?)),
-        })
+    fn partition_by<'py>(
+        mut slf: PyRefMut<'py, Self>,
+        cols: Vec<Bound<'_, PyAny>>,
+    ) -> PyResult<PyRefMut<'py, Self>> {
+        slf.inner = slf.inner.clone().partition_by(flatten_str_cols(cols)?);
+        Ok(slf)
     }
 
     #[pyo3(name = "bucketBy", signature = (numBuckets, col, *cols))]
-    fn bucket_by(
-        &mut self,
+    fn bucket_by<'py>(
+        mut slf: PyRefMut<'py, Self>,
         numBuckets: i32,
         col: String,
         cols: Vec<String>,
-    ) -> PyResult<PyDataFrameWriter> {
+    ) -> PyResult<PyRefMut<'py, Self>> {
         let mut all = vec![col];
         all.extend(cols);
-        Ok(PyDataFrameWriter {
-            inner: Some(self.take()?.bucket_by(numBuckets, all)),
-        })
+        slf.inner = slf.inner.clone().bucket_by(numBuckets, all);
+        Ok(slf)
     }
 
     #[pyo3(name = "sortBy", signature = (col, *cols))]
-    fn sort_by(&mut self, col: String, cols: Vec<String>) -> 
PyResult<PyDataFrameWriter> {
+    fn sort_by<'py>(
+        mut slf: PyRefMut<'py, Self>,
+        col: String,
+        cols: Vec<String>,
+    ) -> PyResult<PyRefMut<'py, Self>> {
         let mut all = vec![col];
         all.extend(cols);
-        Ok(PyDataFrameWriter {
-            inner: Some(self.take()?.sort_by(all)),
-        })
+        slf.inner = slf.inner.clone().sort_by(all);
+        Ok(slf)
     }
 
     #[pyo3(name = "clusterBy", signature = (*cols))]
-    fn cluster_by(&mut self, cols: Vec<String>) -> PyResult<PyDataFrameWriter> 
{
-        Ok(PyDataFrameWriter {
-            inner: Some(self.take()?.cluster_by(cols)),
-        })
+    fn cluster_by<'py>(
+        mut slf: PyRefMut<'py, Self>,
+        cols: Vec<String>,
+    ) -> PyResult<PyRefMut<'py, Self>> {
+        slf.inner = slf.inner.clone().cluster_by(cols);
+        Ok(slf)
     }
 
     /// Write to an external JDBC table. Connection details go through 
`properties`.
@@ -2046,17 +2034,17 @@ impl PyDataFrameWriter {
         mode: Option<&str>,
         properties: Option<std::collections::HashMap<String, String>>,
     ) -> PyResult<()> {
-        let mut w = self
-            .take()?
-            .format("jdbc")
-            .option("url", url)
-            .option("dbtable", table);
+        // Same order as PySpark: format, mode, properties, then url/dbtable 
(which
+        // win over same-named properties), all kept on the writer, then save.
+        let mut w = self.inner.clone().format("jdbc");
         if let Some(m) = mode {
             w = w.mode(m);
         }
         if let Some(props) = properties {
             w = w.options(props);
         }
+        w = w.option("url", url).option("dbtable", table);
+        self.inner = w.clone();
         w.save(None).to_pyerr()
     }
 
@@ -2070,7 +2058,9 @@ impl PyDataFrameWriter {
         partitionBy: Option<&Bound<'_, PyAny>>,
         options: Option<&Bound<'_, PyDict>>,
     ) -> PyResult<()> {
-        let w = apply_write_common(self.take()?, format, mode, partitionBy, 
options)?;
+        let w = apply_write_common(self.inner.clone(), format, mode, 
partitionBy, options)?;
+        // PySpark sets the inline format/mode/partitionBy/options on the 
writer itself.
+        self.inner = w.clone();
         w.save(path.as_deref()).to_pyerr()
     }
 
@@ -2084,7 +2074,8 @@ impl PyDataFrameWriter {
         partitionBy: Option<&Bound<'_, PyAny>>,
         options: Option<&Bound<'_, PyDict>>,
     ) -> PyResult<()> {
-        let w = apply_write_common(self.take()?, format, mode, partitionBy, 
options)?;
+        let w = apply_write_common(self.inner.clone(), format, mode, 
partitionBy, options)?;
+        self.inner = w.clone();
         w.save_as_table(name).to_pyerr()
     }
 
@@ -2092,10 +2083,12 @@ impl PyDataFrameWriter {
     /// mirroring reference `DataFrameWriter.insertInto(tableName, 
overwrite=None)`.
     #[pyo3(name = "insertInto", signature = (tableName, overwrite=None))]
     fn insert_into(&mut self, tableName: &str, overwrite: Option<bool>) -> 
PyResult<()> {
-        let mut w = self.take()?;
+        let mut w = self.inner.clone();
         if let Some(ow) = overwrite {
             w = w.mode(if ow { "overwrite" } else { "append" });
         }
+        // PySpark sets the overwrite/append mode on the writer itself.
+        self.inner = w.clone();
         w.insert_into(tableName).to_pyerr()
     }
 
@@ -2110,7 +2103,7 @@ impl PyDataFrameWriter {
         partitionBy: Option<&Bound<'_, PyAny>>,
         compression: Option<&Bound<'_, PyAny>>,
     ) -> PyResult<()> {
-        let mut w = self.take()?;
+        let mut w = self.inner.clone();
         if let Some(m) = mode {
             w = w.mode(&m.str()?.to_string());
         }
@@ -2123,6 +2116,8 @@ impl PyDataFrameWriter {
             w = w.partition_by(cols);
         }
         w = wset_opt(w, "compression", compression)?;
+        // PySpark sets the mode/options/format on the writer itself, then 
saves.
+        self.inner = w.clone().format("parquet");
         w.parquet(path).to_pyerr()
     }
 
@@ -2141,7 +2136,7 @@ impl PyDataFrameWriter {
         encoding: Option<&Bound<'_, PyAny>>,
         ignoreNullFields: Option<&Bound<'_, PyAny>>,
     ) -> PyResult<()> {
-        let mut w = self.take()?;
+        let mut w = self.inner.clone();
         if let Some(m) = mode {
             w = w.mode(&m.str()?.to_string());
         }
@@ -2151,6 +2146,8 @@ impl PyDataFrameWriter {
         w = wset_opt(w, "lineSep", lineSep)?;
         w = wset_opt(w, "encoding", encoding)?;
         w = wset_opt(w, "ignoreNullFields", ignoreNullFields)?;
+        // PySpark sets the mode/options/format on the writer itself, then 
saves.
+        self.inner = w.clone().format("json");
         w.json(path).to_pyerr()
     }
 
@@ -2179,7 +2176,7 @@ impl PyDataFrameWriter {
         emptyValue: Option<&Bound<'_, PyAny>>,
         lineSep: Option<&Bound<'_, PyAny>>,
     ) -> PyResult<()> {
-        let mut w = self.take()?;
+        let mut w = self.inner.clone();
         if let Some(m) = mode {
             w = w.mode(&m.str()?.to_string());
         }
@@ -2199,6 +2196,8 @@ impl PyDataFrameWriter {
         w = wset_opt(w, "encoding", encoding)?;
         w = wset_opt(w, "emptyValue", emptyValue)?;
         w = wset_opt(w, "lineSep", lineSep)?;
+        // PySpark sets the mode/options/format on the writer itself, then 
saves.
+        self.inner = w.clone().format("csv");
         w.csv(path).to_pyerr()
     }
 
@@ -2213,7 +2212,7 @@ impl PyDataFrameWriter {
         partitionBy: Option<&Bound<'_, PyAny>>,
         compression: Option<&Bound<'_, PyAny>>,
     ) -> PyResult<()> {
-        let mut w = self.take()?;
+        let mut w = self.inner.clone();
         if let Some(m) = mode {
             w = w.mode(&m.str()?.to_string());
         }
@@ -2226,6 +2225,8 @@ impl PyDataFrameWriter {
             w = w.partition_by(cols);
         }
         w = wset_opt(w, "compression", compression)?;
+        // PySpark sets the mode/options/format on the writer itself, then 
saves.
+        self.inner = w.clone().format("orc");
         w.orc(path).to_pyerr()
     }
 
@@ -2239,9 +2240,11 @@ impl PyDataFrameWriter {
         compression: Option<&Bound<'_, PyAny>>,
         lineSep: Option<&Bound<'_, PyAny>>,
     ) -> PyResult<()> {
-        let mut w = self.take()?;
+        let mut w = self.inner.clone();
         w = wset_opt(w, "compression", compression)?;
         w = wset_opt(w, "lineSep", lineSep)?;
+        // PySpark sets the mode/options/format on the writer itself, then 
saves.
+        self.inner = w.clone().format("text");
         w.text(path).to_pyerr()
     }
 
@@ -2266,7 +2269,7 @@ impl PyDataFrameWriter {
         encoding: Option<&Bound<'_, PyAny>>,
         validateName: Option<&Bound<'_, PyAny>>,
     ) -> PyResult<()> {
-        let mut w = self.take()?;
+        let mut w = self.inner.clone();
         w = wset_opt(w, "rowTag", rowTag)?;
         if let Some(m) = mode {
             w = w.mode(&m.str()?.to_string());
@@ -2282,6 +2285,8 @@ impl PyDataFrameWriter {
         w = wset_opt(w, "compression", compression)?;
         w = wset_opt(w, "encoding", encoding)?;
         w = wset_opt(w, "validateName", validateName)?;
+        // PySpark sets the mode/options/format on the writer itself, then 
saves.
+        self.inner = w.clone().format("xml");
         w.xml(path).to_pyerr()
     }
 }
@@ -2289,41 +2294,40 @@ impl PyDataFrameWriter {
 /// Python wrapper for the V2 `DataFrameWriter` (`df.writeTo`).
 #[pyclass(name = "DataFrameWriterV2", module = "pyspark.sql.readwriter")]
 pub struct PyDataFrameWriterV2 {
-    inner: Option<spark_connect::readwriter::DataFrameWriterV2>,
+    inner: spark_connect::readwriter::DataFrameWriterV2,
 }
 
-impl PyDataFrameWriterV2 {
-    fn take(&mut self) -> 
PyResult<spark_connect::readwriter::DataFrameWriterV2> {
-        self.inner.take().ok_or_else(|| {
-            PyErr::new::<pyo3::exceptions::PyRuntimeError, 
_>("DataFrameWriterV2 already consumed")
-        })
-    }
-}
+impl PyDataFrameWriterV2 {}
 
 #[pymethods]
 impl PyDataFrameWriterV2 {
-    fn using(&mut self, provider: &str) -> PyResult<PyDataFrameWriterV2> {
-        Ok(PyDataFrameWriterV2 {
-            inner: Some(self.take()?.using(provider)),
-        })
+    fn using<'py>(mut slf: PyRefMut<'py, Self>, provider: &str) -> 
PyResult<PyRefMut<'py, Self>> {
+        slf.inner = slf.inner.clone().using(provider);
+        Ok(slf)
     }
 
     // None -> option left unset; bools -> "true"/"false" (reference 
`to_str`), same as
     // the other reader/writer option bindings.
-    fn option(&mut self, key: &str, value: &Bound<'_, PyAny>) -> 
PyResult<PyDataFrameWriterV2> {
+    fn option<'py>(
+        mut slf: PyRefMut<'py, Self>,
+        key: &str,
+        value: &Bound<'_, PyAny>,
+    ) -> PyResult<PyRefMut<'py, Self>> {
         match crate::coerce_option_value(value)? {
-            Some(v) => Ok(PyDataFrameWriterV2 {
-                inner: Some(self.take()?.option(key, &v)),
-            }),
-            None => Ok(PyDataFrameWriterV2 {
-                inner: Some(self.take()?),
-            }),
+            Some(v) => {
+                slf.inner = slf.inner.clone().option(key, &v);
+                Ok(slf)
+            }
+            None => Ok(slf),
         }
     }
 
     // Mirrors reference `DataFrameWriterV2.options(**options)`; None skipped, 
bools lowercased.
     #[pyo3(signature = (**options))]
-    fn options(&mut self, options: Option<&Bound<'_, PyDict>>) -> 
PyResult<PyDataFrameWriterV2> {
+    fn options<'py>(
+        mut slf: PyRefMut<'py, Self>,
+        options: Option<&Bound<'_, PyDict>>,
+    ) -> PyResult<PyRefMut<'py, Self>> {
         let mut map = std::collections::HashMap::new();
         if let Some(options) = options {
             for (k, v) in options.iter() {
@@ -2332,65 +2336,72 @@ impl PyDataFrameWriterV2 {
                 }
             }
         }
-        Ok(PyDataFrameWriterV2 {
-            inner: Some(self.take()?.options(map)),
-        })
+        slf.inner = slf.inner.clone().options(map);
+        Ok(slf)
     }
 
     #[pyo3(name = "tableProperty")]
-    fn table_property(&mut self, property: &str, value: &str) -> 
PyResult<PyDataFrameWriterV2> {
-        Ok(PyDataFrameWriterV2 {
-            inner: Some(self.take()?.table_property(property, value)),
-        })
+    fn table_property<'py>(
+        mut slf: PyRefMut<'py, Self>,
+        property: &str,
+        value: &str,
+    ) -> PyResult<PyRefMut<'py, Self>> {
+        slf.inner = slf.inner.clone().table_property(property, value);
+        Ok(slf)
     }
 
     #[pyo3(name = "partitionedBy", signature = (col, *cols))]
-    fn partitioned_by(
-        &mut self,
+    fn partitioned_by<'py>(
+        mut slf: PyRefMut<'py, Self>,
         col: Bound<'_, PyAny>,
         cols: Vec<Bound<'_, PyAny>>,
-    ) -> PyResult<PyDataFrameWriterV2> {
+    ) -> PyResult<PyRefMut<'py, Self>> {
         let mut all = vec![col];
         all.extend(cols);
         let columns = to_column_list(all)?;
-        Ok(PyDataFrameWriterV2 {
-            inner: Some(self.take()?.partition_by(columns)),
-        })
+        slf.inner = slf.inner.clone().partition_by(columns);
+        Ok(slf)
     }
 
     #[pyo3(name = "clusterBy", signature = (col, *cols))]
-    fn cluster_by(&mut self, col: String, cols: Vec<String>) -> 
PyResult<PyDataFrameWriterV2> {
+    fn cluster_by<'py>(
+        mut slf: PyRefMut<'py, Self>,
+        col: String,
+        cols: Vec<String>,
+    ) -> PyResult<PyRefMut<'py, Self>> {
         let mut all = vec![col];
         all.extend(cols);
-        Ok(PyDataFrameWriterV2 {
-            inner: Some(self.take()?.cluster_by(all)),
-        })
+        slf.inner = slf.inner.clone().cluster_by(all);
+        Ok(slf)
     }
 
     fn create(&mut self) -> PyResult<()> {
-        self.take()?.create().to_pyerr()
+        self.inner.clone().create().to_pyerr()
     }
 
     fn replace(&mut self) -> PyResult<()> {
-        self.take()?.replace().to_pyerr()
+        self.inner.clone().replace().to_pyerr()
     }
 
     #[pyo3(name = "createOrReplace")]
     fn create_or_replace(&mut self) -> PyResult<()> {
-        self.take()?.create_or_replace().to_pyerr()
+        self.inner.clone().create_or_replace().to_pyerr()
     }
 
     fn append(&mut self) -> PyResult<()> {
-        self.take()?.append().to_pyerr()
+        self.inner.clone().append().to_pyerr()
     }
 
     fn overwrite(&mut self, condition: &PyColumn) -> PyResult<()> {
-        self.take()?.overwrite(condition.column.clone()).to_pyerr()
+        self.inner
+            .clone()
+            .overwrite(condition.column.clone())
+            .to_pyerr()
     }
 
     #[pyo3(name = "overwritePartitions")]
     fn overwrite_partitions(&mut self) -> PyResult<()> {
-        self.take()?.overwrite_partitions().to_pyerr()
+        self.inner.clone().overwrite_partitions().to_pyerr()
     }
 }
 
diff --git a/crates/pyspark-rs/src/readwriter.rs 
b/crates/pyspark-rs/src/readwriter.rs
index 3ef57a5..3c0db1b 100644
--- a/crates/pyspark-rs/src/readwriter.rs
+++ b/crates/pyspark-rs/src/readwriter.rs
@@ -8,31 +8,26 @@ use spark_connect::readwriter::DataFrameReader;
 
 use crate::dataframe::PyDataFrame;
 
-/// Python wrapper for the batch DataFrameReader. The core reader is a 
consuming
-/// builder, so each step takes the inner value and returns a fresh wrapper 
(mirrors
-/// the existing PyDataStreamReader).
+/// Python wrapper for the batch DataFrameReader.
+///
+/// PySpark's reader is mutable: `format`/`schema`/`option`/`options` change 
`self`
+/// and return it, and a reader can be used again after a read. The core 
reader is a
+/// consuming builder, so the setters replace the wrapped value in place and 
return
+/// the same Python object, and the terminal reads build their DataFrame from 
a clone.
 #[pyclass(name = "DataFrameReader", module = "pyspark.sql.readwriter")]
 pub struct PyDataFrameReader {
-    inner: Option<DataFrameReader>,
+    inner: DataFrameReader,
 }
 
 impl PyDataFrameReader {
     pub fn new(reader: DataFrameReader) -> Self {
-        PyDataFrameReader {
-            inner: Some(reader),
-        }
-    }
-
-    fn take(&mut self) -> PyResult<DataFrameReader> {
-        self.inner.take().ok_or_else(|| {
-            PyErr::new::<pyo3::exceptions::PyRuntimeError, _>("DataFrameReader 
already consumed")
-        })
+        PyDataFrameReader { inner: reader }
     }
 
-    /// Take the inner reader and apply per-call `**options` (skipping None,
-    /// lowercasing bools) before a format-specific read.
-    fn take_with_opts(&mut self, options: Option<&Bound<'_, PyDict>>) -> 
PyResult<DataFrameReader> {
-        let mut r = self.take()?;
+    /// A copy of the current reader with per-call `**options` applied 
(skipping
+    /// None, lowercasing bools), for a format-specific read.
+    fn with_opts(&self, options: Option<&Bound<'_, PyDict>>) -> 
PyResult<DataFrameReader> {
+        let mut r = self.inner.clone();
         if let Some(opts) = options {
             for (k, v) in opts.iter() {
                 if let Some(val) = crate::coerce_option_value(&v)? {
@@ -42,6 +37,14 @@ impl PyDataFrameReader {
         }
         Ok(r)
     }
+
+    /// Keep the options of a format-specific read on this reader, as PySpark 
does
+    /// (its `json`/`csv`/... set the options and the format on `self`, then 
`load`).
+    /// Called only after every argument was validated, so a failed call 
leaves the
+    /// reader unchanged.
+    fn commit(&mut self, r: &DataFrameReader, format: &str) {
+        self.inner = r.clone().format(format);
+    }
 }
 
 /// Apply a single reader option (skipping None; bool->"true"/"false").
@@ -69,30 +72,37 @@ fn apply_reader_schema(r: DataFrameReader, v: &Bound<'_, 
PyAny>) -> PyResult<Dat
 #[pymethods]
 impl PyDataFrameReader {
     /// Set the source format (e.g. "parquet", "json", "csv").
-    fn format(&mut self, source: &str) -> PyResult<PyDataFrameReader> {
-        Ok(PyDataFrameReader::new(self.take()?.format(source)))
+    fn format<'py>(mut slf: PyRefMut<'py, Self>, source: &str) -> 
PyRefMut<'py, Self> {
+        slf.inner = slf.inner.clone().format(source);
+        slf
     }
 
     /// Set the schema (a DDL string).
-    fn schema(&mut self, schema: &str) -> PyResult<PyDataFrameReader> {
-        Ok(PyDataFrameReader::new(
-            self.take()?.schema(schema.to_string()),
-        ))
+    fn schema<'py>(mut slf: PyRefMut<'py, Self>, schema: &str) -> 
PyRefMut<'py, Self> {
+        slf.inner = slf.inner.clone().schema(schema.to_string());
+        slf
     }
 
     /// Set a single read option. `None` leaves the option unset (not the 
string
     /// "None"); booleans lowercase to "true"/"false" (reference `to_str` 
semantics).
-    fn option(&mut self, key: &str, value: &Bound<'_, PyAny>) -> 
PyResult<PyDataFrameReader> {
-        match crate::coerce_option_value(value)? {
-            Some(v) => Ok(PyDataFrameReader::new(self.take()?.option(key, 
&v))),
-            None => Ok(PyDataFrameReader::new(self.take()?)),
+    fn option<'py>(
+        mut slf: PyRefMut<'py, Self>,
+        key: &str,
+        value: &Bound<'_, PyAny>,
+    ) -> PyResult<PyRefMut<'py, Self>> {
+        if let Some(v) = crate::coerce_option_value(value)? {
+            slf.inner = slf.inner.clone().option(key, &v);
         }
+        Ok(slf)
     }
 
     /// Set multiple read options. Mirrors reference 
`DataFrameReader.options(**options)`
     /// - keyword args; `None` values are skipped and booleans lowercased.
     #[pyo3(signature = (**options))]
-    fn options(&mut self, options: Option<&Bound<'_, PyDict>>) -> 
PyResult<PyDataFrameReader> {
+    fn options<'py>(
+        mut slf: PyRefMut<'py, Self>,
+        options: Option<&Bound<'_, PyDict>>,
+    ) -> PyResult<PyRefMut<'py, Self>> {
         let mut map = HashMap::new();
         if let Some(options) = options {
             for (k, v) in options.iter() {
@@ -101,7 +111,26 @@ impl PyDataFrameReader {
                 }
             }
         }
-        Ok(PyDataFrameReader::new(self.take()?.options(map)))
+        slf.inner = slf.inner.clone().options(map);
+        Ok(slf)
+    }
+
+    /// The configured source format (PySpark Connect's 
`DataFrameReader._format`).
+    #[getter]
+    fn _format(&self) -> Option<String> {
+        self.inner.get_format().map(str::to_string)
+    }
+
+    /// The configured schema string, empty when unset 
(`DataFrameReader._schema`).
+    #[getter]
+    fn _schema(&self) -> String {
+        self.inner.get_schema().to_string()
+    }
+
+    /// The configured read options (`DataFrameReader._options`); a copy.
+    #[getter]
+    fn _options(&self) -> HashMap<String, String> {
+        self.inner.get_options().clone()
     }
 
     /// Load data from the (optional) path. Mirrors the reference
@@ -116,25 +145,27 @@ impl PyDataFrameReader {
         schema: Option<&Bound<'_, PyAny>>,
         options: Option<&Bound<'_, PyDict>>,
     ) -> PyResult<PyDataFrame> {
-        let mut r = self.take_with_opts(options)?;
+        let mut r = self.with_opts(options)?;
         if let Some(f) = format {
             r = r.format(f);
         }
         if let Some(s) = schema {
             r = apply_reader_schema(r, s)?;
         }
+        // PySpark's one-shot load sets format/schema/options on the reader 
itself.
+        self.inner = r.clone();
         Ok(PyDataFrame::new(r.load(path)))
     }
 
     /// Read a table by name.
-    fn table(&mut self, tableName: &str) -> PyResult<PyDataFrame> {
-        Ok(PyDataFrame::new(self.take()?.table(tableName)))
+    fn table(&self, tableName: &str) -> PyResult<PyDataFrame> {
+        Ok(PyDataFrame::new(self.inner.clone().table(tableName)))
     }
 
     /// Read the CDC changes of a named table. Mirrors 
`DataFrameReader.changes`.
     #[allow(non_snake_case)]
-    fn changes(&mut self, tableName: &str) -> PyResult<PyDataFrame> {
-        Ok(PyDataFrame::new(self.take()?.changes(tableName)))
+    fn changes(&self, tableName: &str) -> PyResult<PyDataFrame> {
+        Ok(PyDataFrame::new(self.inner.clone().changes(tableName)))
     }
 
     /// Read JSON file(s) - full pyspark signature; each named option is
@@ -170,7 +201,7 @@ impl PyDataFrameReader {
         allowNonNumericNumbers: Option<&Bound<'_, PyAny>>,
         useUnsafeRow: Option<&Bound<'_, PyAny>>,
     ) -> PyResult<PyDataFrame> {
-        let mut r = self.take()?;
+        let mut r = self.inner.clone();
         if let Some(v) = schema {
             r = apply_reader_schema(r, v)?;
         }
@@ -202,6 +233,7 @@ impl PyDataFrameReader {
         r = set_opt(r, "modifiedAfter", modifiedAfter)?;
         r = set_opt(r, "allowNonNumericNumbers", allowNonNumericNumbers)?;
         r = set_opt(r, "useUnsafeRow", useUnsafeRow)?;
+        self.commit(&r, "json");
         Ok(PyDataFrame::new(r.json(path)))
     }
 
@@ -219,9 +251,9 @@ impl PyDataFrameReader {
                 "parquet() currently supports exactly one path",
             ));
         }
-        Ok(PyDataFrame::new(
-            self.take_with_opts(options)?.parquet(&paths[0]),
-        ))
+        let r = self.with_opts(options)?;
+        self.commit(&r, "parquet");
+        Ok(PyDataFrame::new(r.parquet(&paths[0])))
     }
 
     /// Read CSV file(s) - full pyspark signature; each named option is
@@ -265,7 +297,7 @@ impl PyDataFrameReader {
         modifiedAfter: Option<&Bound<'_, PyAny>>,
         unescapedQuoteHandling: Option<&Bound<'_, PyAny>>,
     ) -> PyResult<PyDataFrame> {
-        let mut r = self.take()?;
+        let mut r = self.inner.clone();
         if let Some(v) = schema {
             r = apply_reader_schema(r, v)?;
         }
@@ -305,6 +337,7 @@ impl PyDataFrameReader {
         r = set_opt(r, "modifiedBefore", modifiedBefore)?;
         r = set_opt(r, "modifiedAfter", modifiedAfter)?;
         r = set_opt(r, "unescapedQuoteHandling", unescapedQuoteHandling)?;
+        self.commit(&r, "csv");
         Ok(PyDataFrame::new(r.csv(path)))
     }
 
@@ -321,12 +354,13 @@ impl PyDataFrameReader {
         modifiedBefore: Option<&Bound<'_, PyAny>>,
         modifiedAfter: Option<&Bound<'_, PyAny>>,
     ) -> PyResult<PyDataFrame> {
-        let mut r = self.take()?;
+        let mut r = self.inner.clone();
         r = set_opt(r, "mergeSchema", mergeSchema)?;
         r = set_opt(r, "pathGlobFilter", pathGlobFilter)?;
         r = set_opt(r, "recursiveFileLookup", recursiveFileLookup)?;
         r = set_opt(r, "modifiedBefore", modifiedBefore)?;
         r = set_opt(r, "modifiedAfter", modifiedAfter)?;
+        self.commit(&r, "orc");
         Ok(PyDataFrame::new(r.orc(path)))
     }
 
@@ -344,13 +378,14 @@ impl PyDataFrameReader {
         modifiedBefore: Option<&Bound<'_, PyAny>>,
         modifiedAfter: Option<&Bound<'_, PyAny>>,
     ) -> PyResult<PyDataFrame> {
-        let mut r = self.take()?;
+        let mut r = self.inner.clone();
         r = set_opt(r, "wholetext", wholetext)?;
         r = set_opt(r, "lineSep", lineSep)?;
         r = set_opt(r, "pathGlobFilter", pathGlobFilter)?;
         r = set_opt(r, "recursiveFileLookup", recursiveFileLookup)?;
         r = set_opt(r, "modifiedBefore", modifiedBefore)?;
         r = set_opt(r, "modifiedAfter", modifiedAfter)?;
+        self.commit(&r, "text");
         Ok(PyDataFrame::new(r.text(paths)))
     }
 
@@ -381,7 +416,7 @@ impl PyDataFrameReader {
         samplingRatio: Option<&Bound<'_, PyAny>>,
         locale: Option<&Bound<'_, PyAny>>,
     ) -> PyResult<PyDataFrame> {
-        let mut r = self.take()?;
+        let mut r = self.inner.clone();
         r = set_opt(r, "rowTag", rowTag)?;
         if let Some(v) = schema {
             r = apply_reader_schema(r, v)?;
@@ -403,6 +438,7 @@ impl PyDataFrameReader {
         r = set_opt(r, "multiLine", multiLine)?;
         r = set_opt(r, "samplingRatio", samplingRatio)?;
         r = set_opt(r, "locale", locale)?;
+        self.commit(&r, "xml");
         Ok(PyDataFrame::new(r.xml(path)))
     }
 
@@ -424,7 +460,7 @@ impl PyDataFrameReader {
         predicates: Option<Vec<String>>,
         properties: Option<HashMap<String, String>>,
     ) -> PyResult<PyDataFrame> {
-        let mut r = self.take()?;
+        let mut r = self.inner.clone();
         if let Some(c) = column {
             r = r.option("partitionColumn", &c);
         }
@@ -442,6 +478,12 @@ impl PyDataFrameReader {
                 r = r.option(&k, &v);
             }
         }
+        // PySpark's jdbc also leaves format/url/dbtable on the reader.
+        self.inner = r
+            .clone()
+            .format("jdbc")
+            .option("url", url)
+            .option("dbtable", table);
         Ok(PyDataFrame::new(r.jdbc(url, table, predicates)))
     }
 }
diff --git a/crates/pyspark-rs/src/streaming.rs 
b/crates/pyspark-rs/src/streaming.rs
index 09977bc..88d3baf 100644
--- a/crates/pyspark-rs/src/streaming.rs
+++ b/crates/pyspark-rs/src/streaming.rs
@@ -40,7 +40,6 @@ impl StreamingQueryListener for PyListenerAdapter {
     }
 }
 
-/// Python wrapper for DataStreamReader.
 /// Apply a named read option to a streaming reader when supplied (mirrors the 
batch
 /// `set_opt` in readwriter.rs): `None` leaves the option unset (not the 
string "None");
 /// booleans lowercase to "true"/"false" via `coerce_option_value`.
@@ -74,55 +73,56 @@ fn extract_partition_cols(v: &Bound<'_, PyAny>) -> 
PyResult<Vec<String>> {
     v.extract::<Vec<String>>()
 }
 
+/// Python wrapper for DataStreamReader.
+///
+/// Like PySpark's reader, the setters change this object and return it, and 
the
+/// reader can be used again after a read: the core reader is a consuming 
builder, so
+/// the setters replace the wrapped value in place and the reads work on a 
clone.
 #[pyclass(name = "DataStreamReader", module = 
"pyspark.sql.streaming.readwriter")]
 pub struct PyDataStreamReader {
-    inner: Option<DataStreamReader>,
+    inner: DataStreamReader,
 }
 
 impl PyDataStreamReader {
     pub fn new(reader: DataStreamReader) -> Self {
-        PyDataStreamReader {
-            inner: Some(reader),
-        }
-    }
-
-    fn take(&mut self) -> PyResult<DataStreamReader> {
-        self.inner.take().ok_or_else(|| {
-            PyErr::new::<pyo3::exceptions::PyRuntimeError, 
_>("DataStreamReader already consumed")
-        })
+        PyDataStreamReader { inner: reader }
     }
 }
 
 #[pymethods]
 impl PyDataStreamReader {
-    fn format(&mut self, source: &str) -> PyResult<PyDataStreamReader> {
-        Ok(PyDataStreamReader {
-            inner: Some(self.take()?.format(source)),
-        })
+    fn format<'py>(mut slf: PyRefMut<'py, Self>, source: &str) -> 
PyResult<PyRefMut<'py, Self>> {
+        slf.inner = slf.inner.clone().format(source);
+        Ok(slf)
     }
 
-    fn schema(&mut self, schema: &str) -> PyResult<PyDataStreamReader> {
-        Ok(PyDataStreamReader {
-            inner: Some(self.take()?.schema(schema)),
-        })
+    fn schema<'py>(mut slf: PyRefMut<'py, Self>, schema: &str) -> 
PyResult<PyRefMut<'py, Self>> {
+        slf.inner = slf.inner.clone().schema(schema);
+        Ok(slf)
     }
 
-    fn option(&mut self, key: &str, value: &Bound<'_, PyAny>) -> 
PyResult<PyDataStreamReader> {
+    fn option<'py>(
+        mut slf: PyRefMut<'py, Self>,
+        key: &str,
+        value: &Bound<'_, PyAny>,
+    ) -> PyResult<PyRefMut<'py, Self>> {
         // None -> option left unset; bools -> "true"/"false" (reference 
`to_str`).
         match crate::coerce_option_value(value)? {
-            Some(v) => Ok(PyDataStreamReader {
-                inner: Some(self.take()?.option(key, &v)),
-            }),
-            None => Ok(PyDataStreamReader {
-                inner: Some(self.take()?),
-            }),
+            Some(v) => {
+                slf.inner = slf.inner.clone().option(key, &v);
+                Ok(slf)
+            }
+            None => Ok(slf),
         }
     }
 
     // Mirrors reference `DataStreamReader.options(**options)`: keyword args; 
None values
     // skipped, booleans lowercased.
     #[pyo3(signature = (**options))]
-    fn options(&mut self, options: Option<&Bound<'_, PyDict>>) -> 
PyResult<PyDataStreamReader> {
+    fn options<'py>(
+        mut slf: PyRefMut<'py, Self>,
+        options: Option<&Bound<'_, PyDict>>,
+    ) -> PyResult<PyRefMut<'py, Self>> {
         let mut opts = HashMap::new();
         if let Some(options) = options {
             for (k, v) in options.iter() {
@@ -131,9 +131,26 @@ impl PyDataStreamReader {
                 }
             }
         }
-        Ok(PyDataStreamReader {
-            inner: Some(self.take()?.options(opts)),
-        })
+        slf.inner = slf.inner.clone().options(opts);
+        Ok(slf)
+    }
+
+    /// The configured source format (PySpark Connect's 
`DataStreamReader._format`).
+    #[getter]
+    fn _format(&self) -> Option<String> {
+        self.inner.get_format().map(str::to_string)
+    }
+
+    /// The configured schema string, empty when unset 
(`DataStreamReader._schema`).
+    #[getter]
+    fn _schema(&self) -> String {
+        self.inner.get_schema().to_string()
+    }
+
+    /// The configured read options (`DataStreamReader._options`); a copy.
+    #[getter]
+    fn _options(&self) -> HashMap<String, String> {
+        self.inner.get_options().clone()
     }
 
     #[pyo3(signature = (path=None, format=None, schema=None, **options))]
@@ -144,7 +161,7 @@ impl PyDataStreamReader {
         schema: Option<&Bound<'_, PyAny>>,
         options: Option<&Bound<'_, PyDict>>,
     ) -> PyResult<PyDataFrame> {
-        let mut r = self.take()?;
+        let mut r = self.inner.clone();
         if let Some(fmt) = format {
             r = r.format(fmt);
         }
@@ -160,12 +177,14 @@ impl PyDataStreamReader {
             }
             r = r.options(opts);
         }
+        // PySpark's one-shot load sets format/schema/options on the reader 
itself.
+        self.inner = r.clone();
         Ok(PyDataFrame::new(r.load(path)))
     }
 
     /// Set the source name (for checkpoint stability). Mirrors 
`DataStreamReader.name`,
     /// which validates the name is a non-empty `[A-Za-z0-9_]+` string.
-    fn name(&mut self, source_name: &str) -> PyResult<PyDataStreamReader> {
+    fn name<'py>(mut slf: PyRefMut<'py, Self>, source_name: &str) -> 
PyResult<PyRefMut<'py, Self>> {
         if source_name.is_empty()
             || !source_name
                 .chars()
@@ -175,14 +194,13 @@ impl PyDataStreamReader {
                 "Invalid streaming source name: {source_name:?}; only ASCII 
letters, digits, and underscores are allowed"
             )));
         }
-        Ok(PyDataStreamReader {
-            inner: Some(self.take()?.name(source_name)),
-        })
+        slf.inner = slf.inner.clone().name(source_name);
+        Ok(slf)
     }
 
     /// Read the streaming CDC changes of a named table. Mirrors 
`DataStreamReader.changes`.
-    fn changes(&mut self, tableName: &str) -> PyResult<PyDataFrame> {
-        Ok(PyDataFrame::new(self.take()?.changes(tableName)))
+    fn changes(&self, tableName: &str) -> PyResult<PyDataFrame> {
+        Ok(PyDataFrame::new(self.inner.clone().changes(tableName)))
     }
 
     /// Load an XML streaming source. Mirrors `DataStreamReader.xml(...)`: set 
each named
@@ -212,7 +230,7 @@ impl PyDataStreamReader {
         samplingRatio: Option<&Bound<'_, PyAny>>,
         locale: Option<&Bound<'_, PyAny>>,
     ) -> PyResult<PyDataFrame> {
-        let mut r = self.take()?.format("xml");
+        let mut r = self.inner.clone().format("xml");
         r = set_ropt(r, "rowTag", rowTag)?;
         if let Some(v) = schema {
             r = apply_stream_schema(r, v)?;
@@ -234,11 +252,13 @@ impl PyDataStreamReader {
         r = set_ropt(r, "multiLine", multiLine)?;
         r = set_ropt(r, "samplingRatio", samplingRatio)?;
         r = set_ropt(r, "locale", locale)?;
+        // Like PySpark, the options and the xml format stay on the reader.
+        self.inner = r.clone();
         Ok(PyDataFrame::new(r.load(Some(path))))
     }
 
-    fn table(&mut self, tableName: &str) -> PyResult<PyDataFrame> {
-        let df = self.take()?.table(tableName);
+    fn table(&self, tableName: &str) -> PyResult<PyDataFrame> {
+        let df = self.inner.clone().table(tableName);
         Ok(PyDataFrame::new(df))
     }
 
@@ -270,7 +290,7 @@ impl PyDataStreamReader {
         allowNonNumericNumbers: Option<&Bound<'_, PyAny>>,
         useUnsafeRow: Option<&Bound<'_, PyAny>>,
     ) -> PyResult<PyDataFrame> {
-        let mut r = self.take()?;
+        let mut r = self.inner.clone();
         if let Some(v) = schema {
             r = apply_stream_schema(r, v)?;
         }
@@ -299,6 +319,8 @@ impl PyDataStreamReader {
         r = set_ropt(r, "recursiveFileLookup", recursiveFileLookup)?;
         r = set_ropt(r, "allowNonNumericNumbers", allowNonNumericNumbers)?;
         r = set_ropt(r, "useUnsafeRow", useUnsafeRow)?;
+        // Like PySpark, the options and the format stay on the reader.
+        self.inner = r.clone().format("json");
         Ok(PyDataFrame::new(r.json(path)))
     }
 
@@ -313,12 +335,14 @@ impl PyDataStreamReader {
         datetimeRebaseMode: Option<&Bound<'_, PyAny>>,
         int96RebaseMode: Option<&Bound<'_, PyAny>>,
     ) -> PyResult<PyDataFrame> {
-        let mut r = self.take()?;
+        let mut r = self.inner.clone();
         r = set_ropt(r, "mergeSchema", mergeSchema)?;
         r = set_ropt(r, "pathGlobFilter", pathGlobFilter)?;
         r = set_ropt(r, "recursiveFileLookup", recursiveFileLookup)?;
         r = set_ropt(r, "datetimeRebaseMode", datetimeRebaseMode)?;
         r = set_ropt(r, "int96RebaseMode", int96RebaseMode)?;
+        // Like PySpark, the options and the format stay on the reader.
+        self.inner = r.clone().format("parquet");
         Ok(PyDataFrame::new(r.parquet(path)))
     }
 
@@ -358,7 +382,7 @@ impl PyDataStreamReader {
         recursiveFileLookup: Option<&Bound<'_, PyAny>>,
         unescapedQuoteHandling: Option<&Bound<'_, PyAny>>,
     ) -> PyResult<PyDataFrame> {
-        let mut r = self.take()?;
+        let mut r = self.inner.clone();
         if let Some(v) = schema {
             r = apply_stream_schema(r, v)?;
         }
@@ -395,6 +419,8 @@ impl PyDataStreamReader {
         r = set_ropt(r, "pathGlobFilter", pathGlobFilter)?;
         r = set_ropt(r, "recursiveFileLookup", recursiveFileLookup)?;
         r = set_ropt(r, "unescapedQuoteHandling", unescapedQuoteHandling)?;
+        // Like PySpark, the options and the format stay on the reader.
+        self.inner = r.clone().format("csv");
         Ok(PyDataFrame::new(r.csv(path)))
     }
 
@@ -407,10 +433,12 @@ impl PyDataStreamReader {
         pathGlobFilter: Option<&Bound<'_, PyAny>>,
         recursiveFileLookup: Option<&Bound<'_, PyAny>>,
     ) -> PyResult<PyDataFrame> {
-        let mut r = self.take()?;
+        let mut r = self.inner.clone();
         r = set_ropt(r, "mergeSchema", mergeSchema)?;
         r = set_ropt(r, "pathGlobFilter", pathGlobFilter)?;
         r = set_ropt(r, "recursiveFileLookup", recursiveFileLookup)?;
+        // Like PySpark, the options and the format stay on the reader.
+        self.inner = r.clone().format("orc");
         Ok(PyDataFrame::new(r.orc(path)))
     }
 
@@ -424,11 +452,13 @@ impl PyDataStreamReader {
         pathGlobFilter: Option<&Bound<'_, PyAny>>,
         recursiveFileLookup: Option<&Bound<'_, PyAny>>,
     ) -> PyResult<PyDataFrame> {
-        let mut r = self.take()?;
+        let mut r = self.inner.clone();
         r = set_ropt(r, "wholetext", wholetext)?;
         r = set_ropt(r, "lineSep", lineSep)?;
         r = set_ropt(r, "pathGlobFilter", pathGlobFilter)?;
         r = set_ropt(r, "recursiveFileLookup", recursiveFileLookup)?;
+        // Like PySpark, the options and the format stay on the reader.
+        self.inner = r.clone().format("text");
         Ok(PyDataFrame::new(r.text(path)))
     }
 }
@@ -480,20 +510,12 @@ impl PyTrigger {
 /// Python wrapper for DataStreamWriter.
 #[pyclass(name = "DataStreamWriter", module = 
"pyspark.sql.streaming.readwriter")]
 pub struct PyDataStreamWriter {
-    inner: Option<DataStreamWriter>,
+    inner: DataStreamWriter,
 }
 
 impl PyDataStreamWriter {
     pub fn new(writer: DataStreamWriter) -> Self {
-        PyDataStreamWriter {
-            inner: Some(writer),
-        }
-    }
-
-    fn take(&mut self) -> PyResult<DataStreamWriter> {
-        self.inner.take().ok_or_else(|| {
-            PyErr::new::<pyo3::exceptions::PyRuntimeError, 
_>("DataStreamWriter already consumed")
-        })
+        PyDataStreamWriter { inner: writer }
     }
 
     /// Apply the one-shot writer options accepted by `start`/`toTable`
@@ -507,7 +529,7 @@ impl PyDataStreamWriter {
         query_name: Option<&str>,
         options: Option<&Bound<'_, PyDict>>,
     ) -> PyResult<DataStreamWriter> {
-        let mut w = self.take()?;
+        let mut w = self.inner.clone();
         if let Some(fmt) = format {
             w = w.format(fmt);
         }
@@ -531,40 +553,49 @@ impl PyDataStreamWriter {
             }
             w = w.options(opts);
         }
+        // PySpark's start/toTable set these on the writer itself before 
starting.
+        self.inner = w.clone();
         Ok(w)
     }
 }
 
 #[pymethods]
 impl PyDataStreamWriter {
-    fn outputMode(&mut self, outputMode: &str) -> PyResult<PyDataStreamWriter> 
{
-        Ok(PyDataStreamWriter {
-            inner: Some(self.take()?.output_mode(outputMode)),
-        })
+    fn outputMode<'py>(
+        mut slf: PyRefMut<'py, Self>,
+        outputMode: &str,
+    ) -> PyResult<PyRefMut<'py, Self>> {
+        slf.inner = slf.inner.clone().output_mode(outputMode);
+        Ok(slf)
     }
 
-    fn format(&mut self, source: &str) -> PyResult<PyDataStreamWriter> {
-        Ok(PyDataStreamWriter {
-            inner: Some(self.take()?.format(source)),
-        })
+    fn format<'py>(mut slf: PyRefMut<'py, Self>, source: &str) -> 
PyResult<PyRefMut<'py, Self>> {
+        slf.inner = slf.inner.clone().format(source);
+        Ok(slf)
     }
 
-    fn option(&mut self, key: &str, value: &Bound<'_, PyAny>) -> 
PyResult<PyDataStreamWriter> {
+    fn option<'py>(
+        mut slf: PyRefMut<'py, Self>,
+        key: &str,
+        value: &Bound<'_, PyAny>,
+    ) -> PyResult<PyRefMut<'py, Self>> {
         // None -> option left unset; bools -> "true"/"false" (reference 
`to_str`).
         match crate::coerce_option_value(value)? {
-            Some(v) => Ok(PyDataStreamWriter {
-                inner: Some(self.take()?.option(key, &v)),
-            }),
-            None => Ok(PyDataStreamWriter {
-                inner: Some(self.take()?),
-            }),
+            Some(v) => {
+                slf.inner = slf.inner.clone().option(key, &v);
+                Ok(slf)
+            }
+            None => Ok(slf),
         }
     }
 
     // Mirrors reference `DataStreamWriter.options(**options)`: keyword args; 
None values
     // skipped, booleans lowercased.
     #[pyo3(signature = (**options))]
-    fn options(&mut self, options: Option<&Bound<'_, PyDict>>) -> 
PyResult<PyDataStreamWriter> {
+    fn options<'py>(
+        mut slf: PyRefMut<'py, Self>,
+        options: Option<&Bound<'_, PyDict>>,
+    ) -> PyResult<PyRefMut<'py, Self>> {
         let mut opts = HashMap::new();
         if let Some(options) = options {
             for (k, v) in options.iter() {
@@ -573,44 +604,49 @@ impl PyDataStreamWriter {
                 }
             }
         }
-        Ok(PyDataStreamWriter {
-            inner: Some(self.take()?.options(opts)),
-        })
+        slf.inner = slf.inner.clone().options(opts);
+        Ok(slf)
     }
 
     #[pyo3(signature = (*cols))]
-    fn partitionBy(&mut self, cols: Vec<String>) -> 
PyResult<PyDataStreamWriter> {
+    fn partitionBy<'py>(
+        mut slf: PyRefMut<'py, Self>,
+        cols: Vec<String>,
+    ) -> PyResult<PyRefMut<'py, Self>> {
         let col_refs: Vec<&str> = cols.iter().map(|s| s.as_str()).collect();
-        Ok(PyDataStreamWriter {
-            inner: Some(self.take()?.partition_by(col_refs)),
-        })
+        slf.inner = slf.inner.clone().partition_by(col_refs);
+        Ok(slf)
     }
 
     #[pyo3(signature = (*cols))]
-    fn clusterBy(&mut self, cols: Vec<String>) -> PyResult<PyDataStreamWriter> 
{
+    fn clusterBy<'py>(
+        mut slf: PyRefMut<'py, Self>,
+        cols: Vec<String>,
+    ) -> PyResult<PyRefMut<'py, Self>> {
         let col_refs: Vec<&str> = cols.iter().map(|s| s.as_str()).collect();
-        Ok(PyDataStreamWriter {
-            inner: Some(self.take()?.cluster_by(col_refs)),
-        })
+        slf.inner = slf.inner.clone().cluster_by(col_refs);
+        Ok(slf)
     }
 
-    fn queryName(&mut self, queryName: &str) -> PyResult<PyDataStreamWriter> {
-        Ok(PyDataStreamWriter {
-            inner: Some(self.take()?.query_name(queryName)),
-        })
+    fn queryName<'py>(
+        mut slf: PyRefMut<'py, Self>,
+        queryName: &str,
+    ) -> PyResult<PyRefMut<'py, Self>> {
+        slf.inner = slf.inner.clone().query_name(queryName);
+        Ok(slf)
     }
 
     /// `DataStreamWriter.trigger(...)`: mirrors the reference keyword API — 
exactly one
     /// of `processingTime` / `once` / `availableNow` / `continuous` is given.
     #[pyo3(signature = (*, processingTime=None, once=None, continuous=None, 
availableNow=None, realTime=None))]
-    fn trigger(
-        &mut self,
+    fn trigger<'py>(
+        mut slf: PyRefMut<'py, Self>,
         processingTime: Option<&str>,
         once: Option<bool>,
         continuous: Option<&str>,
         availableNow: Option<bool>,
         realTime: Option<&str>,
-    ) -> PyResult<PyDataStreamWriter> {
+    ) -> PyResult<PyRefMut<'py, Self>> {
         let trigger = if let Some(interval) = processingTime {
             Trigger::ProcessingTime(interval.to_string())
         } else if once == Some(true) {
@@ -630,9 +666,8 @@ impl PyDataStreamWriter {
                 "trigger() requires exactly one of processingTime, once, 
continuous, availableNow, realTime",
             ));
         };
-        Ok(PyDataStreamWriter {
-            inner: Some(self.take()?.trigger(trigger)),
-        })
+        slf.inner = slf.inner.clone().trigger(trigger);
+        Ok(slf)
     }
 
     #[pyo3(signature = (path=None, format=None, outputMode=None, 
partitionBy=None, queryName=None, **options))]
@@ -672,11 +707,11 @@ impl PyDataStreamWriter {
     /// function (via the bundled `pyspark.cloudpickle`) and attach it as the
     /// foreach-batch sink. Pickling happens here so the Python skin can 
re-export this
     /// class directly rather than subclassing a (non-subclassable) PyO3 type.
-    fn foreachBatch(
-        &mut self,
+    fn foreachBatch<'py>(
+        mut slf: PyRefMut<'py, Self>,
         py: Python<'_>,
         func: &Bound<'_, PyAny>,
-    ) -> PyResult<PyDataStreamWriter> {
+    ) -> PyResult<PyRefMut<'py, Self>> {
         let command = crate::dataframe::py_cloudpickle(py, func)?;
         let payload = PythonUDFPayload::new(
             spark_connect::types::DataType::Struct { fields: vec![] },
@@ -684,16 +719,19 @@ impl PyDataStreamWriter {
             command,
             crate::dataframe::py_version(py),
         );
-        Ok(PyDataStreamWriter {
-            inner: Some(self.take()?.foreach_batch(payload)),
-        })
+        slf.inner = slf.inner.clone().foreach_batch(payload);
+        Ok(slf)
     }
 
     /// `DataStreamWriter.foreach(f)`: wrap the row handler as the reference 
client does
     /// — `(f, None, serializer, serializer)` with 
`AutoBatchedSerializer(CPickleSerializer())`
     /// — and cloudpickle it so the worker deserializes it against its own
     /// `pyspark.serializers`. Built here to avoid a Python subclass of the 
PyO3 class.
-    fn foreach(&mut self, py: Python<'_>, f: &Bound<'_, PyAny>) -> 
PyResult<PyDataStreamWriter> {
+    fn foreach<'py>(
+        mut slf: PyRefMut<'py, Self>,
+        py: Python<'_>,
+        f: &Bound<'_, PyAny>,
+    ) -> PyResult<PyRefMut<'py, Self>> {
         let serializers = py.import("pyspark.serializers")?;
         let cpickle = serializers.getattr("CPickleSerializer")?.call0()?;
         let serializer = serializers
@@ -717,9 +755,8 @@ impl PyDataStreamWriter {
             command,
             crate::dataframe::py_version(py),
         );
-        Ok(PyDataStreamWriter {
-            inner: Some(self.take()?.foreach(payload)),
-        })
+        slf.inner = slf.inner.clone().foreach(payload);
+        Ok(slf)
     }
 }
 
diff --git a/crates/spark-connect/src/merge.rs 
b/crates/spark-connect/src/merge.rs
index 1c723db..fc57ad5 100644
--- a/crates/spark-connect/src/merge.rs
+++ b/crates/spark-connect/src/merge.rs
@@ -38,6 +38,7 @@ fn merge_action(
 }
 
 /// Fluent builder for a MERGE INTO command. Create it via 
[`crate::dataframe::DataFrame::merge_into`].
+#[derive(Clone)]
 pub struct MergeIntoWriter {
     session: SparkSession,
     source_plan: Arc<LogicalPlan>,
diff --git a/crates/spark-connect/src/readwriter.rs 
b/crates/spark-connect/src/readwriter.rs
index 54fafeb..f103bb1 100644
--- a/crates/spark-connect/src/readwriter.rs
+++ b/crates/spark-connect/src/readwriter.rs
@@ -16,6 +16,7 @@ use crate::session::SparkSession;
 /// DataFrameReader for reading data from various sources.
 ///
 /// Mirrors `pyspark.sql.connect.readwriter.DataFrameReader`.
+#[derive(Clone)]
 pub struct DataFrameReader {
     session: SparkSession,
     format: Option<String>,
@@ -58,6 +59,21 @@ impl DataFrameReader {
         self
     }
 
+    /// The configured source format, if any.
+    pub fn get_format(&self) -> Option<&str> {
+        self.format.as_deref()
+    }
+
+    /// The configured schema (a DDL/JSON string; empty when unset).
+    pub fn get_schema(&self) -> &str {
+        &self.schema
+    }
+
+    /// The configured read options.
+    pub fn get_options(&self) -> &HashMap<String, String> {
+        &self.options
+    }
+
     /// Load data from the specified path(s) with the configured format, 
schema, and options.
     pub fn load(self, path: Option<&str>) -> DataFrame {
         let paths = path.map(|p| vec![p.to_string()]);
@@ -330,6 +346,7 @@ impl TableSaveMethod {
 /// DataFrameWriter for writing DataFrames to various destinations.
 ///
 /// Mirrors `pyspark.sql.connect.readwriter.DataFrameWriter`.
+#[derive(Clone)]
 pub struct DataFrameWriter {
     session: SparkSession,
     input_plan: Arc<LogicalPlan>,
@@ -515,6 +532,7 @@ impl DataFrameWriter {
 /// DataFrameWriterV2 for the v2 write API (`DataFrame.writeTo`).
 ///
 /// Mirrors `pyspark.sql.connect.readwriter.DataFrameWriterV2`.
+#[derive(Clone)]
 pub struct DataFrameWriterV2 {
     session: SparkSession,
     input_plan: Arc<LogicalPlan>,
diff --git a/crates/spark-connect/src/streaming.rs 
b/crates/spark-connect/src/streaming.rs
index 600ae28..9a03323 100644
--- a/crates/spark-connect/src/streaming.rs
+++ b/crates/spark-connect/src/streaming.rs
@@ -21,6 +21,7 @@ use crate::udf::PythonUDFPayload;
 /// DataStreamReader for reading streaming data from various sources.
 ///
 /// Mirrors `pyspark.sql.connect.streaming.DataStreamReader`.
+#[derive(Clone)]
 pub struct DataStreamReader {
     session: SparkSession,
     format: Option<String>,
@@ -71,6 +72,21 @@ impl DataStreamReader {
         self
     }
 
+    /// The configured source format, if any.
+    pub fn get_format(&self) -> Option<&str> {
+        self.format.as_deref()
+    }
+
+    /// The configured schema (a DDL/JSON string; empty when unset).
+    pub fn get_schema(&self) -> &str {
+        &self.schema
+    }
+
+    /// The configured read options.
+    pub fn get_options(&self) -> &HashMap<String, String> {
+        &self.options
+    }
+
     /// Load streaming data from the specified path(s).
     pub fn load(self, path: Option<&str>) -> DataFrame {
         let paths = path.map(|p| vec![p.to_string()]);
@@ -242,6 +258,7 @@ pub enum Trigger {
 /// DataStreamWriter for writing streaming data to various sinks.
 ///
 /// Mirrors `pyspark.sql.connect.streaming.DataStreamWriter`.
+#[derive(Clone)]
 pub struct DataStreamWriter {
     session: SparkSession,
     plan: Arc<LogicalPlan>,
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 0fb6533..01c2c44 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -4418,3 +4418,194 @@ def 
test_connect_registry_sends_auto_cdc_define_flow(mock_connect):
     assert 4 not in plain
     assert 10 not in details and 11 not in details and 12 not in details
 
+
+# ------------------------------------------------- reader/writer in-place 
setters
+
[email protected]
+def offline_spark():
+    # The gRPC channel connects lazily, so building readers/writers and lazy
+    # DataFrames needs no server.
+    from pyspark.sql import SparkSession
+
+    return SparkSession.builder.remote("sc://localhost:1").getOrCreate()
+
+
+def test_reader_setters_mutate_and_return_self(offline_spark):
+    # PySpark: the setters change the reader and return it; the return value 
may be
+    # ignored (the pattern pandas-on-Spark read_sql_table uses).
+    r = offline_spark.read
+    assert r.option("a", "b") is r
+    assert r.options(c="d", skipped=None, flag=True) is r
+    assert r.schema("x INT") is r
+    assert r.format("jdbc") is r
+    assert r._format == "jdbc"
+    assert r._schema == "x INT"
+    assert r._options == {"a": "b", "c": "d", "flag": "true"}
+    assert type(r.load()).__name__ == "DataFrame"
+
+
+def test_reader_read_sql_table_pattern(offline_spark):
+    # Mirrors pyspark/pandas/namespace.py read_sql_table, which discards every
+    # setter's return value before loading.
+    reader = offline_spark.read
+    reader.option("dbtable", "t")
+    reader.option("url", "jdbc:postgresql:db")
+    reader.schema("id INT")
+    reader.options(fetchsize=10)
+    sdf = reader.format("jdbc").load()
+    assert type(sdf).__name__ == "DataFrame"
+    assert reader._options == {
+        "dbtable": "t", "url": "jdbc:postgresql:db", "fetchsize": "10",
+    }
+
+
+def test_pandas_read_sql_table_reader_stage(offline_spark, monkeypatch):
+    # Drive the vendored read_sql_table itself up to the point it hands the 
loaded
+    # frame on. default_session() asks the server for a config, so point it at 
the
+    # offline session, and stop at _get_index_map (the first step after the 
read).
+    # pandas-on-Spark does not import under pandas >= 3.0 (pyspark.pandas.frame
+    # needs pandas.tseries.frequencies.DateOffset), so skip on any ImportError.
+    ns = pytest.importorskip("pyspark.pandas.namespace", exc_type=ImportError)
+
+    class _Loaded(Exception):
+        pass
+
+    seen = {}
+
+    def fake_index_map(sdf, index_col):
+        seen["sdf"] = sdf
+        raise _Loaded
+
+    monkeypatch.setattr(ns, "default_session", lambda: offline_spark)
+    monkeypatch.setattr(ns, "_get_index_map", fake_index_map)
+    with pytest.raises(_Loaded):
+        ns.read_sql_table(
+            "t", "jdbc:postgresql:db", schema="id INT", options={"fetchsize": 
"1"}
+        )
+    assert type(seen["sdf"]).__name__ == "DataFrame"
+
+
+def test_reader_reusable_after_load(offline_spark):
+    r = offline_spark.read.format("parquet")
+    r.load("/a")
+    r.load("/b")
+    r.table("t")
+    r.csv("/c", header=True)
+    # Like PySpark, a format method keeps its options and format on the reader.
+    assert r._format == "csv"
+    assert r._options == {"header": "true"}
+    r.json("/d")
+    r.parquet("/e", mergeSchema=False)
+    r.jdbc("jdbc:x", "tbl", properties={"user": "u"})
+    assert r._format == "jdbc"
+    assert r._options["dbtable"] == "tbl" and r._options["user"] == "u"
+    assert r.option("k", "v") is r
+    assert type(r.load()).__name__ == "DataFrame"
+
+
+def test_reader_reusable_after_failed_read(offline_spark):
+    r = offline_spark.read.option("a", "b")
+    with pytest.raises(Exception):
+        r.csv("/p", schema=object())
+    with pytest.raises(Exception):
+        r.json("/p", schema=12)
+    # A failed call leaves the reader as it was.
+    assert r._options == {"a": "b"}
+    assert r._format is None
+    assert type(r.format("csv").load("/p")).__name__ == "DataFrame"
+
+
+def test_stream_reader_setters_mutate_and_return_self(offline_spark):
+    r = offline_spark.readStream
+    assert r.format("rate") is r
+    assert r.option("rowsPerSecond", 5) is r
+    assert r.options(numPartitions=2, skipped=None) is r
+    assert r.schema("v LONG") is r
+    assert r.name("src_1") is r
+    with pytest.raises(ValueError):
+        r.name("bad name")
+    assert r._format == "rate"
+    assert r._schema == "v LONG"
+    assert r._options == {"rowsPerSecond": "5", "numPartitions": "2"}
+    r.load()
+    r.json("/j")
+    assert r._format == "json"
+    r.xml("/x", rowTag="row")
+    assert r._format == "xml" and r._options["rowTag"] == "row"
+    r.table("t")
+    assert type(r.load()).__name__ == "DataFrame"
+
+
+def test_writer_setters_mutate_and_return_self(offline_spark):
+    w = offline_spark.range(3).write
+    assert w.mode("overwrite") is w
+    assert w.format("parquet") is w
+    assert w.option("a", "b") is w
+    assert w.option("skipped", None) is w
+    assert w.options(c="d") is w
+    assert w.partitionBy("id") is w
+    assert w.bucketBy(2, "id") is w
+    assert w.sortBy("id") is w
+    assert w.clusterBy("id") is w
+    # A failed format call (bad partitionBy) leaves the writer usable.
+    with pytest.raises(Exception):
+        w.parquet("/p", partitionBy=123)
+    assert w.mode("append") is w
+
+
+def test_writer_v2_setters_mutate_and_return_self(offline_spark):
+    from pyspark.sql import functions as F
+
+    w = offline_spark.range(3).writeTo("t")
+    assert w.using("parquet") is w
+    assert w.option("a", "b") is w
+    assert w.options(c="d") is w
+    assert w.tableProperty("p", "v") is w
+    assert w.partitionedBy(F.col("id")) is w
+    assert w.clusterBy("id") is w
+
+
+def test_stream_writer_setters_mutate_and_return_self(offline_spark):
+    w = offline_spark.readStream.format("rate").load().writeStream
+    assert w.outputMode("append") is w
+    assert w.format("memory") is w
+    assert w.option("a", "b") is w
+    assert w.options(c="d") is w
+    assert w.partitionBy("value") is w
+    assert w.clusterBy("value") is w
+    assert w.queryName("q") is w
+    with pytest.raises(ValueError):
+        w.trigger()
+    assert w.trigger(processingTime="1 second") is w
+    assert w.foreachBatch(lambda df, i: None) is w
+    assert w.foreach(lambda row: None) is w
+
+
+def test_merge_into_writer_mutates_and_returns_self(offline_spark):
+    # PySpark: withSchemaEvolution and every clause action change the writer 
and
+    # return that same writer; a clause object can be used more than once.
+    from pyspark.sql import functions as F
+
+    src = offline_spark.range(3)
+    m = src.mergeInto("target", F.col("id") == F.col("target.id"))
+    assert m.withSchemaEvolution() is m
+
+    matched = m.whenMatched(F.col("id") > 1)
+    assert matched.updateAll() is m
+    assert matched.update({"id": F.col("id")}) is m
+    assert matched.delete() is m
+    assert m.whenMatched().delete() is m
+
+    not_matched = m.whenNotMatched()
+    assert not_matched.insertAll() is m
+    assert not_matched.insert({"id": F.col("id")}) is m
+
+    by_source = m.whenNotMatchedBySource(F.col("target.id") < 0)
+    assert by_source.updateAll() is m
+    assert by_source.update({"id": F.lit(0)}) is m
+    assert by_source.delete() is m
+
+    # The ignore-the-return-value form keeps working on the original object.
+    m.whenNotMatched().insertAll()
+    m.withSchemaEvolution()
+    assert m.whenMatched().updateAll() is m


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to