This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 46522d78 [SPARK-61145][CONNECT][PYTHON] Let several threads use one
writer at once, in the Python client as PySpark does
46522d78 is described below
commit 46522d781c4d4ed327e14d9f8d3b8fb5b2ec4e52
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 11:30:14 2026 +0900
[SPARK-61145][CONNECT][PYTHON] Let several threads use one writer at once,
in the Python client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61145](https://issues.apache.org/jira/browse/SPARK-61145): Let
several threads use one writer at once, in the Python client as PySpark does.
Since SPARK-61052 (7b7ebbdb), the requests of the writers release the GIL
while they wait for the server. Their PyO3 methods took `&mut self`, so PyO3
held an exclusive borrow of the Python object for the whole request, and a
second thread that called a method of the same writer got `RuntimeError:
Already borrowed`.
- `DataFrameWriter`, `DataFrameWriterV2`, `MergeIntoWriter` and
`DataStreamWriter` keep their state in a `WriterState`, a `Mutex` that is held
only to copy or replace the state. Each request copies the state under the
lock, runs on the copy with the GIL released, and holds no lock while it waits.
- The builder methods (`format`, `mode`, `option`, `partitionBy`, `using`,
`withSchemaEvolution`, `trigger`, ...) and the writer methods take `&self` and
return the same Python object, as before.
- The clause methods of `MergeIntoWriter` (`whenMatched().updateAll()`,
...) apply their action to the state under the lock.
- Official PySpark 4.3.0 runs the same calls: three threads that each call
`parquet` on one `DataFrameWriter` all write, and a `format` call of the main
thread during a write returns the writer.
### Why are the changes needed?
With `w = spark.range(300000).write.mode("overwrite")`, three threads that
each run `w.parquet(path_i)` wrote one path and raised `RuntimeError: Already
borrowed` for the other two. Likewise, `w.format("parquet")` on the main thread
failed while another thread was in `w.parquet(...)`. Official PySpark writes
all three paths and both calls return.
### Does this PR introduce _any_ user-facing change?
Yes. Threads may call the methods of one writer object at once, as in
PySpark. Single-threaded behavior is unchanged.
### How was this patch tested?
- Offline `test_writers_shared_by_threads_run_their_requests_as_pyspark`:
with the mock server holding the requests, two threads call `parquet` on one
`DataFrameWriter` and one thread calls `append` on a `DataFrameWriterV2`, while
the main thread calls `format` and `using`. It fails with `Already borrowed`
before this change (the request then never reaches the server).
- Live `test_writers_shared_by_threads_write_one_at_a_time_as_pyspark`:
three threads write 300000 rows with one `DataFrameWriter` and one thread
creates a table with one `DataFrameWriterV2`, while the main thread sets the
format. All writes and the table have the rows. It fails with `Already
borrowed` before this change. Official PySpark 4.3.0 gives the same results for
the same calls on the live server (checked with the two-thread and
writer-versus-main-thread reproductions: all wri [...]
- The lib tests of both crates (590 passed), the build of all tests, also
with `wasm-udf`, and the offline Python suites (2202 passed, 8 skipped), and
the live tests of the writers, tables and streaming queries (29 passed) against
a local Spark 4.3.0 server; rustfmt passes, and clippy reports no diagnostics
in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #334 from HyukjinKwon/SPARK-61145-shared-writers-threads.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/dataframe.rs | 280 ++++++++++++++++++++----------------
crates/pyspark-rs/src/streaming.rs | 99 +++++++------
python/tests/test_dropin_e2e.py | 23 +++
python/tests/test_dropin_offline.py | 27 ++++
4 files changed, 256 insertions(+), 173 deletions(-)
diff --git a/crates/pyspark-rs/src/dataframe.rs
b/crates/pyspark-rs/src/dataframe.rs
index 52aa457d..26a96332 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -2419,7 +2419,7 @@ impl PyDataFrame {
#[getter]
fn write(&self) -> PyDataFrameWriter {
PyDataFrameWriter {
- inner: self.dataframe.write(),
+ inner: WriterState::new(self.dataframe.write()),
}
}
@@ -2427,7 +2427,7 @@ impl PyDataFrame {
#[pyo3(name = "writeTo")]
fn write_to(&self, table: &str) -> PyDataFrameWriterV2 {
PyDataFrameWriterV2 {
- inner: self.dataframe.write_to(table),
+ inner: WriterState::new(self.dataframe.write_to(table)),
}
}
@@ -2453,7 +2453,7 @@ impl PyDataFrame {
#[pyo3(name = "mergeInto")]
fn merge_into(&self, table: &str, condition: &PyColumn) ->
PyMergeIntoWriter {
PyMergeIntoWriter {
- inner: self.dataframe.merge_into(table, condition.column.clone()),
+ inner: WriterState::new(self.dataframe.merge_into(table,
condition.column.clone())),
}
}
@@ -3004,7 +3004,7 @@ pub(crate) fn column_or_name(col: &Bound<'_, PyAny>) ->
PyResult<spark_connect::
/// replaced in place and `merge` runs on a clone.
#[pyclass(name = "MergeIntoWriter", module = "pyspark.sql.merge")]
pub struct PyMergeIntoWriter {
- inner: spark_connect::MergeIntoWriter,
+ inner: WriterState<spark_connect::MergeIntoWriter>,
}
/// Apply a clause action to the parent writer in place and return that same
writer
@@ -3014,10 +3014,7 @@ fn merge_clause_action(
writer: &Py<PyMergeIntoWriter>,
action: impl FnOnce(spark_connect::MergeIntoWriter) ->
spark_connect::MergeIntoWriter,
) -> PyResult<Py<PyMergeIntoWriter>> {
- {
- let mut w = writer.bind(py).try_borrow_mut()?;
- w.inner = action(w.inner.clone());
- }
+ writer.bind(py).borrow().inner.update(action);
Ok(writer.clone_ref(py))
}
@@ -3070,13 +3067,13 @@ impl PyMergeIntoWriter {
}
#[pyo3(name = "withSchemaEvolution")]
- fn with_schema_evolution(mut slf: PyRefMut<'_, Self>) -> PyRefMut<'_,
Self> {
- slf.inner = slf.inner.clone().with_schema_evolution();
- slf
+ fn with_schema_evolution<'py>(slf: &Bound<'py, Self>) -> Bound<'py, Self> {
+ slf.borrow().inner.update(|w| w.with_schema_evolution());
+ slf.clone()
}
fn merge(&self, py: Python<'_>) -> PyResult<()> {
- let writer = self.inner.clone();
+ let writer = self.inner.get();
detached(py, || writer.merge().map_err(Box::new))
}
}
@@ -3191,7 +3188,7 @@ impl PyWhenNotMatchedBySource {
/// Python wrapper for the V1 `DataFrameWriter` (`df.write`).
#[pyclass(name = "DataFrameWriter", module = "pyspark.sql.readwriter")]
pub struct PyDataFrameWriter {
- inner: spark_connect::readwriter::DataFrameWriter,
+ inner: WriterState<spark_connect::readwriter::DataFrameWriter>,
}
impl PyDataFrameWriter {}
@@ -3235,41 +3232,35 @@ fn apply_write_common(
#[pymethods]
impl PyDataFrameWriter {
/// The save mode, as PySpark's: `None` keeps the default, "error".
- fn mode<'py>(
- mut slf: PyRefMut<'py, Self>,
- saveMode: Option<&str>,
- ) -> PyResult<PyRefMut<'py, Self>> {
+ fn mode<'py>(slf: &Bound<'py, Self>, saveMode: Option<&str>) ->
PyResult<Bound<'py, Self>> {
if let Some(save_mode) = saveMode {
- slf.inner = slf.inner.clone().mode(save_mode);
+ slf.borrow().inner.update(|w| w.mode(save_mode));
}
- Ok(slf)
+ Ok(slf.clone())
}
/// The format of the write. As PySpark's, None unsets it, so that the
server writes its
/// default source.
- fn format<'py>(
- mut slf: PyRefMut<'py, Self>,
- source: Option<&str>,
- ) -> PyResult<PyRefMut<'py, Self>> {
- slf.inner = match source {
- Some(source) => slf.inner.clone().format(source),
- None => slf.inner.clone().clear_format(),
- };
- Ok(slf)
+ fn format<'py>(slf: &Bound<'py, Self>, source: Option<&str>) ->
PyResult<Bound<'py, Self>> {
+ match source {
+ Some(source) => slf.borrow().inner.update(|w| w.format(source)),
+ None => slf.borrow().inner.update(|w| w.clear_format()),
+ }
+ Ok(slf.clone())
}
fn option<'py>(
- mut slf: PyRefMut<'py, Self>,
+ slf: &Bound<'py, Self>,
key: &str,
value: &Bound<'_, PyAny>,
- ) -> PyResult<PyRefMut<'py, Self>> {
+ ) -> PyResult<Bound<'py, Self>> {
// None -> option left unset; bools -> "true"/"false" (reference
`to_str`).
match crate::coerce_option_value(value)? {
Some(v) => {
- slf.inner = slf.inner.clone().option(key, &v);
- Ok(slf)
+ slf.borrow().inner.update(|w| w.option(key, &v));
+ Ok(slf.clone())
}
- None => Ok(slf),
+ None => Ok(slf.clone()),
}
}
@@ -3277,9 +3268,9 @@ impl PyDataFrameWriter {
// skipped, booleans lowercased.
#[pyo3(signature = (**options))]
fn options<'py>(
- mut slf: PyRefMut<'py, Self>,
+ slf: &Bound<'py, Self>,
options: Option<&Bound<'_, PyDict>>,
- ) -> PyResult<PyRefMut<'py, Self>> {
+ ) -> PyResult<Bound<'py, Self>> {
let mut map = std::collections::HashMap::new();
if let Some(options) = options {
for (k, v) in options.iter() {
@@ -3288,50 +3279,53 @@ impl PyDataFrameWriter {
}
}
}
- slf.inner = slf.inner.clone().options(map);
- Ok(slf)
+ slf.borrow().inner.update(|w| w.options(map));
+ Ok(slf.clone())
}
#[pyo3(name = "partitionBy", signature = (*cols))]
fn partition_by<'py>(
- mut slf: PyRefMut<'py, Self>,
+ slf: &Bound<'py, Self>,
cols: Vec<Bound<'_, PyAny>>,
- ) -> PyResult<PyRefMut<'py, Self>> {
- slf.inner = slf.inner.clone().partition_by(flatten_str_cols(cols)?);
- Ok(slf)
+ ) -> PyResult<Bound<'py, Self>> {
+ let cols = flatten_str_cols(cols)?;
+ slf.borrow().inner.update(|w| w.partition_by(cols));
+ Ok(slf.clone())
}
#[pyo3(name = "bucketBy", signature = (numBuckets, col, *cols))]
fn bucket_by<'py>(
- mut slf: PyRefMut<'py, Self>,
+ slf: &Bound<'py, Self>,
numBuckets: &Bound<'_, PyAny>,
col: &Bound<'_, PyAny>,
cols: Vec<Bound<'_, PyAny>>,
- ) -> PyResult<PyRefMut<'py, Self>> {
+ ) -> PyResult<Bound<'py, Self>> {
let num_buckets = crate::readwriter::bucket_count(numBuckets)?;
let columns = crate::readwriter::writer_columns(col, &cols)?;
- slf.inner = slf.inner.clone().bucket_by(num_buckets, columns);
- Ok(slf)
+ slf.borrow()
+ .inner
+ .update(|w| w.bucket_by(num_buckets, columns));
+ Ok(slf.clone())
}
#[pyo3(name = "sortBy", signature = (col, *cols))]
fn sort_by<'py>(
- mut slf: PyRefMut<'py, Self>,
+ slf: &Bound<'py, Self>,
col: &Bound<'_, PyAny>,
cols: Vec<Bound<'_, PyAny>>,
- ) -> PyResult<PyRefMut<'py, Self>> {
+ ) -> PyResult<Bound<'py, Self>> {
let columns = crate::readwriter::writer_columns(col, &cols)?;
- slf.inner = slf.inner.clone().sort_by(columns);
- Ok(slf)
+ slf.borrow().inner.update(|w| w.sort_by(columns));
+ Ok(slf.clone())
}
/// The clustering columns, as PySpark's: column names, or a list or tuple
of them, at
/// least one.
#[pyo3(name = "clusterBy", signature = (*cols))]
fn cluster_by<'py>(
- mut slf: PyRefMut<'py, Self>,
+ slf: &Bound<'py, Self>,
cols: Vec<Bound<'_, PyAny>>,
- ) -> PyResult<PyRefMut<'py, Self>> {
+ ) -> PyResult<Bound<'py, Self>> {
let cols: Vec<String> = match cols.as_slice() {
[only] if only.is_instance_of::<PyList>() ||
only.is_instance_of::<PyTuple>() => {
only.extract()?
@@ -3343,15 +3337,15 @@ impl PyDataFrameWriter {
"clusterBy needs one or more clustering columns.",
));
}
- slf.inner = slf.inner.clone().cluster_by(cols);
- Ok(slf)
+ slf.borrow().inner.update(|w| w.cluster_by(cols));
+ Ok(slf.clone())
}
/// Write to an external JDBC table. Connection details go through
`properties`, options of
/// any type (None skipped, bools lowercased), as PySpark's.
#[pyo3(signature = (url, table, mode=None, properties=None))]
fn jdbc(
- &mut self,
+ &self,
py: Python<'_>,
url: &str,
table: &str,
@@ -3360,7 +3354,7 @@ impl PyDataFrameWriter {
) -> PyResult<()> {
// Same order as PySpark: format, mode, properties, then url/dbtable
(which
// win over same-named properties), all kept on the writer, then save.
- let mut w = self.inner.clone().format("jdbc");
+ let mut w = self.inner.get().format("jdbc");
if let Some(m) = mode {
w = w.mode(m);
}
@@ -3372,14 +3366,14 @@ impl PyDataFrameWriter {
}
}
w = w.option("url", url).option("dbtable", table);
- self.inner = w.clone();
+ self.inner.set(w.clone());
detached(py, || w.save(None).map_err(Box::new))
}
#[pyo3(signature = (path=None, format=None, mode=None, partitionBy=None,
**options))]
#[allow(non_snake_case)]
fn save(
- &mut self,
+ &self,
py: Python<'_>,
path: Option<String>,
format: Option<&str>,
@@ -3387,16 +3381,16 @@ impl PyDataFrameWriter {
partitionBy: Option<&Bound<'_, PyAny>>,
options: Option<&Bound<'_, PyDict>>,
) -> PyResult<()> {
- let w = apply_write_common(self.inner.clone(), format, mode,
partitionBy, options)?;
+ let w = apply_write_common(self.inner.get(), format, mode,
partitionBy, options)?;
// PySpark sets the inline format/mode/partitionBy/options on the
writer itself.
- self.inner = w.clone();
+ self.inner.set(w.clone());
detached(py, || w.save(path.as_deref()).map_err(Box::new))
}
#[pyo3(name = "saveAsTable", signature = (name, format=None, mode=None,
partitionBy=None, **options))]
#[allow(non_snake_case)]
fn save_as_table(
- &mut self,
+ &self,
py: Python<'_>,
name: &str,
format: Option<&str>,
@@ -3404,8 +3398,8 @@ impl PyDataFrameWriter {
partitionBy: Option<&Bound<'_, PyAny>>,
options: Option<&Bound<'_, PyDict>>,
) -> PyResult<()> {
- let w = apply_write_common(self.inner.clone(), format, mode,
partitionBy, options)?;
- self.inner = w.clone();
+ let w = apply_write_common(self.inner.get(), format, mode,
partitionBy, options)?;
+ self.inner.set(w.clone());
detached(py, || w.save_as_table(name).map_err(Box::new))
}
@@ -3413,17 +3407,17 @@ impl PyDataFrameWriter {
/// mirroring reference `DataFrameWriter.insertInto(tableName,
overwrite=None)`.
#[pyo3(name = "insertInto", signature = (tableName, overwrite=None))]
fn insert_into(
- &mut self,
+ &self,
py: Python<'_>,
tableName: &str,
overwrite: Option<bool>,
) -> PyResult<()> {
- let mut w = self.inner.clone();
+ let mut w = self.inner.get();
if let Some(ow) = overwrite {
w = w.mode(if ow { "overwrite" } else { "append" });
}
// PySpark sets the overwrite/append mode on the writer itself.
- self.inner = w.clone();
+ self.inner.set(w.clone());
detached(py, || w.insert_into(tableName).map_err(Box::new))
}
@@ -3432,14 +3426,14 @@ impl PyDataFrameWriter {
#[pyo3(signature = (path, mode=None, partitionBy=None, compression=None))]
#[allow(non_snake_case, clippy::too_many_arguments)]
fn parquet(
- &mut self,
+ &self,
py: Python<'_>,
path: &str,
mode: Option<&Bound<'_, PyAny>>,
partitionBy: Option<&Bound<'_, PyAny>>,
compression: Option<&Bound<'_, PyAny>>,
) -> PyResult<()> {
- let mut w = self.inner.clone();
+ let mut w = self.inner.get();
if let Some(m) = mode {
w = w.mode(&m.str()?.to_string());
}
@@ -3453,7 +3447,7 @@ impl PyDataFrameWriter {
}
w = wset_opt(w, "compression", compression)?;
// PySpark sets the mode/options/format on the writer itself, then
saves.
- self.inner = w.clone().format("parquet");
+ self.inner.set(w.clone().format("parquet"));
detached(py, || w.parquet(path).map_err(Box::new))
}
@@ -3462,7 +3456,7 @@ impl PyDataFrameWriter {
#[pyo3(signature = (path, mode=None, compression=None, dateFormat=None,
timestampFormat=None, lineSep=None, encoding=None, ignoreNullFields=None))]
#[allow(non_snake_case, clippy::too_many_arguments)]
fn json(
- &mut self,
+ &self,
py: Python<'_>,
path: &str,
mode: Option<&Bound<'_, PyAny>>,
@@ -3473,7 +3467,7 @@ impl PyDataFrameWriter {
encoding: Option<&Bound<'_, PyAny>>,
ignoreNullFields: Option<&Bound<'_, PyAny>>,
) -> PyResult<()> {
- let mut w = self.inner.clone();
+ let mut w = self.inner.get();
if let Some(m) = mode {
w = w.mode(&m.str()?.to_string());
}
@@ -3484,7 +3478,7 @@ impl PyDataFrameWriter {
w = wset_opt(w, "encoding", encoding)?;
w = wset_opt(w, "ignoreNullFields", ignoreNullFields)?;
// PySpark sets the mode/options/format on the writer itself, then
saves.
- self.inner = w.clone().format("json");
+ self.inner.set(w.clone().format("json"));
detached(py, || w.json(path).map_err(Box::new))
}
@@ -3493,7 +3487,7 @@ impl PyDataFrameWriter {
#[pyo3(signature = (path, mode=None, compression=None, sep=None,
quote=None, escape=None, header=None, nullValue=None, escapeQuotes=None,
quoteAll=None, dateFormat=None, timestampFormat=None,
ignoreLeadingWhiteSpace=None, ignoreTrailingWhiteSpace=None,
charToEscapeQuoteEscaping=None, encoding=None, emptyValue=None, lineSep=None))]
#[allow(non_snake_case, clippy::too_many_arguments)]
fn csv(
- &mut self,
+ &self,
py: Python<'_>,
path: &str,
mode: Option<&Bound<'_, PyAny>>,
@@ -3514,7 +3508,7 @@ impl PyDataFrameWriter {
emptyValue: Option<&Bound<'_, PyAny>>,
lineSep: Option<&Bound<'_, PyAny>>,
) -> PyResult<()> {
- let mut w = self.inner.clone();
+ let mut w = self.inner.get();
if let Some(m) = mode {
w = w.mode(&m.str()?.to_string());
}
@@ -3535,7 +3529,7 @@ impl PyDataFrameWriter {
w = wset_opt(w, "emptyValue", emptyValue)?;
w = wset_opt(w, "lineSep", lineSep)?;
// PySpark sets the mode/options/format on the writer itself, then
saves.
- self.inner = w.clone().format("csv");
+ self.inner.set(w.clone().format("csv"));
detached(py, || w.csv(path).map_err(Box::new))
}
@@ -3544,14 +3538,14 @@ impl PyDataFrameWriter {
#[pyo3(signature = (path, mode=None, partitionBy=None, compression=None))]
#[allow(non_snake_case, clippy::too_many_arguments)]
fn orc(
- &mut self,
+ &self,
py: Python<'_>,
path: &str,
mode: Option<&Bound<'_, PyAny>>,
partitionBy: Option<&Bound<'_, PyAny>>,
compression: Option<&Bound<'_, PyAny>>,
) -> PyResult<()> {
- let mut w = self.inner.clone();
+ let mut w = self.inner.get();
if let Some(m) = mode {
w = w.mode(&m.str()?.to_string());
}
@@ -3565,7 +3559,7 @@ impl PyDataFrameWriter {
}
w = wset_opt(w, "compression", compression)?;
// PySpark sets the mode/options/format on the writer itself, then
saves.
- self.inner = w.clone().format("orc");
+ self.inner.set(w.clone().format("orc"));
detached(py, || w.orc(path).map_err(Box::new))
}
@@ -3574,17 +3568,17 @@ impl PyDataFrameWriter {
#[pyo3(signature = (path, compression=None, lineSep=None))]
#[allow(non_snake_case, clippy::too_many_arguments)]
fn text(
- &mut self,
+ &self,
py: Python<'_>,
path: &str,
compression: Option<&Bound<'_, PyAny>>,
lineSep: Option<&Bound<'_, PyAny>>,
) -> PyResult<()> {
- let mut w = self.inner.clone();
+ let mut w = self.inner.get();
w = wset_opt(w, "compression", compression)?;
w = wset_opt(w, "lineSep", lineSep)?;
// PySpark sets the mode/options/format on the writer itself, then
saves.
- self.inner = w.clone().format("text");
+ self.inner.set(w.clone().format("text"));
detached(py, || w.text(path).map_err(Box::new))
}
@@ -3593,7 +3587,7 @@ impl PyDataFrameWriter {
#[pyo3(signature = (path, rowTag=None, mode=None, attributePrefix=None,
valueTag=None, rootTag=None, declaration=None, arrayElementName=None,
nullValue=None, dateFormat=None, timestampFormat=None, compression=None,
encoding=None, validateName=None))]
#[allow(non_snake_case, clippy::too_many_arguments)]
fn xml(
- &mut self,
+ &self,
py: Python<'_>,
path: &str,
rowTag: Option<&Bound<'_, PyAny>>,
@@ -3610,7 +3604,7 @@ impl PyDataFrameWriter {
encoding: Option<&Bound<'_, PyAny>>,
validateName: Option<&Bound<'_, PyAny>>,
) -> PyResult<()> {
- let mut w = self.inner.clone();
+ let mut w = self.inner.get();
w = wset_opt(w, "rowTag", rowTag)?;
if let Some(m) = mode {
w = w.mode(&m.str()?.to_string());
@@ -3627,7 +3621,7 @@ impl PyDataFrameWriter {
w = wset_opt(w, "encoding", encoding)?;
w = wset_opt(w, "validateName", validateName)?;
// PySpark sets the mode/options/format on the writer itself, then
saves.
- self.inner = w.clone().format("xml");
+ self.inner.set(w.clone().format("xml"));
detached(py, || w.xml(path).map_err(Box::new))
}
}
@@ -3635,40 +3629,40 @@ impl PyDataFrameWriter {
/// Python wrapper for the V2 `DataFrameWriter` (`df.writeTo`).
#[pyclass(name = "DataFrameWriterV2", module = "pyspark.sql.readwriter")]
pub struct PyDataFrameWriterV2 {
- inner: spark_connect::readwriter::DataFrameWriterV2,
+ inner: WriterState<spark_connect::readwriter::DataFrameWriterV2>,
}
impl PyDataFrameWriterV2 {}
#[pymethods]
impl PyDataFrameWriterV2 {
- fn using<'py>(mut slf: PyRefMut<'py, Self>, provider: &str) ->
PyResult<PyRefMut<'py, Self>> {
- slf.inner = slf.inner.clone().using(provider);
- Ok(slf)
+ fn using<'py>(slf: &Bound<'py, Self>, provider: &str) ->
PyResult<Bound<'py, Self>> {
+ slf.borrow().inner.update(|w| w.using(provider));
+ Ok(slf.clone())
}
// None -> option left unset; bools -> "true"/"false" (reference
`to_str`), same as
// the other reader/writer option bindings.
fn option<'py>(
- mut slf: PyRefMut<'py, Self>,
+ slf: &Bound<'py, Self>,
key: &str,
value: &Bound<'_, PyAny>,
- ) -> PyResult<PyRefMut<'py, Self>> {
+ ) -> PyResult<Bound<'py, Self>> {
match crate::coerce_option_value(value)? {
Some(v) => {
- slf.inner = slf.inner.clone().option(key, &v);
- Ok(slf)
+ slf.borrow().inner.update(|w| w.option(key, &v));
+ Ok(slf.clone())
}
- None => Ok(slf),
+ None => Ok(slf.clone()),
}
}
// Mirrors reference `DataFrameWriterV2.options(**options)`; None skipped,
bools lowercased.
#[pyo3(signature = (**options))]
fn options<'py>(
- mut slf: PyRefMut<'py, Self>,
+ slf: &Bound<'py, Self>,
options: Option<&Bound<'_, PyDict>>,
- ) -> PyResult<PyRefMut<'py, Self>> {
+ ) -> PyResult<Bound<'py, Self>> {
let mut map = std::collections::HashMap::new();
if let Some(options) = options {
for (k, v) in options.iter() {
@@ -3677,87 +3671,127 @@ impl PyDataFrameWriterV2 {
}
}
}
- slf.inner = slf.inner.clone().options(map);
- Ok(slf)
+ slf.borrow().inner.update(|w| w.options(map));
+ Ok(slf.clone())
}
/// Set the table property `property` to `value`, or leave it unset for
None, as PySpark's.
#[pyo3(name = "tableProperty")]
fn table_property<'py>(
- mut slf: PyRefMut<'py, Self>,
+ slf: &Bound<'py, Self>,
property: &str,
value: Option<&str>,
- ) -> PyResult<PyRefMut<'py, Self>> {
- slf.inner = match value {
- Some(value) => slf.inner.clone().table_property(property, value),
- None => slf.inner.clone().unset_table_property(property),
- };
- Ok(slf)
+ ) -> PyResult<Bound<'py, Self>> {
+ match value {
+ Some(value) => slf
+ .borrow()
+ .inner
+ .update(|w| w.table_property(property, value)),
+ None => slf
+ .borrow()
+ .inner
+ .update(|w| w.unset_table_property(property)),
+ }
+ Ok(slf.clone())
}
/// Partition by the columns, each a Column or a column name, as PySpark's
`F._to_col` of
/// each, else `NOT_EXPECTED_TYPE` of `col`, also for a list.
#[pyo3(name = "partitionedBy", signature = (col, *cols))]
fn partitioned_by<'py>(
- mut slf: PyRefMut<'py, Self>,
+ slf: &Bound<'py, Self>,
col: Bound<'_, PyAny>,
cols: Vec<Bound<'_, PyAny>>,
- ) -> PyResult<PyRefMut<'py, Self>> {
+ ) -> PyResult<Bound<'py, Self>> {
let columns = std::iter::once(&col)
.chain(&cols)
.map(column_or_name)
.collect::<PyResult<Vec<_>>>()?;
- slf.inner = slf.inner.clone().partition_by(columns);
- Ok(slf)
+ slf.borrow().inner.update(|w| w.partition_by(columns));
+ Ok(slf.clone())
}
#[pyo3(name = "clusterBy", signature = (col, *cols))]
fn cluster_by<'py>(
- mut slf: PyRefMut<'py, Self>,
+ slf: &Bound<'py, Self>,
col: String,
cols: Vec<String>,
- ) -> PyResult<PyRefMut<'py, Self>> {
+ ) -> PyResult<Bound<'py, Self>> {
let mut all = vec![col];
all.extend(cols);
- slf.inner = slf.inner.clone().cluster_by(all);
- Ok(slf)
+ slf.borrow().inner.update(|w| w.cluster_by(all));
+ Ok(slf.clone())
}
- fn create(&mut self, py: Python<'_>) -> PyResult<()> {
- let writer = self.inner.clone();
+ fn create(&self, py: Python<'_>) -> PyResult<()> {
+ let writer = self.inner.get();
detached(py, || writer.create().map_err(Box::new))
}
- fn replace(&mut self, py: Python<'_>) -> PyResult<()> {
- let writer = self.inner.clone();
+ fn replace(&self, py: Python<'_>) -> PyResult<()> {
+ let writer = self.inner.get();
detached(py, || writer.replace().map_err(Box::new))
}
#[pyo3(name = "createOrReplace")]
- fn create_or_replace(&mut self, py: Python<'_>) -> PyResult<()> {
- let writer = self.inner.clone();
+ fn create_or_replace(&self, py: Python<'_>) -> PyResult<()> {
+ let writer = self.inner.get();
detached(py, || writer.create_or_replace().map_err(Box::new))
}
- fn append(&mut self, py: Python<'_>) -> PyResult<()> {
- let writer = self.inner.clone();
+ fn append(&self, py: Python<'_>) -> PyResult<()> {
+ let writer = self.inner.get();
detached(py, || writer.append().map_err(Box::new))
}
/// Overwrite the rows of `condition`, a Column or a column name, as
PySpark's `F._to_col`,
/// else `NOT_EXPECTED_TYPE` of `col`.
- fn overwrite(&mut self, py: Python<'_>, condition: &Bound<'_, PyAny>) ->
PyResult<()> {
- let (writer, condition) = (self.inner.clone(),
column_or_name(condition)?);
+ fn overwrite(&self, py: Python<'_>, condition: &Bound<'_, PyAny>) ->
PyResult<()> {
+ let (writer, condition) = (self.inner.get(),
column_or_name(condition)?);
detached(py, || writer.overwrite(condition).map_err(Box::new))
}
#[pyo3(name = "overwritePartitions")]
- fn overwrite_partitions(&mut self, py: Python<'_>) -> PyResult<()> {
- let writer = self.inner.clone();
+ fn overwrite_partitions(&self, py: Python<'_>) -> PyResult<()> {
+ let writer = self.inner.get();
detached(py, || writer.overwrite_partitions().map_err(Box::new))
}
}
+/// The state of a Python writer (`df.write`, `df.writeTo`, `df.mergeInto`,
`df.writeStream`).
+/// PySpark's writers are plain Python objects, so their methods may run on
several threads at
+/// once; a method that releases the GIL for its request must not hold a Rust
borrow of the
+/// object (that would make the calls of the other threads fail with "Already
borrowed"). So
+/// the lock is held only to copy or replace the state, and a request runs on
its own copy.
+pub(crate) struct WriterState<T>(std::sync::Mutex<T>);
+
+impl<T: Clone> WriterState<T> {
+ pub(crate) fn new(state: T) -> Self {
+ WriterState(std::sync::Mutex::new(state))
+ }
+
+ fn lock(&self) -> std::sync::MutexGuard<'_, T> {
+ self.0
+ .lock()
+ .unwrap_or_else(std::sync::PoisonError::into_inner)
+ }
+
+ /// A copy of the state, to run a request or a builder step on.
+ pub(crate) fn get(&self) -> T {
+ self.lock().clone()
+ }
+
+ pub(crate) fn set(&self, state: T) {
+ *self.lock() = state;
+ }
+
+ /// Replace the state by `f` of it, as the builder methods do (`f` only
builds, no Python).
+ pub(crate) fn update(&self, f: impl FnOnce(T) -> T) {
+ let mut state = self.lock();
+ *state = f(state.clone());
+ }
+}
+
/// `f` without the GIL, as PySpark's requests to the server release it while
they wait, so
/// that other Python threads run. `f` boxes its error, which keeps its result
small.
pub(crate) fn detached<T: Send>(
diff --git a/crates/pyspark-rs/src/streaming.rs
b/crates/pyspark-rs/src/streaming.rs
index 5de4b144..cc47079a 100644
--- a/crates/pyspark-rs/src/streaming.rs
+++ b/crates/pyspark-rs/src/streaming.rs
@@ -10,7 +10,7 @@ use spark_connect::udf::PythonUDFPayload;
use std::collections::HashMap;
use std::sync::Arc;
-use crate::dataframe::PyDataFrame;
+use crate::dataframe::{PyDataFrame, WriterState};
use crate::errors::ResultExt;
/// Adapts a Python `StreamingQueryListener` to the native Rust listener
trait: on each
@@ -604,26 +604,28 @@ impl PyTrigger {
/// Python wrapper for DataStreamWriter.
#[pyclass(name = "DataStreamWriter", module =
"pyspark.sql.streaming.readwriter")]
pub struct PyDataStreamWriter {
- inner: DataStreamWriter,
+ inner: WriterState<DataStreamWriter>,
}
impl PyDataStreamWriter {
pub fn new(writer: DataStreamWriter) -> Self {
- PyDataStreamWriter { inner: writer }
+ PyDataStreamWriter {
+ inner: WriterState::new(writer),
+ }
}
/// Apply the one-shot writer options accepted by `start`/`toTable`
/// (`format`, `outputMode`, `partitionBy`, `queryName`, `**options`)
before
/// terminating the stream. Mirrors the reference, where these are keyword
args.
fn apply_write_opts(
- &mut self,
+ &self,
format: Option<&str>,
output_mode: Option<&str>,
partition_by: Option<&Bound<'_, PyAny>>,
query_name: Option<&Bound<'_, PyAny>>,
options: Option<&Bound<'_, PyDict>>,
) -> PyResult<DataStreamWriter> {
- let mut w = self.inner.clone();
+ let mut w = self.inner.get();
if let Some(fmt) = format {
w = w.format(fmt);
}
@@ -648,38 +650,35 @@ impl PyDataStreamWriter {
w = w.options(opts);
}
// PySpark's start/toTable set these on the writer itself before
starting.
- self.inner = w.clone();
+ self.inner.set(w.clone());
Ok(w)
}
}
#[pymethods]
impl PyDataStreamWriter {
- fn outputMode<'py>(
- mut slf: PyRefMut<'py, Self>,
- outputMode: &str,
- ) -> PyResult<PyRefMut<'py, Self>> {
- slf.inner = slf.inner.clone().output_mode(outputMode);
- Ok(slf)
+ fn outputMode<'py>(slf: &Bound<'py, Self>, outputMode: &str) ->
PyResult<Bound<'py, Self>> {
+ slf.borrow().inner.update(|w| w.output_mode(outputMode));
+ Ok(slf.clone())
}
- fn format<'py>(mut slf: PyRefMut<'py, Self>, source: &str) ->
PyResult<PyRefMut<'py, Self>> {
- slf.inner = slf.inner.clone().format(source);
- Ok(slf)
+ fn format<'py>(slf: &Bound<'py, Self>, source: &str) ->
PyResult<Bound<'py, Self>> {
+ slf.borrow().inner.update(|w| w.format(source));
+ Ok(slf.clone())
}
fn option<'py>(
- mut slf: PyRefMut<'py, Self>,
+ slf: &Bound<'py, Self>,
key: &str,
value: &Bound<'_, PyAny>,
- ) -> PyResult<PyRefMut<'py, Self>> {
+ ) -> PyResult<Bound<'py, Self>> {
// None -> option left unset; bools -> "true"/"false" (reference
`to_str`).
match crate::coerce_option_value(value)? {
Some(v) => {
- slf.inner = slf.inner.clone().option(key, &v);
- Ok(slf)
+ slf.borrow().inner.update(|w| w.option(key, &v));
+ Ok(slf.clone())
}
- None => Ok(slf),
+ None => Ok(slf.clone()),
}
}
@@ -687,9 +686,9 @@ impl PyDataStreamWriter {
// skipped, booleans lowercased.
#[pyo3(signature = (**options))]
fn options<'py>(
- mut slf: PyRefMut<'py, Self>,
+ slf: &Bound<'py, Self>,
options: Option<&Bound<'_, PyDict>>,
- ) -> PyResult<PyRefMut<'py, Self>> {
+ ) -> PyResult<Bound<'py, Self>> {
let mut opts = HashMap::new();
if let Some(options) = options {
for (k, v) in options.iter() {
@@ -698,42 +697,42 @@ impl PyDataStreamWriter {
}
}
}
- slf.inner = slf.inner.clone().options(opts);
- Ok(slf)
+ slf.borrow().inner.update(|w| w.options(opts));
+ Ok(slf.clone())
}
/// Partition by the column names, or a list or tuple of them alone, as
PySpark's.
#[pyo3(signature = (*cols))]
fn partitionBy<'py>(
- mut slf: PyRefMut<'py, Self>,
+ slf: &Bound<'py, Self>,
cols: &Bound<'_, PyTuple>,
- ) -> PyResult<PyRefMut<'py, Self>> {
+ ) -> PyResult<Bound<'py, Self>> {
let cols = column_names(cols)?;
let col_refs: Vec<&str> = cols.iter().map(|s| s.as_str()).collect();
- slf.inner = slf.inner.clone().partition_by(col_refs);
- Ok(slf)
+ slf.borrow().inner.update(|w| w.partition_by(col_refs));
+ Ok(slf.clone())
}
/// Cluster by the column names, or a list or tuple of them alone, as
PySpark's.
#[pyo3(signature = (*cols))]
fn clusterBy<'py>(
- mut slf: PyRefMut<'py, Self>,
+ slf: &Bound<'py, Self>,
cols: &Bound<'_, PyTuple>,
- ) -> PyResult<PyRefMut<'py, Self>> {
+ ) -> PyResult<Bound<'py, Self>> {
let cols = column_names(cols)?;
let col_refs: Vec<&str> = cols.iter().map(|s| s.as_str()).collect();
- slf.inner = slf.inner.clone().cluster_by(col_refs);
- Ok(slf)
+ slf.borrow().inner.update(|w| w.cluster_by(col_refs));
+ Ok(slf.clone())
}
/// Name the query, a non-empty str, else `VALUE_NOT_NON_EMPTY_STR`, as
PySpark's.
fn queryName<'py>(
- mut slf: PyRefMut<'py, Self>,
+ slf: &Bound<'py, Self>,
queryName: &Bound<'_, PyAny>,
- ) -> PyResult<PyRefMut<'py, Self>> {
+ ) -> PyResult<Bound<'py, Self>> {
let name = query_name_of(queryName)?;
- slf.inner = slf.inner.clone().query_name(&name);
- Ok(slf)
+ slf.borrow().inner.update(|w| w.query_name(&name));
+ Ok(slf.clone())
}
/// `DataStreamWriter.trigger(...)`, as PySpark's: exactly one of
`processingTime`, `once`,
@@ -743,13 +742,13 @@ impl PyDataStreamWriter {
/// `VALUE_NOT_ALLOWED`.
#[pyo3(signature = (*, processingTime=None, once=None, continuous=None,
availableNow=None, realTime=None))]
fn trigger<'py>(
- mut slf: PyRefMut<'py, Self>,
+ slf: &Bound<'py, Self>,
processingTime: Option<Bound<'py, PyAny>>,
once: Option<Bound<'py, PyAny>>,
continuous: Option<Bound<'py, PyAny>>,
availableNow: Option<Bound<'py, PyAny>>,
realTime: Option<Bound<'py, PyAny>>,
- ) -> PyResult<PyRefMut<'py, Self>> {
+ ) -> PyResult<Bound<'py, Self>> {
use crate::column::raise_pyspark;
let py = slf.py();
let given = [
@@ -813,14 +812,14 @@ impl PyDataStreamWriter {
}
Trigger::AvailableNow
};
- slf.inner = slf.inner.clone().trigger(trigger);
- Ok(slf)
+ slf.borrow().inner.update(|w| w.trigger(trigger));
+ Ok(slf.clone())
}
#[pyo3(signature = (path=None, format=None, outputMode=None,
partitionBy=None, queryName=None, **options))]
#[allow(non_snake_case, clippy::too_many_arguments)]
fn start(
- &mut self,
+ &self,
py: Python<'_>,
path: Option<&str>,
format: Option<&str>,
@@ -839,7 +838,7 @@ impl PyDataStreamWriter {
#[pyo3(signature = (tableName, format=None, outputMode=None,
partitionBy=None, queryName=None, **options))]
#[allow(non_snake_case, clippy::too_many_arguments)]
fn toTable(
- &mut self,
+ &self,
py: Python<'_>,
tableName: &str,
format: Option<&str>,
@@ -859,10 +858,10 @@ impl PyDataStreamWriter {
/// foreach-batch sink. Pickling happens here so the Python skin can
re-export this
/// class directly rather than subclassing a (non-subclassable) PyO3 type.
fn foreachBatch<'py>(
- mut slf: PyRefMut<'py, Self>,
+ slf: &Bound<'py, Self>,
py: Python<'_>,
func: &Bound<'_, PyAny>,
- ) -> PyResult<PyRefMut<'py, Self>> {
+ ) -> PyResult<Bound<'py, Self>> {
let command = streaming_function_command(py, func, "foreachBatch")?;
let payload = PythonUDFPayload::new(
spark_connect::types::DataType::Struct { fields: vec![] },
@@ -870,8 +869,8 @@ impl PyDataStreamWriter {
command,
crate::dataframe::py_version(py),
);
- slf.inner = slf.inner.clone().foreach_batch(payload);
- Ok(slf)
+ slf.borrow().inner.update(|w| w.foreach_batch(payload));
+ Ok(slf.clone())
}
/// `DataStreamWriter.foreach(f)`, as PySpark's: the function of the
partition and the
@@ -881,10 +880,10 @@ impl PyDataStreamWriter {
/// which the worker of the server deserializes with its own
`pyspark.serializers`. Built
/// here to avoid a Python subclass of the PyO3 class.
fn foreach<'py>(
- mut slf: PyRefMut<'py, Self>,
+ slf: &Bound<'py, Self>,
py: Python<'_>,
f: &Bound<'_, PyAny>,
- ) -> PyResult<PyRefMut<'py, Self>> {
+ ) -> PyResult<Bound<'py, Self>> {
let func = py
.import("pyspark.sql.streaming.readwriter")?
.getattr("_construct_foreach_function")?
@@ -912,8 +911,8 @@ impl PyDataStreamWriter {
command,
crate::dataframe::py_version(py),
);
- slf.inner = slf.inner.clone().foreach(payload);
- Ok(slf)
+ slf.borrow().inner.update(|w| w.foreach(payload));
+ Ok(slf.clone())
}
}
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 18191c31..e70e9fbe 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -695,6 +695,29 @@ def
test_transform_with_state_in_pandas_takes_a_ddl_output_type(session, tmp_pat
session.conf.unset(key)
+def test_writers_shared_by_threads_write_one_at_a_time_as_pyspark(session,
tmp_path):
+ # As PySpark's: a writer is a plain Python object, so threads may write
with one
+ # DataFrameWriter or DataFrameWriterV2 at once, while the main thread sets
its format.
+ from concurrent.futures import ThreadPoolExecutor
+
+ df = session.range(300000)
+ writer = df.write.mode("overwrite")
+ table = f"r48_shared_writer_{os.getpid()}"
+ writer_v2 = df.writeTo(table).using("parquet")
+ try:
+ with ThreadPoolExecutor(max_workers=4) as pool:
+ writes = [pool.submit(writer.parquet, str(tmp_path / f"p{i}")) for
i in range(3)]
+ writes.append(pool.submit(writer_v2.create))
+ assert writer.format("parquet") is writer
+ for write in writes:
+ write.result(timeout=300)
+ for i in range(3):
+ assert session.read.parquet(str(tmp_path / f"p{i}")).count() ==
300000
+ assert session.table(table).count() == 300000
+ finally:
+ session.sql(f"DROP TABLE IF EXISTS {table}")
+
+
def test_transform_with_state_returns_the_rows_of_its_output_schema(session,
tmp_path):
# As PySpark's, transformWithState sends its output schema, a DDL string
or a
# StructType, which the server reads as the schema of the rows of the
processor.
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index bd9e48e2..a8d89486 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -1642,6 +1642,33 @@ def
test_map_functions_send_the_names_of_their_functions_as_pyspark(mock_connect
assert sent_name(stateful, 34, 3) == "with_state"
+def test_writers_shared_by_threads_run_their_requests_as_pyspark(mock_connect):
+ # As PySpark's: a writer is a plain Python object, so threads may call its
methods while
+ # a request waits for the server, which releases the GIL. Two threads save
with one
+ # DataFrameWriter and one thread appends with a DataFrameWriterV2, while
the main thread
+ # sets the format and the provider of the writers meanwhile.
+ spark, service = mock_connect
+ writer = spark.range(1).write.mode("overwrite")
+ writer_v2 = spark.range(1).writeTo("shared_writer")
+ service.block_execute = True
+ with ThreadPoolExecutor(max_workers=3) as pool:
+ requests = [pool.submit(writer.parquet, f"/tmp/shared_writer/p{i}")
for i in range(2)]
+ requests.append(pool.submit(writer_v2.append))
+ try:
+ for _ in range(500):
+ if service.execute_calls >= 3:
+ break
+ time.sleep(0.01)
+ else:
+ raise AssertionError("The writes never reached the server")
+ assert writer.format("parquet") is writer
+ assert writer_v2.using("parquet") is writer_v2
+ finally:
+ service.release_response()
+ for request in requests:
+ request.result(timeout=10)
+
+
def test_try_cast_and_isin_take_their_arguments_as_pyspark():
# As PySpark Connect's: try_cast raises NOT_EXPECTED_TYPE for a type that
is neither a
# DataType nor a str, and isin unpacks a single list or set, but not a
tuple, which is a
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]