This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 373489e [SPARK-61056][CONNECT][PYTHON] Send the function of
DataStreamWriter.foreach in the Python client as PySpark does
373489e is described below
commit 373489e8dd0e052fee004fc91c5274311fc6298c
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 12:16:07 2026 +0900
[SPARK-61056][CONNECT][PYTHON] Send the function of
DataStreamWriter.foreach in the Python client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61056](https://issues.apache.org/jira/browse/SPARK-61056):
`df.writeStream.foreach(f)` now sends the function of PySpark, so the queries
of `foreach` run.
As PySpark Connect's, `foreach(f)` sends the function of a partition and
its rows of `_construct_foreach_function(f)`, which the Python worker of the
server runs:
- for a function `f`, one that calls `f` with each row
- for an object, one that calls its `open(partition_id, epoch_id)`, its
`process(row)` with each row, and its `close(error)`
It raises `ATTRIBUTE_NOT_CALLABLE` for an object without a callable
`process`, or with an `open` or `close` that is not callable. The function is
ported from PySpark, but it imports `TaskContext` on the worker, which has it.
`foreach` and `foreachBatch` now pickle their functions with
`CloudPickleSerializer`, as PySpark's. They raise
`STREAMING_CONNECT_SERIALIZATION_ERROR` for a function that does not pickle.
### Why are the changes needed?
The client sent `f` itself, which the worker calls with a partition and its
rows:
- the query of an object failed with `TypeError: 'Writer' object is not
callable`
- that of a function failed without a batch
So `foreach` did not work.
### Does this PR introduce _any_ user-facing change?
Yes. The queries of `foreach` run, as in PySpark 4.3.0.
### How was this patch tested?
- A new live test runs `foreach` of a function and of an object of `open`,
`process` and `close`, which write a file of each row on the local server, and
checks the files. It also checks the `ATTRIBUTE_NOT_CALLABLE` of an int. It
passes with official PySpark 4.3.0, and fails without this change.
- A new offline test unpickles the command that `foreach` sends to a mock
Connect server, runs its function on rows, and checks the errors of objects
without callable methods and of functions that do not pickle.
- The lib tests of both crates (577 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (1817 passed, 8 skipped), and the
live drop-in and Arrow suites (116 passed, without the test of
`StreamingQuery.explain`, which hangs on a slow machine and #260 fixes) and
live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no
diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #264 from HyukjinKwon/SPARK-61056-foreach.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/streaming.rs | 50 ++++++++++++---
python/pyspark/sql/streaming/readwriter.py | 100 ++++++++++++++++++++++++++++-
python/tests/test_dropin_e2e.py | 45 +++++++++++++
python/tests/test_dropin_offline.py | 66 +++++++++++++++++++
4 files changed, 250 insertions(+), 11 deletions(-)
diff --git a/crates/pyspark-rs/src/streaming.rs
b/crates/pyspark-rs/src/streaming.rs
index e75107a..1a0dc49 100644
--- a/crates/pyspark-rs/src/streaming.rs
+++ b/crates/pyspark-rs/src/streaming.rs
@@ -754,7 +754,7 @@ impl PyDataStreamWriter {
}
/// `DataStreamWriter.foreachBatch(func)`: cloudpickle the `(batch_df,
batch_id)`
- /// function (via the bundled `pyspark.cloudpickle`) and attach it as the
+ /// function (with PySpark's `CloudPickleSerializer`) and attach it as the
/// foreach-batch sink. Pickling happens here so the Python skin can
re-export this
/// class directly rather than subclassing a (non-subclassable) PyO3 type.
fn foreachBatch<'py>(
@@ -762,7 +762,7 @@ impl PyDataStreamWriter {
py: Python<'_>,
func: &Bound<'_, PyAny>,
) -> PyResult<PyRefMut<'py, Self>> {
- let command = crate::dataframe::py_cloudpickle(py, func)?;
+ let command = streaming_function_command(py, func, "foreachBatch")?;
let payload = PythonUDFPayload::new(
spark_connect::types::DataType::Struct { fields: vec![] },
0, // eval type is unused for the streaming foreach sinks
@@ -773,15 +773,21 @@ impl PyDataStreamWriter {
Ok(slf)
}
- /// `DataStreamWriter.foreach(f)`: wrap the row handler as the reference
client does
- /// — `(f, None, serializer, serializer)` with
`AutoBatchedSerializer(CPickleSerializer())`
- /// — and cloudpickle it so the worker deserializes it against its own
- /// `pyspark.serializers`. Built here to avoid a Python subclass of the
PyO3 class.
+ /// `DataStreamWriter.foreach(f)`, as PySpark's: the function of the
partition and the
+ /// rows of each partition of `_construct_foreach_function(f)`, which
raises
+ /// `ATTRIBUTE_NOT_CALLABLE` for an object without a callable `process`,
in the command
+ /// `(func, None, serializer, serializer)` of
`AutoBatchedSerializer(CPickleSerializer())`,
+ /// which the worker of the server deserializes with its own
`pyspark.serializers`. Built
+ /// here to avoid a Python subclass of the PyO3 class.
fn foreach<'py>(
mut slf: PyRefMut<'py, Self>,
py: Python<'_>,
f: &Bound<'_, PyAny>,
) -> PyResult<PyRefMut<'py, Self>> {
+ let func = py
+ .import("pyspark.sql.streaming.readwriter")?
+ .getattr("_construct_foreach_function")?
+ .call1((f,))?;
let serializers = py.import("pyspark.serializers")?;
let cpickle = serializers.getattr("CPickleSerializer")?.call0()?;
let serializer = serializers
@@ -792,13 +798,13 @@ impl PyDataStreamWriter {
let command_tuple = pyo3::types::PyTuple::new(
py,
[
- f.clone(),
+ func,
py.None().into_bound(py),
serializer.clone(),
serializer,
],
)?;
- let command = crate::dataframe::py_cloudpickle(py,
command_tuple.as_any())?;
+ let command = streaming_function_command(py, command_tuple.as_any(),
"foreach")?;
let payload = PythonUDFPayload::new(
spark_connect::types::DataType::Struct { fields: vec![] },
0,
@@ -853,6 +859,34 @@ impl<'a, 'py> FromPyObject<'a, 'py> for ProtoBool {
}
}
+/// The command of the function of `foreach` or `foreachBatch`, as PySpark's:
of
+/// `CloudPickleSerializer().dumps`, which raises a `pickle.PicklingError` for
an object that
+/// does not pickle, as `STREAMING_CONNECT_SERIALIZATION_ERROR` of `name`.
+fn streaming_function_command(
+ py: Python<'_>,
+ function: &Bound<'_, PyAny>,
+ name: &str,
+) -> PyResult<Vec<u8>> {
+ let serializer = py
+ .import("pyspark.serializers")?
+ .getattr("CloudPickleSerializer")?
+ .call0()?;
+ match serializer.call_method1("dumps", (function,)) {
+ Ok(command) => command.extract(),
+ Err(error) if error.matches(py,
py.import("pickle")?.getattr("PicklingError")?)? => {
+ let raised = crate::column::raise_pyspark(
+ py,
+ "PySparkPicklingError",
+ "STREAMING_CONNECT_SERIALIZATION_ERROR",
+ &[("name", name)],
+ );
+ raised.set_cause(py, Some(error));
+ Err(raised)
+ }
+ Err(error) => Err(error),
+ }
+}
+
/// Python wrapper for StreamingQuery.
#[pyclass(name = "StreamingQuery", module = "pyspark.sql.streaming.query")]
pub struct PyStreamingQuery {
diff --git a/python/pyspark/sql/streaming/readwriter.py
b/python/pyspark/sql/streaming/readwriter.py
index 030e1bb..d48d3dc 100644
--- a/python/pyspark/sql/streaming/readwriter.py
+++ b/python/pyspark/sql/streaming/readwriter.py
@@ -19,19 +19,113 @@
Streaming data readers and writers.
Re-exports the PyO3-implemented streaming classes.
`DataStreamWriter.foreachBatch`
-and `.foreach` accept a Python callable and cloudpickle it inside the extension
-(via the bundled ``pyspark.cloudpickle``), so no Python subclass of the (non-
-subclassable) PyO3 class is needed.
+and `.foreach` accept a Python callable and cloudpickle it inside the
extension (with
+``pyspark.serializers.CloudPickleSerializer``), so no Python subclass of the
(non-
+subclassable) PyO3 class is needed; `.foreach` sends the function of
+`_construct_foreach_function`, as PySpark does.
"""
+from typing import Any, Callable, Iterator, Union, cast
+
from pyspark._pyspark import (
DataStreamReader as _DataStreamReader,
DataStreamWriter as _DataStreamWriter,
Trigger as _Trigger,
)
+from pyspark.errors import PySparkAttributeError, PySparkRuntimeError
DataStreamReader = _DataStreamReader
DataStreamWriter = _DataStreamWriter
Trigger = _Trigger
+
+def _construct_foreach_function(
+ f: Union[Callable[[Any], None], Any],
+) -> Callable[[Any, Iterator], Iterator]:
+ """
+ The function of the partition and the rows of each partition that
``foreach(f)`` sends,
+ which the Python worker of the server runs, as PySpark's
+ ``DataStreamWriter._construct_foreach_function``: for a function ``f``,
one that calls ``f``
+ with each row, and for an object, one that calls its ``open(partition_id,
epoch_id)``, its
+ ``process(row)`` with each row, and its ``close(error)``.
+
+ Ported from PySpark, but for ``TaskContext``, which the function imports
on the worker.
+ """
+ if callable(f):
+ # The provided object is a callable function that is supposed to be
called on each row.
+ # Construct a function that takes an iterator and calls the provided
function on each
+ # row.
+ def func_without_process(_: Any, iterator: Iterator) -> Iterator:
+ for x in iterator:
+ f(x)
+ return iter([])
+
+ return func_without_process
+
+ else:
+ # The provided object is not a callable function. Then it is expected
to have a
+ # 'process(row)' method, and optional 'open(partition_id, epoch_id)'
and
+ # 'close(error)' methods.
+
+ if not hasattr(f, "process"):
+ raise PySparkAttributeError(
+ errorClass="ATTRIBUTE_NOT_CALLABLE",
+ messageParameters={"attr_name": "process", "obj_name": "f"},
+ )
+
+ if not callable(getattr(f, "process")):
+ raise PySparkAttributeError(
+ errorClass="ATTRIBUTE_NOT_CALLABLE",
+ messageParameters={"attr_name": "process", "obj_name": "f"},
+ )
+
+ def doesMethodExist(method_name: str) -> bool:
+ exists = hasattr(f, method_name)
+ if exists and not callable(getattr(f, method_name)):
+ raise PySparkAttributeError(
+ errorClass="ATTRIBUTE_NOT_CALLABLE",
+ messageParameters={"attr_name": method_name, "obj_name":
"f"},
+ )
+ return exists
+
+ open_exists = doesMethodExist("open")
+ close_exists = doesMethodExist("close")
+
+ def func_with_open_process_close(partition_id: Any, iterator:
Iterator) -> Iterator:
+ from pyspark.taskcontext import TaskContext
+
+ epoch_id = cast(TaskContext, TaskContext.get()).getLocalProperty(
+ "streaming.sql.batchId"
+ )
+ if epoch_id:
+ int_epoch_id = int(epoch_id)
+ else:
+ raise PySparkRuntimeError(
+ errorClass="CANNOT_GET_BATCH_ID",
+ messageParameters={"obj_name": "TaskContext"},
+ )
+
+ # Check if the data should be processed
+ should_process = True
+ if open_exists:
+ should_process = f.open(partition_id, int_epoch_id)
+
+ error = None
+
+ try:
+ if should_process:
+ for x in iterator:
+ f.process(x)
+ except Exception as ex:
+ error = ex
+ finally:
+ if close_exists:
+ f.close(error)
+ if error:
+ raise error
+
+ return iter([])
+
+ return func_with_open_process_close
+
__all__ = ["DataStreamReader", "DataStreamWriter", "Trigger"]
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 9bb0ff7..4ae90d8 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1874,6 +1874,51 @@ def
test_create_dataframe_merges_the_types_of_every_row_as_pyspark(session):
assert [tuple(r) for r in df.collect()] == [(text,), ("5",)]
+def test_foreach_runs_a_function_and_a_writer_of_each_row_as_pyspark(session,
tmp_path):
+ # As PySpark's foreach: a function of a row, and an object of open,
process and close, run
+ # on each row of each batch, here writing a file of each row on the local
server.
+ from pyspark.errors import PySparkAttributeError
+
+ source, out = tmp_path / "source", tmp_path / "out"
+ session.range(3).write.json(str(source))
+ out.mkdir()
+ directory = str(out)
+
+ def write_row(row):
+ import os
+
+ open(os.path.join(directory, f"function_{row.id}"), "w").close()
+
+ class Writer:
+ def open(self, partition_id, epoch_id):
+ return True
+
+ def process(self, row):
+ import os
+
+ open(os.path.join(directory, f"writer_{row.id}"), "w").close()
+
+ def close(self, error):
+ pass
+
+ for name, function in [("function", write_row), ("writer", Writer())]:
+ query = (
+ session.readStream.schema("id long")
+ .json(str(source))
+ .writeStream.foreach(function)
+ .option("checkpointLocation", str(tmp_path / f"checkpoint_{name}"))
+ .trigger(availableNow=True)
+ .start()
+ )
+ query.awaitTermination()
+ assert query.exception() is None
+ expected = [f"{name}_{i}" for name in ("function", "writer") for i in
range(3)]
+ assert sorted(path.name for path in out.iterdir()) == expected
+ with pytest.raises(PySparkAttributeError) as e:
+ session.readStream.format("rate").load().writeStream.foreach(1)
+ assert e.value.getCondition() == "ATTRIBUTE_NOT_CALLABLE"
+
+
def test_dataframes_of_a_stream_are_streaming_as_pyspark(session):
# As PySpark's, which asks the server: a DataFrame with a streaming source
is streaming.
stream = session.readStream.format("rate").load()
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 6a93c53..8d68cc0 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -9398,6 +9398,72 @@ def
test_stream_writer_triggers_are_checked_as_pyspark(offline_spark, kwargs, co
assert w.trigger(processingTime=None, once=True) is w
+def test_foreach_sends_the_function_of_each_partition_as_pyspark(mock_connect):
+ # As PySpark's foreach: the command (func, None, serializer, serializer)
of the function of
+ # _construct_foreach_function, which calls a function with each row of a
partition;
+ # ATTRIBUTE_NOT_CALLABLE for an object without a callable process, open or
close; and
+ # STREAMING_CONNECT_SERIALIZATION_ERROR for a function that does not
pickle, as of
+ # foreachBatch.
+ import threading
+
+ from pyspark import cloudpickle
+ from pyspark.errors import PySparkAttributeError, PySparkPicklingError
+
+ spark, service = mock_connect
+ # ExecutePlanResponse.write_stream_operation_start_result=8: query_id=1
(id=1, run_id=2).
+ service.execute_payload = message_field(
+ 8, message_field(1, message_field(1, "q") + message_field(2, "r"))
+ )
+ writer = spark.readStream.format("rate").load().writeStream
+
+ def row_function(row):
+ raise ValueError(f"the row {row}")
+
+ writer.foreach(row_function).start()
+ # ExecutePlanRequest.plan=3, Plan.command=2,
Command.write_stream_operation_start=6:
+ # WriteStreamOperationStart.foreach_writer=13 of
StreamingForeachFunction.python_function=1
+ # of PythonUDF.command=3.
+ plan = parse_message(parse_message(service.execute_requests[-1])[3][0])
+ start = parse_message(parse_message(plan[2][0])[6][0])
+ udf = parse_message(parse_message(start[13][0])[1][0])
+ function, return_type, input_serializer, output_serializer =
cloudpickle.loads(udf[3][0])
+ assert return_type is None and input_serializer is output_serializer
+ with pytest.raises(ValueError, match="the row 7"):
+ function(0, iter([7]))
+
+ class NoProcess:
+ pass
+
+ class Uncallable:
+ process = 1
+
+ class UncallableClose:
+ def process(self, row):
+ pass
+
+ close = 1
+
+ for writer_object, attr_name in [
+ (1, "process"),
+ (NoProcess(), "process"),
+ (Uncallable(), "process"),
+ (UncallableClose(), "close"),
+ ]:
+ with pytest.raises(PySparkAttributeError) as e:
+ writer.foreach(writer_object)
+ assert e.value.getCondition() == "ATTRIBUTE_NOT_CALLABLE"
+ assert e.value.getMessageParameters() == {"attr_name": attr_name,
"obj_name": "f"}
+ lock = threading.Lock()
+ for method, name, function in [
+ (writer.foreach, "foreach", lambda row: lock),
+ (writer.foreachBatch, "foreachBatch", lambda df, batch_id: lock),
+ ]:
+ with pytest.raises(PySparkPicklingError) as e:
+ method(function)
+ assert e.value.getCondition() ==
"STREAMING_CONNECT_SERIALIZATION_ERROR"
+ assert e.value.getMessageParameters() == {"name": name}
+
+
def test_stream_writer_sends_its_trigger_as_pyspark(mock_connect):
# As PySpark's: the trigger of a WriteStreamOperationStart, an interval or
duration
# stripped, of the real-time trigger too.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]