This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 373489e  [SPARK-61056][CONNECT][PYTHON] Send the function of 
DataStreamWriter.foreach in the Python client as PySpark does
373489e is described below

commit 373489e8dd0e052fee004fc91c5274311fc6298c
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 12:16:07 2026 +0900

    [SPARK-61056][CONNECT][PYTHON] Send the function of 
DataStreamWriter.foreach in the Python client as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61056](https://issues.apache.org/jira/browse/SPARK-61056): 
`df.writeStream.foreach(f)` now sends the function of PySpark, so the queries 
of `foreach` run.
    
    As PySpark Connect's, `foreach(f)` sends the function of a partition and 
its rows of `_construct_foreach_function(f)`, which the Python worker of the 
server runs:
    - for a function `f`, one that calls `f` with each row
    - for an object, one that calls its `open(partition_id, epoch_id)`, its 
`process(row)` with each row, and its `close(error)`
    
    It raises `ATTRIBUTE_NOT_CALLABLE` for an object without a callable 
`process`, or with an `open` or `close` that is not callable. The function is 
ported from PySpark, but it imports `TaskContext` on the worker, which has it.
    
    `foreach` and `foreachBatch` now pickle their functions with 
`CloudPickleSerializer`, as PySpark's. They raise 
`STREAMING_CONNECT_SERIALIZATION_ERROR` for a function that does not pickle.
    
    ### Why are the changes needed?
    
    The client sent `f` itself, which the worker calls with a partition and its 
rows:
    - the query of an object failed with `TypeError: 'Writer' object is not 
callable`
    - that of a function failed without a batch
    
    So `foreach` did not work.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. The queries of `foreach` run, as in PySpark 4.3.0.
    
    ### How was this patch tested?
    
    - A new live test runs `foreach` of a function and of an object of `open`, 
`process` and `close`, which write a file of each row on the local server, and 
checks the files. It also checks the `ATTRIBUTE_NOT_CALLABLE` of an int. It 
passes with official PySpark 4.3.0, and fails without this change.
    - A new offline test unpickles the command that `foreach` sends to a mock 
Connect server, runs its function on rows, and checks the errors of objects 
without callable methods and of functions that do not pickle.
    - The lib tests of both crates (577 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (1817 passed, 8 skipped), and the 
live drop-in and Arrow suites (116 passed, without the test of 
`StreamingQuery.explain`, which hangs on a slow machine and #260 fixes) and 
live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no 
diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #264 from HyukjinKwon/SPARK-61056-foreach.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/streaming.rs         |  50 ++++++++++++---
 python/pyspark/sql/streaming/readwriter.py | 100 ++++++++++++++++++++++++++++-
 python/tests/test_dropin_e2e.py            |  45 +++++++++++++
 python/tests/test_dropin_offline.py        |  66 +++++++++++++++++++
 4 files changed, 250 insertions(+), 11 deletions(-)

diff --git a/crates/pyspark-rs/src/streaming.rs 
b/crates/pyspark-rs/src/streaming.rs
index e75107a..1a0dc49 100644
--- a/crates/pyspark-rs/src/streaming.rs
+++ b/crates/pyspark-rs/src/streaming.rs
@@ -754,7 +754,7 @@ impl PyDataStreamWriter {
     }
 
     /// `DataStreamWriter.foreachBatch(func)`: cloudpickle the `(batch_df, 
batch_id)`
-    /// function (via the bundled `pyspark.cloudpickle`) and attach it as the
+    /// function (with PySpark's `CloudPickleSerializer`) and attach it as the
     /// foreach-batch sink. Pickling happens here so the Python skin can 
re-export this
     /// class directly rather than subclassing a (non-subclassable) PyO3 type.
     fn foreachBatch<'py>(
@@ -762,7 +762,7 @@ impl PyDataStreamWriter {
         py: Python<'_>,
         func: &Bound<'_, PyAny>,
     ) -> PyResult<PyRefMut<'py, Self>> {
-        let command = crate::dataframe::py_cloudpickle(py, func)?;
+        let command = streaming_function_command(py, func, "foreachBatch")?;
         let payload = PythonUDFPayload::new(
             spark_connect::types::DataType::Struct { fields: vec![] },
             0, // eval type is unused for the streaming foreach sinks
@@ -773,15 +773,21 @@ impl PyDataStreamWriter {
         Ok(slf)
     }
 
-    /// `DataStreamWriter.foreach(f)`: wrap the row handler as the reference 
client does
-    /// — `(f, None, serializer, serializer)` with 
`AutoBatchedSerializer(CPickleSerializer())`
-    /// — and cloudpickle it so the worker deserializes it against its own
-    /// `pyspark.serializers`. Built here to avoid a Python subclass of the 
PyO3 class.
+    /// `DataStreamWriter.foreach(f)`, as PySpark's: the function of the 
partition and the
+    /// rows of each partition of `_construct_foreach_function(f)`, which 
raises
+    /// `ATTRIBUTE_NOT_CALLABLE` for an object without a callable `process`, 
in the command
+    /// `(func, None, serializer, serializer)` of 
`AutoBatchedSerializer(CPickleSerializer())`,
+    /// which the worker of the server deserializes with its own 
`pyspark.serializers`. Built
+    /// here to avoid a Python subclass of the PyO3 class.
     fn foreach<'py>(
         mut slf: PyRefMut<'py, Self>,
         py: Python<'_>,
         f: &Bound<'_, PyAny>,
     ) -> PyResult<PyRefMut<'py, Self>> {
+        let func = py
+            .import("pyspark.sql.streaming.readwriter")?
+            .getattr("_construct_foreach_function")?
+            .call1((f,))?;
         let serializers = py.import("pyspark.serializers")?;
         let cpickle = serializers.getattr("CPickleSerializer")?.call0()?;
         let serializer = serializers
@@ -792,13 +798,13 @@ impl PyDataStreamWriter {
         let command_tuple = pyo3::types::PyTuple::new(
             py,
             [
-                f.clone(),
+                func,
                 py.None().into_bound(py),
                 serializer.clone(),
                 serializer,
             ],
         )?;
-        let command = crate::dataframe::py_cloudpickle(py, 
command_tuple.as_any())?;
+        let command = streaming_function_command(py, command_tuple.as_any(), 
"foreach")?;
         let payload = PythonUDFPayload::new(
             spark_connect::types::DataType::Struct { fields: vec![] },
             0,
@@ -853,6 +859,34 @@ impl<'a, 'py> FromPyObject<'a, 'py> for ProtoBool {
     }
 }
 
+/// The command of the function of `foreach` or `foreachBatch`, as PySpark's: 
of
+/// `CloudPickleSerializer().dumps`, which raises a `pickle.PicklingError` for 
an object that
+/// does not pickle, as `STREAMING_CONNECT_SERIALIZATION_ERROR` of `name`.
+fn streaming_function_command(
+    py: Python<'_>,
+    function: &Bound<'_, PyAny>,
+    name: &str,
+) -> PyResult<Vec<u8>> {
+    let serializer = py
+        .import("pyspark.serializers")?
+        .getattr("CloudPickleSerializer")?
+        .call0()?;
+    match serializer.call_method1("dumps", (function,)) {
+        Ok(command) => command.extract(),
+        Err(error) if error.matches(py, 
py.import("pickle")?.getattr("PicklingError")?)? => {
+            let raised = crate::column::raise_pyspark(
+                py,
+                "PySparkPicklingError",
+                "STREAMING_CONNECT_SERIALIZATION_ERROR",
+                &[("name", name)],
+            );
+            raised.set_cause(py, Some(error));
+            Err(raised)
+        }
+        Err(error) => Err(error),
+    }
+}
+
 /// Python wrapper for StreamingQuery.
 #[pyclass(name = "StreamingQuery", module = "pyspark.sql.streaming.query")]
 pub struct PyStreamingQuery {
diff --git a/python/pyspark/sql/streaming/readwriter.py 
b/python/pyspark/sql/streaming/readwriter.py
index 030e1bb..d48d3dc 100644
--- a/python/pyspark/sql/streaming/readwriter.py
+++ b/python/pyspark/sql/streaming/readwriter.py
@@ -19,19 +19,113 @@
 Streaming data readers and writers.
 
 Re-exports the PyO3-implemented streaming classes. 
`DataStreamWriter.foreachBatch`
-and `.foreach` accept a Python callable and cloudpickle it inside the extension
-(via the bundled ``pyspark.cloudpickle``), so no Python subclass of the (non-
-subclassable) PyO3 class is needed.
+and `.foreach` accept a Python callable and cloudpickle it inside the 
extension (with
+``pyspark.serializers.CloudPickleSerializer``), so no Python subclass of the 
(non-
+subclassable) PyO3 class is needed; `.foreach` sends the function of
+`_construct_foreach_function`, as PySpark does.
 """
 
+from typing import Any, Callable, Iterator, Union, cast
+
 from pyspark._pyspark import (
     DataStreamReader as _DataStreamReader,
     DataStreamWriter as _DataStreamWriter,
     Trigger as _Trigger,
 )
+from pyspark.errors import PySparkAttributeError, PySparkRuntimeError
 
 DataStreamReader = _DataStreamReader
 DataStreamWriter = _DataStreamWriter
 Trigger = _Trigger
 
+
+def _construct_foreach_function(
+    f: Union[Callable[[Any], None], Any],
+) -> Callable[[Any, Iterator], Iterator]:
+    """
+    The function of the partition and the rows of each partition that 
``foreach(f)`` sends,
+    which the Python worker of the server runs, as PySpark's
+    ``DataStreamWriter._construct_foreach_function``: for a function ``f``, 
one that calls ``f``
+    with each row, and for an object, one that calls its ``open(partition_id, 
epoch_id)``, its
+    ``process(row)`` with each row, and its ``close(error)``.
+
+    Ported from PySpark, but for ``TaskContext``, which the function imports 
on the worker.
+    """
+    if callable(f):
+        # The provided object is a callable function that is supposed to be 
called on each row.
+        # Construct a function that takes an iterator and calls the provided 
function on each
+        # row.
+        def func_without_process(_: Any, iterator: Iterator) -> Iterator:
+            for x in iterator:
+                f(x)
+            return iter([])
+
+        return func_without_process
+
+    else:
+        # The provided object is not a callable function. Then it is expected 
to have a
+        # 'process(row)' method, and optional 'open(partition_id, epoch_id)' 
and
+        # 'close(error)' methods.
+
+        if not hasattr(f, "process"):
+            raise PySparkAttributeError(
+                errorClass="ATTRIBUTE_NOT_CALLABLE",
+                messageParameters={"attr_name": "process", "obj_name": "f"},
+            )
+
+        if not callable(getattr(f, "process")):
+            raise PySparkAttributeError(
+                errorClass="ATTRIBUTE_NOT_CALLABLE",
+                messageParameters={"attr_name": "process", "obj_name": "f"},
+            )
+
+        def doesMethodExist(method_name: str) -> bool:
+            exists = hasattr(f, method_name)
+            if exists and not callable(getattr(f, method_name)):
+                raise PySparkAttributeError(
+                    errorClass="ATTRIBUTE_NOT_CALLABLE",
+                    messageParameters={"attr_name": method_name, "obj_name": 
"f"},
+                )
+            return exists
+
+        open_exists = doesMethodExist("open")
+        close_exists = doesMethodExist("close")
+
+        def func_with_open_process_close(partition_id: Any, iterator: 
Iterator) -> Iterator:
+            from pyspark.taskcontext import TaskContext
+
+            epoch_id = cast(TaskContext, TaskContext.get()).getLocalProperty(
+                "streaming.sql.batchId"
+            )
+            if epoch_id:
+                int_epoch_id = int(epoch_id)
+            else:
+                raise PySparkRuntimeError(
+                    errorClass="CANNOT_GET_BATCH_ID",
+                    messageParameters={"obj_name": "TaskContext"},
+                )
+
+            # Check if the data should be processed
+            should_process = True
+            if open_exists:
+                should_process = f.open(partition_id, int_epoch_id)
+
+            error = None
+
+            try:
+                if should_process:
+                    for x in iterator:
+                        f.process(x)
+            except Exception as ex:
+                error = ex
+            finally:
+                if close_exists:
+                    f.close(error)
+                if error:
+                    raise error
+
+            return iter([])
+
+        return func_with_open_process_close
+
 __all__ = ["DataStreamReader", "DataStreamWriter", "Trigger"]
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 9bb0ff7..4ae90d8 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1874,6 +1874,51 @@ def 
test_create_dataframe_merges_the_types_of_every_row_as_pyspark(session):
         assert [tuple(r) for r in df.collect()] == [(text,), ("5",)]
 
 
+def test_foreach_runs_a_function_and_a_writer_of_each_row_as_pyspark(session, 
tmp_path):
+    # As PySpark's foreach: a function of a row, and an object of open, 
process and close, run
+    # on each row of each batch, here writing a file of each row on the local 
server.
+    from pyspark.errors import PySparkAttributeError
+
+    source, out = tmp_path / "source", tmp_path / "out"
+    session.range(3).write.json(str(source))
+    out.mkdir()
+    directory = str(out)
+
+    def write_row(row):
+        import os
+
+        open(os.path.join(directory, f"function_{row.id}"), "w").close()
+
+    class Writer:
+        def open(self, partition_id, epoch_id):
+            return True
+
+        def process(self, row):
+            import os
+
+            open(os.path.join(directory, f"writer_{row.id}"), "w").close()
+
+        def close(self, error):
+            pass
+
+    for name, function in [("function", write_row), ("writer", Writer())]:
+        query = (
+            session.readStream.schema("id long")
+            .json(str(source))
+            .writeStream.foreach(function)
+            .option("checkpointLocation", str(tmp_path / f"checkpoint_{name}"))
+            .trigger(availableNow=True)
+            .start()
+        )
+        query.awaitTermination()
+        assert query.exception() is None
+    expected = [f"{name}_{i}" for name in ("function", "writer") for i in 
range(3)]
+    assert sorted(path.name for path in out.iterdir()) == expected
+    with pytest.raises(PySparkAttributeError) as e:
+        session.readStream.format("rate").load().writeStream.foreach(1)
+    assert e.value.getCondition() == "ATTRIBUTE_NOT_CALLABLE"
+
+
 def test_dataframes_of_a_stream_are_streaming_as_pyspark(session):
     # As PySpark's, which asks the server: a DataFrame with a streaming source 
is streaming.
     stream = session.readStream.format("rate").load()
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 6a93c53..8d68cc0 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -9398,6 +9398,72 @@ def 
test_stream_writer_triggers_are_checked_as_pyspark(offline_spark, kwargs, co
     assert w.trigger(processingTime=None, once=True) is w
 
 
+def test_foreach_sends_the_function_of_each_partition_as_pyspark(mock_connect):
+    # As PySpark's foreach: the command (func, None, serializer, serializer) 
of the function of
+    # _construct_foreach_function, which calls a function with each row of a 
partition;
+    # ATTRIBUTE_NOT_CALLABLE for an object without a callable process, open or 
close; and
+    # STREAMING_CONNECT_SERIALIZATION_ERROR for a function that does not 
pickle, as of
+    # foreachBatch.
+    import threading
+
+    from pyspark import cloudpickle
+    from pyspark.errors import PySparkAttributeError, PySparkPicklingError
+
+    spark, service = mock_connect
+    # ExecutePlanResponse.write_stream_operation_start_result=8: query_id=1 
(id=1, run_id=2).
+    service.execute_payload = message_field(
+        8, message_field(1, message_field(1, "q") + message_field(2, "r"))
+    )
+    writer = spark.readStream.format("rate").load().writeStream
+
+    def row_function(row):
+        raise ValueError(f"the row {row}")
+
+    writer.foreach(row_function).start()
+    # ExecutePlanRequest.plan=3, Plan.command=2, 
Command.write_stream_operation_start=6:
+    # WriteStreamOperationStart.foreach_writer=13 of 
StreamingForeachFunction.python_function=1
+    # of PythonUDF.command=3.
+    plan = parse_message(parse_message(service.execute_requests[-1])[3][0])
+    start = parse_message(parse_message(plan[2][0])[6][0])
+    udf = parse_message(parse_message(start[13][0])[1][0])
+    function, return_type, input_serializer, output_serializer = 
cloudpickle.loads(udf[3][0])
+    assert return_type is None and input_serializer is output_serializer
+    with pytest.raises(ValueError, match="the row 7"):
+        function(0, iter([7]))
+
+    class NoProcess:
+        pass
+
+    class Uncallable:
+        process = 1
+
+    class UncallableClose:
+        def process(self, row):
+            pass
+
+        close = 1
+
+    for writer_object, attr_name in [
+        (1, "process"),
+        (NoProcess(), "process"),
+        (Uncallable(), "process"),
+        (UncallableClose(), "close"),
+    ]:
+        with pytest.raises(PySparkAttributeError) as e:
+            writer.foreach(writer_object)
+        assert e.value.getCondition() == "ATTRIBUTE_NOT_CALLABLE"
+        assert e.value.getMessageParameters() == {"attr_name": attr_name, 
"obj_name": "f"}
+    lock = threading.Lock()
+    for method, name, function in [
+        (writer.foreach, "foreach", lambda row: lock),
+        (writer.foreachBatch, "foreachBatch", lambda df, batch_id: lock),
+    ]:
+        with pytest.raises(PySparkPicklingError) as e:
+            method(function)
+        assert e.value.getCondition() == 
"STREAMING_CONNECT_SERIALIZATION_ERROR"
+        assert e.value.getMessageParameters() == {"name": name}
+
+
 def test_stream_writer_sends_its_trigger_as_pyspark(mock_connect):
     # As PySpark's: the trigger of a WriteStreamOperationStart, an interval or 
duration
     # stripped, of the real-time trigger too.


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to