This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 75273625 [SPARK-61155][CONNECT][PYTHON] Unpickle the DataFrames that 
foreachBatch functions capture on the Python workers of the server, in the 
Python client as PySpark does
75273625 is described below

commit 752736252088a4e17c818922359261264225cdcc
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 14:14:53 2026 +0900

    [SPARK-61155][CONNECT][PYTHON] Unpickle the DataFrames that foreachBatch 
functions capture on the Python workers of the server, in the Python client as 
PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61155](https://issues.apache.org/jira/browse/SPARK-61155): A 
`foreachBatch` function runs on a Python worker of the server. The worker has 
PySpark and the active session of the query, where a DataFrame that the 
function captures unpickles as PySpark Connect's DataFrame, of that session and 
of its plan.
    
    The client pickles a DataFrame as a function, by value, of its session and 
of the bytes of its relation. On the client, that function rebuilds the 
DataFrame with the Rust client, as before. On a worker without the Rust client, 
it now gives back PySpark Connect's DataFrame of the relation, as it was sent:
    - its plan is a `LogicalPlan` of the relation, of its plan ID, so the 
columns that refer to the DataFrame by that ID resolve against it
    - everything that the function calls there is of PySpark: its own classes 
are made in the function
    
    ### Why are the changes needed?
    
    The function imported the Rust client, which the worker does not have, so a 
`foreachBatch` function that captured a DataFrame failed to start:
    
    ```
    StreamingPythonRunnerInitializationException: 
[STREAMING_PYTHON_RUNNER_INITIALIZATION_FAILURE] ...
    ImportError: cannot import name '_pyspark' from 'pyspark'
    ```
    
    Four tests of Apache Spark's `test_parity_foreach_batch.py` failed on this 
client: `test_accessing_spark_session_through_df`, `test_nested_dataframes`, 
`test_batch_dataframe_join_with_root_session_dataframe_is_unsupported` and 
`test_disabling_fix_restores_root_session_behavior`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `foreachBatch` functions that capture DataFrames run, as with PySpark 
4.3.0.
    
    ### How was this patch tested?
    
    - A new live test runs a `foreachBatch` function that captures a filtered 
DataFrame, whose filter refers to the DataFrame by its plan ID, and makes a 
temporary view of it in the session of the query, which the client reads. It 
passes with official PySpark 4.3.0, and fails without this change.
    - The four tests of Apache Spark above run on this client against a local 
server: 3 pass, and `test_disabling_fix_restores_root_session_behavior` fails 
on `StreamingQueryProgress.numInputRows`, which SPARK-61151 fixes.
    - The lib tests of both crates (596 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2212 passed, 8 skipped), and the 
live drop-in and Arrow suites (197 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #351 from HyukjinKwon/SPARK-61155-captured-dataframes.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 python/pyspark/sql/connect/dataframe.py | 35 ++++++++++++++++++++++++++++++---
 python/tests/test_dropin_e2e.py         | 22 +++++++++++++++++++++
 2 files changed, 54 insertions(+), 3 deletions(-)

diff --git a/python/pyspark/sql/connect/dataframe.py 
b/python/pyspark/sql/connect/dataframe.py
index 5297f0d4..04546dac 100644
--- a/python/pyspark/sql/connect/dataframe.py
+++ b/python/pyspark/sql/connect/dataframe.py
@@ -26,10 +26,39 @@ __all__ = ["DataFrame", "DataFrameNaFunctions", 
"DataFrameStatFunctions"]
 def _reduce_dataframe(session: Any, relation: bytes) -> Tuple[Callable[..., 
Any], Tuple[Any, ...]]:
     """The ``__reduce__`` of a DataFrame of ``session`` and the bytes of its 
``relation``, as
     PySpark Connect's, of its session and its plan: a function, which 
cloudpickle pickles by
-    value, of the session, which unpickles first, and of the relation."""
+    value, of the session, which unpickles first, and of the relation.
 
-    def creator(session: Any, relation: bytes) -> "DataFrame":
-        from pyspark import _pyspark
+    On a Python worker of the server, whose PySpark has no Rust client, such 
as the worker of a
+    ``foreachBatch`` function, which has the active session of the query, the 
function gives
+    back PySpark Connect's DataFrame of the relation, as PySpark's DataFrame 
of its plan unpickles
+    there. So everything that it calls there is of PySpark, which the worker 
has."""
+
+    def creator(session: Any, relation: bytes) -> Any:
+        try:
+            from pyspark import _pyspark
+        except ImportError:
+            from pyspark.sql.connect import proto
+            from pyspark.sql.connect.dataframe import DataFrame as 
ConnectDataFrame
+            from pyspark.sql.connect.plan import LogicalPlan
+
+            # Without annotations: the class body looks its names up by name, 
which cloudpickle
+            # does not pickle with the function.
+            class Relation(LogicalPlan):
+                """The plan of a relation as it was sent, of its plan ID, 
which the columns
+                that refer to it by that ID resolve against."""
+
+                def __init__(self, relation):  # type: ignore[no-untyped-def]
+                    super().__init__(None)
+                    self._relation = relation
+                    if relation.common.HasField("plan_id"):
+                        self._root_plan_id = relation.common.plan_id
+
+                def plan(self, session):  # type: ignore[no-untyped-def]
+                    sent = proto.Relation()
+                    sent.CopyFrom(self._relation)
+                    return sent
+
+            return 
ConnectDataFrame(Relation(proto.Relation.FromString(relation)), session)
 
         return _pyspark.DataFrame._of_relation(session, relation)
 
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index c644ed2d..af4ab5a1 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -2938,6 +2938,28 @@ def 
test_create_dataframe_merges_the_types_of_every_row_as_pyspark(session):
         assert [tuple(r) for r in df.collect()] == [(text,), ("5",)]
 
 
+def 
test_foreach_batch_functions_that_capture_dataframes_run_as_pyspark(session, 
tmp_path):
+    # As PySpark's: a DataFrame that a foreachBatch function captures 
unpickles on the Python
+    # worker of the server, of the session of the query there, with its plan, 
whose columns refer
+    # to it by its plan ID.
+    (tmp_path / "input.txt").write_text("a\nb\n")
+    captured = session.createDataFrame([("you", "can"), ("serialize", 
"dataframe")])
+    kept = captured.filter(captured["_1"] == "you")
+    name = "foreach_batch_captured_dataframe"
+
+    def func(df, batch_id):
+        kept.createOrReplaceTempView(name)
+
+    query = 
session.readStream.format("text").load(str(tmp_path)).writeStream.foreachBatch(func)
+    query = query.start()
+    try:
+        query.processAllAvailable()
+        assert session.table(name).collect() == [Row(_1="you", _2="can")]
+    finally:
+        query.stop()
+        session.catalog.dropTempView(name)
+
+
 def test_foreach_runs_a_function_and_a_writer_of_each_row_as_pyspark(session, 
tmp_path):
     # As PySpark's foreach: a function of a row, and an object of open, 
process and close, run
     # on each row of each batch, here writing a file of each row on the local 
server.


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to