This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 75273625 [SPARK-61155][CONNECT][PYTHON] Unpickle the DataFrames that
foreachBatch functions capture on the Python workers of the server, in the
Python client as PySpark does
75273625 is described below
commit 752736252088a4e17c818922359261264225cdcc
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 14:14:53 2026 +0900
[SPARK-61155][CONNECT][PYTHON] Unpickle the DataFrames that foreachBatch
functions capture on the Python workers of the server, in the Python client as
PySpark does
### What changes were proposed in this pull request?
[SPARK-61155](https://issues.apache.org/jira/browse/SPARK-61155): A
`foreachBatch` function runs on a Python worker of the server. The worker has
PySpark and the active session of the query, where a DataFrame that the
function captures unpickles as PySpark Connect's DataFrame, of that session and
of its plan.
The client pickles a DataFrame as a function, by value, of its session and
of the bytes of its relation. On the client, that function rebuilds the
DataFrame with the Rust client, as before. On a worker without the Rust client,
it now gives back PySpark Connect's DataFrame of the relation, as it was sent:
- its plan is a `LogicalPlan` of the relation, of its plan ID, so the
columns that refer to the DataFrame by that ID resolve against it
- everything that the function calls there is of PySpark: its own classes
are made in the function
### Why are the changes needed?
The function imported the Rust client, which the worker does not have, so a
`foreachBatch` function that captured a DataFrame failed to start:
```
StreamingPythonRunnerInitializationException:
[STREAMING_PYTHON_RUNNER_INITIALIZATION_FAILURE] ...
ImportError: cannot import name '_pyspark' from 'pyspark'
```
Four tests of Apache Spark's `test_parity_foreach_batch.py` failed on this
client: `test_accessing_spark_session_through_df`, `test_nested_dataframes`,
`test_batch_dataframe_join_with_root_session_dataframe_is_unsupported` and
`test_disabling_fix_restores_root_session_behavior`.
### Does this PR introduce _any_ user-facing change?
Yes. `foreachBatch` functions that capture DataFrames run, as with PySpark
4.3.0.
### How was this patch tested?
- A new live test runs a `foreachBatch` function that captures a filtered
DataFrame, whose filter refers to the DataFrame by its plan ID, and makes a
temporary view of it in the session of the query, which the client reads. It
passes with official PySpark 4.3.0, and fails without this change.
- The four tests of Apache Spark above run on this client against a local
server: 3 pass, and `test_disabling_fix_restores_root_session_behavior` fails
on `StreamingQueryProgress.numInputRows`, which SPARK-61151 fixes.
- The lib tests of both crates (596 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2212 passed, 8 skipped), and the
live drop-in and Arrow suites (197 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #351 from HyukjinKwon/SPARK-61155-captured-dataframes.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
python/pyspark/sql/connect/dataframe.py | 35 ++++++++++++++++++++++++++++++---
python/tests/test_dropin_e2e.py | 22 +++++++++++++++++++++
2 files changed, 54 insertions(+), 3 deletions(-)
diff --git a/python/pyspark/sql/connect/dataframe.py
b/python/pyspark/sql/connect/dataframe.py
index 5297f0d4..04546dac 100644
--- a/python/pyspark/sql/connect/dataframe.py
+++ b/python/pyspark/sql/connect/dataframe.py
@@ -26,10 +26,39 @@ __all__ = ["DataFrame", "DataFrameNaFunctions",
"DataFrameStatFunctions"]
def _reduce_dataframe(session: Any, relation: bytes) -> Tuple[Callable[...,
Any], Tuple[Any, ...]]:
"""The ``__reduce__`` of a DataFrame of ``session`` and the bytes of its
``relation``, as
PySpark Connect's, of its session and its plan: a function, which
cloudpickle pickles by
- value, of the session, which unpickles first, and of the relation."""
+ value, of the session, which unpickles first, and of the relation.
- def creator(session: Any, relation: bytes) -> "DataFrame":
- from pyspark import _pyspark
+ On a Python worker of the server, whose PySpark has no Rust client, such
as the worker of a
+ ``foreachBatch`` function, which has the active session of the query, the
function gives
+ back PySpark Connect's DataFrame of the relation, as PySpark's DataFrame
of its plan unpickles
+ there. So everything that it calls there is of PySpark, which the worker
has."""
+
+ def creator(session: Any, relation: bytes) -> Any:
+ try:
+ from pyspark import _pyspark
+ except ImportError:
+ from pyspark.sql.connect import proto
+ from pyspark.sql.connect.dataframe import DataFrame as
ConnectDataFrame
+ from pyspark.sql.connect.plan import LogicalPlan
+
+ # Without annotations: the class body looks its names up by name,
which cloudpickle
+ # does not pickle with the function.
+ class Relation(LogicalPlan):
+ """The plan of a relation as it was sent, of its plan ID,
which the columns
+ that refer to it by that ID resolve against."""
+
+ def __init__(self, relation): # type: ignore[no-untyped-def]
+ super().__init__(None)
+ self._relation = relation
+ if relation.common.HasField("plan_id"):
+ self._root_plan_id = relation.common.plan_id
+
+ def plan(self, session): # type: ignore[no-untyped-def]
+ sent = proto.Relation()
+ sent.CopyFrom(self._relation)
+ return sent
+
+ return
ConnectDataFrame(Relation(proto.Relation.FromString(relation)), session)
return _pyspark.DataFrame._of_relation(session, relation)
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index c644ed2d..af4ab5a1 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -2938,6 +2938,28 @@ def
test_create_dataframe_merges_the_types_of_every_row_as_pyspark(session):
assert [tuple(r) for r in df.collect()] == [(text,), ("5",)]
+def
test_foreach_batch_functions_that_capture_dataframes_run_as_pyspark(session,
tmp_path):
+ # As PySpark's: a DataFrame that a foreachBatch function captures
unpickles on the Python
+ # worker of the server, of the session of the query there, with its plan,
whose columns refer
+ # to it by its plan ID.
+ (tmp_path / "input.txt").write_text("a\nb\n")
+ captured = session.createDataFrame([("you", "can"), ("serialize",
"dataframe")])
+ kept = captured.filter(captured["_1"] == "you")
+ name = "foreach_batch_captured_dataframe"
+
+ def func(df, batch_id):
+ kept.createOrReplaceTempView(name)
+
+ query =
session.readStream.format("text").load(str(tmp_path)).writeStream.foreachBatch(func)
+ query = query.start()
+ try:
+ query.processAllAvailable()
+ assert session.table(name).collect() == [Row(_1="you", _2="can")]
+ finally:
+ query.stop()
+ session.catalog.dropTempView(name)
+
+
def test_foreach_runs_a_function_and_a_writer_of_each_row_as_pyspark(session,
tmp_path):
# As PySpark's foreach: a function of a row, and an object of open,
process and close, run
# on each row of each batch, here writing a file of each row on the local
server.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]