Hyukjin Kwon created SPARK-61155:
------------------------------------
Summary: Unpickle the DataFrames that foreachBatch functions
capture on the Python workers of the server, in the Python client as PySpark
does
Key: SPARK-61155
URL: https://issues.apache.org/jira/browse/SPARK-61155
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
A foreachBatch function runs on a Python worker of the server, which has
PySpark and an active session of the query. A DataFrame that the function
captures is pickled with it: PySpark Connect's unpickles there, of that session
and of its plan.
The Python client pickles a DataFrame as a function, by value, that rebuilds it
with the Rust client (pyspark._pyspark), which the worker does not have:
StreamingPythonRunnerInitializationException:
[STREAMING_PYTHON_RUNNER_INITIALIZATION_FAILURE] ... ImportError: cannot import
name '_pyspark' from 'pyspark'
So any foreachBatch function that uses a DataFrame defined outside of it fails
to start. Four tests of Apache Spark's test_parity_foreach_batch.py fail on
this client: test_accessing_spark_session_through_df, test_nested_dataframes,
test_batch_dataframe_join_with_root_session_dataframe_is_unsupported and
test_disabling_fix_restores_root_session_behavior.
On a worker without the Rust client, the function should give back PySpark
Connect's DataFrame of the relation, of its plan ID.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
