Deependra Patel created SPARK-50550:
---------------------------------------
Summary: [Connect] Support sc.broadcast on Connect
Key: SPARK-50550
URL: https://issues.apache.org/jira/browse/SPARK-50550
Project: Spark
Issue Type: Improvement
Components: Connect, PySpark
Affects Versions: 4.0.0
Reporter: Deependra Patel
Spark Context methods are not supported in Spark Connect. Broadcast is a common
API used by developers to send over some data to all executors. Eg. machine
learning model weights can be broadcasted, so no need to fetch individually.
This makes migration of workloads to Spark Connect complex.
I will try to come up with a design doc & PR after that if approach looks good.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
