[
https://issues.apache.org/jira/browse/SPARK-61848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61848:
---------------------------------
Fix Version/s: connect-rust-4.4.0
> Vendor the modules of pyspark-client 4.3.0 that the drop-in lacks: PySpark's
> RDD helpers, tools, server-side workers and gRPC client modules
> --------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61848
> URL: https://issues.apache.org/jira/browse/SPARK-61848
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.4.0
>
>
> The pyspark-client 4.3.0 package ships 22 modules that the Python drop-in
> lacks; importing each raises ModuleNotFoundError here, while it works on
> pyspark-client:
> - PySpark's RDD helpers: pyspark.join, rddsampler, resultiterable, shuffle,
> statcounter;
> - its tools: pyspark.errors_doc_gen, install, threaddump,
> pyspark.sql.transpile, pyspark.sql.connect.local_server;
> - the modules that the server runs in Python: pyspark.worker, pyspark.daemon,
> pyspark.worker_message, pyspark.sql.streaming.python_streaming_source_runner,
> pyspark.sql.connect.streaming.worker.foreach_batch_worker and
> listener_worker, pyspark.sql.streaming.proto.StateMessage_pb2;
> - the modules of PySpark's gRPC client: pyspark.sql.connect.client.retries,
> artifact, reattach, and pyspark.sql.connect.proto.base_pb2_grpc, which import
> grpcio.
> Vendor them verbatim from Apache Spark v4.3.0, under the drift gate, so that
> they import as in pyspark-client (the gRPC ones where grpcio is installed).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]