Hyukjin Kwon created SPARK-61848:
------------------------------------
Summary: Vendor the modules of pyspark-client 4.3.0 that the
drop-in lacks: PySpark's RDD helpers, tools, server-side workers and gRPC
client modules
Key: SPARK-61848
URL: https://issues.apache.org/jira/browse/SPARK-61848
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
The pyspark-client 4.3.0 package ships 22 modules that the Python drop-in
lacks; importing each raises ModuleNotFoundError here, while it works on
pyspark-client:
- PySpark's RDD helpers: pyspark.join, rddsampler, resultiterable, shuffle,
statcounter;
- its tools: pyspark.errors_doc_gen, install, threaddump,
pyspark.sql.transpile, pyspark.sql.connect.local_server;
- the modules that the server runs in Python: pyspark.worker, pyspark.daemon,
pyspark.worker_message, pyspark.sql.streaming.python_streaming_source_runner,
pyspark.sql.connect.streaming.worker.foreach_batch_worker and listener_worker,
pyspark.sql.streaming.proto.StateMessage_pb2;
- the modules of PySpark's gRPC client: pyspark.sql.connect.client.retries,
artifact, reattach, and pyspark.sql.connect.proto.base_pb2_grpc, which import
grpcio.
Vendor them verbatim from Apache Spark v4.3.0, under the drift gate, so that
they import as in pyspark-client (the gRPC ones where grpcio is installed).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]