Hyukjin Kwon created SPARK-61115:
------------------------------------
Summary: Add the module-level names and the connect modules that
the Python client lacks, as PySpark 4.3.0 has them
Key: SPARK-61115
URL: https://issues.apache.org/jira/browse/SPARK-61115
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
The {{pyspark-client}} package of Apache Spark v4.3.0 ({{connect_packages}} of
{{python/packaging/client/setup.py}}) has module-level names and modules that
the Python client lacks:
* {{pyspark.util}}: {{fail_on_stopiteration}}, {{disable_gc}},
{{spark_connect_mode}}, {{default_api_mode}}, {{FuncT}} and its {{__all__}}.
* The {{__all__}} of {{pyspark.sql.udf}}, {{pyspark.sql.udtf}},
{{pyspark.sql.avro}} and {{pyspark.sql.protobuf}}. Without them, {{from
pyspark.sql.udtf import *}} exports the other public names of the module too.
* {{pyspark.sql.functions.builtin}} and
{{pyspark.sql.connect.functions.builtin}} lack {{approxCountDistinct}},
{{bitwiseNOT}}, {{countDistinct}}, {{shiftLeft}}, {{shiftRight}},
{{shiftRightUnsigned}}, {{sumDistinct}}, {{toDegrees}}, {{toRadians}}, {{chr}},
{{random}} and {{uuid}}, which {{pyspark.sql.functions}} has. They also have an
{{__all__}}, which PySpark's {{builtin}} does not. So {{from
pyspark.sql.functions.builtin import countDistinct}} raises {{ImportError}}.
* The connect import path lacks {{pyspark.sql.connect.functions.partitioning}},
{{pyspark.sql.connect.avro.functions}},
{{pyspark.sql.connect.protobuf.functions}}, {{pyspark.sql.connect.table_arg}},
{{pyspark.sql.connect.datasource}}, {{pyspark.sql.connect.utils}},
{{pyspark.sql.connect.logging}} and {{pyspark.sql.connect._typing}}. Importing
them raises {{ModuleNotFoundError}}.
The fix is to add them, as PySpark has them. Pure-Python modules are vendored.
The connect modules re-export the functions and classes of the client, and
{{pyspark.sql.connect.utils.check_dependencies}} checks the dependencies of
this client, which do not include gRPC.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
