Hyukjin Kwon created SPARK-62134:
------------------------------------
Summary: Move the client of the Python client into
pyspark.sql.connect.client.core, of PySpark's layout and names of the package
and its core
Key: SPARK-62134
URL: https://issues.apache.org/jira/browse/SPARK-62134
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
In pyspark-client 4.3.0, pyspark/sql/connect/client/core.py defines the client:
SparkConnectClient, ChannelBuilder, DefaultChannelBuilder and RpcDeadlines (its
__all__), PlanObservedMetrics, AnalyzeResult, ConfigResult and its other names;
the package's __init__.py has no __all__ and runs `from
pyspark.sql.connect.client.core import *` and `from pyspark.sql.connect.logging
import getLogLevel`. The Python client (pyspark-client-rust) has it the other
way around: its package __init__.py defines the client and its helpers, of an
__all__ of its own, and core.py is a shim to which the package assigns its
classes. So the class of SparkSession.client is
pyspark.sql.connect.client.SparkConnectClient, where PySpark's is
pyspark.sql.connect.client.core.SparkConnectClient, which type checkers,
help(), inspect.getsource and the pickles of its functions show, and the
package has names that PySpark's has not.
Expected: as pyspark-client, the client in core.py of PySpark's names and
__all__, and the package's __init__.py as PySpark's.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]