Hyukjin Kwon created SPARK-62134:
------------------------------------

             Summary: Move the client of the Python client into 
pyspark.sql.connect.client.core, of PySpark's layout and names of the package 
and its core
                 Key: SPARK-62134
                 URL: https://issues.apache.org/jira/browse/SPARK-62134
             Project: Spark
          Issue Type: Improvement
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


In pyspark-client 4.3.0, pyspark/sql/connect/client/core.py defines the client: 
SparkConnectClient, ChannelBuilder, DefaultChannelBuilder and RpcDeadlines (its 
__all__), PlanObservedMetrics, AnalyzeResult, ConfigResult and its other names; 
the package's __init__.py has no __all__ and runs `from 
pyspark.sql.connect.client.core import *` and `from pyspark.sql.connect.logging 
import getLogLevel`. The Python client (pyspark-client-rust) has it the other 
way around: its package __init__.py defines the client and its helpers, of an 
__all__ of its own, and core.py is a shim to which the package assigns its 
classes. So the class of SparkSession.client is 
pyspark.sql.connect.client.SparkConnectClient, where PySpark's is 
pyspark.sql.connect.client.core.SparkConnectClient, which type checkers, 
help(), inspect.getsource and the pickles of its functions show, and the 
package has names that PySpark's has not.

Expected: as pyspark-client, the client in core.py of PySpark's names and 
__all__, and the package's __init__.py as PySpark's.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to