Hyukjin Kwon created SPARK-61852:
------------------------------------
Summary: Give the remaining objects of a session PySpark Connect's
classes, modules and reprs, and make the classic conf, tvf, merge and
observation classes PySpark's
Key: SPARK-61852
URL: https://issues.apache.org/jira/browse/SPARK-61852
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
Follow-ups of SPARK-61758 and SPARK-61769 (the classes and modules of the
objects of a session), found by a review of the last 100 commits, each checked
against official pyspark-client 4.3.0:
- spark.dataSource is a native class that claims
pyspark.sql.datasource.DataSourceRegistration; PySpark gives
pyspark.sql.connect.datasource.DataSourceRegistration.
- pyspark.sql.connect.avro.functions and protobuf.functions re-export the
classic functions.
- pickle of pyspark.sql.connect.functions.random fails, and F.column, negate
and power are not the aliases col, negative and pow.
- repr(spark.catalog) is Catalog() instead of the default object repr.
- The Connect builder's enableHiveSupport() is a no-op where PySpark raises
NOT_IMPLEMENTED.
- SparkSession.Hook is not nested (qualname Hook), and neither its class nor a
subclass instance pickles.
- The SparkSession.builder descriptor is a native class that claims
pyspark.sql.session.classproperty.
- ChannelBuilder, DefaultChannelBuilder and PlanObservedMetrics are not of
pyspark.sql.connect.client.core.
- The classic RuntimeConfig, TableValuedFunction, MergeIntoWriter and
Observation name the Connect classes, so isinstance is True where PySpark gives
False; pyspark.sql.connect.window.Window is the classic class.
- The connect banner names pyspark.__version__ (4.3.0) instead of the package
version (4.4.0).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]