Hyukjin Kwon created SPARK-61897:
------------------------------------
Summary: Make the session's objects PySpark's: a cached catalog,
PySpark's public names and members, subclassable Observation and readers, and
the bytearray literal repr
Key: SPARK-61897
URL: https://issues.apache.org/jira/browse/SPARK-61897
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
Object-model gaps of the drop-in against PySpark 4.3.0:
- spark.catalog is a new object at each access, where PySpark's is a
functools.cached_property, so its attributes and mock.patch of its methods are
lost.
- DataFrame.column_names is a public method of the drop-in, so it hides a
column of that name, which PySpark's __getattr__ gives.
- Public names that PySpark lacks: SparkSession.sessionId(),
SparkSession.buildResourceProfile(),
StreamingQueryManager.streamListenerEvents(), UserDefinedTableFunction.command
and python_ver, and operation_id on every PySpark error class.
- Missing members: the writer of the merge clauses, TableArg.to_plan, and
Column.__getattr__ as a class attribute.
- repr(F.lit(bytearray(b"xy"))) is Column<'b'xy''>, where PySpark's is
Column<'bytearray(b'xy')'>.
- The connect Observation, DataFrameReader and DataStreamReader cannot be
subclassed or constructed as PySpark's plain classes, and PythonEvalType is not
PySpark's class; the classes that are not subclassable are not documented.
- Docs: the pickles of Observation subclasses, StorageLevel at protocols 0 and
1, the type-valued attributes, the subquery repr and the list of inputFiles().
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]