Hyukjin Kwon created SPARK-61897:
------------------------------------

             Summary: Make the session's objects PySpark's: a cached catalog, 
PySpark's public names and members, subclassable Observation and readers, and 
the bytearray literal repr
                 Key: SPARK-61897
                 URL: https://issues.apache.org/jira/browse/SPARK-61897
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


Object-model gaps of the drop-in against PySpark 4.3.0:
- spark.catalog is a new object at each access, where PySpark's is a 
functools.cached_property, so its attributes and mock.patch of its methods are 
lost.
- DataFrame.column_names is a public method of the drop-in, so it hides a 
column of that name, which PySpark's __getattr__ gives.
- Public names that PySpark lacks: SparkSession.sessionId(), 
SparkSession.buildResourceProfile(), 
StreamingQueryManager.streamListenerEvents(), UserDefinedTableFunction.command 
and python_ver, and operation_id on every PySpark error class.
- Missing members: the writer of the merge clauses, TableArg.to_plan, and 
Column.__getattr__ as a class attribute.
- repr(F.lit(bytearray(b"xy"))) is Column<'b'xy''>, where PySpark's is 
Column<'bytearray(b'xy')'>.
- The connect Observation, DataFrameReader and DataStreamReader cannot be 
subclassed or constructed as PySpark's plain classes, and PythonEvalType is not 
PySpark's class; the classes that are not subclassable are not documented.
- Docs: the pickles of Observation subclasses, StorageLevel at protocols 0 and 
1, the type-valued attributes, the subquery repr and the list of inputFiles().




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to