[ 
https://issues.apache.org/jira/browse/SPARK-61897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-61897.
----------------------------------
    Resolution: Fixed

Issue resolved by pull request 799
[https://github.com/apache/spark-connect-rust/pull/799]

> Make the session's objects PySpark's: a cached catalog, PySpark's public 
> names and members, subclassable Observation and readers, and the bytearray 
> literal repr
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61897
>                 URL: https://issues.apache.org/jira/browse/SPARK-61897
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> Object-model gaps of the drop-in against PySpark 4.3.0:
> - spark.catalog is a new object at each access, where PySpark's is a 
> functools.cached_property, so its attributes and mock.patch of its methods 
> are lost.
> - DataFrame.column_names is a public method of the drop-in, so it hides a 
> column of that name, which PySpark's __getattr__ gives.
> - Public names that PySpark lacks: SparkSession.sessionId(), 
> SparkSession.buildResourceProfile(), 
> StreamingQueryManager.streamListenerEvents(), 
> UserDefinedTableFunction.command and python_ver, and operation_id on every 
> PySpark error class.
> - Missing members: the writer of the merge clauses, TableArg.to_plan, and 
> Column.__getattr__ as a class attribute.
> - repr(F.lit(bytearray(b"xy"))) is Column<'b'xy''>, where PySpark's is 
> Column<'bytearray(b'xy')'>.
> - The connect Observation, DataFrameReader and DataStreamReader cannot be 
> subclassed or constructed as PySpark's plain classes, and PythonEvalType is 
> not PySpark's class; the classes that are not subclassable are not documented.
> - Docs: the pickles of Observation subclasses, StorageLevel at protocols 0 
> and 1, the type-valued attributes, the subquery repr and the list of 
> inputFiles().



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to