[
https://issues.apache.org/jira/browse/SPARK-61897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61897:
---------------------------------
Fix Version/s: connect-rust-4.4.0
> Make the session's objects PySpark's: a cached catalog, PySpark's public
> names and members, subclassable Observation and readers, and the bytearray
> literal repr
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61897
> URL: https://issues.apache.org/jira/browse/SPARK-61897
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.4.0
>
>
> Object-model gaps of the drop-in against PySpark 4.3.0:
> - spark.catalog is a new object at each access, where PySpark's is a
> functools.cached_property, so its attributes and mock.patch of its methods
> are lost.
> - DataFrame.column_names is a public method of the drop-in, so it hides a
> column of that name, which PySpark's __getattr__ gives.
> - Public names that PySpark lacks: SparkSession.sessionId(),
> SparkSession.buildResourceProfile(),
> StreamingQueryManager.streamListenerEvents(),
> UserDefinedTableFunction.command and python_ver, and operation_id on every
> PySpark error class.
> - Missing members: the writer of the merge clauses, TableArg.to_plan, and
> Column.__getattr__ as a class attribute.
> - repr(F.lit(bytearray(b"xy"))) is Column<'b'xy''>, where PySpark's is
> Column<'bytearray(b'xy')'>.
> - The connect Observation, DataFrameReader and DataStreamReader cannot be
> subclassed or constructed as PySpark's plain classes, and PythonEvalType is
> not PySpark's class; the classes that are not subclassable are not documented.
> - Docs: the pickles of Observation subclasses, StorageLevel at protocols 0
> and 1, the type-valued attributes, the subquery repr and the list of
> inputFiles().
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]