[
https://issues.apache.org/jira/browse/SPARK-61089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DB Tsai updated SPARK-61089:
----------------------------
Fix Version/s: connect-rust-4.3.0
> Allow UDTF AnalyzeResult subclasses in the Rust Python client
> -------------------------------------------------------------
>
> Key: SPARK-61089
> URL: https://issues.apache.org/jira/browse/SPARK-61089
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> The Rust-backed Python client exposes AnalyzeResult as a PyO3 class that
> cannot be subclassed. Official PySpark deliberately exposes a mutable
> dataclass so UDTF handlers can extend the analysis result with custom state
> passed to the handler constructor.
> Minimal reproduction with the Rust Python client:
> {code:python}
> from dataclasses import dataclass
> from pyspark.sql.udtf import AnalyzeResult
> @dataclass
> class CustomAnalyzeResult(AnalyzeResult):
> buffer: str = ""
> {code}
> The class definition fails with:
> {code}
> TypeError: type 'pyspark.sql.udtf.AnalyzeResult' is not an acceptable base
> type
> {code}
> Expected: the subclass inherits schema and the other standard analysis fields
> and can add custom fields such as buffer. A UDTF's analyze method can return
> this subclass and its constructor can consume the custom state.
> Proposed fix: expose a mutable Python AnalyzeResult dataclass with the same
> field order and defaults as official PySpark. Preserve the existing internal
> native symbol. Add regression coverage for dataclass inheritance,
> custom-state serialization, handler command serialization and live
> analysis-to-constructor state transfer.
> Pull request: https://github.com/apache/spark-connect-rust/pull/286
> This is an inherited compatibility gap present on master
> 62080cbe5ac168a112dea656134c580b9ae4df03, discovered during the post-merge
> review of PR #119; PR #119 did not introduce it.
> Validation: the new regression reproduces the baseline TypeError and the
> existing family test plus new subclass test bodies pass with a private older
> native helper and installed official PySpark 4.2 type/serializer
> dependencies. Dataclass fields match official Spark 4.3.0 source. The
> exact-current native build, full suites, live Spark 4.3 execution and
> mandatory Claude review remain pending; PR #286 is a draft.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
