DB Tsai created SPARK-61089:
-------------------------------
Summary: Allow UDTF AnalyzeResult subclasses in the Rust Python
client
Key: SPARK-61089
URL: https://issues.apache.org/jira/browse/SPARK-61089
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
The Rust-backed Python client exposes AnalyzeResult as a PyO3 class that cannot
be subclassed. Official PySpark deliberately exposes a mutable dataclass so
UDTF handlers can extend the analysis result with custom state passed to the
handler constructor.
Minimal reproduction with the Rust Python client:
{code:python}
from dataclasses import dataclass
from pyspark.sql.udtf import AnalyzeResult
@dataclass
class CustomAnalyzeResult(AnalyzeResult):
buffer: str = ""
{code}
The class definition fails with:
{code}
TypeError: type 'pyspark.sql.udtf.AnalyzeResult' is not an acceptable base type
{code}
Expected: the subclass inherits schema and the other standard analysis fields
and can add custom fields such as buffer. A UDTF's analyze method can return
this subclass and its constructor can consume the custom state.
Proposed fix: expose a mutable Python AnalyzeResult dataclass with the same
field order and defaults as official PySpark. Preserve the existing internal
native symbol. Add regression coverage for dataclass inheritance, custom-state
serialization, handler command serialization and live analysis-to-constructor
state transfer.
Pull request: https://github.com/apache/spark-connect-rust/pull/286
This is an inherited compatibility gap present on master
62080cbe5ac168a112dea656134c580b9ae4df03, discovered during the post-merge
review of PR #119; PR #119 did not introduce it.
Validation: the new regression reproduces the baseline TypeError and the
existing family test plus new subclass test bodies pass with a private older
native helper and installed official PySpark 4.2 type/serializer dependencies.
Dataclass fields match official Spark 4.3.0 source. The exact-current native
build, full suites, live Spark 4.3 execution and mandatory Claude review remain
pending; PR #286 is a draft.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
