DB Tsai created SPARK-61217:
-------------------------------
Summary: Support independent Python DataFrame subclass
construction in Spark Connect Rust
Key: SPARK-61217
URL: https://issues.apache.org/jira/browse/SPARK-61217
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
The public pyspark.sql.DataFrame currently reexports the native allocator.
Official pyspark.testing.mlutils.MockDataset subclasses it and calls
object.__new__(cls); construction raises TypeError before Python-owned state is
initialized. MockDataset().index should be 0, and official pipeline tests use
this pattern.
Provide a Python public DataFrame base/factory that supports independent
subclass state and overridden operations, while ordinary construction returns
the concrete native Connect frame. Preserve call-time dispatch, temporary
public overrides and restoration, and public-base fallbacks. Add subclass,
dispatch, restoration and annotation regression coverage.
Final revision validation: 13 selected subclass/dispatch/cloudpickle regression
cases passed, with 81 deselected and zero skips/failures/errors. Pytest exited
0; the outer harness's exit 1 came from an incorrect minimum-22-case assertion
for that narrower selection. The counting failure is retained and assessed
separately.
Before the final two private-helper renames and comments, full offline
validation passed 2,676 Python tests, including all 22 required subclass cases
and annotation assertions, with eight separate live-server skips. These results
carry forward as historical execution: inverse renaming yields an identical AST
and all other source/tests/build inputs and the extension are unchanged.
Historical identical-input validation also passed 141 core Rust tests, 495 API
Rust tests, formatting, extension build and all-test/wasm-udf compilation.
Advisory Clippy exited 101 at source sites assessed as inherited; no clean lint
result is claimed.
The unchanged official/native high-level pipeline, tuning and one-vs-rest
selection passed under predeclared PYTHONHASHSEED=0: 16 passed and four exact
upstream-disabled GC skips per client. This remains bounded historical
execution. The original default-hash native Random Forest failure on a
one-class training fold is retained; its cause is unproven. Separately,
official high-level PySpark through this artifact's Rust transport plugin
recorded 39 passes and five skips: configuration/creation 23 passes and one
skip, plus ML 16 passes and four GC skips. This artifact execution also
predates only the final private-helper renames/comments.
Cloudpickle probe 3 passed ordinary local-class/captured-closure portability
through an official-only consumer without native imports before those
renames/comments. API assessment covered all 500 canonical mismatch records and
171 controlled fields, plus 12 central API groups and the Row-name gate. The
raw API comparison exited 1 for assessed differences; no public names are
missing. Public/Connect Column aliases and return annotation are the same
native object; its module metadata differs from official public PySpark.
Common transport regression evidence comes from a separate artifact with 53
unchanged transport/build/dependency inputs: 87 ordinary modules produced 2,586
passes, 641 skips, one manifest deselection and one retained expected
catalog-drift failure; four cluster modules produced 15 passes and 14 skips. No
unexpected failures remained. This candidate executed its own plugin checks
above; it did not execute both full suites. Source equality is not binary
equivalence. Complete dependency-tree start/end attestations, per-case skip
provenance and per-module pytest exit codes are unavailable from the shared
harness.
Scope limits: native concrete membership in the public base is virtual; its
actual MRO remains native, so super(DataFrame, native_frame) is unsupported.
Direct native subclasses must initialize their native payload. The public base
uses an ABCMeta subclass; ordinary subclasses use standard abc.ABCMeta and
unrelated metaclasses need a combined metaclass. Inherited signatures and
factory/metaclass differences remain. Portability coverage excludes independent
instances, explicit user metaclasses, native DataFrame serialization and server
UDFs. Full DataFrame/ML parity is not claimed.
Final independent full-patch review completed with no remaining findings. All
earlier findings and fixes, including the final helper-name and comment
changes, were assessed. The execution boundaries and retained failures above
remain part of the validation record.
This pull request and its description were written by OpenAI Codex.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
