[
https://issues.apache.org/jira/browse/SPARK-61217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DB Tsai updated SPARK-61217:
----------------------------
Fix Version/s: connect-rust-4.3.0
> Support independent Python DataFrame subclass construction in Spark Connect
> Rust
> --------------------------------------------------------------------------------
>
> Key: SPARK-61217
> URL: https://issues.apache.org/jira/browse/SPARK-61217
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> The public pyspark.sql.DataFrame currently reexports the native allocator.
> Official pyspark.testing.mlutils.MockDataset subclasses it and calls
> object.__new__(cls); construction raises TypeError before Python-owned state
> is initialized. MockDataset().index should be 0, and official pipeline tests
> use this pattern.
> Provide a Python public DataFrame base/factory that supports independent
> subclass state and overridden operations, while ordinary construction returns
> the concrete native Connect frame. Preserve call-time dispatch, temporary
> public overrides and restoration, and public-base fallbacks. Add subclass,
> dispatch, restoration and annotation regression coverage.
> Final revision validation: 13 selected subclass/dispatch/cloudpickle
> regression cases passed, with 81 deselected and zero skips/failures/errors.
> Pytest exited 0; the outer harness's exit 1 came from an incorrect
> minimum-22-case assertion for that narrower selection. The counting failure
> is retained and assessed separately.
> Before the final two private-helper renames and comments, full offline
> validation passed 2,676 Python tests, including all 22 required subclass
> cases and annotation assertions, with eight separate live-server skips. These
> results carry forward as historical execution: inverse renaming yields an
> identical AST and all other source/tests/build inputs and the extension are
> unchanged. Historical identical-input validation also passed 141 core Rust
> tests, 495 API Rust tests, formatting, extension build and all-test/wasm-udf
> compilation. Advisory Clippy exited 101 at source sites assessed as
> inherited; no clean lint result is claimed.
> The unchanged official/native high-level pipeline, tuning and one-vs-rest
> selection passed under predeclared PYTHONHASHSEED=0: 16 passed and four exact
> upstream-disabled GC skips per client. This remains bounded historical
> execution. The original default-hash native Random Forest failure on a
> one-class training fold is retained; its cause is unproven. Separately,
> official high-level PySpark through this artifact's Rust transport plugin
> recorded 39 passes and five skips: configuration/creation 23 passes and one
> skip, plus ML 16 passes and four GC skips. This artifact execution also
> predates only the final private-helper renames/comments.
> Cloudpickle probe 3 passed ordinary local-class/captured-closure portability
> through an official-only consumer without native imports before those
> renames/comments. API assessment covered all 500 canonical mismatch records
> and 171 controlled fields, plus 12 central API groups and the Row-name gate.
> The raw API comparison exited 1 for assessed differences; no public names are
> missing. Public/Connect Column aliases and return annotation are the same
> native object; its module metadata differs from official public PySpark.
> Common transport regression evidence comes from a separate artifact with 53
> unchanged transport/build/dependency inputs: 87 ordinary modules produced
> 2,586 passes, 641 skips, one manifest deselection and one retained expected
> catalog-drift failure; four cluster modules produced 15 passes and 14 skips.
> No unexpected failures remained. This candidate executed its own plugin
> checks above; it did not execute both full suites. Source equality is not
> binary equivalence. Complete dependency-tree start/end attestations, per-case
> skip provenance and per-module pytest exit codes are unavailable from the
> shared harness.
> Scope limits: native concrete membership in the public base is virtual; its
> actual MRO remains native, so super(DataFrame, native_frame) is unsupported.
> Direct native subclasses must initialize their native payload. The public
> base uses an ABCMeta subclass; ordinary subclasses use standard abc.ABCMeta
> and unrelated metaclasses need a combined metaclass. Inherited signatures and
> factory/metaclass differences remain. Portability coverage excludes
> independent instances, explicit user metaclasses, native DataFrame
> serialization and server UDFs. Full DataFrame/ML parity is not claimed.
> Final independent full-patch review completed with no remaining findings. All
> earlier findings and fixes, including the final helper-name and comment
> changes, were assessed. The execution boundaries and retained failures above
> remain part of the validation record.
> This pull request and its description were written by OpenAI Codex.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
