[
https://issues.apache.org/jira/browse/SPARK-61216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DB Tsai updated SPARK-61216:
----------------------------
Fix Version/s: connect-rust-4.3.0
> Validate native random_split weights in Spark Connect Rust
> ----------------------------------------------------------
>
> Key: SPARK-61216
> URL: https://issues.apache.org/jira/browse/SPARK-61216
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> Native Rust DataFrame::random_split constructs sampling plans for negative
> weights and nonpositive totals, while PySpark randomSplit raises
> VALUE_NOT_POSITIVE before constructing splits. An empty list silently returns
> no frames, and a zero total creates invalid normalized bounds.
> Return Result<Vec<DataFrame>>, reject the first negative weight and a
> nonpositive total (including empty input), and propagate the structured error
> through the Python wrapper. All in-repository native callers are updated;
> downstream native callers must use ? or handle the result explicitly.
> Preserve the Python wrapper's original-input error formatting, Rust-native
> float formatting, shared seeds, deterministic ordering, positive-total
> zero-weight splits and PySpark's NaN/infinity comparison semantics. Server
> acceptance of every nonfinite plan remains outside this fix.
> On master 0ac7a0f4e8c98a2d13157175b20c9ebb3c536c28, formatting, release
> library tests, integration-test compilation with and without wasm-udf, and
> the extension build passed. Offline Python suites recorded 2579 passed and 8
> inherited live-server skips. All three new native unit tests and both updated
> native live callers passed; dataframe_metadata_json_semantics and
> replace_to_with_metadata_random_split each recorded 1 passed / 0 ignored
> against Spark. The 12-group API public-name gate and no-stubs audit passed.
> Complete source review round 7 reported no findings. Review and validation
> source manifests match, and the saved extension hash matches its validation
> receipt. API-name checks do not establish complete signatures or behavior.
> This artifact's official configuration and DataFrame-creation plugin smoke
> recorded 5 + 18 passed and 1 skipped against a live Spark 4.3 JVM. Four
> passes are remote-mode guards. The missing-pandas test was skipped because
> pandas was installed. These tests execute the official Python client through
> this artifact's raw-transport plugin; the changed native sampling behavior
> has the separate unit/live evidence above.
> Shared raw-transport coverage is carried from the pandas artifact: the full
> ordinary phase recorded 87 modules, 2586 passed, 641 skipped and 1 manifest
> deselection; the separate cluster/resource phase recorded 4 modules, 15
> passed and 14 skipped, with 10 functional passes and 5 guards. Both had zero
> unexpected failures. A separate ordinary drift check retained one expected
> missing-catalog failure. All 53 assessed shared transport/build inputs match
> this candidate. This artifact has no executed full official-suite or cluster
> pass; the carry combines common-source evidence with its own smoke/native
> checks and does not establish binary equivalence.
> Fresh workspace/all-target Clippy exited 101. All 483 emitted source
> diagnostics and 732 primary/secondary spans map to unchanged saved-master
> statements, with no emitted span on an own-changed statement. This
> inherited-source failure remains recorded; no passing Clippy or executed
> pristine-master lint result is claimed.
> Retained receipts lack complete official-source/dependency whole-tree
> start/end attestations, raw per-module pytest return codes and per-case skip
> provenance. Result-log hashes are assessment-time observations. Rejected
> static fixture overrides mean the plugin tests do not establish forced
> short-stream reattachment. Donor totals count executions; these scoped
> results do not establish complete API or whole-client parity.
> Generated-by: OpenAI Codex
> This pull request and its description were written by OpenAI Codex.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
