DB Tsai created SPARK-61216:
-------------------------------
Summary: Validate native random_split weights in Spark Connect Rust
Key: SPARK-61216
URL: https://issues.apache.org/jira/browse/SPARK-61216
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Native Rust DataFrame::random_split constructs sampling plans for negative
weights and nonpositive totals, while PySpark randomSplit raises
VALUE_NOT_POSITIVE before constructing splits. An empty list silently returns
no frames, and a zero total creates invalid normalized bounds.
Return Result<Vec<DataFrame>>, reject the first negative weight and a
nonpositive total (including empty input), and propagate the structured error
through the Python wrapper. All in-repository native callers are updated;
downstream native callers must use ? or handle the result explicitly. Preserve
the Python wrapper's original-input error formatting, Rust-native float
formatting, shared seeds, deterministic ordering, positive-total zero-weight
splits and PySpark's NaN/infinity comparison semantics. Server acceptance of
every nonfinite plan remains outside this fix.
On master 0ac7a0f4e8c98a2d13157175b20c9ebb3c536c28, formatting, release library
tests, integration-test compilation with and without wasm-udf, and the
extension build passed. Offline Python suites recorded 2579 passed and 8
inherited live-server skips. All three new native unit tests and both updated
native live callers passed; dataframe_metadata_json_semantics and
replace_to_with_metadata_random_split each recorded 1 passed / 0 ignored
against Spark. The 12-group API public-name gate and no-stubs audit passed.
Complete source review round 7 reported no findings. Review and validation
source manifests match, and the saved extension hash matches its validation
receipt. API-name checks do not establish complete signatures or behavior.
This artifact's official configuration and DataFrame-creation plugin smoke
recorded 5 + 18 passed and 1 skipped against a live Spark 4.3 JVM. Four passes
are remote-mode guards. The missing-pandas test was skipped because pandas was
installed. These tests execute the official Python client through this
artifact's raw-transport plugin; the changed native sampling behavior has the
separate unit/live evidence above.
Shared raw-transport coverage is carried from the pandas artifact: the full
ordinary phase recorded 87 modules, 2586 passed, 641 skipped and 1 manifest
deselection; the separate cluster/resource phase recorded 4 modules, 15 passed
and 14 skipped, with 10 functional passes and 5 guards. Both had zero
unexpected failures. A separate ordinary drift check retained one expected
missing-catalog failure. All 53 assessed shared transport/build inputs match
this candidate. This artifact has no executed full official-suite or cluster
pass; the carry combines common-source evidence with its own smoke/native
checks and does not establish binary equivalence.
Fresh workspace/all-target Clippy exited 101. All 483 emitted source
diagnostics and 732 primary/secondary spans map to unchanged saved-master
statements, with no emitted span on an own-changed statement. This
inherited-source failure remains recorded; no passing Clippy or executed
pristine-master lint result is claimed.
Retained receipts lack complete official-source/dependency whole-tree start/end
attestations, raw per-module pytest return codes and per-case skip provenance.
Result-log hashes are assessment-time observations. Rejected static fixture
overrides mean the plugin tests do not establish forced short-stream
reattachment. Donor totals count executions; these scoped results do not
establish complete API or whole-client parity.
Generated-by: OpenAI Codex
This pull request and its description were written by OpenAI Codex.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
