This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 57598d3f [SPARK-61216][CONNECT][RUST] Validate native random_split
weights
57598d3f is described below
commit 57598d3f9a8d250d24083f43cbf9a380d07084db
Author: DB Tsai <[email protected]>
AuthorDate: Mon Nov 9 15:48:58 2026 +0900
[SPARK-61216][CONNECT][RUST] Validate native random_split weights
### What changes were proposed in this pull request?
Validate native Rust `DataFrame::random_split` weights before constructing
sampling plans. Return structured `VALUE_NOT_POSITIVE` errors for the first
negative weight or a nonpositive total, including empty and all-zero inputs.
Update the Python binding and every in-repository native caller to handle the
result.
The independent patch is based on master
0ac7a0f4e8c98a2d13157175b20c9ebb3c536c28.
### Why are the changes needed?
The Rust method currently constructs plans for negative weights and zero
totals, and an empty list silently returns no frames. PySpark rejects these
inputs before constructing splits. For example, `random_split(vec![-1.0, 2.0],
Some(42))` now returns `VALUE_NOT_POSITIVE` with `arg_name=weights`; an empty
input reports `arg_name=sum(weights)`.
### Does this PR introduce _any_ user-facing change?
Yes. The Rust return type changes from `Vec<DataFrame>` to
`Result<Vec<DataFrame>>`. Downstream native callers must use `?` or handle the
error explicitly. All in-repository callers are updated.
The Python wrapper retains its validation first, preserving error values
formatted from the original Python input. Native errors use Rust float
formatting. Valid weights retain shared seeds, deterministic ordering and
zero-weight splits when the total is positive. NaN and positive infinity retain
PySpark's comparison semantics; server acceptance of every nonfinite plan
remains outside this fix.
### How was this patch tested?
- Formatting, release core/API library tests, integration-test compilation
with and without `wasm-udf`, and the native extension build passed. The offline
Python suites recorded **2,579 passed and 8 inherited live-server skips**.
- All three new native unit tests passed: invalid weights before plan
construction, serialized normalized bounds/shared seed/ordering, and nonfinite
comparison semantics. Both updated native live callers passed against Spark:
`dataframe_metadata_json_semantics` and
`replace_to_with_metadata_random_split`, each **1 passed / 0 ignored**.
- Introspection completed across 12 API groups and passed the public-name
gate. This checks names, not complete signatures or behavior. The no-stubs
audit passed. Complete source review round 7 reported no findings. Review and
validation source manifests match, and the saved extension hash matches its
validation receipt.
- This artifact's official configuration and DataFrame-creation plugin
smoke passed against a live Spark 4.3 JVM: **5 + 18 passed, 1 skipped**,
including four remote-mode guard passes. The missing-pandas test was skipped
because pandas was installed. These tests exercise the official Python client
through this artifact's raw-transport plugin; the changed native sampling
behavior is covered separately above.
- Shared raw-transport regression coverage is carried from the pandas
artifact, which executed the full ordinary phase (**87 modules: 2,586 passed,
641 skipped, 1 manifest deselection**) and the separate cluster/resource phase
(**4 modules: 15 passed, 14 skipped**, including 10 functional passes and five
guards), with zero unexpected failures. The ordinary drift check retained one
expected missing-catalog failure. All 53 assessed shared transport/build inputs
match this candidate. Thi [...]
- Fresh workspace/all-target Clippy exited **101**. Its **483 source
diagnostics and 732 primary/secondary spans** map to unchanged statements in
saved master source, with no emitted span on an own-changed statement. This
inherited-source failure remains recorded; source attribution establishes
neither a passing Clippy run nor an executed pristine-master lint comparison.
Retained receipts do not provide complete official-source/dependency
whole-tree start/end attestations, raw per-module pytest return codes or
per-case skip provenance. Result-log hashes are assessment-time observations.
Rejected static fixture overrides mean the plugin tests do not establish forced
short-stream reattachment. Donor totals count executions; these scoped results
do not establish complete API or whole-client parity.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex
This pull request and its description were written by OpenAI Codex.
Closes #399 from
dbtsai/codex/residual-random-split-validation-master-0ac7a0f.
Authored-by: DB Tsai <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/dataframe.rs | 3 +
crates/spark-connect/src/dataframe.rs | 87 +++++++++++++++++++++++++--
crates/spark-connect/tests/e2e_dataframe.rs | 2 +-
crates/spark-connect/tests/e2e_new_methods.rs | 2 +-
4 files changed, 88 insertions(+), 6 deletions(-)
diff --git a/crates/pyspark-rs/src/dataframe.rs
b/crates/pyspark-rs/src/dataframe.rs
index 39d1789c..f47a4688 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -1914,6 +1914,8 @@ impl PyDataFrame {
],
))
};
+ // Keep Python's checks so arg_value uses str() of the original
objects, as in
+ // PySpark. The core also validates the extracted weights for native
Rust callers.
for weight in weights.try_iter()? {
let weight = weight?;
if weight.lt(0.0)? {
@@ -1928,6 +1930,7 @@ impl PyDataFrame {
Ok(self
.dataframe
.random_split(weights, seed)
+ .to_pyerr()?
.into_iter()
.map(PyDataFrame::new)
.collect())
diff --git a/crates/spark-connect/src/dataframe.rs
b/crates/spark-connect/src/dataframe.rs
index 6807a12c..bdda219c 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -1932,11 +1932,39 @@ impl DataFrame {
}
/// Randomly split this DataFrame into multiple parts.
- pub fn random_split(&self, weights: Vec<f64>, seed: Option<i64>) ->
Vec<DataFrame> {
+ ///
+ /// # Errors
+ ///
+ /// Returns `VALUE_NOT_POSITIVE` for the first negative weight, or for
weights whose
+ /// sum is nonpositive, including an empty list. These are the same checks
as PySpark's
+ /// `randomSplit`; zero weights are allowed when the total is positive.
+ /// The native `arg_value` uses Rust's `f64` display format (for example,
`-1`);
+ /// the Python wrapper retains PySpark's `str()` formatting of the
original inputs.
+ ///
+ /// This method returns a `Result`: callers must handle the error (for
example, with
+ /// `?`) before using the split DataFrames.
+ pub fn random_split(&self, weights: Vec<f64>, seed: Option<i64>) ->
Result<Vec<DataFrame>> {
+ for weight in &weights {
+ if *weight < 0.0 {
+ return Err(SparkError::value(
+ "VALUE_NOT_POSITIVE",
+ &[("arg_name", "weights"), ("arg_value",
&weight.to_string())],
+ ));
+ }
+ }
// As PySpark's: one seed of every split, of the client without a
seed, and the input
// in a deterministic order, so that the splits are disjoint and have
every row.
let seed = Some(seed.unwrap_or_else(random_seed));
- let total: f64 = weights.iter().sum();
+ let total = weights.iter().fold(0.0, |sum, weight| sum + weight);
+ if total <= 0.0 {
+ return Err(SparkError::value(
+ "VALUE_NOT_POSITIVE",
+ &[
+ ("arg_name", "sum(weights)"),
+ ("arg_value", &total.to_string()),
+ ],
+ ));
+ }
let normalized: Vec<f64> = weights.iter().map(|w| w / total).collect();
let mut results = vec![];
@@ -1956,7 +1984,7 @@ impl DataFrame {
cumulative = upper;
}
- results
+ Ok(results)
}
/// Print the schema of this DataFrame.
@@ -4663,6 +4691,7 @@ mod plan_construction_tests {
let df = spark.range(10).unwrap();
let seeds: Vec<(Option<i64>, bool)> = df
.random_split(vec![0.5, 0.5], None)
+ .unwrap()
.iter()
.map(|split| match split.plan.as_ref() {
LogicalPlan::Sample {
@@ -4690,7 +4719,7 @@ mod plan_construction_tests {
fn random_split_plan() {
let spark = session();
let df = spark.range(10).unwrap();
- let dfs = df.random_split(vec![0.7, 0.3], None);
+ let dfs = df.random_split(vec![0.7, 0.3], None).unwrap();
assert_eq!(dfs.len(), 2);
// Each split is a different DataFrame with its own plan
for split_df in &dfs {
@@ -4706,6 +4735,56 @@ mod plan_construction_tests {
}
}
+ #[test]
+ fn random_split_rejects_invalid_weights_before_building_plans() {
+ let df = session().range(10).unwrap();
+ for (weights, argument, value) in [
+ (vec![-1.0, 2.0], "weights", "-1"),
+ (vec![0.0, -2.0, -1.0], "weights", "-2"),
+ (vec![f64::NAN, -1.0], "weights", "-1"),
+ (vec![0.0, 0.0], "sum(weights)", "0"),
+ (vec![], "sum(weights)", "0"),
+ ] {
+ let error = match df.random_split(weights.clone(), Some(42)) {
+ Err(error) => error,
+ Ok(_) => panic!("expected invalid weights to fail:
{weights:?}"),
+ };
+ assert!(matches!(error.kind, SparkErrorKind::ValueError));
+ assert_eq!(error.error_class, "VALUE_NOT_POSITIVE", "{weights:?}");
+ assert_eq!(error.params["arg_name"], argument, "{weights:?}");
+ assert_eq!(error.params["arg_value"], value, "{weights:?}");
+ }
+ }
+
+ #[test]
+ fn random_split_serializes_normalized_bounds_shared_seed_and_ordering() {
+ use proto::relation::RelType;
+
+ let df = session().range(10).unwrap();
+ let splits = df.random_split(vec![1.0, 0.0, 3.0], Some(42)).unwrap();
+ assert_eq!(splits.len(), 3);
+ for (split, (lower, upper)) in splits.iter().zip([(0.0, 0.25), (0.25,
0.25), (0.25, 1.0)]) {
+ let Some(RelType::Sample(sample)) = split.relation().rel_type else
{
+ panic!("expected a serialized Sample relation");
+ };
+ assert_eq!(sample.lower_bound, lower);
+ assert_eq!(sample.upper_bound, upper);
+ assert_eq!(sample.seed, Some(42));
+ assert_eq!(sample.with_replacement, Some(false));
+ assert!(sample.deterministic_order);
+ }
+ }
+
+ #[test]
+ fn random_split_keeps_pyspark_nonfinite_comparison_semantics() {
+ let df = session().range(10).unwrap();
+ // PySpark checks w < 0 and total <= 0; neither comparison rejects NaN
or +inf.
+ // Any server-side handling of those plans remains separate from
client validation.
+ for weights in [vec![f64::NAN], vec![f64::INFINITY]] {
+ assert_eq!(df.random_split(weights, Some(42)).unwrap().len(), 1);
+ }
+ }
+
#[test]
fn replace_literals_plan() {
use crate::expression::LiteralExpression;
diff --git a/crates/spark-connect/tests/e2e_dataframe.rs
b/crates/spark-connect/tests/e2e_dataframe.rs
index d242ffe0..46fb12eb 100644
--- a/crates/spark-connect/tests/e2e_dataframe.rs
+++ b/crates/spark-connect/tests/e2e_dataframe.rs
@@ -131,7 +131,7 @@ fn dataframe_metadata_json_semantics() {
assert_eq!(s.table("cov_df_view").unwrap().count().unwrap(), 4);
let _ = df.create_or_replace_global_temp_view("cov_df_gview");
// random_split returns the requested number of frames
- let parts = df.random_split(vec![0.5, 0.5], Some(1));
+ let parts = df.random_split(vec![0.5, 0.5], Some(1)).unwrap();
assert_eq!(parts.len(), 2);
}
diff --git a/crates/spark-connect/tests/e2e_new_methods.rs
b/crates/spark-connect/tests/e2e_new_methods.rs
index 3fa54e3f..0bb20b90 100644
--- a/crates/spark-connect/tests/e2e_new_methods.rs
+++ b/crates/spark-connect/tests/e2e_new_methods.rs
@@ -135,7 +135,7 @@ fn replace_to_with_metadata_random_split() {
assert_eq!(with_md.collect().expect("with_metadata collect").len(), 5);
// randomSplit returns multiple DataFrames whose row counts sum to the
input.
- let parts = df.random_split(vec![0.5, 0.5], Some(1));
+ let parts = df.random_split(vec![0.5, 0.5], Some(1)).unwrap();
let total: usize = parts
.iter()
.map(|p| p.clone().collect().expect("split collect").len())
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]