This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 57598d3f [SPARK-61216][CONNECT][RUST] Validate native random_split 
weights
57598d3f is described below

commit 57598d3f9a8d250d24083f43cbf9a380d07084db
Author: DB Tsai <[email protected]>
AuthorDate: Mon Nov 9 15:48:58 2026 +0900

    [SPARK-61216][CONNECT][RUST] Validate native random_split weights
    
    ### What changes were proposed in this pull request?
    
    Validate native Rust `DataFrame::random_split` weights before constructing 
sampling plans. Return structured `VALUE_NOT_POSITIVE` errors for the first 
negative weight or a nonpositive total, including empty and all-zero inputs. 
Update the Python binding and every in-repository native caller to handle the 
result.
    
    The independent patch is based on master 
0ac7a0f4e8c98a2d13157175b20c9ebb3c536c28.
    
    ### Why are the changes needed?
    
    The Rust method currently constructs plans for negative weights and zero 
totals, and an empty list silently returns no frames. PySpark rejects these 
inputs before constructing splits. For example, `random_split(vec![-1.0, 2.0], 
Some(42))` now returns `VALUE_NOT_POSITIVE` with `arg_name=weights`; an empty 
input reports `arg_name=sum(weights)`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. The Rust return type changes from `Vec<DataFrame>` to 
`Result<Vec<DataFrame>>`. Downstream native callers must use `?` or handle the 
error explicitly. All in-repository callers are updated.
    
    The Python wrapper retains its validation first, preserving error values 
formatted from the original Python input. Native errors use Rust float 
formatting. Valid weights retain shared seeds, deterministic ordering and 
zero-weight splits when the total is positive. NaN and positive infinity retain 
PySpark's comparison semantics; server acceptance of every nonfinite plan 
remains outside this fix.
    
    ### How was this patch tested?
    
    - Formatting, release core/API library tests, integration-test compilation 
with and without `wasm-udf`, and the native extension build passed. The offline 
Python suites recorded **2,579 passed and 8 inherited live-server skips**.
    - All three new native unit tests passed: invalid weights before plan 
construction, serialized normalized bounds/shared seed/ordering, and nonfinite 
comparison semantics. Both updated native live callers passed against Spark: 
`dataframe_metadata_json_semantics` and 
`replace_to_with_metadata_random_split`, each **1 passed / 0 ignored**.
    - Introspection completed across 12 API groups and passed the public-name 
gate. This checks names, not complete signatures or behavior. The no-stubs 
audit passed. Complete source review round 7 reported no findings. Review and 
validation source manifests match, and the saved extension hash matches its 
validation receipt.
    - This artifact's official configuration and DataFrame-creation plugin 
smoke passed against a live Spark 4.3 JVM: **5 + 18 passed, 1 skipped**, 
including four remote-mode guard passes. The missing-pandas test was skipped 
because pandas was installed. These tests exercise the official Python client 
through this artifact's raw-transport plugin; the changed native sampling 
behavior is covered separately above.
    - Shared raw-transport regression coverage is carried from the pandas 
artifact, which executed the full ordinary phase (**87 modules: 2,586 passed, 
641 skipped, 1 manifest deselection**) and the separate cluster/resource phase 
(**4 modules: 15 passed, 14 skipped**, including 10 functional passes and five 
guards), with zero unexpected failures. The ordinary drift check retained one 
expected missing-catalog failure. All 53 assessed shared transport/build inputs 
match this candidate. Thi [...]
    - Fresh workspace/all-target Clippy exited **101**. Its **483 source 
diagnostics and 732 primary/secondary spans** map to unchanged statements in 
saved master source, with no emitted span on an own-changed statement. This 
inherited-source failure remains recorded; source attribution establishes 
neither a passing Clippy run nor an executed pristine-master lint comparison.
    
    Retained receipts do not provide complete official-source/dependency 
whole-tree start/end attestations, raw per-module pytest return codes or 
per-case skip provenance. Result-log hashes are assessment-time observations. 
Rejected static fixture overrides mean the plugin tests do not establish forced 
short-stream reattachment. Donor totals count executions; these scoped results 
do not establish complete API or whole-client parity.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex
    
    This pull request and its description were written by OpenAI Codex.
    
    Closes #399 from 
dbtsai/codex/residual-random-split-validation-master-0ac7a0f.
    
    Authored-by: DB Tsai <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/dataframe.rs            |  3 +
 crates/spark-connect/src/dataframe.rs         | 87 +++++++++++++++++++++++++--
 crates/spark-connect/tests/e2e_dataframe.rs   |  2 +-
 crates/spark-connect/tests/e2e_new_methods.rs |  2 +-
 4 files changed, 88 insertions(+), 6 deletions(-)

diff --git a/crates/pyspark-rs/src/dataframe.rs 
b/crates/pyspark-rs/src/dataframe.rs
index 39d1789c..f47a4688 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -1914,6 +1914,8 @@ impl PyDataFrame {
                 ],
             ))
         };
+        // Keep Python's checks so arg_value uses str() of the original 
objects, as in
+        // PySpark. The core also validates the extracted weights for native 
Rust callers.
         for weight in weights.try_iter()? {
             let weight = weight?;
             if weight.lt(0.0)? {
@@ -1928,6 +1930,7 @@ impl PyDataFrame {
         Ok(self
             .dataframe
             .random_split(weights, seed)
+            .to_pyerr()?
             .into_iter()
             .map(PyDataFrame::new)
             .collect())
diff --git a/crates/spark-connect/src/dataframe.rs 
b/crates/spark-connect/src/dataframe.rs
index 6807a12c..bdda219c 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -1932,11 +1932,39 @@ impl DataFrame {
     }
 
     /// Randomly split this DataFrame into multiple parts.
-    pub fn random_split(&self, weights: Vec<f64>, seed: Option<i64>) -> 
Vec<DataFrame> {
+    ///
+    /// # Errors
+    ///
+    /// Returns `VALUE_NOT_POSITIVE` for the first negative weight, or for 
weights whose
+    /// sum is nonpositive, including an empty list. These are the same checks 
as PySpark's
+    /// `randomSplit`; zero weights are allowed when the total is positive.
+    /// The native `arg_value` uses Rust's `f64` display format (for example, 
`-1`);
+    /// the Python wrapper retains PySpark's `str()` formatting of the 
original inputs.
+    ///
+    /// This method returns a `Result`: callers must handle the error (for 
example, with
+    /// `?`) before using the split DataFrames.
+    pub fn random_split(&self, weights: Vec<f64>, seed: Option<i64>) -> 
Result<Vec<DataFrame>> {
+        for weight in &weights {
+            if *weight < 0.0 {
+                return Err(SparkError::value(
+                    "VALUE_NOT_POSITIVE",
+                    &[("arg_name", "weights"), ("arg_value", 
&weight.to_string())],
+                ));
+            }
+        }
         // As PySpark's: one seed of every split, of the client without a 
seed, and the input
         // in a deterministic order, so that the splits are disjoint and have 
every row.
         let seed = Some(seed.unwrap_or_else(random_seed));
-        let total: f64 = weights.iter().sum();
+        let total = weights.iter().fold(0.0, |sum, weight| sum + weight);
+        if total <= 0.0 {
+            return Err(SparkError::value(
+                "VALUE_NOT_POSITIVE",
+                &[
+                    ("arg_name", "sum(weights)"),
+                    ("arg_value", &total.to_string()),
+                ],
+            ));
+        }
         let normalized: Vec<f64> = weights.iter().map(|w| w / total).collect();
 
         let mut results = vec![];
@@ -1956,7 +1984,7 @@ impl DataFrame {
             cumulative = upper;
         }
 
-        results
+        Ok(results)
     }
 
     /// Print the schema of this DataFrame.
@@ -4663,6 +4691,7 @@ mod plan_construction_tests {
         let df = spark.range(10).unwrap();
         let seeds: Vec<(Option<i64>, bool)> = df
             .random_split(vec![0.5, 0.5], None)
+            .unwrap()
             .iter()
             .map(|split| match split.plan.as_ref() {
                 LogicalPlan::Sample {
@@ -4690,7 +4719,7 @@ mod plan_construction_tests {
     fn random_split_plan() {
         let spark = session();
         let df = spark.range(10).unwrap();
-        let dfs = df.random_split(vec![0.7, 0.3], None);
+        let dfs = df.random_split(vec![0.7, 0.3], None).unwrap();
         assert_eq!(dfs.len(), 2);
         // Each split is a different DataFrame with its own plan
         for split_df in &dfs {
@@ -4706,6 +4735,56 @@ mod plan_construction_tests {
         }
     }
 
+    #[test]
+    fn random_split_rejects_invalid_weights_before_building_plans() {
+        let df = session().range(10).unwrap();
+        for (weights, argument, value) in [
+            (vec![-1.0, 2.0], "weights", "-1"),
+            (vec![0.0, -2.0, -1.0], "weights", "-2"),
+            (vec![f64::NAN, -1.0], "weights", "-1"),
+            (vec![0.0, 0.0], "sum(weights)", "0"),
+            (vec![], "sum(weights)", "0"),
+        ] {
+            let error = match df.random_split(weights.clone(), Some(42)) {
+                Err(error) => error,
+                Ok(_) => panic!("expected invalid weights to fail: 
{weights:?}"),
+            };
+            assert!(matches!(error.kind, SparkErrorKind::ValueError));
+            assert_eq!(error.error_class, "VALUE_NOT_POSITIVE", "{weights:?}");
+            assert_eq!(error.params["arg_name"], argument, "{weights:?}");
+            assert_eq!(error.params["arg_value"], value, "{weights:?}");
+        }
+    }
+
+    #[test]
+    fn random_split_serializes_normalized_bounds_shared_seed_and_ordering() {
+        use proto::relation::RelType;
+
+        let df = session().range(10).unwrap();
+        let splits = df.random_split(vec![1.0, 0.0, 3.0], Some(42)).unwrap();
+        assert_eq!(splits.len(), 3);
+        for (split, (lower, upper)) in splits.iter().zip([(0.0, 0.25), (0.25, 
0.25), (0.25, 1.0)]) {
+            let Some(RelType::Sample(sample)) = split.relation().rel_type else 
{
+                panic!("expected a serialized Sample relation");
+            };
+            assert_eq!(sample.lower_bound, lower);
+            assert_eq!(sample.upper_bound, upper);
+            assert_eq!(sample.seed, Some(42));
+            assert_eq!(sample.with_replacement, Some(false));
+            assert!(sample.deterministic_order);
+        }
+    }
+
+    #[test]
+    fn random_split_keeps_pyspark_nonfinite_comparison_semantics() {
+        let df = session().range(10).unwrap();
+        // PySpark checks w < 0 and total <= 0; neither comparison rejects NaN 
or +inf.
+        // Any server-side handling of those plans remains separate from 
client validation.
+        for weights in [vec![f64::NAN], vec![f64::INFINITY]] {
+            assert_eq!(df.random_split(weights, Some(42)).unwrap().len(), 1);
+        }
+    }
+
     #[test]
     fn replace_literals_plan() {
         use crate::expression::LiteralExpression;
diff --git a/crates/spark-connect/tests/e2e_dataframe.rs 
b/crates/spark-connect/tests/e2e_dataframe.rs
index d242ffe0..46fb12eb 100644
--- a/crates/spark-connect/tests/e2e_dataframe.rs
+++ b/crates/spark-connect/tests/e2e_dataframe.rs
@@ -131,7 +131,7 @@ fn dataframe_metadata_json_semantics() {
     assert_eq!(s.table("cov_df_view").unwrap().count().unwrap(), 4);
     let _ = df.create_or_replace_global_temp_view("cov_df_gview");
     // random_split returns the requested number of frames
-    let parts = df.random_split(vec![0.5, 0.5], Some(1));
+    let parts = df.random_split(vec![0.5, 0.5], Some(1)).unwrap();
     assert_eq!(parts.len(), 2);
 }
 
diff --git a/crates/spark-connect/tests/e2e_new_methods.rs 
b/crates/spark-connect/tests/e2e_new_methods.rs
index 3fa54e3f..0bb20b90 100644
--- a/crates/spark-connect/tests/e2e_new_methods.rs
+++ b/crates/spark-connect/tests/e2e_new_methods.rs
@@ -135,7 +135,7 @@ fn replace_to_with_metadata_random_split() {
     assert_eq!(with_md.collect().expect("with_metadata collect").len(), 5);
 
     // randomSplit returns multiple DataFrames whose row counts sum to the 
input.
-    let parts = df.random_split(vec![0.5, 0.5], Some(1));
+    let parts = df.random_split(vec![0.5, 0.5], Some(1)).unwrap();
     let total: usize = parts
         .iter()
         .map(|p| p.clone().collect().expect("split collect").len())


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to