This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new b54c3cd  [SPARK-61092][CONNECT][PYTHON] Take a client seed in sample 
and randomSplit, and the arguments of withMetadata, withWatermark and 
randomSplit and the attributes of a DataFrame, in the Python client as PySpark 
does
b54c3cd is described below

commit b54c3cd7bbb223cddf7925e7d2dd781ee266bd33
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 16:11:15 2026 +0900

    [SPARK-61092][CONNECT][PYTHON] Take a client seed in sample and 
randomSplit, and the arguments of withMetadata, withWatermark and randomSplit 
and the attributes of a DataFrame, in the Python client as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61092](https://issues.apache.org/jira/browse/SPARK-61092): `sample` 
and `randomSplit` now choose their seed in the client, and several methods and 
attributes of a DataFrame take their arguments as PySpark Connect's do.
    
    As PySpark Connect's:
    - `sample` and `randomSplit` without a seed choose a random seed in the 
client (`random.randint(0, sys.maxsize)`).
      - Every action of a sample returns the same rows.
      - All splits of `randomSplit` use one seed and an input in a 
deterministic order (the `deterministic_order` of their `Sample`), so they are 
disjoint and together hold every row.
    - `randomSplit` raises `VALUE_NOT_POSITIVE` for a negative weight 
(`arg_name` "weights") and for weights whose sum is not positive (`arg_name` 
"sum(weights)").
    - `withMetadata` takes a dict of any JSON values, which it sends as 
`json.dumps(metadata)`. It raises `NOT_EXPECTED_TYPE` for metadata that is not 
a dict.
    - `withWatermark` raises `NOT_EXPECTED_TYPE` for an event time or delay 
that is empty or not a str.
    - `df.name` is the column `name` for any name except a dunder one. Unless 
`PYSPARK_VALIDATE_COLUMN_NAME_LEGACY` is `1`, a name is only checked against 
the columns when it is a dunder name. A checked name that is not a column 
raises `ATTRIBUTE_NOT_SUPPORTED`.
    - `_jseq`, `_jdf`, `_jmap` and `_jcols` raise 
`JVM_ATTRIBUTE_NOT_SUPPORTED`, and `rdd` raises `NOT_IMPLEMENTED`.
    
    In the core, the `Sample` plan gains `deterministic_order`, and `sample` 
and `random_split` choose a seed when none is given. 
`DataFrame::with_metadata_json` sets the metadata of a column from its JSON.
    
    ### Why are the changes needed?
    
    The client sent no seed, so the server chose a new one at each execution:
    - each action of `df.sample(0.5)` returned other rows
    - the splits of `randomSplit([0.5, 0.5])` lost and repeated rows: they had 
2003, 1994 and 2060 rows of 2000 in three runs
    
    The client also:
    - sent negative weights to the server (`IllegalArgumentException`), and an 
empty event time (`PARSE_EMPTY_STATEMENT`)
    - accepted only str values of metadata
    - raised plain `TypeError`s for these arguments, and an `AttributeError` 
for any name starting with `_`
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Samples and splits are consistent, and these methods take the 
arguments, and raise the errors, of PySpark 4.3.0.
    
    ### How was this patch tested?
    
    - A new live test checks that two actions of a sample return the same rows, 
that the splits of `randomSplit` are disjoint and hold every row, and the 
metadata of `withMetadata` of non-str values. It passes with official PySpark 
4.3.0 and fails without this change.
    - A new offline test checks the errors of `randomSplit`, `withMetadata` and 
`withWatermark`, and the attributes of a DataFrame, including 
`PYSPARK_VALIDATE_COLUMN_NAME_LEGACY`.
    - A new lib test checks the seed of the splits and of a sample, and the 
deterministic order of the splits.
    - On a local server, the argument errors of 64 cases of DataFrame methods, 
and 14 attributes, give the same results as official PySpark 4.3.0. The only 
exceptions are the messages of `TypeError`s and `AssertionError`s that come 
from protobuf or from Python internals.
    - The lib tests of both crates (595 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2045 passed, 12 skipped), and the 
live drop-in and Arrow suites (148 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #296 from HyukjinKwon/SPARK-61092-df-arguments.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/dataframe.rs    | 120 +++++++++++++++++++++++++++++-----
 crates/spark-connect/src/dataframe.rs |  55 +++++++++++++++-
 crates/spark-connect/src/plan.rs      |   6 ++
 python/tests/test_dropin_e2e.py       |  16 +++++
 python/tests/test_dropin_offline.py   |  50 ++++++++++++++
 5 files changed, 226 insertions(+), 21 deletions(-)

diff --git a/crates/pyspark-rs/src/dataframe.rs 
b/crates/pyspark-rs/src/dataframe.rs
index 1de5ccc..1d47c1a 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -503,6 +503,19 @@ fn cond_column(condition: &Bound<'_, PyAny>) -> 
PyResult<spark_connect::column::
     }
 }
 
+/// An argument of `withWatermark`, as PySpark's: `NOT_EXPECTED_TYPE` for an 
empty one, or one
+/// whose type is not str.
+fn watermark_argument(value: &Bound<'_, PyAny>, name: &str) -> 
PyResult<String> {
+    if !value.is_truthy()?
+        || !value
+            .get_type()
+            .is(value.py().get_type::<pyo3::types::PyString>())
+    {
+        return Err(crate::stat_arguments::not_expected_type(value, name, 
"str"));
+    }
+    value.extract()
+}
+
 #[pymethods]
 impl PyDataFrame {
     /// Select columns (accepts Column objects or string names).
@@ -1593,30 +1606,82 @@ impl PyDataFrame {
         Ok(PyDataFrame::new(self.dataframe.to(dt)))
     }
 
-    /// Set metadata (a dict) on a column.
+    /// Set the metadata of a column, as PySpark's: a dict of any JSON values, 
which it sends as
+    /// `json.dumps(metadata)`, and `NOT_EXPECTED_TYPE` for one that is not a 
dict.
     #[pyo3(name = "withMetadata")]
     fn with_metadata(
         &self,
+        py: Python<'_>,
         columnName: &str,
-        metadata: std::collections::HashMap<String, String>,
-    ) -> PyDataFrame {
-        PyDataFrame::new(self.dataframe.with_metadata(columnName, metadata))
+        metadata: &Bound<'_, PyAny>,
+    ) -> PyResult<PyDataFrame> {
+        if !metadata.is_instance_of::<PyDict>() {
+            return Err(crate::stat_arguments::not_expected_type(
+                metadata, "metadata", "dict",
+            ));
+        }
+        let json: String = py
+            .import("json")?
+            .call_method1("dumps", (metadata,))?
+            .extract()?;
+        Ok(PyDataFrame::new(
+            self.dataframe.with_metadata_json(columnName, &json),
+        ))
     }
 
-    /// Define an event-time watermark.
+    /// Define an event-time watermark, as PySpark's: `NOT_EXPECTED_TYPE` for 
an empty event time
+    /// or delay, or one that is not a str.
     #[pyo3(name = "withWatermark")]
-    fn with_watermark(&self, eventTime: &str, delayThreshold: &str) -> 
PyDataFrame {
-        PyDataFrame::new(self.dataframe.with_watermark(eventTime, 
delayThreshold))
+    fn with_watermark(
+        &self,
+        eventTime: &Bound<'_, PyAny>,
+        delayThreshold: &Bound<'_, PyAny>,
+    ) -> PyResult<PyDataFrame> {
+        let event_time = watermark_argument(eventTime, "eventTime")?;
+        let delay_threshold = watermark_argument(delayThreshold, 
"delayThreshold")?;
+        Ok(PyDataFrame::new(
+            self.dataframe.with_watermark(&event_time, &delay_threshold),
+        ))
     }
 
     /// Randomly split into multiple DataFrames by the given weights.
+    /// As PySpark's: `VALUE_NOT_POSITIVE` for a negative weight and for 
weights of a sum that is
+    /// not positive, of their `str`; without a seed, the client chooses one 
of every split.
     #[pyo3(name = "randomSplit", signature = (weights, seed=None))]
-    fn random_split(&self, weights: Vec<f64>, seed: Option<i64>) -> 
Vec<PyDataFrame> {
-        self.dataframe
+    fn random_split(
+        &self,
+        weights: &Bound<'_, PyAny>,
+        seed: Option<i64>,
+    ) -> PyResult<Vec<PyDataFrame>> {
+        let py = weights.py();
+        let not_positive = |arg_name: &str, value: &Bound<'_, PyAny>| -> 
PyResult<PyErr> {
+            Ok(crate::column::raise_pyspark(
+                py,
+                "PySparkValueError",
+                "VALUE_NOT_POSITIVE",
+                &[
+                    ("arg_name", arg_name),
+                    ("arg_value", &value.str()?.to_cow()?),
+                ],
+            ))
+        };
+        for weight in weights.try_iter()? {
+            let weight = weight?;
+            if weight.lt(0.0)? {
+                return Err(not_positive("weights", &weight)?);
+            }
+        }
+        let total = py.import("builtins")?.call_method1("sum", (weights,))?;
+        if total.le(0)? {
+            return Err(not_positive("sum(weights)", &total)?);
+        }
+        let weights: Vec<f64> = weights.extract()?;
+        Ok(self
+            .dataframe
             .random_split(weights, seed)
             .into_iter()
             .map(PyDataFrame::new)
-            .collect()
+            .collect())
     }
 
     /// The list of files that make up this DataFrame.
@@ -1670,11 +1735,15 @@ impl PyDataFrame {
         crate::session::PySparkSession::new(self.dataframe.spark_session())
     }
 
-    /// Not supported on Spark Connect — `DataFrame.rdd` requires the legacy 
RDD API.
+    /// Not supported on Spark Connect, as PySpark Connect's: 
`NOT_IMPLEMENTED` of `rdd`, which
+    /// needs the RDD API of a classic session.
     #[getter]
-    fn rdd(&self) -> PyResult<()> {
-        Err(PyErr::new::<pyo3::exceptions::PyNotImplementedError, _>(
-            "rdd() is not supported for Spark Connect",
+    fn rdd(&self, py: Python<'_>) -> PyResult<()> {
+        Err(crate::column::raise_pyspark(
+            py,
+            "PySparkNotImplementedError",
+            "NOT_IMPLEMENTED",
+            &[("feature", "rdd")],
         ))
     }
 
@@ -1789,10 +1858,25 @@ impl PyDataFrame {
     /// Column access by attribute (`df.colname` -> Column), mirroring 
reference pyspark: the
     /// column of this DataFrame, by its plan ID. Names beginning with `_` 
raise
     /// AttributeError so normal attribute lookup works.
-    fn __getattr__(&self, name: &str) -> PyResult<PyColumn> {
-        if name.starts_with('_') {
-            return Err(PyErr::new::<pyo3::exceptions::PyAttributeError, _>(
-                name.to_string(),
+    fn __getattr__(&self, py: Python<'_>, name: &str) -> PyResult<PyColumn> {
+        // As PySpark Connect's: the attributes of the JVM of a classic 
DataFrame are not
+        // supported, and a name is validated as a column only for a dunder 
name, which Python
+        // may look up for an attribute, or with 
PYSPARK_VALIDATE_COLUMN_NAME_LEGACY.
+        if ["_jseq", "_jdf", "_jmap", "_jcols", "rdd"].contains(&name) {
+            return Err(crate::column::raise_pyspark(
+                py,
+                "PySparkAttributeError",
+                "JVM_ATTRIBUTE_NOT_SUPPORTED",
+                &[("attr_name", name)],
+            ));
+        }
+        let legacy = 
std::env::var("PYSPARK_VALIDATE_COLUMN_NAME_LEGACY").as_deref() == Ok("1");
+        if (legacy || name.starts_with("__")) && 
!self.columns(py)?.iter().any(|c| c == name) {
+            return Err(crate::column::raise_pyspark(
+                py,
+                "PySparkAttributeError",
+                "ATTRIBUTE_NOT_SUPPORTED",
+                &[("attr_name", name)],
             ));
         }
         Ok(PyColumn::new(self.dataframe.col(name)))
diff --git a/crates/spark-connect/src/dataframe.rs 
b/crates/spark-connect/src/dataframe.rs
index 838789b..587635c 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -758,7 +758,8 @@ impl DataFrame {
         Ok(())
     }
 
-    /// Sample rows.
+    /// Sample rows. Without a seed, the client chooses a random one, as 
PySpark's, so that every
+    /// execution of the sample returns the same rows.
     pub fn sample(&self, fraction: f64, seed: Option<i64>) -> DataFrame {
         self.sample_opt(fraction, false, seed)
     }
@@ -776,7 +777,8 @@ impl DataFrame {
             lower_bound: 0.0,
             upper_bound: fraction,
             with_replacement,
-            seed,
+            seed: Some(seed.unwrap_or_else(random_seed)),
+            deterministic_order: false,
         };
         self.with_cached_schema(plan)
     }
@@ -1759,6 +1761,9 @@ impl DataFrame {
 
     /// Randomly split this DataFrame into multiple parts.
     pub fn random_split(&self, weights: Vec<f64>, seed: Option<i64>) -> 
Vec<DataFrame> {
+        // As PySpark's: one seed of every split, of the client without a 
seed, and the input
+        // in a deterministic order, so that the splits are disjoint and have 
every row.
+        let seed = Some(seed.unwrap_or_else(random_seed));
         let total: f64 = weights.iter().sum();
         let normalized: Vec<f64> = weights.iter().map(|w| w / total).collect();
 
@@ -1773,6 +1778,7 @@ impl DataFrame {
                 upper_bound: upper,
                 with_replacement: false,
                 seed,
+                deterministic_order: true,
             };
             results.push(self.with_cached_schema(plan));
             cumulative = upper;
@@ -1962,10 +1968,16 @@ impl DataFrame {
     /// re-selected with the given metadata attached (serialized to a JSON 
map).
     pub fn with_metadata(&self, column_name: &str, metadata: HashMap<String, 
String>) -> DataFrame {
         let metadata_json = 
serde_json::to_string(&metadata).unwrap_or_else(|_| "{}".to_string());
+        self.with_metadata_json(column_name, &metadata_json)
+    }
+
+    /// Set the metadata of a column of its JSON, of any JSON values, as 
PySpark's
+    /// `withMetadata` sends `json.dumps(metadata)`.
+    pub fn with_metadata_json(&self, column_name: &str, metadata_json: &str) 
-> DataFrame {
         let plan = LogicalPlan::WithColumnMetadata {
             input: Arc::clone(&self.plan),
             column_name: column_name.to_string(),
-            metadata_json,
+            metadata_json: metadata_json.to_string(),
         };
         DataFrame::new(self.session.clone(), plan)
     }
@@ -2296,6 +2308,12 @@ fn set_observations(
     }
 }
 
+/// A random seed of a sample without one, as PySpark's `random.randint(0, 
sys.maxsize)`: the
+/// client chooses it, so that every execution of the sample returns the same 
rows.
+fn random_seed() -> i64 {
+    (uuid::Uuid::new_v4().as_u128() >> 65) as i64
+}
+
 /// Decode an Arrow batch into rows.
 pub(crate) fn decode_arrow_batch(
     batch: &proto::execute_plan_response::ArrowBatch,
@@ -4181,6 +4199,37 @@ mod plan_construction_tests {
         }
     }
 
+    /// As PySpark's: without a seed, the client takes one, of every split of 
`random_split`,
+    /// whose input is in a deterministic order, and of a sample.
+    #[test]
+    fn samples_take_a_seed_of_the_client() {
+        let spark = session();
+        let df = spark.range(10).unwrap();
+        let seeds: Vec<(Option<i64>, bool)> = df
+            .random_split(vec![0.5, 0.5], None)
+            .iter()
+            .map(|split| match split.plan.as_ref() {
+                LogicalPlan::Sample {
+                    seed,
+                    deterministic_order,
+                    ..
+                } => (*seed, *deterministic_order),
+                other => panic!("expected a Sample plan, got {other:?}"),
+            })
+            .collect();
+        assert!(seeds[0].0.is_some_and(|seed| seed >= 0));
+        assert_eq!(seeds[0], seeds[1]);
+        assert!(seeds[0].1);
+        match df.sample(0.5, None).plan.as_ref() {
+            LogicalPlan::Sample {
+                seed: Some(_),
+                deterministic_order: false,
+                ..
+            } => {}
+            other => panic!("expected a Sample plan of a seed, got {other:?}"),
+        }
+    }
+
     #[test]
     fn random_split_plan() {
         let spark = session();
diff --git a/crates/spark-connect/src/plan.rs b/crates/spark-connect/src/plan.rs
index bef5092..bce9c66 100644
--- a/crates/spark-connect/src/plan.rs
+++ b/crates/spark-connect/src/plan.rs
@@ -126,6 +126,9 @@ pub enum LogicalPlan {
         upper_bound: f64,
         with_replacement: bool,
         seed: Option<i64>,
+        /// Whether the input is sorted, or cached, to be in a deterministic 
order, as the
+        /// splits of `randomSplit` are, so that they are disjoint.
+        deterministic_order: bool,
     },
     /// Repartition: `df.repartition(num_partitions)`.
     Repartition {
@@ -921,6 +924,7 @@ impl LogicalPlan {
                 upper_bound,
                 with_replacement,
                 seed,
+                deterministic_order,
             } => {
                 let mut sample = proto::Sample::default();
                 sample.input = Some(Box::new(relation_of(input, plan_ids)));
@@ -930,6 +934,7 @@ impl LogicalPlan {
                 if let Some(s) = seed {
                     sample.seed = Some(*s);
                 }
+                sample.deterministic_order = *deterministic_order;
                 relation.rel_type = 
Some(proto::relation::RelType::Sample(Box::new(sample)));
             }
 
@@ -1841,6 +1846,7 @@ pub fn sample(
         upper_bound,
         with_replacement,
         seed,
+        deterministic_order: false,
     }
 }
 
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 328547a..868dfc4 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -2531,6 +2531,22 @@ def 
test_create_dataframe_raises_merge_errors_as_pyspark(session):
         assert error.value.getMessageParameters() == parameters, data
 
 
+def test_samples_and_splits_take_a_seed_of_the_client_as_pyspark(session):
+    # As PySpark's: without a seed, the client takes one, so every action of a 
sample returns
+    # the same rows, and the splits of randomSplit are disjoint and have every 
row; and the
+    # metadata of withMetadata takes any JSON value.
+    df = session.range(0, 2000, 1, 8)
+    sample = df.sample(0.5)
+    assert sorted(row.id for row in sample.collect()) == sorted(row.id for row 
in sample.collect())
+    first, second = df.randomSplit([0.5, 0.5])
+    assert first.count() + second.count() == 2000
+    assert first.intersect(second).count() == 0
+    metadata = {"k": 1, "l": [1, 2], "s": "v"}
+    assert df.withMetadata("id", metadata).schema["id"].metadata == metadata
+
+
+
+
 def 
test_persist_cache_and_unpersist_return_the_dataframe_itself_as_pyspark(session):
     # As PySpark's: the DataFrame itself, cached at the storage level given.
     from pyspark.storagelevel import StorageLevel
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 4d9a75d..378d796 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -7373,6 +7373,56 @@ def 
test_schema_cache_set_operations_require_equal_warm_schemas(mock_connect, op
     assert left.columns == ["id"]
 
 
+def 
test_dataframes_take_their_arguments_and_attributes_as_pyspark(mock_connect, 
monkeypatch):
+    # As PySpark Connect's: the errors of randomSplit, withMetadata and 
withWatermark, and the
+    # attributes of a DataFrame: a column of any name but a dunder one, which 
must be a column,
+    # and JVM_ATTRIBUTE_NOT_SUPPORTED of those of a classic DataFrame.
+    from pyspark.errors import (
+        PySparkAttributeError,
+        PySparkNotImplementedError,
+        PySparkTypeError,
+        PySparkValueError,
+    )
+
+    spark, _ = mock_connect
+    df = spark.range(3)
+    for weights, arg_name, arg_value in (([1.0, -1.0], "weights", "-1.0"),
+                                          ([0, 0], "sum(weights)", "0")):
+        with pytest.raises(PySparkValueError) as e:
+            df.randomSplit(weights)
+        assert e.value.getCondition() == "VALUE_NOT_POSITIVE"
+        assert e.value.getMessageParameters() == {"arg_name": arg_name, 
"arg_value": arg_value}
+    with pytest.raises(PySparkTypeError) as e:
+        df.withMetadata("id", 1)
+    assert e.value.getMessageParameters() == {
+        "expected_type": "dict", "arg_name": "metadata", "arg_type": "int"}
+    df.withMetadata("id", {"k": 1, "l": [1, 2]})
+    for event_time, delay, arg_name, arg_type in ((1, "1 minute", "eventTime", 
"int"),
+                                                  ("", "1 minute", 
"eventTime", "str"),
+                                                  ("id", None, 
"delayThreshold", "NoneType")):
+        with pytest.raises(PySparkTypeError) as e:
+            df.withWatermark(event_time, delay)
+        assert e.value.getMessageParameters() == {
+            "arg_name": arg_name, "expected_type": "str", "arg_type": arg_type}
+    assert type(df._not_a_dunder).__name__ == "Column"
+    for name in ("_jseq", "_jdf", "_jmap", "_jcols"):
+        with pytest.raises(PySparkAttributeError) as e:
+            getattr(df, name)
+        assert e.value.getCondition() == "JVM_ATTRIBUTE_NOT_SUPPORTED"
+    with pytest.raises(PySparkNotImplementedError) as e:
+        df.rdd
+    assert e.value.getMessageParameters() == {"feature": "rdd"}
+    with pytest.raises(PySparkAttributeError) as e:
+        df.__not_a_column__
+    assert e.value.getCondition() == "ATTRIBUTE_NOT_SUPPORTED"
+    monkeypatch.setenv("PYSPARK_VALIDATE_COLUMN_NAME_LEGACY", "1")
+    with pytest.raises(PySparkAttributeError) as e:
+        df.not_a_column
+    assert e.value.getMessageParameters() == {"attr_name": "not_a_column"}
+
+
+
+
 def test_schema_cache_to_uses_supplied_struct_schema(mock_connect):
     spark, service = mock_connect
     df = spark.range(10)


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to