This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new b54c3cd [SPARK-61092][CONNECT][PYTHON] Take a client seed in sample
and randomSplit, and the arguments of withMetadata, withWatermark and
randomSplit and the attributes of a DataFrame, in the Python client as PySpark
does
b54c3cd is described below
commit b54c3cd7bbb223cddf7925e7d2dd781ee266bd33
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 16:11:15 2026 +0900
[SPARK-61092][CONNECT][PYTHON] Take a client seed in sample and
randomSplit, and the arguments of withMetadata, withWatermark and randomSplit
and the attributes of a DataFrame, in the Python client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61092](https://issues.apache.org/jira/browse/SPARK-61092): `sample`
and `randomSplit` now choose their seed in the client, and several methods and
attributes of a DataFrame take their arguments as PySpark Connect's do.
As PySpark Connect's:
- `sample` and `randomSplit` without a seed choose a random seed in the
client (`random.randint(0, sys.maxsize)`).
- Every action of a sample returns the same rows.
- All splits of `randomSplit` use one seed and an input in a
deterministic order (the `deterministic_order` of their `Sample`), so they are
disjoint and together hold every row.
- `randomSplit` raises `VALUE_NOT_POSITIVE` for a negative weight
(`arg_name` "weights") and for weights whose sum is not positive (`arg_name`
"sum(weights)").
- `withMetadata` takes a dict of any JSON values, which it sends as
`json.dumps(metadata)`. It raises `NOT_EXPECTED_TYPE` for metadata that is not
a dict.
- `withWatermark` raises `NOT_EXPECTED_TYPE` for an event time or delay
that is empty or not a str.
- `df.name` is the column `name` for any name except a dunder one. Unless
`PYSPARK_VALIDATE_COLUMN_NAME_LEGACY` is `1`, a name is only checked against
the columns when it is a dunder name. A checked name that is not a column
raises `ATTRIBUTE_NOT_SUPPORTED`.
- `_jseq`, `_jdf`, `_jmap` and `_jcols` raise
`JVM_ATTRIBUTE_NOT_SUPPORTED`, and `rdd` raises `NOT_IMPLEMENTED`.
In the core, the `Sample` plan gains `deterministic_order`, and `sample`
and `random_split` choose a seed when none is given.
`DataFrame::with_metadata_json` sets the metadata of a column from its JSON.
### Why are the changes needed?
The client sent no seed, so the server chose a new one at each execution:
- each action of `df.sample(0.5)` returned other rows
- the splits of `randomSplit([0.5, 0.5])` lost and repeated rows: they had
2003, 1994 and 2060 rows of 2000 in three runs
The client also:
- sent negative weights to the server (`IllegalArgumentException`), and an
empty event time (`PARSE_EMPTY_STATEMENT`)
- accepted only str values of metadata
- raised plain `TypeError`s for these arguments, and an `AttributeError`
for any name starting with `_`
### Does this PR introduce _any_ user-facing change?
Yes. Samples and splits are consistent, and these methods take the
arguments, and raise the errors, of PySpark 4.3.0.
### How was this patch tested?
- A new live test checks that two actions of a sample return the same rows,
that the splits of `randomSplit` are disjoint and hold every row, and the
metadata of `withMetadata` of non-str values. It passes with official PySpark
4.3.0 and fails without this change.
- A new offline test checks the errors of `randomSplit`, `withMetadata` and
`withWatermark`, and the attributes of a DataFrame, including
`PYSPARK_VALIDATE_COLUMN_NAME_LEGACY`.
- A new lib test checks the seed of the splits and of a sample, and the
deterministic order of the splits.
- On a local server, the argument errors of 64 cases of DataFrame methods,
and 14 attributes, give the same results as official PySpark 4.3.0. The only
exceptions are the messages of `TypeError`s and `AssertionError`s that come
from protobuf or from Python internals.
- The lib tests of both crates (595 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2045 passed, 12 skipped), and the
live drop-in and Arrow suites (148 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #296 from HyukjinKwon/SPARK-61092-df-arguments.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/dataframe.rs | 120 +++++++++++++++++++++++++++++-----
crates/spark-connect/src/dataframe.rs | 55 +++++++++++++++-
crates/spark-connect/src/plan.rs | 6 ++
python/tests/test_dropin_e2e.py | 16 +++++
python/tests/test_dropin_offline.py | 50 ++++++++++++++
5 files changed, 226 insertions(+), 21 deletions(-)
diff --git a/crates/pyspark-rs/src/dataframe.rs
b/crates/pyspark-rs/src/dataframe.rs
index 1de5ccc..1d47c1a 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -503,6 +503,19 @@ fn cond_column(condition: &Bound<'_, PyAny>) ->
PyResult<spark_connect::column::
}
}
+/// An argument of `withWatermark`, as PySpark's: `NOT_EXPECTED_TYPE` for an
empty one, or one
+/// whose type is not str.
+fn watermark_argument(value: &Bound<'_, PyAny>, name: &str) ->
PyResult<String> {
+ if !value.is_truthy()?
+ || !value
+ .get_type()
+ .is(value.py().get_type::<pyo3::types::PyString>())
+ {
+ return Err(crate::stat_arguments::not_expected_type(value, name,
"str"));
+ }
+ value.extract()
+}
+
#[pymethods]
impl PyDataFrame {
/// Select columns (accepts Column objects or string names).
@@ -1593,30 +1606,82 @@ impl PyDataFrame {
Ok(PyDataFrame::new(self.dataframe.to(dt)))
}
- /// Set metadata (a dict) on a column.
+ /// Set the metadata of a column, as PySpark's: a dict of any JSON values,
which it sends as
+ /// `json.dumps(metadata)`, and `NOT_EXPECTED_TYPE` for one that is not a
dict.
#[pyo3(name = "withMetadata")]
fn with_metadata(
&self,
+ py: Python<'_>,
columnName: &str,
- metadata: std::collections::HashMap<String, String>,
- ) -> PyDataFrame {
- PyDataFrame::new(self.dataframe.with_metadata(columnName, metadata))
+ metadata: &Bound<'_, PyAny>,
+ ) -> PyResult<PyDataFrame> {
+ if !metadata.is_instance_of::<PyDict>() {
+ return Err(crate::stat_arguments::not_expected_type(
+ metadata, "metadata", "dict",
+ ));
+ }
+ let json: String = py
+ .import("json")?
+ .call_method1("dumps", (metadata,))?
+ .extract()?;
+ Ok(PyDataFrame::new(
+ self.dataframe.with_metadata_json(columnName, &json),
+ ))
}
- /// Define an event-time watermark.
+ /// Define an event-time watermark, as PySpark's: `NOT_EXPECTED_TYPE` for
an empty event time
+ /// or delay, or one that is not a str.
#[pyo3(name = "withWatermark")]
- fn with_watermark(&self, eventTime: &str, delayThreshold: &str) ->
PyDataFrame {
- PyDataFrame::new(self.dataframe.with_watermark(eventTime,
delayThreshold))
+ fn with_watermark(
+ &self,
+ eventTime: &Bound<'_, PyAny>,
+ delayThreshold: &Bound<'_, PyAny>,
+ ) -> PyResult<PyDataFrame> {
+ let event_time = watermark_argument(eventTime, "eventTime")?;
+ let delay_threshold = watermark_argument(delayThreshold,
"delayThreshold")?;
+ Ok(PyDataFrame::new(
+ self.dataframe.with_watermark(&event_time, &delay_threshold),
+ ))
}
/// Randomly split into multiple DataFrames by the given weights.
+ /// As PySpark's: `VALUE_NOT_POSITIVE` for a negative weight and for
weights of a sum that is
+ /// not positive, of their `str`; without a seed, the client chooses one
of every split.
#[pyo3(name = "randomSplit", signature = (weights, seed=None))]
- fn random_split(&self, weights: Vec<f64>, seed: Option<i64>) ->
Vec<PyDataFrame> {
- self.dataframe
+ fn random_split(
+ &self,
+ weights: &Bound<'_, PyAny>,
+ seed: Option<i64>,
+ ) -> PyResult<Vec<PyDataFrame>> {
+ let py = weights.py();
+ let not_positive = |arg_name: &str, value: &Bound<'_, PyAny>| ->
PyResult<PyErr> {
+ Ok(crate::column::raise_pyspark(
+ py,
+ "PySparkValueError",
+ "VALUE_NOT_POSITIVE",
+ &[
+ ("arg_name", arg_name),
+ ("arg_value", &value.str()?.to_cow()?),
+ ],
+ ))
+ };
+ for weight in weights.try_iter()? {
+ let weight = weight?;
+ if weight.lt(0.0)? {
+ return Err(not_positive("weights", &weight)?);
+ }
+ }
+ let total = py.import("builtins")?.call_method1("sum", (weights,))?;
+ if total.le(0)? {
+ return Err(not_positive("sum(weights)", &total)?);
+ }
+ let weights: Vec<f64> = weights.extract()?;
+ Ok(self
+ .dataframe
.random_split(weights, seed)
.into_iter()
.map(PyDataFrame::new)
- .collect()
+ .collect())
}
/// The list of files that make up this DataFrame.
@@ -1670,11 +1735,15 @@ impl PyDataFrame {
crate::session::PySparkSession::new(self.dataframe.spark_session())
}
- /// Not supported on Spark Connect — `DataFrame.rdd` requires the legacy
RDD API.
+ /// Not supported on Spark Connect, as PySpark Connect's:
`NOT_IMPLEMENTED` of `rdd`, which
+ /// needs the RDD API of a classic session.
#[getter]
- fn rdd(&self) -> PyResult<()> {
- Err(PyErr::new::<pyo3::exceptions::PyNotImplementedError, _>(
- "rdd() is not supported for Spark Connect",
+ fn rdd(&self, py: Python<'_>) -> PyResult<()> {
+ Err(crate::column::raise_pyspark(
+ py,
+ "PySparkNotImplementedError",
+ "NOT_IMPLEMENTED",
+ &[("feature", "rdd")],
))
}
@@ -1789,10 +1858,25 @@ impl PyDataFrame {
/// Column access by attribute (`df.colname` -> Column), mirroring
reference pyspark: the
/// column of this DataFrame, by its plan ID. Names beginning with `_`
raise
/// AttributeError so normal attribute lookup works.
- fn __getattr__(&self, name: &str) -> PyResult<PyColumn> {
- if name.starts_with('_') {
- return Err(PyErr::new::<pyo3::exceptions::PyAttributeError, _>(
- name.to_string(),
+ fn __getattr__(&self, py: Python<'_>, name: &str) -> PyResult<PyColumn> {
+ // As PySpark Connect's: the attributes of the JVM of a classic
DataFrame are not
+ // supported, and a name is validated as a column only for a dunder
name, which Python
+ // may look up for an attribute, or with
PYSPARK_VALIDATE_COLUMN_NAME_LEGACY.
+ if ["_jseq", "_jdf", "_jmap", "_jcols", "rdd"].contains(&name) {
+ return Err(crate::column::raise_pyspark(
+ py,
+ "PySparkAttributeError",
+ "JVM_ATTRIBUTE_NOT_SUPPORTED",
+ &[("attr_name", name)],
+ ));
+ }
+ let legacy =
std::env::var("PYSPARK_VALIDATE_COLUMN_NAME_LEGACY").as_deref() == Ok("1");
+ if (legacy || name.starts_with("__")) &&
!self.columns(py)?.iter().any(|c| c == name) {
+ return Err(crate::column::raise_pyspark(
+ py,
+ "PySparkAttributeError",
+ "ATTRIBUTE_NOT_SUPPORTED",
+ &[("attr_name", name)],
));
}
Ok(PyColumn::new(self.dataframe.col(name)))
diff --git a/crates/spark-connect/src/dataframe.rs
b/crates/spark-connect/src/dataframe.rs
index 838789b..587635c 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -758,7 +758,8 @@ impl DataFrame {
Ok(())
}
- /// Sample rows.
+ /// Sample rows. Without a seed, the client chooses a random one, as
PySpark's, so that every
+ /// execution of the sample returns the same rows.
pub fn sample(&self, fraction: f64, seed: Option<i64>) -> DataFrame {
self.sample_opt(fraction, false, seed)
}
@@ -776,7 +777,8 @@ impl DataFrame {
lower_bound: 0.0,
upper_bound: fraction,
with_replacement,
- seed,
+ seed: Some(seed.unwrap_or_else(random_seed)),
+ deterministic_order: false,
};
self.with_cached_schema(plan)
}
@@ -1759,6 +1761,9 @@ impl DataFrame {
/// Randomly split this DataFrame into multiple parts.
pub fn random_split(&self, weights: Vec<f64>, seed: Option<i64>) ->
Vec<DataFrame> {
+ // As PySpark's: one seed of every split, of the client without a
seed, and the input
+ // in a deterministic order, so that the splits are disjoint and have
every row.
+ let seed = Some(seed.unwrap_or_else(random_seed));
let total: f64 = weights.iter().sum();
let normalized: Vec<f64> = weights.iter().map(|w| w / total).collect();
@@ -1773,6 +1778,7 @@ impl DataFrame {
upper_bound: upper,
with_replacement: false,
seed,
+ deterministic_order: true,
};
results.push(self.with_cached_schema(plan));
cumulative = upper;
@@ -1962,10 +1968,16 @@ impl DataFrame {
/// re-selected with the given metadata attached (serialized to a JSON
map).
pub fn with_metadata(&self, column_name: &str, metadata: HashMap<String,
String>) -> DataFrame {
let metadata_json =
serde_json::to_string(&metadata).unwrap_or_else(|_| "{}".to_string());
+ self.with_metadata_json(column_name, &metadata_json)
+ }
+
+ /// Set the metadata of a column of its JSON, of any JSON values, as
PySpark's
+ /// `withMetadata` sends `json.dumps(metadata)`.
+ pub fn with_metadata_json(&self, column_name: &str, metadata_json: &str)
-> DataFrame {
let plan = LogicalPlan::WithColumnMetadata {
input: Arc::clone(&self.plan),
column_name: column_name.to_string(),
- metadata_json,
+ metadata_json: metadata_json.to_string(),
};
DataFrame::new(self.session.clone(), plan)
}
@@ -2296,6 +2308,12 @@ fn set_observations(
}
}
+/// A random seed of a sample without one, as PySpark's `random.randint(0,
sys.maxsize)`: the
+/// client chooses it, so that every execution of the sample returns the same
rows.
+fn random_seed() -> i64 {
+ (uuid::Uuid::new_v4().as_u128() >> 65) as i64
+}
+
/// Decode an Arrow batch into rows.
pub(crate) fn decode_arrow_batch(
batch: &proto::execute_plan_response::ArrowBatch,
@@ -4181,6 +4199,37 @@ mod plan_construction_tests {
}
}
+ /// As PySpark's: without a seed, the client takes one, of every split of
`random_split`,
+ /// whose input is in a deterministic order, and of a sample.
+ #[test]
+ fn samples_take_a_seed_of_the_client() {
+ let spark = session();
+ let df = spark.range(10).unwrap();
+ let seeds: Vec<(Option<i64>, bool)> = df
+ .random_split(vec![0.5, 0.5], None)
+ .iter()
+ .map(|split| match split.plan.as_ref() {
+ LogicalPlan::Sample {
+ seed,
+ deterministic_order,
+ ..
+ } => (*seed, *deterministic_order),
+ other => panic!("expected a Sample plan, got {other:?}"),
+ })
+ .collect();
+ assert!(seeds[0].0.is_some_and(|seed| seed >= 0));
+ assert_eq!(seeds[0], seeds[1]);
+ assert!(seeds[0].1);
+ match df.sample(0.5, None).plan.as_ref() {
+ LogicalPlan::Sample {
+ seed: Some(_),
+ deterministic_order: false,
+ ..
+ } => {}
+ other => panic!("expected a Sample plan of a seed, got {other:?}"),
+ }
+ }
+
#[test]
fn random_split_plan() {
let spark = session();
diff --git a/crates/spark-connect/src/plan.rs b/crates/spark-connect/src/plan.rs
index bef5092..bce9c66 100644
--- a/crates/spark-connect/src/plan.rs
+++ b/crates/spark-connect/src/plan.rs
@@ -126,6 +126,9 @@ pub enum LogicalPlan {
upper_bound: f64,
with_replacement: bool,
seed: Option<i64>,
+ /// Whether the input is sorted, or cached, to be in a deterministic
order, as the
+ /// splits of `randomSplit` are, so that they are disjoint.
+ deterministic_order: bool,
},
/// Repartition: `df.repartition(num_partitions)`.
Repartition {
@@ -921,6 +924,7 @@ impl LogicalPlan {
upper_bound,
with_replacement,
seed,
+ deterministic_order,
} => {
let mut sample = proto::Sample::default();
sample.input = Some(Box::new(relation_of(input, plan_ids)));
@@ -930,6 +934,7 @@ impl LogicalPlan {
if let Some(s) = seed {
sample.seed = Some(*s);
}
+ sample.deterministic_order = *deterministic_order;
relation.rel_type =
Some(proto::relation::RelType::Sample(Box::new(sample)));
}
@@ -1841,6 +1846,7 @@ pub fn sample(
upper_bound,
with_replacement,
seed,
+ deterministic_order: false,
}
}
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 328547a..868dfc4 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -2531,6 +2531,22 @@ def
test_create_dataframe_raises_merge_errors_as_pyspark(session):
assert error.value.getMessageParameters() == parameters, data
+def test_samples_and_splits_take_a_seed_of_the_client_as_pyspark(session):
+ # As PySpark's: without a seed, the client takes one, so every action of a
sample returns
+ # the same rows, and the splits of randomSplit are disjoint and have every
row; and the
+ # metadata of withMetadata takes any JSON value.
+ df = session.range(0, 2000, 1, 8)
+ sample = df.sample(0.5)
+ assert sorted(row.id for row in sample.collect()) == sorted(row.id for row
in sample.collect())
+ first, second = df.randomSplit([0.5, 0.5])
+ assert first.count() + second.count() == 2000
+ assert first.intersect(second).count() == 0
+ metadata = {"k": 1, "l": [1, 2], "s": "v"}
+ assert df.withMetadata("id", metadata).schema["id"].metadata == metadata
+
+
+
+
def
test_persist_cache_and_unpersist_return_the_dataframe_itself_as_pyspark(session):
# As PySpark's: the DataFrame itself, cached at the storage level given.
from pyspark.storagelevel import StorageLevel
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 4d9a75d..378d796 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -7373,6 +7373,56 @@ def
test_schema_cache_set_operations_require_equal_warm_schemas(mock_connect, op
assert left.columns == ["id"]
+def
test_dataframes_take_their_arguments_and_attributes_as_pyspark(mock_connect,
monkeypatch):
+ # As PySpark Connect's: the errors of randomSplit, withMetadata and
withWatermark, and the
+ # attributes of a DataFrame: a column of any name but a dunder one, which
must be a column,
+ # and JVM_ATTRIBUTE_NOT_SUPPORTED of those of a classic DataFrame.
+ from pyspark.errors import (
+ PySparkAttributeError,
+ PySparkNotImplementedError,
+ PySparkTypeError,
+ PySparkValueError,
+ )
+
+ spark, _ = mock_connect
+ df = spark.range(3)
+ for weights, arg_name, arg_value in (([1.0, -1.0], "weights", "-1.0"),
+ ([0, 0], "sum(weights)", "0")):
+ with pytest.raises(PySparkValueError) as e:
+ df.randomSplit(weights)
+ assert e.value.getCondition() == "VALUE_NOT_POSITIVE"
+ assert e.value.getMessageParameters() == {"arg_name": arg_name,
"arg_value": arg_value}
+ with pytest.raises(PySparkTypeError) as e:
+ df.withMetadata("id", 1)
+ assert e.value.getMessageParameters() == {
+ "expected_type": "dict", "arg_name": "metadata", "arg_type": "int"}
+ df.withMetadata("id", {"k": 1, "l": [1, 2]})
+ for event_time, delay, arg_name, arg_type in ((1, "1 minute", "eventTime",
"int"),
+ ("", "1 minute",
"eventTime", "str"),
+ ("id", None,
"delayThreshold", "NoneType")):
+ with pytest.raises(PySparkTypeError) as e:
+ df.withWatermark(event_time, delay)
+ assert e.value.getMessageParameters() == {
+ "arg_name": arg_name, "expected_type": "str", "arg_type": arg_type}
+ assert type(df._not_a_dunder).__name__ == "Column"
+ for name in ("_jseq", "_jdf", "_jmap", "_jcols"):
+ with pytest.raises(PySparkAttributeError) as e:
+ getattr(df, name)
+ assert e.value.getCondition() == "JVM_ATTRIBUTE_NOT_SUPPORTED"
+ with pytest.raises(PySparkNotImplementedError) as e:
+ df.rdd
+ assert e.value.getMessageParameters() == {"feature": "rdd"}
+ with pytest.raises(PySparkAttributeError) as e:
+ df.__not_a_column__
+ assert e.value.getCondition() == "ATTRIBUTE_NOT_SUPPORTED"
+ monkeypatch.setenv("PYSPARK_VALIDATE_COLUMN_NAME_LEGACY", "1")
+ with pytest.raises(PySparkAttributeError) as e:
+ df.not_a_column
+ assert e.value.getMessageParameters() == {"attr_name": "not_a_column"}
+
+
+
+
def test_schema_cache_to_uses_supplied_struct_schema(mock_connect):
spark, service = mock_connect
df = spark.range(10)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]