This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new fd759f4a [SPARK-61144][CONNECT][PYTHON] Repartition by range without a
number of partitions in repartitionByRange of columns, in the Python client as
PySpark does
fd759f4a is described below
commit fd759f4a23d5b0faf6f61dfb803df02dbf457d7f
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 11:13:58 2026 +0900
[SPARK-61144][CONNECT][PYTHON] Repartition by range without a number of
partitions in repartitionByRange of columns, in the Python client as PySpark
does
### What changes were proposed in this pull request?
[SPARK-61144](https://issues.apache.org/jira/browse/SPARK-61144):
Repartition by range without a number of partitions in `repartitionByRange` of
columns, in the Python client as PySpark does.
`DataFrame.repartitionByRange(numPartitions, *cols)` in the Python client
sent a partition count of 0 when the first argument is a column name or a
`Column`. The server rejects a count of 0 with "Number of partitions (0) must
be positive.". PySpark sends `numPartitions=None` for these forms, so the
server picks the count.
- The Rust core gets `DataFrame::repartition_by_range_opt(Option<i32>,
columns)`, following the `union_by_name_opt` pattern. `repartition_by_range(n,
columns)` keeps its signature and delegates with `Some(n)`.
- The PyO3 `repartitionByRange` sends `None` for the str/Column forms, and
`Some(n)` for the int form.
- `repartition` and `repartitionById` already send no count (or a positive
one) and are unchanged.
Official PySpark 4.3.0 returns the rows for `df.repartitionByRange("id")`,
`df.repartitionByRange(F.col("id"))` and `df.repartitionByRange("id", "id")`,
the same as this change.
### Why are the changes needed?
Since 2f173672, these calls fail on the server:
```python
spark.range(5).repartitionByRange("id").collect()
# requirement failed: Number of partitions (0) must be positive.
```
### Does this PR introduce _any_ user-facing change?
Yes. `repartitionByRange(col)` and `repartitionByRange(col, *cols)` work
and return the rows, as in PySpark.
### How was this patch tested?
- Offline
`test_repartition_by_range_without_a_count_sends_none_as_pyspark`: the
RepartitionByExpression sent for the str/Column forms has no `num_partitions`,
and `repartitionByRange(2, "id")` sends 2.
- Live `test_repartition_by_range_without_a_count_lets_the_server_pick_it`:
the three forms return the rows of `range(5)`. The same results with official
PySpark 4.3.0.
- Rust `repartition_by_range_without_a_count_sends_none` in
`coverage_plan_extra.rs`: `None` is not sent, `Some(4)` is.
- The lib tests of both crates (590 passed), the build of all tests, also
with `wasm-udf`, and the offline Python suites (2202 passed, 8 skipped);
rustfmt passes, and clippy reports no diagnostics in the changed lines. CI runs
the e2e and live suites.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #333 from HyukjinKwon/SPARK-61144-repartition-by-range-no-count.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/dataframe.rs | 8 +++++---
crates/spark-connect/src/dataframe.rs | 13 ++++++++++++-
crates/spark-connect/tests/coverage_plan_extra.rs | 21 +++++++++++++++++++++
python/tests/test_dropin_e2e.py | 9 +++++++++
python/tests/test_dropin_offline.py | 22 ++++++++++++++++++++++
5 files changed, 69 insertions(+), 4 deletions(-)
diff --git a/crates/pyspark-rs/src/dataframe.rs
b/crates/pyspark-rs/src/dataframe.rs
index 4f59a5f4..52aa457d 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -1011,13 +1011,14 @@ impl PyDataFrame {
&[("item", "cols")],
));
}
- (n, cols)
+ (Some(n), cols)
} else if numPartitions.is_instance_of::<pyo3::types::PyString>()
|| numPartitions.extract::<PyRef<'_, PyColumn>>().is_ok()
{
+ // No partition count is sent, so the server picks it, as
PySpark's.
let mut all = vec![numPartitions];
all.extend(cols);
- (0, all)
+ (None, all)
} else {
return Err(not_expected_type(
&numPartitions,
@@ -1033,7 +1034,8 @@ impl PyDataFrame {
})
.collect();
Ok(PyDataFrame::new(
- self.dataframe.repartition_by_range(num_partitions, exprs),
+ self.dataframe
+ .repartition_by_range_opt(num_partitions, exprs),
))
}
diff --git a/crates/spark-connect/src/dataframe.rs
b/crates/spark-connect/src/dataframe.rs
index 14496d6b..5ed29850 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -1432,9 +1432,20 @@ impl DataFrame {
/// Repartition this DataFrame by range.
pub fn repartition_by_range(&self, num_partitions: i32, columns:
Vec<Expression>) -> DataFrame {
+ self.repartition_by_range_opt(Some(num_partitions), columns)
+ }
+
+ /// Repartition by range on the given columns, with the partition count
optional: `None`
+ /// sends no count, so the server picks it. Mirrors
`DataFrame.repartitionByRange(col)`
+ /// and `repartitionByRange(col, *cols)`, which PySpark sends with
`numPartitions=None`.
+ pub fn repartition_by_range_opt(
+ &self,
+ num_partitions: Option<i32>,
+ columns: Vec<Expression>,
+ ) -> DataFrame {
let plan = LogicalPlan::RepartitionByRange {
input: Arc::clone(&self.plan),
- num_partitions: Some(num_partitions),
+ num_partitions,
partition_exprs: columns,
};
self.with_cached_schema(plan)
diff --git a/crates/spark-connect/tests/coverage_plan_extra.rs
b/crates/spark-connect/tests/coverage_plan_extra.rs
index 21ed11aa..10736a00 100644
--- a/crates/spark-connect/tests/coverage_plan_extra.rs
+++ b/crates/spark-connect/tests/coverage_plan_extra.rs
@@ -115,6 +115,27 @@ fn repartition_by_range_to_proto() {
assert!(p.to_proto().rel_type.is_some());
}
+/// `RepartitionByRange` without a partition count (`repartitionByRange(col)`
as PySpark sends
+/// it) sends `num_partitions` unset, so the server picks the count; `Some(n)`
sends it.
+#[test]
+fn repartition_by_range_without_a_count_sends_none() {
+ use spark_connect_proto::relation::RelType;
+
+ let count_of = |num_partitions: Option<i32>| {
+ let p = LogicalPlan::RepartitionByRange {
+ input: Arc::new(base()),
+ num_partitions,
+ partition_exprs: vec![],
+ };
+ match p.to_proto().rel_type {
+ Some(RelType::RepartitionByExpression(r)) => r.num_partitions,
+ other => panic!("unexpected relation {other:?}"),
+ }
+ };
+ assert_eq!(count_of(None), None);
+ assert_eq!(count_of(Some(4)), Some(4));
+}
+
/// `Read::to_proto` for the DataSource read type with
format/schema/source_name set.
#[test]
fn read_datasource_to_proto() {
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 7c6326fb..18191c31 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -413,6 +413,15 @@ def
test_year_month_intervals_are_not_collected_as_pyspark(session):
datetime.timedelta(days=1, seconds=7384))
+def test_repartition_by_range_without_a_count_lets_the_server_pick_it(session):
+ # As PySpark's: the str/Column forms send no partition count, so the
server picks it
+ # and returns the rows.
+ df = session.range(5)
+ for repartitioned in (df.repartitionByRange("id"),
df.repartitionByRange(F.col("id")),
+ df.repartitionByRange("id", "id")):
+ assert sorted(row.id for row in repartitioned.collect()) == [0, 1, 2,
3, 4]
+
+
def test_stream_reader_checks_its_arguments_as_pyspark(session, tmp_path):
# As PySpark's DataStreamReader, which checks a path, a source name and
the schema of
# changes in the client, rather than sending them to the server.
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index f98579c6..bd9e48e2 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -1031,6 +1031,28 @@ def
test_sessions_raise_the_errors_of_their_arguments_as_pyspark(mock_connect, t
+def
test_repartition_by_range_without_a_count_sends_none_as_pyspark(mock_connect):
+ # As PySpark's repartitionByRange(col) and repartitionByRange(col, *cols):
a
+ # RepartitionByExpression with num_partitions unset, so the server picks
the count;
+ # repartitionByRange(n, *cols) sends n.
+ from pyspark.sql import functions as F
+
+ spark, service = mock_connect
+ df = spark.range(1)
+
+ def sent(dataframe):
+ dataframe.collect()
+ # ExecutePlanRequest.plan=3, Plan.root=1;
Relation.repartition_by_expression=27,
+ # RepartitionByExpression.num_partitions=3, which is absent when unset.
+ plan =
parse_message(parse_message(service.execute_requests[-1])[3][0])[1][0]
+ return parse_message(parse_message(plan)[27][0]).get(3)
+
+ assert sent(df.repartitionByRange("id")) is None
+ assert sent(df.repartitionByRange(F.col("id"))) is None
+ assert sent(df.repartitionByRange("id", "id")) is None
+ assert sent(df.repartitionByRange(2, "id")) == [2]
+
+
def test_channel_builder_uses_endpoint():
# channelBuilder reconstructs an sc:// URL from the builder's host/port (+
params)
# for the native transport, returning a configured Builder (no longer
raises).
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]