This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new fd759f4a [SPARK-61144][CONNECT][PYTHON] Repartition by range without a 
number of partitions in repartitionByRange of columns, in the Python client as 
PySpark does
fd759f4a is described below

commit fd759f4a23d5b0faf6f61dfb803df02dbf457d7f
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 11:13:58 2026 +0900

    [SPARK-61144][CONNECT][PYTHON] Repartition by range without a number of 
partitions in repartitionByRange of columns, in the Python client as PySpark 
does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61144](https://issues.apache.org/jira/browse/SPARK-61144): 
Repartition by range without a number of partitions in `repartitionByRange` of 
columns, in the Python client as PySpark does.
    
    `DataFrame.repartitionByRange(numPartitions, *cols)` in the Python client 
sent a partition count of 0 when the first argument is a column name or a 
`Column`. The server rejects a count of 0 with "Number of partitions (0) must 
be positive.". PySpark sends `numPartitions=None` for these forms, so the 
server picks the count.
    
    - The Rust core gets `DataFrame::repartition_by_range_opt(Option<i32>, 
columns)`, following the `union_by_name_opt` pattern. `repartition_by_range(n, 
columns)` keeps its signature and delegates with `Some(n)`.
    - The PyO3 `repartitionByRange` sends `None` for the str/Column forms, and 
`Some(n)` for the int form.
    - `repartition` and `repartitionById` already send no count (or a positive 
one) and are unchanged.
    
    Official PySpark 4.3.0 returns the rows for `df.repartitionByRange("id")`, 
`df.repartitionByRange(F.col("id"))` and `df.repartitionByRange("id", "id")`, 
the same as this change.
    
    ### Why are the changes needed?
    
    Since 2f173672, these calls fail on the server:
    
    ```python
    spark.range(5).repartitionByRange("id").collect()
    # requirement failed: Number of partitions (0) must be positive.
    ```
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `repartitionByRange(col)` and `repartitionByRange(col, *cols)` work 
and return the rows, as in PySpark.
    
    ### How was this patch tested?
    
    - Offline 
`test_repartition_by_range_without_a_count_sends_none_as_pyspark`: the 
RepartitionByExpression sent for the str/Column forms has no `num_partitions`, 
and `repartitionByRange(2, "id")` sends 2.
    - Live `test_repartition_by_range_without_a_count_lets_the_server_pick_it`: 
the three forms return the rows of `range(5)`. The same results with official 
PySpark 4.3.0.
    - Rust `repartition_by_range_without_a_count_sends_none` in 
`coverage_plan_extra.rs`: `None` is not sent, `Some(4)` is.
    - The lib tests of both crates (590 passed), the build of all tests, also 
with `wasm-udf`, and the offline Python suites (2202 passed, 8 skipped); 
rustfmt passes, and clippy reports no diagnostics in the changed lines. CI runs 
the e2e and live suites.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #333 from HyukjinKwon/SPARK-61144-repartition-by-range-no-count.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/dataframe.rs                |  8 +++++---
 crates/spark-connect/src/dataframe.rs             | 13 ++++++++++++-
 crates/spark-connect/tests/coverage_plan_extra.rs | 21 +++++++++++++++++++++
 python/tests/test_dropin_e2e.py                   |  9 +++++++++
 python/tests/test_dropin_offline.py               | 22 ++++++++++++++++++++++
 5 files changed, 69 insertions(+), 4 deletions(-)

diff --git a/crates/pyspark-rs/src/dataframe.rs 
b/crates/pyspark-rs/src/dataframe.rs
index 4f59a5f4..52aa457d 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -1011,13 +1011,14 @@ impl PyDataFrame {
                     &[("item", "cols")],
                 ));
             }
-            (n, cols)
+            (Some(n), cols)
         } else if numPartitions.is_instance_of::<pyo3::types::PyString>()
             || numPartitions.extract::<PyRef<'_, PyColumn>>().is_ok()
         {
+            // No partition count is sent, so the server picks it, as 
PySpark's.
             let mut all = vec![numPartitions];
             all.extend(cols);
-            (0, all)
+            (None, all)
         } else {
             return Err(not_expected_type(
                 &numPartitions,
@@ -1033,7 +1034,8 @@ impl PyDataFrame {
             })
             .collect();
         Ok(PyDataFrame::new(
-            self.dataframe.repartition_by_range(num_partitions, exprs),
+            self.dataframe
+                .repartition_by_range_opt(num_partitions, exprs),
         ))
     }
 
diff --git a/crates/spark-connect/src/dataframe.rs 
b/crates/spark-connect/src/dataframe.rs
index 14496d6b..5ed29850 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -1432,9 +1432,20 @@ impl DataFrame {
 
     /// Repartition this DataFrame by range.
     pub fn repartition_by_range(&self, num_partitions: i32, columns: 
Vec<Expression>) -> DataFrame {
+        self.repartition_by_range_opt(Some(num_partitions), columns)
+    }
+
+    /// Repartition by range on the given columns, with the partition count 
optional: `None`
+    /// sends no count, so the server picks it. Mirrors 
`DataFrame.repartitionByRange(col)`
+    /// and `repartitionByRange(col, *cols)`, which PySpark sends with 
`numPartitions=None`.
+    pub fn repartition_by_range_opt(
+        &self,
+        num_partitions: Option<i32>,
+        columns: Vec<Expression>,
+    ) -> DataFrame {
         let plan = LogicalPlan::RepartitionByRange {
             input: Arc::clone(&self.plan),
-            num_partitions: Some(num_partitions),
+            num_partitions,
             partition_exprs: columns,
         };
         self.with_cached_schema(plan)
diff --git a/crates/spark-connect/tests/coverage_plan_extra.rs 
b/crates/spark-connect/tests/coverage_plan_extra.rs
index 21ed11aa..10736a00 100644
--- a/crates/spark-connect/tests/coverage_plan_extra.rs
+++ b/crates/spark-connect/tests/coverage_plan_extra.rs
@@ -115,6 +115,27 @@ fn repartition_by_range_to_proto() {
     assert!(p.to_proto().rel_type.is_some());
 }
 
+/// `RepartitionByRange` without a partition count (`repartitionByRange(col)` 
as PySpark sends
+/// it) sends `num_partitions` unset, so the server picks the count; `Some(n)` 
sends it.
+#[test]
+fn repartition_by_range_without_a_count_sends_none() {
+    use spark_connect_proto::relation::RelType;
+
+    let count_of = |num_partitions: Option<i32>| {
+        let p = LogicalPlan::RepartitionByRange {
+            input: Arc::new(base()),
+            num_partitions,
+            partition_exprs: vec![],
+        };
+        match p.to_proto().rel_type {
+            Some(RelType::RepartitionByExpression(r)) => r.num_partitions,
+            other => panic!("unexpected relation {other:?}"),
+        }
+    };
+    assert_eq!(count_of(None), None);
+    assert_eq!(count_of(Some(4)), Some(4));
+}
+
 /// `Read::to_proto` for the DataSource read type with 
format/schema/source_name set.
 #[test]
 fn read_datasource_to_proto() {
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 7c6326fb..18191c31 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -413,6 +413,15 @@ def 
test_year_month_intervals_are_not_collected_as_pyspark(session):
         datetime.timedelta(days=1, seconds=7384))
 
 
+def test_repartition_by_range_without_a_count_lets_the_server_pick_it(session):
+    # As PySpark's: the str/Column forms send no partition count, so the 
server picks it
+    # and returns the rows.
+    df = session.range(5)
+    for repartitioned in (df.repartitionByRange("id"), 
df.repartitionByRange(F.col("id")),
+                          df.repartitionByRange("id", "id")):
+        assert sorted(row.id for row in repartitioned.collect()) == [0, 1, 2, 
3, 4]
+
+
 def test_stream_reader_checks_its_arguments_as_pyspark(session, tmp_path):
     # As PySpark's DataStreamReader, which checks a path, a source name and 
the schema of
     # changes in the client, rather than sending them to the server.
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index f98579c6..bd9e48e2 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -1031,6 +1031,28 @@ def 
test_sessions_raise_the_errors_of_their_arguments_as_pyspark(mock_connect, t
 
 
 
+def 
test_repartition_by_range_without_a_count_sends_none_as_pyspark(mock_connect):
+    # As PySpark's repartitionByRange(col) and repartitionByRange(col, *cols): 
a
+    # RepartitionByExpression with num_partitions unset, so the server picks 
the count;
+    # repartitionByRange(n, *cols) sends n.
+    from pyspark.sql import functions as F
+
+    spark, service = mock_connect
+    df = spark.range(1)
+
+    def sent(dataframe):
+        dataframe.collect()
+        # ExecutePlanRequest.plan=3, Plan.root=1; 
Relation.repartition_by_expression=27,
+        # RepartitionByExpression.num_partitions=3, which is absent when unset.
+        plan = 
parse_message(parse_message(service.execute_requests[-1])[3][0])[1][0]
+        return parse_message(parse_message(plan)[27][0]).get(3)
+
+    assert sent(df.repartitionByRange("id")) is None
+    assert sent(df.repartitionByRange(F.col("id"))) is None
+    assert sent(df.repartitionByRange("id", "id")) is None
+    assert sent(df.repartitionByRange(2, "id")) == [2]
+
+
 def test_channel_builder_uses_endpoint():
     # channelBuilder reconstructs an sc:// URL from the builder's host/port (+ 
params)
     # for the native transport, returning a configured Builder (no longer 
raises).


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to