This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.1 by this push:
     new 5c0eb439f529 [SPARK-59011][PS] Fix descending rank first tie ordering
5c0eb439f529 is described below

commit 5c0eb439f5295721d72509ad868fe601dc77c283
Author: NaVis <[email protected]>
AuthorDate: Sat Sep 5 13:14:28 2026 +0000

    [SPARK-59011][PS] Fix descending rank first tie ordering
    
    ### What changes were proposed in this pull request?
    
    This pull request fixes the tie-breaking order of pandas-on-Spark 
`Series.rank(method="first", ascending=False)`.
    
    The value column follows the requested ascending direction, while the 
natural-order column is always ordered ascending so that equal values are 
ranked according to their original occurrence order.
    
    A regression test for descending ranking with `method="first"` is added.
    
    ### Why are the changes needed?
    
    The existing implementation applies descending order to both the value 
column and the natural-order tie-breaker. This reverses the original order of 
equal values and produces results inconsistent with pandas.
    
    For `pd.Series([1, 2, 3, 1])`, pandas returns:
    
    `[3.0, 2.0, 1.0, 4.0]`
    
    The previous pandas-on-Spark result was:
    
    `[4.0, 2.0, 1.0, 3.0]`
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `Series.rank(method="first", ascending=False)` now preserves the 
original occurrence order when breaking ties, matching pandas behavior.
    
    ### How was this patch tested?
    
    A regression assertion was added to `SeriesStatTests.test_rank`.
    
    The test failed before the fix with a 50% value difference and passed after 
the fix:
    
    ./python/run-tests --python-executables python3 \
     --testnames 'pyspark.pandas.tests.series.test_stat 
SeriesStatTests.test_rank' \
      -p 1
    
    The Spark assembly build, Ruff checks, Ruff formatting check, and `git diff 
--check` also passed.
    
    This contribution was prepared and reviewed by our university project team, 
including Phan Trung Nhut and Bui Nam Viet. We license this contribution to the 
project under the Apache License, Version 2.0.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex (GPT-5)
    
    Closes #58337 from namviet157/fix-SPARK-59011.
    
    Lead-authored-by: NaVis <[email protected]>
    Co-authored-by: Phan Trung Nhá»±t 
<[email protected]>
    Signed-off-by: Yicong-Huang <[email protected]>
    (cherry picked from commit d7cb6592d67f92d11239e1cead153d6fc80fce18)
    Signed-off-by: Yicong-Huang <[email protected]>
    (cherry picked from commit 76ba3bb7c4c0c75db00d54fb31686668f01e6ae5)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 python/pyspark/pandas/series.py                 | 2 +-
 python/pyspark/pandas/tests/series/test_stat.py | 4 ++++
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/python/pyspark/pandas/series.py b/python/pyspark/pandas/series.py
index e5c8890b646c..0087d0896dc7 100644
--- a/python/pyspark/pandas/series.py
+++ b/python/pyspark/pandas/series.py
@@ -4231,7 +4231,7 @@ class Series(Frame, IndexOpsMixin, Generic[T]):
             window = (
                 Window.orderBy(
                     asc_func(self.spark.column),
-                    asc_func(F.col(NATURAL_ORDER_COLUMN_NAME)),
+                    F.col(NATURAL_ORDER_COLUMN_NAME).asc(),
                 )
                 .partitionBy(*part_cols)
                 .rowsBetween(Window.unboundedPreceding, Window.currentRow)
diff --git a/python/pyspark/pandas/tests/series/test_stat.py 
b/python/pyspark/pandas/tests/series/test_stat.py
index 077a8e31d20a..2931e1782c41 100644
--- a/python/pyspark/pandas/tests/series/test_stat.py
+++ b/python/pyspark/pandas/tests/series/test_stat.py
@@ -321,6 +321,10 @@ class SeriesStatMixin:
         self.assert_eq(pser.rank(method="min"), 
psser.rank(method="min").sort_index())
         self.assert_eq(pser.rank(method="max"), 
psser.rank(method="max").sort_index())
         self.assert_eq(pser.rank(method="first"), 
psser.rank(method="first").sort_index())
+        self.assert_eq(
+            pser.rank(method="first", ascending=False),
+            psser.rank(method="first", ascending=False).sort_index(),
+        )
         self.assert_eq(pser.rank(method="dense"), 
psser.rank(method="dense").sort_index())
 
         non_numeric_pser = pd.Series(["a", "c", "b", "d"], name="x", 
index=[10, 11, 12, 13])


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to