Jing Zhan created SPARK-59011:
---------------------------------
Summary: Series.rank with method='first' and ascending=False gives
wrong ranks for tied values
Key: SPARK-59011
URL: https://issues.apache.org/jira/browse/SPARK-59011
Project: Spark
Issue Type: Task
Components: Pandas API on Spark
Affects Versions: 4.2.1
Reporter: Jing Zhan
Series.rank(method='first', ascending=False) produces incorrect ranks for tied
values in the pandas API on Spark. The tiebreaker should always favor the
element that appeared first in the original array (lower index), but Spark
incorrectly reverses the tiebreaker direction when ascending=False.
Reproduction:
import pandas as pd
import pyspark.pandas as ps
pser = pd.Series([1, 2, 3, 1], name='x')
psser = ps.from_pandas(pser)
print(pser.rank(method='first', ascending=False).tolist()) # [3.0, 2.0, 1.0,
4.0]
print(psser.rank(method='first', ascending=False).sort_index().tolist()) #
[4.0, 2.0, 1.0, 3.0]
Expected: [3.0, 2.0, 1.0, 4.0] (matches pandas — index 0 gets rank 3, index 3
gets rank 4)
Actual: [4.0, 2.0, 1.0, 3.0] (wrong — index 0 gets rank 4, index 3 gets rank 3)
Root cause: In Series._rank, the natural order tiebreaker column uses the same
direction as the value sort (DESC when ascending=False). This reverses the
tiebreaker for tied values. The fix is to always use ASC for the natural order
tiebreaker since "first" always means first in the original array, regardless
of rank direction.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
