This is an automated email from the ASF dual-hosted git repository.
peter-toth pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 73535081cbe5 [SPARK-57982][SQL] Make exact percentile monotonic for
large values
73535081cbe5 is described below
commit 73535081cbe5d0a572deac7a458d2db29f8ecb8f
Author: Matthew B. <[email protected]>
AuthorDate: Thu Jul 16 14:02:45 2026 +0200
[SPARK-57982][SQL] Make exact percentile monotonic for large values
### What changes were proposed in this pull request?
Rewrite the exact `percentile` linear interpolation in
`PercentileBase.getPercentile` from `(higher - position) * lower + (position -
lower) * higher` to the equivalent `lower + (position - lower) * (higher -
lower)`. This method is shared by `percentile`, `percentile_cont`, `median`,
and their `WITHIN GROUP` forms.
### Why are the changes needed?
The two forms are equal in exact arithmetic, but the old one scales both
(potentially large) endpoint values by the weights independently, so rounding
error on the order of the gap between adjacent doubles can make the result
non-monotonic in the percentage:
```
val df = (0 until 20).map(i => 1e18 + i * 128.0).toDF("x")
df.selectExpr("percentile(x, 0.04)", "percentile(x, 0.05)").show(false)
// before: 1.00000000000000013E18, 1.0E18 (p04 > p05,
not monotonic)
// after: 1.00000000000000013E18, 1.00000000000000013E18
```
The new form scales only the small `(higher - lower)` delta, so the result
stays monotonic.
### Does this PR introduce _any_ user-facing change?
Yes. Exact `percentile` / `percentile_cont` / `median` on large-magnitude
inputs can change in the last ULP and are now monotonic in the percentage.
### How was this patch tested?
Added UT in `PercentileQuerySuite`; existing percentile suites and the
`percentiles.sql` golden file pass unchanged.
### Was this patch authored or co-authored using generative AI tooling?
Yes, co-authored with Claude Opus 4.8
Closes #57241 from Ma77Ball/SPARK-57982-percentile-monotonicity.
Authored-by: Matthew B. <[email protected]>
Signed-off-by: Peter Toth <[email protected]>
---
docs/sql-migration-guide.md | 1 +
.../expressions/aggregate/percentiles.scala | 8 ++++++--
.../apache/spark/sql/DataFrameAggregateSuite.scala | 2 +-
.../org/apache/spark/sql/PercentileQuerySuite.scala | 21 +++++++++++++++++++++
4 files changed, 29 insertions(+), 3 deletions(-)
diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index 39cf3ad35aa5..fe19681d3357 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -28,6 +28,7 @@ license: |
- Since Spark 4.3, the configuration key
`spark.sql.sources.v2.bucketing.allowJoinKeysSubsetOfPartitionKeys.enabled` has
been renamed to
`spark.sql.sources.v2.bucketing.allowKeysSubsetOfPartitionKeys.enabled` to
reflect that it now applies to storage-partitioned joins, aggregates, and
windows. The old key continues to work as an alias.
- Since Spark 4.3, the Spark Thrift Server rejects setting JVM system
properties through the `set:system:` session configuration overlay (for
example, in a JDBC connection string). To restore the previous behavior, set
`spark.sql.legacy.hive.thriftServer.allowSettingSystemProperties` to `true`.
- Since Spark 4.3, the adaptive execution rule
`org.apache.spark.sql.execution.adaptive.DynamicJoinSelection` has been renamed
to `DemoteBroadcastHashJoin`, which now only demotes broadcast hash joins
(emitting `NO_BROADCAST_HASH`). Its selection of shuffled hash join over sort
merge join has moved to a new physical rule gated by
`spark.sql.adaptive.convertSortMergeJoinToShuffledHashJoin.enabled` (default
`true`). If you previously disabled the shuffled-hash-join preference by
listing `o [...]
+- Since Spark 4.3, the exact `percentile`, `percentile_cont`, and `median`
aggregate functions (including their `WITHIN GROUP (ORDER BY ...)` forms)
compute the linear interpolation between two neighboring values as `lower +
fraction * (higher - lower)` instead of `(1 - fraction) * lower + fraction *
higher`. The two are equal in exact arithmetic, but the new form is
monotonically non-decreasing in the requested percentage and avoids a rounding
error the old form could introduce. As a re [...]
## Upgrading from Spark SQL 4.1 to 4.2
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/percentiles.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/percentiles.scala
index ac351db9e471..8bd201478596 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/percentiles.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/percentiles.scala
@@ -223,8 +223,12 @@ abstract class PercentileBase
// We end up here only if spark.sql.legacy.percentileDiscCalculation=true
toDoubleValue(lowerKey)
} else {
- // Linear interpolation to get the exact percentile
- (higher - position) * toDoubleValue(lowerKey) + (position - lower) *
toDoubleValue(higherKey)
+ // Linear interpolation to get the exact percentile. Computed as lower +
fraction *
+ // (higher - lower) to stay monotonic; the delta can overflow to
Infinity only for adjacent
+ // values straddling zero near Double.MaxValue, an extreme case where
the old form was also
+ // inaccurate.
+ toDoubleValue(lowerKey) +
+ (position - lower) * (toDoubleValue(higherKey) -
toDoubleValue(lowerKey))
}
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala
index 97931a007605..36c2a9540109 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala
@@ -1616,7 +1616,7 @@ class DataFrameAggregateSuite extends SharedSparkSession
percentile(col("year"), lit(0.3), lit(2)),
percentile(col("year"), lit(Array(0.25, 0.75)), lit(2))
),
- Row("Java", 2012.2999999999997, Seq(2012.25, 2012.75), 2012.0,
Seq(2012.0, 2013.0)) ::
+ Row("Java", 2012.3, Seq(2012.25, 2012.75), 2012.0, Seq(2012.0, 2013.0))
::
Row("dotNET", 2012.0, Seq(2012.0, 2012.5), 2012.0, Seq(2012.0,
2012.75)) :: Nil
)
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/PercentileQuerySuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/PercentileQuerySuite.scala
index 3f8d38b25216..2004e028079f 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/PercentileQuerySuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/PercentileQuerySuite.scala
@@ -46,4 +46,25 @@ class PercentileQuerySuite extends SharedSparkSession {
Row("INTERVAL '16-8' YEAR TO MONTH", null, "INTERVAL '0 00:03:20' DAY
TO SECOND"))
}
}
+
+ test("SPARK-57982: exact percentile is monotonically non-decreasing in the
percentage") {
+ withTempView(table) {
+ // Values near 1e18, where consecutive doubles are spaced ~128 apart.
+ spark.range(0, 20).selectExpr("1e18 + id * 128.0 AS x")
+ .createOrReplaceTempView(table)
+
+ // percentile(x, 0.04) must not exceed percentile(x, 0.05).
+ checkAnswer(
+ spark.sql(s"SELECT percentile(x, 0.04) <= percentile(x, 0.05) FROM
$table"),
+ Row(true))
+
+ // percentile over increasing percentages must come back non-decreasing.
+ val percentages = (0 to 100).map(_ / 100.0)
+ val arrayLit = percentages.mkString("array(", ", ", ")")
+ val result = spark.sql(s"SELECT percentile(x, $arrayLit) FROM $table")
+ .head().getSeq[Double](0)
+ assert(result.sliding(2).forall(w => w.head <= w.last),
+ s"percentile results are not monotonically non-decreasing: $result")
+ }
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
