Ma77Ball opened a new pull request, #57241:
URL: https://github.com/apache/spark/pull/57241
### What changes were proposed in this pull request?
The exact `percentile` aggregate interpolates linearly between the two
values surrounding the target position. This changes that interpolation in
`PercentileBase.getPercentile` from
```
(higher - position) * lowerValue + (position - lower) * higherValue
```
to the algebraically equivalent
```
lowerValue + (position - lower) * (higherValue - lowerValue)
```
The method is shared by `percentile`, `percentile_cont`, `median`, and their
`WITHIN GROUP` forms.
### Why are the changes needed?
The two forms are identical in exact arithmetic but differ under
floating-point rounding. The old form scales both endpoint values (which can be
very large) by the interpolation weights independently, so the two large
products carry rounding error on the order of the spacing between adjacent
`double`s. When that spacing exceeds the true difference between two nearby
percentiles, the results round in inconsistent directions and the percentile
stops being monotonically non-decreasing in the percentage.
Reproduction:
```scala
val df = (0 until 20).map(i => 1e18 + i * 128.0).toDF("x")
df.selectExpr("percentile(x, 0.04) AS p04", "percentile(x, 0.05) AS
p05").show(false)
```
Before this change:
```
+---------------------+------+
|p04 |p05 |
+---------------------+------+
|1.0000000000000001E18|1.0E18|
+---------------------+------+
```
Here `p04 > p05`, which is not monotonic. A negative interquartile range
(`percentile(x, 0.75) - percentile(x, 0.25)`) is one user-visible consequence.
The new form leaves the large magnitude in `lowerValue` untouched and scales
only the small delta `(higherValue - lowerValue)`, so the rounding error is
proportional to that small delta and the result stays non-decreasing in the
fraction.
### Does this PR introduce _any_ user-facing change?
Yes. Exact `percentile` / `percentile_cont` / `median` results may change in
the last unit(s) in the last place for large-magnitude inputs, and are now
non-decreasing as the percentage increases. After this change the reproduction
above returns `p04 = p05 = 1.0000000000000001E18`. Results for normal-magnitude
inputs are unchanged.
### How was this patch tested?
- New regression test in `PercentileQuerySuite`: the reproduction above plus
a 101-point sweep asserting the results are monotonically non-decreasing.
- Existing `PercentileSuite`, `PercentileQuerySuite`, and the
`percentiles.sql` golden-file suite pass unchanged (no result churn for
normal-magnitude data).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
