[
https://issues.apache.org/jira/browse/SPARK-57982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Toth updated SPARK-57982:
-------------------------------
Issue Type: Improvement (was: Bug)
> percentile function not monotonically increasing
> ------------------------------------------------
>
> Key: SPARK-57982
> URL: https://issues.apache.org/jira/browse/SPARK-57982
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.5.8, 4.1.2
> Reporter: Joel Hooimeyer
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> The {{percentile}} function which is documented as being exact does not
> monotonically increase with certain data sets. This leads to us getting a
> negative IQR and causes other calculations to be incorrect.
> Reproduction:
> {code:scala}
> import spark.implicits._
> val df = (0 until 20).map(i => 1e18 + i * 128.0).toDF("x")
> df.selectExpr("percentile(x, 0.04) AS p04", "percentile(x, 0.05) AS
> p05").show(false)
> {code}
> Outputs:
> {code:none}
> +---------------------+------+
> |p04 |p05 |
> +---------------------+------+
> |1.0000000000000001E18|1.0E18|
> +---------------------+------+
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
