Joel Hooimeyer created SPARK-57982:
--------------------------------------
Summary: percentile function not monotonically increasing
Key: SPARK-57982
URL: https://issues.apache.org/jira/browse/SPARK-57982
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 4.1.2, 3.5.8
Reporter: Joel Hooimeyer
The {{percentile}} function which is documented as being exact does not
monotonically increase with certain data sets. This leads to us getting a
negative IQR and causes other calculations to be incorrect.
Reproduction:
{code:scala}
import spark.implicits._
val df = (0 until 20).map(i => 1e18 + i * 128.0).toDF("x")
df.selectExpr("percentile(x, 0.04) AS p04", "percentile(x, 0.05) AS
p05").show(false)
{code}
Outputs:
{code:none}
+---------------------+------+
|p04 |p05 |
+---------------------+------+
|1.0000000000000001E18|1.0E18|
+---------------------+------+
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
