[
https://issues.apache.org/jira/browse/SPARK-23091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen updated SPARK-23091:
------------------------------
Environment: Agree, go ahead with a PR.
Priority: Minor (was: Major)
Component/s: Tests
> Incorrect unit test for approxQuantile
> --------------------------------------
>
> Key: SPARK-23091
> URL: https://issues.apache.org/jira/browse/SPARK-23091
> Project: Spark
> Issue Type: Improvement
> Components: ML, Tests
> Affects Versions: 2.2.1
> Environment: Agree, go ahead with a PR.
> Reporter: Kuang Chen
> Priority: Minor
>
> Currently, test for `approxQuantile` (quantile estimation algorithm) checks
> whether estimated quantile is within +- 2*`relativeError` from the true
> quantile. See the code below:
> [https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/DataFrameStatSuite.scala#L157]
> However, based on the original paper by Greenwald and Khanna, the estimated
> quantile is guaranteed to be within +- `relativeError` from the true
> quantile. Using the double "tolerance" is misleading and incorrect, and we
> should fix it.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
