[
https://issues.apache.org/jira/browse/SPARK-18111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15610192#comment-15610192
]
Zhenhua Wang edited comment on SPARK-18111 at 10/27/16 12:46 AM:
-----------------------------------------------------------------
[~srowen] The minimum is not only skipped once in the whole data, but *skipped
per partition*.
For example, we have two partitions of data: (1, 1, 3, 3) and (5, 5, 7, 7),
then when we do global merging, the samples in QuantileSummaries is (1, 3, 3,
5, 7, 7), and the percentiles result returned for query percentile_approx(0.25,
0.5, 0.75) is (3.0, 5.0, 7.0), but the correct answer should be (1.0, 3.0,
5.0). Of course we can say it's an approximate algorithm, but this error is
already *beyond the error bound which the algo provides*. And also, we can make
the error even larger if we construct more such partitions and thus more
skipped minimum elements.
was (Author: zenwzh):
[~srowen] The minimum is not only skipped once in the whole data, but skipped
per partition.
For example, we have two partitions of data: (1, 1, 3, 3) and (5, 5, 7, 7),
then when we do global merging, the samples in QuantileSummaries is (1, 3, 3,
5, 7, 7), and the percentiles result returned for query percentile_approx(0.25,
0.5, 0.75) is (3.0, 5.0, 7.0), but the correct answer should be (1.0, 3.0,
5.0). Of course we can say it's an approximate algorithm, but this error is
already *beyond the error bound which the algo provides*. And also, we can make
the error even larger if we construct more such partitions and thus more
skipped minimum elements.
> Wrong ApproximatePercentile answer when multiple records have the minimum
> value
> -------------------------------------------------------------------------------
>
> Key: SPARK-18111
> URL: https://issues.apache.org/jira/browse/SPARK-18111
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.1
> Reporter: Zhenhua Wang
>
> When multiple records have the minimum value, the answer of
> ApproximatePercentile is wrong.
> For example, the following query returns 2.0 for percentile 0.5, but the
> correct answer should be 1.0
> 0: jdbc:hive2://localhost:10000> select key from src2;
> +------+--+
> | key |
> +------+--+
> | 1 |
> | 1 |
> | 2 |
> | 2 |
> +------+--+
> 4 rows selected (0.185 seconds)
> 0: jdbc:hive2://localhost:10000> select percentile_approx(key, array(0.5))
> from src2;
> +------------------------------------------------------------+--+
> | percentile_approx(CAST(key AS DOUBLE), array(0.5), 10000) |
> +------------------------------------------------------------+--+
> | [2.0] |
> +------------------------------------------------------------+--+
> 1 row selected (0.292 seconds)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
