Github user tanejagagan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16497#discussion_r95303304
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Percentile.scala
---
@@ -126,10 +152,15 @@ case class Percentile(
buffer: OpenHashMap[Number, Long],
input: InternalRow): OpenHashMap[Number, Long] = {
val key = child.eval(input).asInstanceOf[Number]
+ val frqValue = frequency.eval(input)
// Null values are ignored in counts map.
- if (key != null) {
- buffer.changeValue(key, 1L, _ + 1L)
+ if (key != null && frqValue != null) {
--- End diff --
Yes this would be wrong to use the default value of 1
Let take a data set of
Age, Count
20, 1
15, 1
10, 0
If we take the default value of 1L when the frq is 0 is then .5 percentile
would become 15 . This is incorrect. I agree with other suggestion of either
failing or disregard those values
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
