Github user setjet commented on a diff in the pull request:
https://github.com/apache/spark/pull/18113#discussion_r118821077
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/typedaggregators.scala
---
@@ -99,3 +97,67 @@ class TypedAverage[IN](val f: IN => Double) extends
Aggregator[IN, (Double, Long
toColumn.asInstanceOf[TypedColumn[IN, java.lang.Double]]
}
}
+
+class TypedMinDouble[IN](val f: IN => Double) extends Aggregator[IN,
Double, Double] {
+ override def zero: Double = Double.PositiveInfinity
--- End diff --
Hi, thanks for having a look. This is actually not an issue because on an
empty dataset, nothing is returned. For more details, you could have a look a
the existing tests: the 'agg' function is called on a 'KeyValueGroupedDataset'
object, which is returned by the 'groupByKey'. This ensures it's only done per
key.
I have added an additional unit test to demonstrate.
Regarding Double.PositiveInfinity, I could change it to Double.Max, to be
in line with Long.Max if you'd prefer that. I personally think Infinity makes
more sense, although that is inconsistent with Long.Max because
Long.PositiiveInfinity is not available
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
