srowen commented on a change in pull request #26596: [SPARK-29959][ML][PYSPARK]
Summarizer support more metrics
URL: https://github.com/apache/spark/pull/26596#discussion_r350228834
##########
File path: mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala
##########
@@ -460,21 +486,52 @@ private[ml] object SummaryBuilderImpl extends Logging {
val realMean = Array.ofDim[Double](n)
var i = 0
while (i < n) {
- realMean(i) = currMean(i) * (weightSum(i) / totalWeightSum)
+ realMean(i) = currMean(i) * (currWeightSum(i) / totalWeightSum)
i += 1
}
Vectors.dense(realMean)
}
+ /**
+ * Sum of each dimension.
+ */
+ def sum: Vector = {
+ require(requestedMetrics.contains(Sum))
+ require(totalWeightSum > 0, s"Nothing has been added to this
summarizer.")
+
+ val realSum = Array.ofDim[Double](n)
+ var i = 0
+ while (i < n) {
+ realSum(i) = currMean(i) * currWeightSum(i)
+ i += 1
+ }
+ Vectors.dense(realSum)
+ }
+
/**
* Unbiased estimate of sample variance of each dimension.
Review comment:
Hm, this comment confuses me. "Unbiased" arises when estimating the
population variance from the sample's. Here there is no sample, just a
population. Normally this amounts to diving by n instead of n-1 when computing
the variance, but I don't quite understand the formula below, so I'm not sure
which one it's doing. Do you happen to know what it is following, as you've
looked at it recently?
It should follow pandas, which is computing population variance. The thing
that confuses me in the comment below about considering a negative denominator,
which leads me to ...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
