huaxingao commented on a change in pull request #26596:
[SPARK-29959][ML][PYSPARK] Summarizer support more metrics
URL: https://github.com/apache/spark/pull/26596#discussion_r348186250
##########
File path: mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala
##########
@@ -484,19 +559,59 @@ private[ml] object SummaryBuilderImpl extends Logging {
val len = currM2n.length
while (i < len) {
// We prevent variance from negative value caused by numerical error.
- realVariance(i) = math.max((currM2n(i) + deltaMean(i) * deltaMean(i)
* weightSum(i) *
- (totalWeightSum - weightSum(i)) / totalWeightSum) / denominator,
0.0)
+ realVariance(i) = math.max((currM2n(i) + deltaMean(i) * deltaMean(i)
* currWeightSum(i) *
+ (totalWeightSum - currWeightSum(i)) / totalWeightSum) /
denominator, 0.0)
i += 1
}
}
Vectors.dense(realVariance)
}
+ /**
+ * Unbiased estimate of standard deviation of each dimension.
+ */
+ def std: Vector = {
+ require(requestedMetrics.contains(Std))
+ require(totalWeightSum > 0, s"Nothing has been added to this
summarizer.")
+
+ val realStd = Array.ofDim[Double](n)
+
+ val denominator = totalWeightSum - (weightSquareSum / totalWeightSum)
+
+ // Sample variance is computed, if the denominator is less than 0, the
variance is just 0.
+ if (denominator > 0.0) {
+ val deltaMean = currMean
+ var i = 0
+ val len = currM2n.length
+ while (i < len) {
+ // We prevent variance from negative value caused by numerical error.
+ val variance = math.max((currM2n(i) + deltaMean(i) * deltaMean(i) *
currWeightSum(i) *
+ (totalWeightSum - currWeightSum(i)) / totalWeightSum) /
denominator, 0.0)
+ realStd(i) = math.sqrt(variance)
+ i += 1
+ }
+ }
+ Vectors.dense(realStd)
+ }
+
Review comment:
Maybe reuse common code between ```std``` and ```variance```?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
