Github user dbtsai commented on a diff in the pull request:
https://github.com/apache/spark/pull/3220#discussion_r20208266
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/stat/MultivariateOnlineSummarizer.scala
---
@@ -50,6 +50,29 @@ class MultivariateOnlineSummarizer extends
MultivariateStatisticalSummary with S
private var currMin: BDV[Double] = _
/**
+ * Adds input value to position i.
+ */
+ private[this] def add(i: Int, value: Double) = {
+ if (value != 0.0) {
--- End diff --
Yes. However, we know the total # of samples, and # of nonzero in each
column, so if # of samples and # of nonzero are different, and we find the min
is some positive number, then the actually min will be zero since we have zero
somewhere which we don't add into summarizer.
For max, the same logic will be applied.
For mean, we can fix this effect by realMean(i) = currMean(i) *
(nnz(i) / totalCnt)
As a result, for sparse dataset, we only need to add the nonzero into the
summarizer, and it will be O(\bar{n}) instead of O(n) where \bar{n} is the
average nonzero elements in one sample.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
