zhengruifeng commented on a change in pull request #26596:
[SPARK-29959][ML][PYSPARK] Summarizer support more metrics
URL: https://github.com/apache/spark/pull/26596#discussion_r348860419
##########
File path: docs/ml-statistics.md
##########
@@ -109,7 +109,8 @@ Refer to the [`ChiSquareTest` Python
docs](api/python/index.html#pyspark.ml.stat
## Summarizer
We provide vector column summary statistics for `Dataframe` through
`Summarizer`.
-Available metrics are the column-wise max, min, mean, variance, and number of
nonzeros, as well as the total count.
+Available metrics are the column-wise max, min, mean, sum, variance, std,
squared sum, and number of nonzeros,
Review comment:
This PR is mainly for convenience:
1, `sum`: existing summarizer does not provide sum of weights, so if I want
to compute `sum` in NaiveBayes, I need to use two udafs like
`.agg(sum(w).as("weightSum"), Summarizer.metrics("mean",
"count").summary(validateUDF(col($(featuresCol))), w))`, and then I need to
call `BLAS.scal(weightSum, mean)`
2, `std`: I noticed that `std` is more widely used than `variance`;
3, `weightSum`: it is provided in `mllib.MultivariateOnlineSummarizer`, I
donot know why `Summarizer` do not expose it.
In general I guess that the exisintg OPs of Vectors are not enough, if we
can provide new OPs like `sqrt`,`square`,`abs` (like
`np.sqrt`,`np.square`,`np.abs` in numpy), then we do not need to add above
`std`,`sumL2`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
