Github user sethah commented on a diff in the pull request:
https://github.com/apache/spark/pull/16149#discussion_r91021502
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/regression/GeneralizedLinearRegression.scala
---
@@ -479,7 +479,12 @@ object GeneralizedLinearRegression extends
DefaultParamsReadable[GeneralizedLine
numInstances: Double,
weightSum: Double): Double = {
-2.0 * predictions.map { case (y: Double, mu: Double, weight:
Double) =>
- weight * dist.Binomial(1, mu).logProbabilityOf(math.round(y).toInt)
+ val wt = math.round(weight).toInt
+ if (wt == 0) {
+ 0.0
+ } else {
+ dist.Binomial(wt, mu).logProbabilityOf(math.round(y *
weight).toInt)
--- End diff --
So I think the real issue here is that we don't currently allow users to
specify a binomial GLM using success/outcome pairs. One way to mash that kind
of grouped data into the format Spark requires is using the process described
above by @actuaryzhang, but then we need to adjust the log-likelihood
computation as was also noted.
So @srowen is correct in saying that this is inaccurate for non-integer
weights. I checked with R's glmnet, and it seems that they obey the semantics
of data weights for a binomial GLM corresponding to the number of successes. So
they log a warning when you input data weights of non-integer values, then
proceed with the method proposed in this patch.
So, this actually _does_ match R's behavior and I am in favor of the
change. But we need to log appropriate warnings and write good unit tests. What
are others' thoughts?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
