Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/10639#issuecomment-172656627
@yanboliang I made a pass on your implementation and read Green's paper
"Iteratively Reweighted Least Squares for Maximum Likelihood Estimation, and
some Robust and Resistant Alternatives". My main question is whether we should
couple IRLS and GLM together. IRLS could be applied to models outside GLM
family. Coupling them together makes it harder to extend and some concepts
harder to understand, e.g., "Family.startingMu". I thought about decoupling the
concepts but didn't find a satisfactory solution. One possible approach is to
make IRLS accept the initial guess `x0` and a reweighting function that updates
weights and offsets at each iteration. Then we can wrap GLM family and link
functions to use that interface. It would be great if the IRLS implementation
could be extended to handle: GLMs, Lp regression, LASSO, and maybe some
non-convex loss functions. If this sounds good to you, we can think more about
the refactoring.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
