While migrating from custom LR implementation to MLLib's LR implementation
my colleagues noticed that prediction quality dropped (accoring to
different business metrics).
It's turned out that this issue caused by features standardization perfomed
by MLLib's LR: disregard to 'standardization' option's value all features
are scaled during loss and gradient computation (as well as in few other
According to comments in the code, standardization should be implemented
the same way it was implementes in R's glmnet package. I've looked through
corresponding Fortran code, an it seems like glmnet don't scale features
when you're disabling standardisation (but MLLib still does).
Our models contains multiple one-hot encoded features and scaling them is a
pretty bad idea.
Why MLLib's LR always scale all features? From my POV it's a bug.
Thanks in advance,