GitHub user sethah opened a pull request:
https://github.com/apache/spark/pull/15893
[SPARK-18456][ML][FOLLOWUP] Use matrix abstraction for coefficients in
LogisticRegression training
## What changes were proposed in this pull request?
This is a follow up to some of the discussion
[here](https://github.com/apache/spark/pull/15593). During LogisticRegression
training, we store the coefficients combined with intercepts as a flat vector,
but a more natural abstraction is a matrix. Here, we refactor the code to use
matrix where possible, which makes the code more readable and greatly
simplifies the indexing.
Note: We do not use a Breeze matrix for the cost function as was mentioned
in the linked PR. This is because LBFGS/OWLQN require an implicit
`MutableInnerProductModule[DenseMatrix[Double], Double]` which is not natively
defined in Breeze. We would need to extend Breeze in Spark to define it
ourselves. Also, we do not modify the `regParamL1Fun` because OWLQN in Breeze
requires a `MutableEnumeratedCoordinateField[(Int, Int), DenseVector[Double]]`
(since we still use a dense vector for coefficients). Here again we would have
to extend Breeze inside Spark.
## How was this patch tested?
This is internal code refactoring - the current unit tests passing show us
that the change did not break anything. No added functionality in this patch.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sethah/spark logreg_refactor
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15893.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15893
----
commit fef9951904385ed9018cf7c7b526b44389d1b8af
Author: sethah <[email protected]>
Date: 2016-11-15T22:11:34Z
refactor logistic regression to use matrix abstraction where possible
commit 28f67fb27d3a760597fd7281900c756b4dd79c17
Author: sethah <[email protected]>
Date: 2016-11-15T22:59:45Z
clean up some comments and naming
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
