[
https://issues.apache.org/jira/browse/SPARK-15575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15360099#comment-15360099
]
Yanbo Liang edited comment on SPARK-15575 at 7/2/16 9:40 AM:
-------------------------------------------------------------
Yes, {{LinearRegression}}, {{LogisticRegression}}, {{AFTSurvivalRegression}}
and {{MultilayerPerceptronClassifier}} depend on L-BFGS/OWL-QN currently. We
have also implemented Spark owned L-BFGS/SGD optimizers in mllib package. But
the four estimators mentioned above use different implementation.
|| Estimators || Optimizer implementation ||
| LinearRegression | breeze L-BFGS/OWL-QN |
| LogisticRegression | breeze L-BFGS/OWL-QN |
| AFTSurvivalRegression | breeze L-BFGS/OWL-QN |
| MultilayerPerceptronClassifier | mllib L-BFGS/SGD |
The L-BFGS implementation in mllib also calling breeze L-BFGS underneath. We
should figure out a way to make the transformation smoothly. Since I'm also
investigating the scalable version of L-BFGS (SPARK-10078) recently, I can
start to write a draft to track the features and requirements of optimizers
that Spark needed. Then we can discuss and design how to move to the new
implementation. Thanks!
was (Author: yanboliang):
Yes, {{LinearRegression}}, {{LogisticRegression}}, {{AFTSurvivalRegression}}
and {{MultilayerPerceptronClassifier}} depend on L-BFGS/OWL-QN currently. We
have implemented Spark owned L-BFGS/SGD optimizers in mllib package. And the
four estimators mentioned above use different implementation.
|| Estimators || Optimizer implementation ||
| LinearRegression | breeze L-BFGS/OWL-QN |
| LogisticRegression | breeze L-BFGS/OWL-QN |
| AFTSurvivalRegression | breeze L-BFGS/OWL-QN |
| MultilayerPerceptronClassifier | mllib L-BFGS/SGD |
The L-BFGS implementation in mllib also calling breeze L-BFGS underneath. We
should figure out a way to make the transformation smoothly. Since I'm also
investigating the scalable version of L-BFGS (SPARK-10078) recently, I can
start to write a draft to track the features and requirements of optimizers
that Spark needed. Then we can discuss and design how to move to the new
implementation. Thanks!
> Remove breeze from dependencies?
> --------------------------------
>
> Key: SPARK-15575
> URL: https://issues.apache.org/jira/browse/SPARK-15575
> Project: Spark
> Issue Type: Improvement
> Components: ML
> Reporter: Joseph K. Bradley
>
> This JIRA is for discussing whether we should remove Breeze from the
> dependencies of MLlib. The main issues with Breeze are Scala 2.12 support
> and performance issues.
> There are a few paths:
> # Keep dependency. This could be OK, especially if the Scala version issues
> are fixed within Breeze.
> # Remove dependency
> ## Implement our own linear algebra operators as needed
> ## Design a way to build Spark using custom linalg libraries of the user's
> choice. E.g., you could build MLlib using Breeze, or any other library
> supporting the required operations. This might require significant work.
> See [SPARK-6442] for related discussion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
