[
https://issues.apache.org/jira/browse/SPARK-17136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935112#comment-15935112
]
Seth Hendrickson commented on SPARK-17136:
------------------------------------------
The reason to support setting them in both places would be backwards
compatibility mainly. If we still allow users to set {{maxIter}} on the
estimator then we won't break code that previously did this. Specifying the
optimizer, either one built into Spark or a custom one, would be optional and
something mostly advanced users would do. About grid-based CV, this would be a
point that we need to carefully consider and make sure that we get it right.
We'd still allow users to search over grids of {{maxIter}}, {{tol}} etc...
since those params are still there, but additionally users could search over
different optimizers and optimizers with different parameters themselves. I
think that could be a bit clunky, but it's open for design discussion. e.g.
{code}
val paramGrid = new ParamGridBuilder()
.addGrid(lr.minimizer, Array(new LBFGS(), new OWLQN(), new LBFGSB(lb, ub)))
.build()
{code}
Yes, there are cases where users could supply conflicting grids, but AFAICT
this problem already exists, e.g.
{code}
val paramGrid = new ParamGridBuilder()
.addGrid(lr.solver, Array("normal", "l-bfgs"))
.addGrid(lr.maxIter, Array(10, 20)) // maxIter is ignored when solver is
normal
.build()
{code}
About your suggestion of mimicking Spark SQL - would you mind elaborating here
or on the design doc? I'm not as familiar with it, so if you have some design
in mind it would be great to hear that.
> Design optimizer interface for ML algorithms
> --------------------------------------------
>
> Key: SPARK-17136
> URL: https://issues.apache.org/jira/browse/SPARK-17136
> Project: Spark
> Issue Type: Sub-task
> Components: ML
> Reporter: Seth Hendrickson
>
> We should consider designing an interface that allows users to use their own
> optimizers in some of the ML algorithms, similar to MLlib.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
