[
https://issues.apache.org/jira/browse/FLINK-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14723555#comment-14723555
]
Trevor Grant commented on FLINK-1994:
-------------------------------------
Sorry for the delay on this- I've been in the middle of a move.
I got back to it today and was reviewing what I had done and the original post.
I realize there are two ways to go about this and what want some input on
which would be preferable:
Method 1) Several pre-baked ways to calculate the effective learning rate, set
as a switch. E.g.:
val effectiveLearningRate = optimizationMethod match {
// original effective learning rate method for backward
compatability
case 0 => learningRate/Math.sqrt(iteration)
// These come straight from sklearn
case 1 => learningRate
case 2 => 1 / (regularizationConstant * iteration)
case 3 => learningRate / Math.pow(iteration, 0.5)
...
}
Method2) Make the calculation definable by the user. E.g. introduce a function
to the class which maybe overridden.
This is a classic trade-off between ease of use and functionality. Method 1 is
easier for novice users/users who are migrating from sklearn. Method2 will be
more extensible- letting users write any old effective learning rate
calculation they want.
I am leaning toward method 1 because how many people really are writing out
their own custom effective learning rate (as long as there is a fairly good
number of 'prebaked' calculators available, and because if someone really wants
to add a method, it simply requires adding another case.
I want to open this up in case anyone has an opinion, just in case.
Best,
tg
> Add different gain calculation schemes to SGD
> ---------------------------------------------
>
> Key: FLINK-1994
> URL: https://issues.apache.org/jira/browse/FLINK-1994
> Project: Flink
> Issue Type: Improvement
> Components: Machine Learning Library
> Reporter: Till Rohrmann
> Assignee: Trevor Grant
> Priority: Minor
> Labels: ML, Starter
>
> The current SGD implementation uses as gain for the weight updates the
> formula {{stepsize/sqrt(iterationNumber)}}. It would be good to make the gain
> calculation configurable and to provide different strategies for that. For
> example:
> * stepsize/(1 + iterationNumber)
> * stepsize*(1 + regularization * stepsize * iterationNumber)^(-3/4)
> See also how to properly select the gains [1].
> Resources:
> [1] http://arxiv.org/pdf/1107.2490.pdf
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
