Just noticed I got my Greeks wrong. By nu I meant eta everywhere.
On Thu, Sep 11, 2014 at 2:24 PM, F. William High <[email protected]> wrote:
> The Shalev-Schwartz et al. Pegasos update rule on the learning rate
> parameter is
>
> nu_i = 1 / (lambda * i)
>
> where lambda multiplies the regularization term. If this rule is used,
> they show you can converge to an error of epsilon in O(1/(lambda*epsilon))
> iterations at high probability. This differs from SGDClassifier's
> 'optimal' learning_rate update by the added 1/lambda multiplier.
>
> In SGDClassifier this corresponds to eta = nu, alpha = lambda and eta0
> =1/alpha. Just to make it explicit, the learning rate would be
>
> eta0 = 1/alpha
> eta = eta0 / power(t,power_t)
>
> I can effectively apply Pegasos in a single SGDClassifier training run if
> I set the following options (having chosen a value of alpha_this_step):
>
> learning_rate='invscaling'
> eta0=1/alpha_this_step
> alpha=alpha_this_step
> power_t=1
>
> The problem is that I want to do GridSearchCV on a grid of alpha values,
> and at each step I want to use the above learning rate update rules. I
> can't do this inside a grid search, so I propose the following optionality:
>
> learning_rate='pegasos'
> alpha=alpha_this_step
>
> Under the hood, this would set eta0=1/alpha and power_t=1.
>
> If this functionality already exists (grid search of SGD classification
> using Pegasos learning rate update at each step), I'd appreciate if you'd
> point it out to me. Otherwise I'll submit a pull request.
>
> Thanks!
>
> Will
>
>
------------------------------------------------------------------------------
Want excitement?
Manually upgrade your production database.
When you want reliability, choose Perforce
Perforce version control. Predictably reliable.
http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
_______________________________________________
Scikit-learn-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
