That appears to work (with a small modification):
param_grid = [{'eta0':[1/alpha_this_step], 'alpha':[alpha_this_step]} for
alpha_this_step in 10.0**(np.arange(11)-5)]
Neat trick. Thanks.
Unfortunately I don't have time to do the experiment, but eta = 1/t and eta
= 1/(alpha*t) seem to be pretty different update rules if alpha is
significantly different from 1. But I will also say that the number of
iterations is usually pretty forgiving, in my experience.
On Thu, Sep 11, 2014 at 4:28 PM, Andy <[email protected]> wrote:
> You can actually do that using the current grid-search.
> Specify the "grid" as a list of single grid-points. That should do.
> param_grid = [{'eta0':1/alpha_this_step, 'alpha':alpha_this_step} for
> alpha_this_step in my_alphas]
>
> That should do it, right?
> I think for the "optimum" the same guarantee holds as for the pegasos rate
> and I don't think you'll be much faster.
> I'd be interested in the results, though ;)
>
> Andy
>
>
>
> On 09/11/2014 08:24 PM, F. William High wrote:
>
> The Shalev-Schwartz et al. Pegasos update rule on the learning rate
> parameter is
>
> nu_i = 1 / (lambda * i)
>
> where lambda multiplies the regularization term. If this rule is used,
> they show you can converge to an error of epsilon in O(1/(lambda*epsilon))
> iterations at high probability. This differs from SGDClassifier's
> 'optimal' learning_rate update by the added 1/lambda multiplier.
>
> In SGDClassifier this corresponds to eta = nu, alpha = lambda and eta0
> =1/alpha. Just to make it explicit, the learning rate would be
>
> eta0 = 1/alpha
> eta = eta0 / power(t,power_t)
>
> I can effectively apply Pegasos in a single SGDClassifier training run
> if I set the following options (having chosen a value of alpha_this_step):
>
> learning_rate='invscaling'
> eta0=1/alpha_this_step
> alpha=alpha_this_step
> power_t=1
>
> The problem is that I want to do GridSearchCV on a grid of alpha values,
> and at each step I want to use the above learning rate update rules. I
> can't do this inside a grid search, so I propose the following optionality:
>
> learning_rate='pegasos'
> alpha=alpha_this_step
>
> Under the hood, this would set eta0=1/alpha and power_t=1.
>
> If this functionality already exists (grid search of SGD classification
> using Pegasos learning rate update at each step), I'd appreciate if you'd
> point it out to me. Otherwise I'll submit a pull request.
>
> Thanks!
>
> Will
>
>
>
> ------------------------------------------------------------------------------
> Want excitement?
> Manually upgrade your production database.
> When you want reliability, choose Perforce
> Perforce version control. Predictably
> reliable.http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
>
>
>
> _______________________________________________
> Scikit-learn-general mailing
> [email protected]https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
>
>
>
> ------------------------------------------------------------------------------
> Want excitement?
> Manually upgrade your production database.
> When you want reliability, choose Perforce
> Perforce version control. Predictably reliable.
>
> http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
> _______________________________________________
> Scikit-learn-general mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
>
------------------------------------------------------------------------------
Want excitement?
Manually upgrade your production database.
When you want reliability, choose Perforce
Perforce version control. Predictably reliable.
http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
_______________________________________________
Scikit-learn-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
