You can actually do that using the current grid-search.
Specify the "grid" as a list of single grid-points. That should do.
param_grid = [{'eta0':1/alpha_this_step, 'alpha':alpha_this_step} for
alpha_this_step in my_alphas]
That should do it, right?
I think for the "optimum" the same guarantee holds as for the pegasos
rate and I don't think you'll be much faster.
I'd be interested in the results, though ;)
Andy
On 09/11/2014 08:24 PM, F. William High wrote:
The Shalev-Schwartz et al. Pegasos update rule on the learning rate
parameter is
nu_i = 1 / (lambda * i)
where lambda multiplies the regularization term. If this rule is
used, they show you can converge to an error of epsilon in
O(1/(lambda*epsilon)) iterations at high probability. This differs
from SGDClassifier's 'optimal' learning_rate update by the added
1/lambda multiplier.
In SGDClassifier this corresponds to eta = nu, alpha = lambda and eta0
=1/alpha. Just to make it explicit, the learning rate would be
eta0 = 1/alpha
eta = eta0 / power(t,power_t)
I can effectively apply Pegasos in a single SGDClassifier training run
if I set the following options (having chosen a value of alpha_this_step):
learning_rate='invscaling'
eta0=1/alpha_this_step
alpha=alpha_this_step
power_t=1
The problem is that I want to do GridSearchCV on a grid of alpha
values, and at each step I want to use the above learning rate update
rules. I can't do this inside a grid search, so I propose the
following optionality:
learning_rate='pegasos'
alpha=alpha_this_step
Under the hood, this would set eta0=1/alpha and power_t=1.
If this functionality already exists (grid search of SGD
classification using Pegasos learning rate update at each step), I'd
appreciate if you'd point it out to me. Otherwise I'll submit a pull
request.
Thanks!
Will
------------------------------------------------------------------------------
Want excitement?
Manually upgrade your production database.
When you want reliability, choose Perforce
Perforce version control. Predictably reliable.
http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
_______________________________________________
Scikit-learn-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
------------------------------------------------------------------------------
Want excitement?
Manually upgrade your production database.
When you want reliability, choose Perforce
Perforce version control. Predictably reliable.
http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
_______________________________________________
Scikit-learn-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
