The way I implemented it, the learning rate for the intercept should be
0.01 times the learning rate of the other features.
The value of .01 is something that I set empirically, I adopted it from
Leon Buttou's sgd project and experimented with different values. I found
that lower intercept learning rates help a bit but the concrete value is
not too important - so I decided to use a fixed value.
I think the decay value might in fact be a function of the number of
non-zero values per feature. If you have a dataset with sparse and dense
features then intercept decay should be turned off -- alternatively, you
can also scale the dense features to decrease their magnitude.
2014-07-30 11:42 GMT+02:00 Danny Sullivan <[email protected]>:
> I found that for sparse data, the scikit implementation of sgd uses an
> intercept_decay variable set to .01 (SPARSE_INTERCEPT_DECAY) to avoid
> intercept oscillation. Shouldn't this be determined by the learning_rate
> instead? I'm asking because it adds a layer of tuning that the user doesn't
> have control over.
>
> Danny
>
>
> ------------------------------------------------------------------------------
> Infragistics Professional
> Build stunning WinForms apps today!
> Reboot your WinForms applications with our WinForms controls.
> Build a bridge from your legacy apps to the future.
>
> http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk
> _______________________________________________
> Scikit-learn-general mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
>
--
Peter Prettenhofer
------------------------------------------------------------------------------
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls.
Build a bridge from your legacy apps to the future.
http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk
_______________________________________________
Scikit-learn-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
