Hi again,
On Wed, Sep 26, 2012 at 10:27 PM, Gael Varoquaux <
[email protected]> wrote:
> On Wed, Sep 26, 2012 at 09:53:36PM -0700, Ariel Rokem wrote:
> > I haven't tried this yet - I'll try it tomorrow. In a way it sounds like
> it's
> > inadvertently implementing an early stopping criterion,
>
> Yes: maxiter and tol
OK - I tried this. Doesn't seem to have much effect on the results. To be
on the safe side, I set maxiter to 100000 and tol to 1e-64.
Still, the r-squared between the fit and data is only about 0.95 (that's
not just 'numerical error', if you mean floating point rounding kind of
stuff, right?). Again, this is for a case where OLS fitting gives a perfect
correlation of approximately 1.0 (0.999998, to be precise. Now, that's
numerical error!). Even more puzzling, the difference between the default
behavior and the behavior with these rather extreme setting seems
negligible. So it seems that setting these inputs does very little in
pushing the results from the 'early stopping' regularization.
> which is also a form of regularization. That's confusing, considering
> > that I set the regularization to 0.
>
> Indeed. However in the real world of numerical algorithms, this kind of
> problems happen often, and there is no way you avoid them. One option
> would be to set a tiny tol and not limit the amount of iterations, but
> for most people, this ends up being a problem more than a solution.
>
> In practice, it is not recommended to use coordinate descent with a very
> small regularization.
>
Sure - that makes sense. I am just trying to pry open the "black box" of
the algorithm to understand the effects of regularization. At least now I
know that there is always some regularization. That is, I shouldn't take
too literally the alpha setting of 0 (which you guys recommend against
anyway, apparently for good reason).
Thanks for the help!
Ariel
> HTH,
>
> Gaël
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://ad.doubleclick.net/clk;258768047;13503038;j?
> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
> _______________________________________________
> Scikit-learn-general mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
------------------------------------------------------------------------------
Got visibility?
Most devs has no idea what their production app looks like.
Find out how fast your code is with AppDynamics Lite.
http://ad.doubleclick.net/clk;262219671;13503038;y?
http://info.appdynamics.com/FreeJavaPerformanceDownload.html
_______________________________________________
Scikit-learn-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
