My 2 cents ...
The problem is that penalized (l1 o l2) single class Logistic Regression is
not
well calibrated to start with.
In other terms: the penalization param value that optimizes classification
accuracy is not
guaranteed to be the one that maximizes probability estimation accuracy.
Optimizing a cross validated probability estimation loss
(like log-likelihood or brier scorer) might be better but again
you are using a single parameter (the penalization value) to obtain two
goals:
- preventing overfitting by controlling model parsimony
- having a well calibrated estimate of log likelihood
I'd suggest tuning the penalization parameter in order to get the best
classification accuracy,
then run out of bag calibration on the score function
(standard logistic regression if n_sample < 500/1000, isotonic otherwise)
Then I'm +1 with Mathieu on simple normalization of the one-class
probabilities
Paolo
On Mon, Mar 25, 2013 at 5:03 PM, Mathieu Blondel <math...@mblondel.org>wrote:
> On Mon, Mar 25, 2013 at 10:57 PM, Lars Buitinck <l.j.buiti...@uva.nl>
> wrote:
>
> > You mean the score rescaling at the end of section 2? What Zadrozny
> > and Elkan really seem to be recommended is isotonic regression...
>
> The question is how to deal with the *multiclass* case, which isotonic
> regression cannot handle directly.
> In the paper (section 4), Zadrozny and Elkan show that breaking the
> multiclass problem into two-class problems,
> calibrating the probabilities for each problem then normalizing works
> better than other fancier approaches.
>
> Mathieu
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_mar
> _______________________________________________
> Scikit-learn-general mailing list
> Scikit-learn-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_mar
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
