2013/3/25 Paolo Losi <paolo.l...@gmail.com>: > My 2 cents ... > > The problem is that penalized (l1 o l2) single class Logistic Regression is > not > well calibrated to start with. > In other terms: the penalization param value that optimizes classification > accuracy is not > guaranteed to be the one that maximizes probability estimation accuracy. > > Optimizing a cross validated probability estimation loss > (like log-likelihood or brier scorer) might be better but again > you are using a single parameter (the penalization value) to obtain two > goals: > - preventing overfitting by controlling model parsimony > - having a well calibrated estimate of log likelihood > > I'd suggest tuning the penalization parameter in order to get the best > classification accuracy, > then run out of bag calibration on the score function > (standard logistic regression if n_sample < 500/1000, isotonic otherwise) > > Then I'm +1 with Mathieu on simple normalization of the one-class > probabilities
I am also +1 a simple short term solution while still keeping longer terms goal for - proper multinomial penalized LR on one hand, - tooling for calibrating arbitrary decision functions in general (not just penalized LR) using isotonic regression for both binary class and multiclass with one-vs-rest re-normalization) on the other hand. -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel ------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_mar _______________________________________________ Scikit-learn-general mailing list Scikit-learn-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
