On Jan 8, 2007, at 10:51 AM, Tom Fawcett wrote:
Just to add a note here: Ken is correct -- both NB and SVMs are
known to be rather poor at providing accurate probabilities. Their
scores tend to be too extreme. Producing good probabilities from
these scores is called calibrating the classifier, and it's more
complex than just taking a root of the score. There are several
methods for calibrating scores. The good news is that there's an
effective one called isotonic regression (or Pool Adjacent
Violators) which is pretty easy and fast. The bad news is that
there's no plug-in (ie, CPAN-ready) perl implementation of it (I've
got a simple implementation which I should convert and contribute
someday).
If you want to read about classifier calibration, google one of
these titles:
"Transforming classifier scores into accurate multiclass
probability estimates"
by Bianca Zadrozny and Charles Elkan
"Predicting Good Probabilities With Supervised Learning"
by A. Niculescu-Mizil and R. Caruana
Cool, thanks for the references. It might be nice to add somesuch
scheme to Algorithm::NaiveBayes (and friends), so that the user has a
choice of several normalization schemes, including "none". If I get
a surplus of tuits I'll add it, or if you feel like contributing your
stuff that would be great too.
-Ken
