Hi Mathieu, average the ROC curves across folds (train / test splits) is a way:
http://scikit-learn.sourceforge.net/auto_examples/plot_roc_crossval.html then you can compare the mean ROC curves for the different algorithms. Just be careful not to estimate the model parameters using the test set. Alex On Sat, Oct 1, 2011 at 8:37 AM, mathieu lacage <[email protected]> wrote: > hi, > > I am looking for advice on how to pick a classifier among n competing > classifiers when they are evaluated on more than a single training/test data > set. i.e., I would like to compare, for each classifier, the set of roc > curves that are generated from each training/test data set. Is there an > established way of doing this ? > > Mathieu > -- > Mathieu Lacage <[email protected]> > > > ------------------------------------------------------------------------------ > All of the data generated in your IT infrastructure is seriously valuable. > Why? It contains a definitive record of application performance, security > threats, fraudulent activity, and more. Splunk takes this data and makes > sense of it. IT sense. And common sense. > http://p.sf.net/sfu/splunk-d2dcopy2 > _______________________________________________ > Scikit-learn-general mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general > > ------------------------------------------------------------------------------ All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2dcopy2 _______________________________________________ Scikit-learn-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
