Re: [scikit-learn] Plot Cross-validated ROCs for multi-class classification problem

[serafim loukas]

Hello J.B,


I could simply create some ROC curves as shown in the scikit-learn 
documentation by selecting only 2 classes and then repeating by selecting other 
pair of classes (in total I have 3 classes so this would result in 3 different 
ROC figures).

An alternative would be I would like to plot the mean and confidence intervals 
of the 3-class Cohen Kappa metric as estimated by KFolds (k=5) cross-validation.

Any tips about this ?


Cheers,
Makis



On 21 Jul 2018, at 16:02, Brown J.B. via scikit-learn 
mailto:scikit-learn@python.org>> wrote:

Hello Makis,

2018-07-20 23:44 GMT+09:00 Andreas Mueller 
mailto:t3k...@gmail.com>>:
There is no single roc curve for a 3 class problem. So what do you want to plot?

On 07/20/2018 10:40 AM, serafim loukas wrote:
What I want to do is to plot the average(mean) ROC across Folds for a 3-class 
case.

The prototypical ROC curve uses True Positive Rate and False Positive Rate for 
its axes, so it is for 2-class problems, and not for 3+-class problems, as Andy 
mentioned.
Perhaps you are wanting the mean and confidence intervals of the n-class Cohen 
Kappa metric as estimated by either many folds of cross validation, or you want 
to evaluate your classifier by repeated subsampling experiments and Kappa value 
distribution/histogram?

Hope this helps,
J.B.
___
scikit-learn mailing list
scikit-learn@python.org
https://mail.python.org/mailman/listinfo/scikit-learn

___
scikit-learn mailing list
scikit-learn@python.org
https://mail.python.org/mailman/listinfo/scikit-learn

Re: [scikit-learn] Plot Cross-validated ROCs for multi-class classification problem

[Brown J.B. via scikit-learn]

Hello Makis,

2018-07-20 23:44 GMT+09:00 Andreas Mueller :

> There is no single roc curve for a 3 class problem. So what do you want to
> plot?
>
> On 07/20/2018 10:40 AM, serafim loukas wrote:
>
> What I want to do is to plot the average(mean) ROC across Folds for a
> 3-class case.
>
>
The prototypical ROC curve uses True Positive Rate and False Positive Rate
for its axes, so it is for 2-class problems, and not for 3+-class problems,
as Andy mentioned.
Perhaps you are wanting the mean and confidence intervals of the n-class
Cohen Kappa metric as estimated by either many folds of cross validation,
or you want to evaluate your classifier by repeated subsampling experiments
and Kappa value distribution/histogram?

Hope this helps,
J.B.
___
scikit-learn mailing list
scikit-learn@python.org
https://mail.python.org/mailman/listinfo/scikit-learn