hi Paolo,
I think too that a cross_val_confusion or cross_val_confusion_matrix function
in sklearn.cross_validation would be handy.
Maybe you can contribute yourself? or at least open an issue?
Best,
Alex
On Mon, Nov 11, 2013 at 4:55 PM, Paolo Di Prodi
<paolo.dipr...@contextis.co.uk> wrote:
> Hello there,
> correct me if I am wrong, but I couldn't find a method for calculating the
> average confusion matrix from a k-fold validation routine.
> The average confusion matrix is quite handy and is used in many scientific
> papers ... well at least the one I read!
> So I wrote a couple of functions that might be useful for other users (saw
> also some questions on Stack Overflow).
> There are things that I don't know how to implement correctly like knowing
> how many classes there are in total,
> and what happens if the matrix has a different dimension in the case that
> some of the classes are not present in the group set.
> Hopefully somebody can contribute!
>
> Example:
> clf = RandomForestClassifier(n_estimators=10)
> total_classes=list(set(Y))
> kfolder_confusion(clf,X,Y,total_classes=len(total_classes),n_folds=10)
>
>
> def average_matrix(cm):
> """ Given a confusion matrix calculate the average """
> result=numpy.zeros((cm.shape[0],cm.shape[1]))
> for i in range(0,cm.shape[0]):
> for j in range(0,cm.shape[1]):
> result[i][j]=cm[i][j]/ (cm[i , :].sum()+cm[: , j].sum() -
> cm[i][j] )
> return result
>
> def kfolder_confusion(clf,corpus,label_features,total_classes,n_folds=10):
> """ Do a K fold validation and compute the average confusion matrix """
> kf = KFold(len(label_features), n_folds=n_folds, indices=False)
> #initialize an empty confusion matrix
> partial_sum=numpy.zeros((total_classes,total_classes))
>
> for train, test in kf:
>
> train_data = corpus[train==True]
> test_data = corpus[test==False]
> train_label=label_features[train==True]
> test_label=label_features[test==False]
> clf.fit(train_data, train_label)
> label_pred= clf.predict(test_data)
> # Compute confusion matrix for each fold
> cm = confusion_matrix(test_label, label_pred)
> # Keep the temporary sym
> partial_sum=numpy.add(cm,partial_sum)
>
> average_cm=average_matrix(partial_sum)
>
> # Show the average confusion matrix
> pl.matshow(average_cm)
> pl.title('Average confusion matrix')
> pl.colorbar()
> pl.ylabel('True label')
> pl.xlabel('Predicted label')
> pl.show()
>
> ------------------------------------------------------------------------------
> November Webinars for C, C++, Fortran Developers
> Accelerate application performance with scalable programming models. Explore
> techniques for threading, error checking, porting, and tuning. Get the most
> from the latest Intel processors and coprocessors. See abstracts and register
> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
> _______________________________________________
> Scikit-learn-general mailing list
> Scikit-learn-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
------------------------------------------------------------------------------
November Webinars for C, C++, Fortran Developers
Accelerate application performance with scalable programming models. Explore
techniques for threading, error checking, porting, and tuning. Get the most
from the latest Intel processors and coprocessors. See abstracts and register
http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
