On Fri, 18 Jul 2003 15:21:55 +0200, Koen Vermeer wrote: [...] > I would appreciate any comments on this!
I did some more research and came up with this: It is not uncommon to think of an ROC as a function showing the performance of a classifier in general (that is, without restrictions to the number of parameters that are varied or the underlying structure of the classifier). 'On an ROC graph, TP is plotted on the Y axis and FP is plotted on the X axis.' and 'An ROC curve illustrates the error tradeoffs available with a given classifier.' [1]. Also, combining the ROCs of several classifiers to give one ROC of the best possible combination has been done before [1] (ROC convex hull). Cross-validation of an ROC is not new either. In [2], 10-fold cross-validation is used to create the ROCs. The ROC curve of each fold i is considered to be a function f_i, such that TP=f_i(FP). The average is the function f(FP), where f(FP)=mean(f_i(FP)). I'm not sure this last technique is the best, but it is a solution to the problem I was dealing with. [1]Provost F, Fawcett T. Analysis and Visualization of Classifier Performance: Comparison under Imprecise Class and Cost Distributions. KDD-97. [2] Provost F, Fawcett T, Kohavi R. The Case Against Accuracy Estimation for Comparing Induction Algorithms. IMLC-98. Regards, Koen . . ================================================================= Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at: . http://jse.stat.ncsu.edu/ . =================================================================
