"Koen Vermeer" <[EMAIL PROTECTED]> wrote in message news:<[EMAIL PROTECTED]>... > Hi, > > I am having conceptual problems with cross-validating an ROC. The thing is > that for me the only reason to draw an ROC is to show the individual > fpr/tpr pairs, so one can choose the optimal setting for a specific > application (depending on prevalence, cost of FP/FN, etc). So, in fact, > the ROC just shows various algorithms, and you choose one that suits you > best.
A single ROC curve shows the result of a single algorithm when a single parameter (e.g., a decision threshold is varied). However, family of ROC curves can be obtained for different values of a second parameter. The different values of the second parameter (e.g., No. of predictors) can be interpreted to indicate "different" algorithms. Although families of ROC curves can also be obtained for different combinations of multiple parameters, understanding the effect of each parameter becomes too difficult. Each algorithm is created with a design set. The corresponding ROC is created from the algorithm using an independent validation set. To assist understanding, use the same sets for all algorithms. The choice of an algorithm and corresponding operational threshold is chosen from the family of ROC curves. Finally, an estimate of the performance is obtained using an independent test set. > The thing with validation is that it is supposed to be done on the > final algorithm, not on some intermediate result. Validation is done on all the algorithms in order to choose the "best" one. > More detailed: > Consider algorithm A. It tests a number of algorithms (1..N) and chooses > the best one (say number i). I don't understand your terminology. What do you mean by using algorithm A to test other algorithms? What do you mean by cross validation? Hope this helps. Greg >Even if algorithm A uses cross-validation to > train and test all N algorithms, we cannot say that the error rate of > algorithm A is the same as the estimated error rate of algorithm i. So, we > cross-validate algorithm A: We use a data set to train it (and thus to > select i) and an independent set to test its performance. > > Now, if we compare this to the ROC, the ROC is like the outcome of all N > algorithms. Based on the application, one would choose the best algorithm. > Cross-validation is therefore not possible before this selection has been > made. > > On the other hand, one could ofcourse 'cross-validate' the ROC. For > example, the ROCs of the several folds could be averaged in some way, or > the individual tpr/fpr pairs could be cross-validated. > > I would appreciate any comments on this! > > Regards, > Koen Vermeer . . ================================================================= Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at: . http://jse.stat.ncsu.edu/ . =================================================================
