The best way to compare is to include an independent set of wrong hits (decoys) that are unknown to the algorithm. Then you can calculate the error rate based on the number of wrong hits that pass any threshold of each algorithm. The mixture models of each dataset are learned independently from that dataset, by PeptideProphet and by iProphet. Therefore, the minimum threshold to achieve a fixed error rate (and FDR) changes depending on the dataset. This method will also allow you to test the accuracy of the FDRs reported by the validation tools.
-David On Tue, Dec 20, 2016 at 4:27 PM, Ali <[email protected]> wrote: > Dear all > > I would like to compare the identification rate of a set of > PeptideProphet/iProphet validated results from Comet/X!Tandem, with > MaxQuant's results. For MaxQuant I have PEPs. What is the fair way to > compare the results with these two types of validations? At what threshold > should I cut the peptideprohet/iprophet to be comparable with a PEP of say > 0.1? > > Is it fair if I compare a PEP=x with PepPro/iPro=1-x? One thing that I > have noticed is that FDR calculated by mean(all PEPs) OR 1-mean(all > PepPro), is always much less for PepPro. > > And one more question, what is an acceptable PepPro/iPro probability? Is > there a way that I allow a higher FDR without changing the probability > threshold? > > Thank you very much in advance. > > Ali > > -- > You received this message because you are subscribed to the Google Groups > "spctools-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/spctools-discuss. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "spctools-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/spctools-discuss. For more options, visit https://groups.google.com/d/optout.
