Thank you very much Eric for the insights. I think the MQ search uses decoys too and search parameters are almost similar, at least they are not more relaxed for MQ. I am not sure about the behavior of PEP and the tiny peaks after the highest confidence and that is something that I need to investigate more carefully.
I have another unrelated question which I am going to ask in another topic. Sincerely, Ali On Wednesday, January 4, 2017 at 6:28:24 PM UTC-5, Eric Deutsch wrote: > > Hi Ali, my thoughts are that for this dataset and analysis, > PeptideProphet+iProphet (PP+iP) are discriminating between correct and > incorrect much better than the other analysis. The vast majority of > peptides are deemed by PP+iP as very likely to be correct or very likely > incorrect with a rather small population of IDs at intermediate confidence. > The other analysis seems to be unsure about many more. In fact the curve is > extremely steep indicating not very good discriminating power between > correct and incorrect. There are many more in the intermediate range. > > > > The PP+iP analysis looks quite similar to what we are used to for modern > datasets. The steepness of the other analysis seems a bit worrying to me. > How are the PEPs calculated? Can you perform an independent analysis of the > PEPs to see if they agree with decoys or some other measure of errors? I > would say that in the range that matters (FDR < 1%) PP+iP is giving you a > better result. It is puzzling that the total number of correct hits at FDR > 2.5% is so much lower fir PP+iP. I couldn’t say from the data shown why > that is. Maybe the PEPs are not correct or MQ is finding more IDs because > of different search parameters? I don’t know. > > > > Regards, > > Eric > > > > > > *From:* Ali [mailto:[email protected] <javascript:>] > *Sent:* Wednesday, January 04, 2017 1:06 PM > *To:* spctools-discuss > *Cc:* [email protected] <javascript:>; > [email protected] <javascript:> > *Subject:* Re: [spctools-discuss] PeptideProphet/iProphet Probability > Distribution > > > > > > > > Dear Eric and David > > > > By accepting lower thresholds I actually mean accepting higher FDRs but > that is not so useful here because there is not so many hits after the very > high probabilities and I will not be gaining much by using lower > thresholds. PeptideProphet gives me a lot of PSMs with very high confidence > and very low FDR. But as I allow higher FDRs like 2-3% it falls behind PEP > from MaxQuant. > > > > Here is another plot to better show my point. I have compared the combined > results above with Max Quant's results as I increase the FDR. As it can be > seen, number of unique peptides is more with PepPro/iPro for low FDR, but > in the end it falls behind PEP. > > > > > > > <https://lh3.googleusercontent.com/-VZgUOiAvvno/WG1ggpOR3uI/AAAAAAAABBA/nyrjsBbLrGQ60eo3iqB3beMOVs4ZCXCAACLcB/s1600/Picture1.png> > > > > And this can be understood from the following plot: > > > > > <https://lh3.googleusercontent.com/-jmsbfmBfbXA/WG1hNLP8e1I/AAAAAAAABBI/SBnRc5-naTw09vjKk0K4m9xcCBgS-1Z8QCLcB/s1600/Picture2.png> > > > > Do you have any thoughts on this? > > > > Thank you very much, > > Ali > > > > > > On Wednesday, January 4, 2017 at 3:08:14 PM UTC-5, Ali wrote: > > Dear David and Eric > > > > Thanks for your replies and suggestions. I thought that I might be able to > increase identification by accepting hits with lower thresholds. I do > understand what you mentioned here. > > > > Thanks again, > > Ali > > On Tuesday, January 3, 2017 at 1:57:38 PM UTC-5, Eric Deutsch wrote: > > In the ideal case where the PeptideProphet and iProphet had perfect > discriminating power, i.e. the correct and incorrect PSM distributions do > not overlap, you would expect a large peak at P=0 and P=1 and nothing in > between. As a dataset diverges from this ideal and the correct and > incorrect distributions begin to overlap, you will still have very large > peaks near P=0 and near P=1, with a small number of intermediate values > where the probability is between 0 and 1. > > > > So, in short, your distribution is exactly what you would expect based on > what you did. It is indicative of a great dataset. You should follow > David’s suggestion, and then if you replot, you will see huge peaks at 0 > and 1 with rather little in between. > > > > Regards, > > Eric > > > > > > *From:* [email protected] [mailto:[email protected]] > *On Behalf Of *David Shteynberg > *Sent:* Tuesday, January 03, 2017 10:36 AM > *To:* spctools-discuss > *Subject:* Re: [spctools-discuss] PeptideProphet/iProphet Probability > Distribution > > > > These results are showing you that iProphet has great discriminating power > even when you restrict the input to only non-low probability PSMs > (filtering with PeptideProphet probability). The best way to run iProphet > is to give it all the data from PeptideProphet (use a PeptideProphet > minimum probability of 0). > > > > -David > > > > On Mon, Jan 2, 2017 at 9:26 PM, Ali <[email protected]> wrote: > > Dear all > > > > I am plotting the distribution of the assigned PeptideProphet and iProphet > probabilities to my search results. I am seeing a rather strange pattern > shown below. Why am I seeing so many hits with very high confidence > (probability >0.98) but suddenly after that threshold the number of > assigned probabilities to the PSMs decrease abruptly? What could be the > reason for this pattern? One would expect to see all ranges of > probabilities for PSMs. > > > > > <https://lh3.googleusercontent.com/-BeikGpozjkw/WGsygFV83UI/AAAAAAAABAg/DcG6lRxucM4IRclvdadal-eFSaA05sd6QCLcB/s1600/Picture1.png> > > > > > > This figure is the combination of different search engines: > > - Comet (min PepPro=0.5) > > - X! Tandem GPM (min PepPro=0.05) > > - X! Tandem TPP (min PepPro=0.05) > > > > The horizontal axis is between 0.5 and 1. > > > > and I can see the same pattern for the PeptideProphet distribution of each > one of these search engine results including SpectraST (Except for X! > Tandem GPM). > > > > Thank you very much, > > > > Ali > > > > > > -- > You received this message because you are subscribed to the Google Groups > "spctools-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/spctools-discuss. > For more options, visit https://groups.google.com/d/optout. > > > > -- > You received this message because you are subscribed to the Google Groups > "spctools-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/spctools-discuss. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "spctools-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/spctools-discuss. For more options, visit https://groups.google.com/d/optout.
