Hi David, Hi Natalie, I just posted the 4 pepxml files which give me the most striking differences in results between TPP-V4.0 and TPP-V4.3: lgillet_pepxml_for_TPP4.3.rar. I also posted the results (interact.pep.xml) which I obtain from running TPP-V4.0, TPP-V4.3 and TPP-V4.3 on scrambled file order (file #4>#3>#2>#1): lgillet_interact- results.rar. I really tried my best to figure out what the problem could be. Maybe you could re-run the same analyses (TPP-V4.0, TPP-V4.3, TPP-V4.3 with the scrambled file order) and let me know if you confirm my results or if there is something wrong maybe with the compiled version we have on our server (could still be a possibility). Finally, to answer Natalie's question, the differences are quite dramatic (to my opinion) between V4.0 and V4.3 (I would not have worried about 1-2% differences in IDs), but here, I am passing from 1% decoy (V4.0) to 23% decoy (V4.3) hits (at the same proba > 0.9). Also the number of unique peptides reported by V4.0 and V4.3 is quite different (2150 and 3161 resp.). Finally, many decoy hits pulled up in V4.3 with a prob>0.9 have actually a very bad MS/MS spectrum and a very low prob<0.01 (only reported if you use -p0 option) on V4.0.
Have a look at those MS/MS spectra for example: 20100422_04_control_07.c.07700.07700.4 20100422_04_control_07.c.02864.02864.3 Let me know if you need any extra information. Thanks a lot for your help on that. Best, Ludovic On May 18, 11:21 pm, Natalie Tasman <[email protected]> wrote: > Ludovic, > > Go ahead and post the files to the newsgroup's file area > (http://groups.google.com/group/spctools-discuss/files), and hopefully > one of the validation experts will take a look. > > I will point out that PeptideProphet uses random initialization for > it's curve fitting (EM algorithm). So it's not out of the question > that you'd see some small differences between runs on the same data > files, regardless of the order. Can you provide some measure of the > differences between runs for the reordered datasets? > > -Natalie > > > > On Tue, May 18, 2010 at 4:35 AM, lgillet <[email protected]> wrote: > > Hi everybody, > > I recently encountered a "bug" I think when people in my lab installed > > the newest TPP (v4.3 JETSTREAM rev 1, Build 201004201202 (linux)), > > especially when I try to confront the result to v4.0 which was our > > former "benchmark" version. > > When searching the same 4 pep.xml files with v4.0 and v4.3, I get an > > incredible difference in decoy hits number. For example, with v4.0, > > p>0.9, I would get my "regular" 1% decoy, while with v4.3, p>0.9, I > > get above 25% of decoys?!?? > > All the interact are run with the following options: xinteract -OApld - > > ddecoy *.pep.xml > > I could nail down the "problem" to the PeptideProphetParser which > > behaves very differently between v4.0 and v4.3, while InteractParser > > (which introduces the "is_rejected=1" tags) and RefreshParser do not > > influence the results. > > But at the moment, I do not know if it is an issue of the decoy > > statistical distribution of prophet or not... > > > One more thing that makes me even more suspicious is the fact that, > > only with TPP version 4.3, if you search those files in a difference > > order (let say: xinteract file1 file2 file3 Vs xinteract file3 file2 > > file1), you do get differences in the results as well?!? > > > I am willing to send the 4 pepxml where those observations are the > > most critical to David or Luis or anybody interested, but I truly > > believe that there might be something going wrong with the TPP v4.3. > > > Let me know to whom I should post the files. > > > Best regards, > > > Ludovic > > > -- > > You received this message because you are subscribed to the Google Groups > > "spctools-discuss" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group > > athttp://groups.google.com/group/spctools-discuss?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "spctools-discuss" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group > athttp://groups.google.com/group/spctools-discuss?hl=en. -- You received this message because you are subscribed to the Google Groups "spctools-discuss" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/spctools-discuss?hl=en.
