Dear James, I love your Star Trek Transporter analogy! "Beam me up Scotty..." Oops! Cheers, Bill
From: "James Holton" <jmhol...@lbl.gov> To: "CCP4BB" <CCP4BB@JISCMAIL.AC.UK> Sent: Saturday, 15 April, 2023 01:28:35 Subject: Re: [ccp4bb] Structure prediction - waiting to happen Very sorry to hear about your grant. I've been there. It is crushing to be rejected, and frustrating when the reason given is ... wrong. My journalist friends wonder why scientists don't like talking to journalists. This is why. I remember when the first results from XFELs were published, and it was immediately declared that there was no longer a need for NASA, whose sole purpose (apparently) was to grow bigger and better crystals in space. (?!) I find the idea that AlphaFold has eliminated the need to solve any more structures equally ludicrous. I think the best analogy for what has happened in structural biology is the same impact a Star Trek style "transporter device" would have on your daily commute. Except this "transporter" is only accurate enough to get you within a mile or two of your house. Most of the time. Don't worry, its not going to beam you inside a rock or into the sky, as it was trained on data with good Clashscores (we think). But, you are on your own getting the rest of the way home. This "Last Mile" of transportation networks is actually the most challenging, and expensive, but also the most critical. In structural biology, the "Last Angstrom" between prediction and actuality is equally important, but also fraught with difficulty. It may seem like a short distance, until you have to walk it. So, despite amazing progress, it is still premature to dismantle infrastructure, and definitely a bad idea to nail your front door shut. Personally, I see this structure prediction revolution as nothing more nor less than the fruition of Structural Genomics. It started in the final days of the 20th century. I was there! The stated goal of that worldwide initiative was to create the data set that would be needed by some future (at the time) homology modelling technology to do exactly what AlphaFold does: get us "close enough". And then Greg Petsko asked: what is "close enough"? He called it "The Grail problem". By what metric do you declare victory? He made an excellent suggestion: "But there is an obvious method of evaluation that will allow any structure prediction method to be assessed. It is simply to demand that the method produce a model that can be used to solve the corresponding protein crystal structure by the method of molecular replacement." -Greg Petsko - June 9, 2000 [ https://doi.org/10.1186/gb-2000-1-1-comment002 | https://doi.org/10.1186/gb-2000-1-1-comment002 ] This is the thing that just changed. Structure prediction has finally crossed the "G-P threshold". Not 100% of the time, but impressively often now, the predictions can be used for MR. This is a massively useful tool! Not the end of the field, but rather the beginning of an exciting new era where success rates skyrocket. Scores like the GDT used in CASP were developed with this Grail Problem criterion in mind, and I think that is what John Moult and others meant when they said things that got quoted like this: "Scores above 90 on the 100-point scale are considered on par with experimental methods, Moult says." [ https://www.science.org/doi/full/10.1126/science.370.6521.1144 | https://www.science.org/doi/full/10.1126/science.370.6521.1144 ] Meaning that the predicted models work as search models for MR about as often as search models derived from homologous (and yes, "experimentally determined") structures. A GDT of 100 does NOT mean the model is better than the data. That is not even how it works. But, unfortunately, this seems to have gotten paraphrased and sensationalized: "generally considered to be competitive with the same results obtained via experimental methods" [ https://www.sciencealert.com/ai-solves-50-year-old-biology-grand-challenge-decades-before-experts-predicted | https://www.sciencealert.com/ai-solves-50-year-old-biology-grand-challenge-decades-before-experts-predicted ] "software predictions finally match structures calculated from experimental data" [ https://www.science.org/doi/full/10.1126/science.370.6521.1144 | https://www.science.org/doi/full/10.1126/science.370.6521.1144 ] "comparable in quality to experimental structures" [ https://www.nature.com/articles/d41586-020-03348-4 | https://www.nature.com/articles/d41586-020-03348-4 ] "accuracy comparable to laboratory experiments" [ https://www.bbc.com/news/science-environment-55133972 | https://www.bbc.com/news/science-environment-55133972 ] <sigh> The only kind of diffraction where prediction is better than experiment is that of monoatomic gasses. These curves can be derived very accurately and completely from fundamental constants of physics. This is where those tables of atomic scattering factors used by refinement programs come from. For a while, the experimentally measured curves were used, but once Hartree, Fock, Slater, Cromer, Mann and others worked out how to do the self-consistent field calculations accurately, by the late 1960s the calculated form factors supplanted the measured ones. You might also say that for "small molecule" crystals the models are better than the data. Indeed, the CSD did not require experimental data to be deposited until fairly recently. The coordinates were considered more accurate than the intensities because publication requirements for chemical crystallography R factors are low enough to be dominated by experimental noise only. Nevertheless, despite the phase problem being cracked by direct methods in the 1980s, your local chemistry department has yet to shut down their diffractometer. Why? Because they need it. And for macromolecular structures, the systematic errors between refined coordinates and their corresponding data are about 4-5x larger than experimental error. So, don't delete your image data! Not for a while yet. -James Holton MAD Scientist On 4/1/2023 7:57 AM, Subramanian, Ramaswamy wrote: Ian, Thank you. This is not an April fools.. Rams [ mailto:subra...@purdue.edu | subra...@purdue.edu ] BQ_BEGIN On Apr 1, 2023, at 10:46 AM, Ian Tickle [ mailto:ianj...@gmail.com | <ianj...@gmail.com> ] wrote: ---- External Email : Use caution with attachments, links, or sharing data ---- Hi Ramaswamy I assume this is an April Fool's but it's still a serious question because many reviewers who are not crystallographers or electron microscopists may not fully appreciate the difference currently between the precision of structures obtained by experimental and predictive methods, though the latter are certainly catching up. The answer of course lies in the mean co-ordinate precision, related to the map resolution. Quoting [ https://people.cryst.bbk.ac.uk/~ubcg05m/precgrant.html | https://people.cryst.bbk.ac.uk/~ubcg05m/precgrant.html ] : " The accuracy and precision required of an experimentally determined model of a macromolecule depends on the biological questions being asked of the structure. Questions involving the overall fold of a protein, or its topological similarity to other proteins, can be answered by structures of fairly low precision such as those obtained from very low resolution X-ray crystal diffraction data [or AlphaFold]. Questions involving reaction mechanisms require much greater accuracy and precision as obtained from well-refined, high-resolution X-ray structures, including proper statistical analyses of the standard uncertainties ( s.u.'s ) of atomic positions and bond lengths.". According to [ https://www.nature.com/articles/s41586-021-03819-2 | https://www.nature.com/articles/s41586-021-03819-2 ] : The accuracy of AlphaFold structures at the time of writing (2021) was around 1.0 Ang. RMSD for main-chain and 1.5 Ang. RMSD for side-chain atoms and probably hasn't changed much since. This is described as "highly accurate"; however this only means that AlphaFold's accuracy is much higher in comparison with other prediction methods, not in comparison with experimental methods. Also note that AlphaFold's accuracy is estimated by comparison with the X-ray structure which remains the "gold standard"; there's no way (AFAIK) of independently assessing AlphaFold's accuracy or precision. Quoting [ https://scripts.iucr.org/cgi-bin/paper?S0907444998012645 | https://scripts.iucr.org/cgi-bin/paper?S0907444998012645 ] : " Data of 0.94 A resolution for the 237-residue protein concanavalin A are used in unrestrained and restrained full-matrix inversions to provide standard uncertainties sigma(r) for positions and sigma(l) for bond lengths. sigma(r) is as small as 0.01 A for atoms with low Debye B values but increases strongly with B." There's a yawning gap between 1.0 - 1.5 Ang. and 0.01 Ang.! Perhaps AlphaFold structures should be deposited using James Holton's new PDB format (now that is an April Fool's !). One final suggestion for a reference in your grant application: [ https://www.biorxiv.org/content/10.1101/2022.03.08.483439v2 | https://www.biorxiv.org/content/10.1101/2022.03.08.483439v2 ] . Cheers -- Ian On Sat, 1 Apr 2023 at 13:06, Subramanian, Ramaswamy < [ mailto:subra...@purdue.edu | subra...@purdue.edu ] > wrote: BQ_BEGIN Dear All, I am unsure if all other groups will get it - but I am sure this group will understand the frustration. My NIH grant did not get funded. A few genuine comments - they make excellent sense. We will fix that. One major comment is, “Structures can be predicted by alpfafold and other software accurately, so the effort put on the grant to get structures by X-ray crystallography/cryo-EM is not justified.” The problem is when a company with billions of $$s develops a method and blasts it everywhere - the message is so pervasive… Question: I s there a canned consensus paragraph that one can add with references to grants with structural biology (especially if the review group is not a structural biology group) to say why the most modern structure prediction programs are not a substitute for structural work? Thanks. Rams [ mailto:subra...@purdue.edu | subra...@purdue.edu ] To unsubscribe from the CCP4BB list, click the following link: [ https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 | https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 ] BQ_END To unsubscribe from the CCP4BB list, click the following link: [ https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 | https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 ] BQ_END To unsubscribe from the CCP4BB list, click the following link: [ https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 | https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 ] ######################################################################## To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/
