Thanks, I will try this.
Also, on CASP website there are such scores as RMS_ALL (can be seen in
tables) and GDC_SC (for side-chains, not visible in tables for some reason).
RMS_ALL presumably includes side-chains and seems good for AlphaFold2
models, between 1 to 2 Angstrom (apart from the same outliers as
RMS_CA), although that is not quite at the experimental level.
Were any scores including side-chains included in ranking/evaluation (as
we hear mostly about GDT_TS)?
If not, how can "experimental level" precision be claimed?
Thanks,
Leonid
On 11.12.20 13:56, Tristan Croll wrote:
I agree the website can be quite cryptic!
You can get all the targets as a tarball from
https://predictioncenter.org/download_area/CASP14/targets/
<https://predictioncenter.org/download_area/CASP14/targets/>. For the
predictions, you can either get them as PDB files on a case-by-case
basis from the results section, or tarballs of all predictions for a
given target from
https://predictioncenter.org/download_area/CASP14/predictions_trimmed_to_domains/
<https://predictioncenter.org/download_area/CASP14/predictions_trimmed_to_domains/>.
In the latter case, each file is essentially a PDB file without the
.pdb extension, except with 4 lines added to the front looking
something like:
PFRMAT TS
TARGET T1049
MODEL 2
PARENT N/A
Depending on your choice of viewer, you may need to remove these lines
before attempting to open it.
The GDT_TS score only considers alpha carbons, so in principle it /is/
possible to get a high score on it while still having a model that's
rubbish in every other respect. It's certainly worth complementing it
with other scores - e.g. good old MolProbity, or SphereGrinder. The
latter is quite good in principle - essentially, it places a 6 A
radius sphere at each CA atom of the target, finds all heavy atoms in
the sphere, and measures their RMSD to the corresponding atoms in the
prediction. The actual implementation for CASP is a bit broad-brush,
though - the score is just the fraction of spheres whose RMSD is under
2 A.
In the last CASP round I pushed for the need to start adding metrics
that directly compared the models in torsion space - far from the
first time that's been suggested, but it's arguably only in the past
few rounds that models have gotten good enough for this to be a useful
discriminating measure. It doesn't appear that this has been added to
the standard measures for CASP14, but if it had I can see that
AlphaFold2 would have done extremely well - I only showed the ribbon
representation for T1049 in my last email, but the sidechains in the
core show pretty amazing agreement with the target.
Best regards,
Tristan
------------------------------------------------------------------------
*From:* Leonid Sazanov <saza...@ist.ac.at>
*Sent:* 11 December 2020 12:32
*To:* Tristan Croll <ti...@cam.ac.uk>; CCP4BB@JISCMAIL.AC.UK
<CCP4BB@JISCMAIL.AC.UK>
*Subject:* Re: [ccp4bb] External: Re: [ccp4bb] AlphaFold: more
thinking and less pipetting (?)
I see, thanks, that looks good.
Where can one download predicted_model+exp_model PDBs together?
I could easily find predicted models but not experimental - CASP
website seems very cryptic.
Also, can you comment on how much GDT_TS depends on CA and how much on
side chains positioning?
E.g. if it is >90, can one be sure that most side-chains are in the
right place?
Thanks.
Leonid
On 11.12.20 13:12, Tristan Croll wrote:
I'm not Randy, but I do have an answer: like this. This is T1049-D1.
AlphaFold prediction in red, experimental structure (6y4f) in green.
Agreement is close to perfect, apart from the C-terminal tail which
is way off - but clearly flexible and only resolved in this
conformation in the crystal due to packing interactions. GDT_TS is
93.1; RMS_CA is 3.68 - but if you exclude those tail residues, it's
0.79. With an alignment cutoff of 1 A, you can align 109 of 134 CAs
with an RMSD of 0.46 A.
------------------------------------------------------------------------
*From:* CCP4 bulletin board <CCP4BB@JISCMAIL.AC.UK>
<mailto:CCP4BB@JISCMAIL.AC.UK> on behalf of Leonid Sazanov
<saza...@ist.ac.at> <mailto:saza...@ist.ac.at>
*Sent:* 11 December 2020 10:36
*To:* CCP4BB@JISCMAIL.AC.UK <mailto:CCP4BB@JISCMAIL.AC.UK>
<CCP4BB@JISCMAIL.AC.UK> <mailto:CCP4BB@JISCMAIL.AC.UK>
*Subject:* Re: [ccp4bb] External: Re: [ccp4bb] AlphaFold: more
thinking and less pipetting (?)
Dear Randy,
Can you comment on why for some of AplhaFold2 models with GDT_TS > 90
(supposedly as good as experimental model) the RMS_CA (backbone) is >
3.0 Angstrom? Such a deviation can hardly be described as good as
experimental. Could it be that GDT_TS is kind of designed to evaluate
how well the general sub-domain level fold is predicted, rather than
overall detail?
Thanks,
Leonid
>>>>>
Several people have mentioned lack of peer review as a reason to
doubt the significance of the AlphaFold2 results. There are
different routes to peer review and, while the results have not been
published in a peer review journal, I would have to say (as someone
who has been an assessor for two CASPs, as well as having editorial
responsibilities for a peer-reviewed journal), the peer review at
CASP is much more rigorous than the peer review that most papers
undergo. The targets are selected from structures that have recently
been solved but not published or disseminated, and even just tweeting
a C-alpha trace is probably enough to get a target cancelled. In
some cases (as we’ve heard here) the people determining the structure
are overly optimistic about when their structure solution will be
finished, so even they may not know the structure at the time it is
predicted. The assessors are blinded to the identities of the
predictors, and they carry out months of calculations and inspections
of the models, computing ranking scores before they find out who made
the predictions. Most assessors try to bring something new to the
assessment, because the criteria should get more stringent as the
predictions get better, and they have new ideas of what to look for,
but there’s always some overlap with “traditional” measures such as
GDT-TS, GDT-HA (more stringent high-accuracy version of GDT) and lDDT.
Of course we’d all like to know the details of how AlphaFold2 works,
and the DeepMind people could have been (and should be) much more
forthcoming, but their results are real. They didn’t have any way of
cheating, being selective about what they reported, or gaming the
system in any other way that the other groups couldn’t do. (And yes,
when we learned that DeepMind was behind the exceptionally good
results two years ago at CASP13, we made the same half-jokes about
whether Gmail had been in the database they were mining!)
Best wishes,
Randy Read
########################################################################
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
<https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1>
This message was issued to members of www.jiscmail.ac.uk/CCP4BB
<http://www.jiscmail.ac.uk/CCP4BB>, a mailing list hosted by
www.jiscmail.ac.uk <http://www.jiscmail.ac.uk>, terms & conditions
are available at https://www.jiscmail.ac.uk/policyandsecurity/
<https://www.jiscmail.ac.uk/policyandsecurity/>
--
Prof. Leonid Sazanov FRS
IST Austria
Am Campus 1
A-3400 Klosterneuburg
Austria
Phone: +43 2243 9000 3026
E-mail:saza...@ist.ac.at <mailto:saza...@ist.ac.at>
--
Prof. Leonid Sazanov FRS
IST Austria
Am Campus 1
A-3400 Klosterneuburg
Austria
Phone: +43 2243 9000 3026
E-mail: saza...@ist.ac.at
########################################################################
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list
hosted by www.jiscmail.ac.uk, terms & conditions are available at
https://www.jiscmail.ac.uk/policyandsecurity/
