Re: [ccp4bb] Completeness question

[Robbie Joosten]

Hi Clemens,

Thanks for pointing out the table in the PDBpeep log file. It has the data I 
need and since the issue only applies to deposited PDB entries it does the 
trick quite nicely.

Cheers,
Robbie

> -Original Message-
> From: Clemens Vonrhein 
> Sent: Wednesday, June 3, 2020 14:18
> To: Robbie Joosten 
> Cc: CCP4BB@JISCMAIL.AC.UK
> Subject: Re: [ccp4bb] Completeness question
> 
> Dear Robbie,
> 
> On Sat, May 30, 2020 at 08:36:06AM +, Robbie Joosten wrote:
> > I've been looking at some recent PDB entries that have much lower
> > spherical) completeness than reported in the coordinate file. One
> > reason for this is that the data were anisotropicly truncated, another
> > reason is some mess-up with the deposition of the reflection data.
> > There is a lot of discussion about the former practice and I don't
> > want to go in to that, but the second one is obviously an error. Now
> > how do I distinguish these cases?
> >
> > Sometimes, you can look at the reported number of reflections and
> > compare that to the deposited reflection file and you will find that
> > something has clearly gone wrong. However, the reported number of
> > reflections is not entirely reliable because of other issues so I'd
> > rather not use it. If you use PDBpeep (e.g. for 6rjy) you can see
> > something is wrong, but that is completely visual. Is there a tool in
> > CCP4 that reports both spherical and ellipsoidal completeness (on
> > merged reflection data)? That would make it easy to distinguish such
> > cases.
> 
> One needs a description of the anisotropy in a form that allows for a simple
> computation ("Is a reflection above a significance limit or
> not?") to relate this to the completeness of reflections above such a
> significance level. That model could e.g. be the ellipsoid fitted to the
> significance (local ) cutoff surface as done by STARANISO [1]. You can
> see that information e.g. at the PDBpeep server [2]:
> 
>   http://staraniso.globalphasing.org/cgi-bin/PDBpeep.cgi?ID=6rjy
> 
> as
> 
>   Diffraction limits & principal axes of ellipsoid fitted to diffraction 
> cut-off
> surface:
> 
>   1.747 1.   0.   0.   a*
>   1.777 0.   1.   0.   b*
>   1.465 0.   0.   1.   c*
> 
> Of course, if this information is already present in deposited PDB entries of
> interest, no additional computation is required. We are currently working
> with our colleagues from the PDBx/mmCIF working group on definitions and
> items that would extend the current PDBx/mmCIF dictionary so that the
> above information could be deposited to and retrieved from the PDB
> database.
> 
> In any case, this information is then used for computation of the spherical
> and ellipsoidal completness e.g. within STARANISO - see link to the
> "complete log file" at the bottom of the page:
> http://staraniso.globalphasing.org/PDB/6rjy.log. It contains a table
> ("Statistics for isotropic, anisotropic & ellipsoidal diffraction
> cut-offs") where Cmeas_sph and Cmeas_ell should give the above
> information (showing clearly that for 6rjy something else went wrong during
> deposition).
> 
> If there is unmerged data deposited, one could give that ellipsoid description
> e.g. to MRFANA [3] via
> 
>   mrfana -ell 1.747 1.0 0.0 0.0 1.777 0.0 1.0 0.0 1.465 0.0 0.0 1.0
> unmerged.mtz
> 
> to get the same information (but maybe use different binning, limit
> resolution ranges etc).
> 
> Cheers
> 
> Clemens, Claus, Ian & Gerard
> 
> [1] http://staraniso.globalphasing.org/
> [2] http://staraniso.globalphasing.org/cgi-bin/PDBpeep.cgi
> [3] https://github.com/githubgphl/MRFANA, a result of the Data Quality
> Metrics Workshop organised by Global Phasing at the ESRF in April
> 2019



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] Completeness question

[Clemens Vonrhein]

Dear Robbie,

On Sat, May 30, 2020 at 08:36:06AM +, Robbie Joosten wrote:
> I've been looking at some recent PDB entries that have much lower
> spherical) completeness than reported in the coordinate file. One
> reason for this is that the data were anisotropicly truncated,
> another reason is some mess-up with the deposition of the reflection
> data. There is a lot of discussion about the former practice and I
> don't want to go in to that, but the second one is obviously an
> error. Now how do I distinguish these cases?
> 
> Sometimes, you can look at the reported number of reflections and
> compare that to the deposited reflection file and you will find that
> something has clearly gone wrong. However, the reported number of
> reflections is not entirely reliable because of other issues so I'd
> rather not use it. If you use PDBpeep (e.g. for 6rjy) you can see
> something is wrong, but that is completely visual. Is there a tool
> in CCP4 that reports both spherical and ellipsoidal completeness (on
> merged reflection data)? That would make it easy to distinguish such
> cases.

One needs a description of the anisotropy in a form that allows for a
simple computation ("Is a reflection above a significance limit or
not?") to relate this to the completeness of reflections above such a
significance level. That model could e.g. be the ellipsoid fitted to
the significance (local ) cutoff surface as done by STARANISO
[1]. You can see that information e.g. at the PDBpeep server [2]:

  http://staraniso.globalphasing.org/cgi-bin/PDBpeep.cgi?ID=6rjy

as

  Diffraction limits & principal axes of ellipsoid fitted to diffraction 
cut-off surface:

  1.747 1.   0.   0.   a*
  1.777 0.   1.   0.   b*
  1.465 0.   0.   1.   c*

Of course, if this information is already present in deposited PDB
entries of interest, no additional computation is required. We are
currently working with our colleagues from the PDBx/mmCIF working
group on definitions and items that would extend the current
PDBx/mmCIF dictionary so that the above information could be deposited
to and retrieved from the PDB database.

In any case, this information is then used for computation of the
spherical and ellipsoidal completness e.g. within STARANISO - see link to
the "complete log file" at the bottom of the page:
http://staraniso.globalphasing.org/PDB/6rjy.log. It contains a table
("Statistics for isotropic, anisotropic & ellipsoidal diffraction
cut-offs") where Cmeas_sph and Cmeas_ell should give the above
information (showing clearly that for 6rjy something else went wrong
during deposition).

If there is unmerged data deposited, one could give that ellipsoid
description e.g. to MRFANA [3] via

  mrfana -ell 1.747 1.0 0.0 0.0 1.777 0.0 1.0 0.0 1.465 0.0 0.0 1.0 unmerged.mtz

to get the same information (but maybe use different binning, limit
resolution ranges etc).

Cheers

Clemens, Claus, Ian & Gerard

[1] http://staraniso.globalphasing.org/
[2] http://staraniso.globalphasing.org/cgi-bin/PDBpeep.cgi
[3] https://github.com/githubgphl/MRFANA, a result of the Data Quality
Metrics Workshop organised by Global Phasing at the ESRF in April
2019



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] Completeness question

[Clemens Vonrhein]

Dear all,

On Mon, Jun 01, 2020 at 11:48:41PM +0200, vincent Chaptal wrote:
> since I've been contacted off line several times about this, I'm
> posting here the protocol to combine maps with 2 mtz files of
> original and truncated data.

If this is of wider interest (and since some of our tools for doing
this were mentioned), maybe we can also give a bit of additional
background information and provide some outlook.

> I went through this procedure as I wanted to include the maps, and
> this was the tricky part of the procedure in my hands. This way,
> everyone can look at the same maps when critically assessing a
> structure (especially in the context of redo-ing and severe
> anisotropy).

I think it is a very good idea to also include into the deposition the
maps that were used/seen during model building/refinement. After all,
these were the basis for the interpretation that produced the final
model and the conclusions drawn fom it in a publication. Of course,
one can always re-create similar maps through some external
(re-)refinement approach (be that PDB-REDO or manually using your own
favourite program). Those maps might be similar, better, worse or just
different, perhaps providing added value and additional information -
but it can be difficult to get the exact same maps as seen by the
depositor, due e.g. to different program versions, or to not using the
exact commands and/or parameter settings etc.

Then there is the question of the handling of missing data in 2mFo-DFc
maps (ignoring it, filling it with DFc up to the highest spherical
diffraction limit, or filling it only within the ellipsoidal model of
anisotropy given by STARANISO). BUSTER will create a single mmCIF file
containing all those different map coefficients in separate data
loops. Other programs are most likely to be doing the same. The
important point is that current refinement programs should generate a
single mmCIF file containing all reflection data (and meta-data)
available at the point of refinement - something we are all working
hard at within the PDBx/mmCIF working group. So the task (as described
below) is then to combine this set of reflection data (all nicely
self-contained from model refinement) with the full set of reflection
data (and meta data) from data processing. There are various caveats
and traps one has to consider here (different indexing possibilities,
different cell settings and conventions, enantiomorphs, initially
unassigned screw axes, C2 vs. I2, H3 vs. R3, etc).

Ideally we want an application that takes the two (self-contained)
mmCIF files from data-processing and model refinement, and combines
them in a consistent way, robustly taking care of all of the above
potential issues.  Because there often is a large time-gap between
original data-processing and final structure deposition, a lot of
additional checks need to be in place to avoid picking up the wrong
files by accident. This is precisely what we are currently working on
and hope to provide in one of our next releases: a tool working
ideally directly on mmCIF files produced by any data processing
package and any refinement program, that will combine these in a
consistent and validated manner. We hope that will become useful - not
only to users of autoPROC/STARANISO and BUSTER, but also to people who
use multiple different packages in their workflow.

Cheers

Clemens

> In a new directory, put all 3 mtz files (original Intentisties, merged
> staraniso reflections, maps from buster)
> 
> /PATH_TO-DIR/BUSTER/Refine45/autoPROC/.
> 
> aP_deposition_prep -p combine -r maps.mtz -f staraniso_output.mtz -a
> Original_Intensities.mtz >aP_deposition_prep.log
> 
> It creates combine_all.cif
> 
> This file has everything but it has a “refln” naming issue.
> 
> To fix the “refln” issue:
> 
> cat combine_all.cif | sed -e "s/refln.F_sigma /refln.F_meas_sigma /g" -e
> "s%[ ]* #.*from dataset.*%%g" | awk 
> '/^#/{if(s){print;next};c++;a[c]=$0;next}/^data/{print;if(s)next;s=1;for(i=1;i<=c;i++){print
> a[i]}next}{print}' > combine_all_refln_fixed.cif
> 
>  DONE.
> 
> 
> Best
> Vincent
> ps: again, the credit of this script goes to the globalphasing team who
> kindly helped when I asked them for help with it.
> 
> 
> 
> Le 01/06/2020 à 11:47, Clemens Vonrhein a écrit :
> > Dear all,
> > 
> > On Sat, May 30, 2020 at 03:40:53PM +0100, Eleanor Dodson wrote:
> > > My pennysworth. If you find your maps look better after the
> > > anisotroy correction use it, but it may be helpful to those wo want to 
> > > mine
> > > your data if you deposit the whole sphere..
> > Agree (which is what e.g. we provide when using STARANISO via autoPROC
> > [1]).
> > 
> > And in the same vein: those depositing isotropically truncated data
> > should consider also providing data to a higher diffraction limit to
> > give a potentially more accurate picture (if there is even a slight
> > indication of anisotropy - which there often is).
> > 
> > I find it very helpful even looking at an idealised 

Re: [ccp4bb] Completeness question

[vincent Chaptal]

Good evening,

since I've been contacted off line several times about this, I'm posting 
here the protocol to combine maps with 2 mtz files of original and 
truncated data.
I went through this procedure as I wanted to include the maps, and this 
was the tricky part of the procedure in my hands. This way, everyone can 
look at the same maps when critically assessing a structure (especially 
in the context of redo-ing and severe anisotropy).


In a new directory, put all 3 mtz files (original Intentisties, merged 
staraniso reflections, maps from buster)


/PATH_TO-DIR/BUSTER/Refine45/autoPROC/.

aP_deposition_prep -p combine -r maps.mtz -f staraniso_output.mtz -a 
Original_Intensities.mtz >aP_deposition_prep.log


It creates combine_all.cif

This file has everything but it has a “refln” naming issue.

To fix the “refln” issue:

cat combine_all.cif | sed -e "s/refln.F_sigma /refln.F_meas_sigma /g" -e 
"s%[ ]* #.*from dataset.*%%g" | awk 
'/^#/{if(s){print;next};c++;a[c]=$0;next}/^data/{print;if(s)next;s=1;for(i=1;i<=c;i++){print 
a[i]}next}{print}' > combine_all_refln_fixed.cif


 DONE.


Best
Vincent
ps: again, the credit of this script goes to the globalphasing team who 
kindly helped when I asked them for help with it.




Le 01/06/2020 à 11:47, Clemens Vonrhein a écrit :

Dear all,

On Sat, May 30, 2020 at 03:40:53PM +0100, Eleanor Dodson wrote:

My pennysworth. If you find your maps look better after the
anisotroy correction use it, but it may be helpful to those wo want to mine
your data if you deposit the whole sphere..

Agree (which is what e.g. we provide when using STARANISO via autoPROC
[1]).

And in the same vein: those depositing isotropically truncated data
should consider also providing data to a higher diffraction limit to
give a potentially more accurate picture (if there is even a slight
indication of anisotropy - which there often is).

I find it very helpful even looking at an idealised (and therefore
simplified) picture of anisotropy as in

   http://staraniso.globalphasing.org/anisotropy_about.html

We can consider

  (1) for refinement:

  (1a) green+red, i.e. spherical (i.e. isotropically) truncated
   data

  (1b) green+blue, i.e. anisotropycally truncated data

  (2) for deposition:

  (2a) green+red => full sphere, but dropping real observations
   (blue)

  (2b) green+blue => all observations, but not providing
   insignificant/weak data (red) in all directions

  (2c) a sphere to the "tip" of blue (i.e. anisotropic diffraction
   limit) => all observations and all insignificant/weak data

Cheers

Clemens

[1] https://www.globalphasing.com/autoproc/ - which gives a mmCIF file
 with (2a), (2b) and (2c) ready for deposition.



eleanor

On Sat, 30 May 2020 at 09:36, Robbie Joosten 
wrote:


Hi Everyone,

I've been looking at some recent PDB entries that have much lower
spherical) completeness than reported in the coordinate file. One reason
for this is that the data were anisotropicly truncated, another reason is
some mess-up with the deposition of the reflection data. There is a lot of
discussion about the former practice and I don't want to go in to that, but
the second one is obviously an error. Now how do I distinguish these cases?

Sometimes, you can look at the reported number of reflections and compare
that to the deposited reflection file and you will find that something has
clearly gone wrong. However, the reported number of reflections is not
entirely reliable because of other issues so I'd rather not use it. If you
use PDBpeep (e.g. for 6rjy) you can see something is wrong, but that is
completely visual. Is there a tool in CCP4 that reports both spherical and
ellipsoidal completeness (on merged reflection data)? That would make it
easy to distinguish such cases.

Cheers,
Robbie



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a
mailing list hosted by www.jiscmail.ac.uk, terms & conditions are
available at https://www.jiscmail.ac.uk/policyandsecurity/




To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/


--

Vincent Chaptal, PhD

Director of GdR APPICOM

Drug Resistance and Membrane Proteins Lab


MMSB -UMR5086

7 passage du Vercors

69007 LYON

FRANCE

+33 4 37 65 29 01

http://www.appicom.cnrs.fr

http://mmsb.cnrs.fr/en/





To unsubscribe from the CCP4BB list, click the following link:

Re: [ccp4bb] Completeness question

[Clemens Vonrhein]

Dear all,

On Sat, May 30, 2020 at 03:40:53PM +0100, Eleanor Dodson wrote:
> My pennysworth. If you find your maps look better after the
> anisotroy correction use it, but it may be helpful to those wo want to mine
> your data if you deposit the whole sphere..

Agree (which is what e.g. we provide when using STARANISO via autoPROC
[1]).

And in the same vein: those depositing isotropically truncated data
should consider also providing data to a higher diffraction limit to
give a potentially more accurate picture (if there is even a slight
indication of anisotropy - which there often is).

I find it very helpful even looking at an idealised (and therefore
simplified) picture of anisotropy as in

  http://staraniso.globalphasing.org/anisotropy_about.html

We can consider

 (1) for refinement:

 (1a) green+red, i.e. spherical (i.e. isotropically) truncated
  data

 (1b) green+blue, i.e. anisotropycally truncated data

 (2) for deposition:

 (2a) green+red => full sphere, but dropping real observations
  (blue)

 (2b) green+blue => all observations, but not providing
  insignificant/weak data (red) in all directions

 (2c) a sphere to the "tip" of blue (i.e. anisotropic diffraction
  limit) => all observations and all insignificant/weak data

Cheers

Clemens

[1] https://www.globalphasing.com/autoproc/ - which gives a mmCIF file
with (2a), (2b) and (2c) ready for deposition.


> eleanor
> 
> On Sat, 30 May 2020 at 09:36, Robbie Joosten 
> wrote:
> 
> > Hi Everyone,
> >
> > I've been looking at some recent PDB entries that have much lower
> > spherical) completeness than reported in the coordinate file. One reason
> > for this is that the data were anisotropicly truncated, another reason is
> > some mess-up with the deposition of the reflection data. There is a lot of
> > discussion about the former practice and I don't want to go in to that, but
> > the second one is obviously an error. Now how do I distinguish these cases?
> >
> > Sometimes, you can look at the reported number of reflections and compare
> > that to the deposited reflection file and you will find that something has
> > clearly gone wrong. However, the reported number of reflections is not
> > entirely reliable because of other issues so I'd rather not use it. If you
> > use PDBpeep (e.g. for 6rjy) you can see something is wrong, but that is
> > completely visual. Is there a tool in CCP4 that reports both spherical and
> > ellipsoidal completeness (on merged reflection data)? That would make it
> > easy to distinguish such cases.
> >
> > Cheers,
> > Robbie
> >
> > 
> >
> > To unsubscribe from the CCP4BB list, click the following link:
> > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1
> >
> > This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a
> > mailing list hosted by www.jiscmail.ac.uk, terms & conditions are
> > available at https://www.jiscmail.ac.uk/policyandsecurity/
> >
> 
> 
> 
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1
> 
> This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing 
> list hosted by www.jiscmail.ac.uk, terms & conditions are available at 
> https://www.jiscmail.ac.uk/policyandsecurity/

-- 

*--
* Clemens Vonrhein, Ph.D. vonrhein AT GlobalPhasing DOT com
* Global Phasing Ltd., Sheraton House, Castle Park 
* Cambridge CB3 0AX, UK   www.globalphasing.com
*--



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] Completeness question

[vincent Chaptal]

Hi,

also agreed, but actually doing it proved a lot more tricky than I 
initially thought.
For my last structure, which was very anisotropic, I deposited a mmcif 
containing 1/ the originial data, 2/the truncated data and 3/ the map. 
It proved impossible to create this file with the tools at hand, and 
thanks to the globalphasing team, they were able to write a script using 
autoproc to do this.
If this routine could be incorporated into the staraniso web site, I'm 
sure it would be very used and help the case described here.


All the best
Vincent

ps: I'll be happy to share my procedure if it is helpful to anyone.


Le 30/05/2020 à 17:17, Ian Tickle a écrit :


Also agree, see http://staraniso.globalphasing.org/deposition_about.html .

Cheers

-- Ian


On Sat, 30 May 2020 at 15:58, Robbie Joosten 
mailto:robbie_joos...@hotmail.com>> wrote:


I fully agree. Unfortunately, not everyone does that so cases like
I described will keep appearing.

Cheers,
Robbie

On 30 May 2020 16:40, Eleanor Dodson mailto:eleanor.dod...@york.ac.uk>> wrote:

My pennysworth. If you find your maps look better after the
anisotroy correction use it, but it may be helpful to those
wo want to mine your data if you deposit the whole sphere..
eleanor

On Sat, 30 May 2020 at 09:36, Robbie Joosten
mailto:robbie_joos...@hotmail.com>> wrote:

Hi Everyone,

I've been looking at some recent PDB entries that have
much lower spherical) completeness than reported in the
coordinate file. One reason for this is that the data were
anisotropicly truncated, another reason is some mess-up
with the deposition of the reflection data. There is a lot
of discussion about the former practice and I don't want
to go in to that, but the second one is obviously an
error. Now how do I distinguish these cases?

Sometimes, you can look at the reported number of
reflections and compare that to the deposited reflection
file and you will find that something has clearly gone
wrong. However, the reported number of reflections is not
entirely reliable because of other issues so I'd rather
not use it. If you use PDBpeep (e.g. for 6rjy) you can see
something is wrong, but that is completely visual. Is
there a tool in CCP4 that reports both spherical and
ellipsoidal completeness (on merged reflection data)? That
would make it easy to distinguish such cases.

Cheers,
Robbie




To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1

This message was issued to members of
www.jiscmail.ac.uk/CCP4BB
, a mailing list hosted
by www.jiscmail.ac.uk , terms &
conditions are available at
https://www.jiscmail.ac.uk/policyandsecurity/





To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1




To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1



--

Vincent Chaptal, PhD

Director of GdR APPICOM

Drug Resistance and Membrane Proteins Lab


MMSB -UMR5086

7 passage du Vercors

69007 LYON

FRANCE

+33 4 37 65 29 01

http://www.appicom.cnrs.fr

http://mmsb.cnrs.fr/en/





To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] Completeness question

[Ian Tickle]

Also agree, see http://staraniso.globalphasing.org/deposition_about.html .

Cheers

-- Ian


On Sat, 30 May 2020 at 15:58, Robbie Joosten 
wrote:

> I fully agree. Unfortunately, not everyone does that so cases like I
> described will keep appearing.
>
> Cheers,
> Robbie
>
> On 30 May 2020 16:40, Eleanor Dodson  wrote:
>
> My pennysworth. If you find your maps look better after the
> anisotroy correction use it, but it may be helpful to those wo want to mine
> your data if you deposit the whole sphere..
> eleanor
>
> On Sat, 30 May 2020 at 09:36, Robbie Joosten 
> wrote:
>
> Hi Everyone,
>
> I've been looking at some recent PDB entries that have much lower
> spherical) completeness than reported in the coordinate file. One reason
> for this is that the data were anisotropicly truncated, another reason is
> some mess-up with the deposition of the reflection data. There is a lot of
> discussion about the former practice and I don't want to go in to that, but
> the second one is obviously an error. Now how do I distinguish these cases?
>
> Sometimes, you can look at the reported number of reflections and compare
> that to the deposited reflection file and you will find that something has
> clearly gone wrong. However, the reported number of reflections is not
> entirely reliable because of other issues so I'd rather not use it. If you
> use PDBpeep (e.g. for 6rjy) you can see something is wrong, but that is
> completely visual. Is there a tool in CCP4 that reports both spherical and
> ellipsoidal completeness (on merged reflection data)? That would make it
> easy to distinguish such cases.
>
> Cheers,
> Robbie
>
> 
>
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1
>
> This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a
> mailing list hosted by www.jiscmail.ac.uk, terms & conditions are
> available at https://www.jiscmail.ac.uk/policyandsecurity/
>
>
>
> --
>
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1
>



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] Completeness question

[Robbie Joosten]

I fully agree. Unfortunately, not everyone does that so cases like I described will keep appearing. Cheers,RobbieOn 30 May 2020 16:40, Eleanor Dodson  wrote:My pennysworth. If you find your maps look better after the anisotroy correction use it, but it may be helpful to those wo want to mine your data if you deposit the whole sphere..eleanorOn Sat, 30 May 2020 at 09:36, Robbie Joosten  wrote:Hi Everyone,

I've been looking at some recent PDB entries that have much lower spherical) completeness than reported in the coordinate file. One reason for this is that the data were anisotropicly truncated, another reason is some mess-up with the deposition of the reflection data. There is a lot of discussion about the former practice and I don't want to go in to that, but the second one is obviously an error. Now how do I distinguish these cases?

Sometimes, you can look at the reported number of reflections and compare that to the deposited reflection file and you will find that something has clearly gone wrong. However, the reported number of reflections is not entirely reliable because of other issues so I'd rather not use it. If you use PDBpeep (e.g. for 6rjy) you can see something is wrong, but that is completely visual. Is there a tool in CCP4 that reports both spherical and ellipsoidal completeness (on merged reflection data)? That would make it easy to distinguish such cases.

Cheers,
Robbie



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/




To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1

Re: [ccp4bb] Completeness question

[Eleanor Dodson]

My pennysworth. If you find your maps look better after the
anisotroy correction use it, but it may be helpful to those wo want to mine
your data if you deposit the whole sphere..
eleanor

On Sat, 30 May 2020 at 09:36, Robbie Joosten 
wrote:

> Hi Everyone,
>
> I've been looking at some recent PDB entries that have much lower
> spherical) completeness than reported in the coordinate file. One reason
> for this is that the data were anisotropicly truncated, another reason is
> some mess-up with the deposition of the reflection data. There is a lot of
> discussion about the former practice and I don't want to go in to that, but
> the second one is obviously an error. Now how do I distinguish these cases?
>
> Sometimes, you can look at the reported number of reflections and compare
> that to the deposited reflection file and you will find that something has
> clearly gone wrong. However, the reported number of reflections is not
> entirely reliable because of other issues so I'd rather not use it. If you
> use PDBpeep (e.g. for 6rjy) you can see something is wrong, but that is
> completely visual. Is there a tool in CCP4 that reports both spherical and
> ellipsoidal completeness (on merged reflection data)? That would make it
> easy to distinguish such cases.
>
> Cheers,
> Robbie
>
> 
>
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1
>
> This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a
> mailing list hosted by www.jiscmail.ac.uk, terms & conditions are
> available at https://www.jiscmail.ac.uk/policyandsecurity/
>



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] [EXTERNAL] Re: [ccp4bb] Completeness question

[Edward Berry]

>>> Ian Tickle  05/30/20 7:14 AM >>>
>>(unless of course the completeness calculations were performed on two
different reflection files)?

EDS is in fact using a different dataset compared to the coordinates,
when I submit the output reflections.cif produced by phenix, when the
input I, sigma-I from a merged scalepack output (.sca) file.

Phenix copies the raw input data (I's) into the output file, and that is
what EDS uses.
Then phenix does French & Wilson conversion of I's to F's, and rejects
reflections with a low probability. 
The resulting dataset is used in refinement, and the output statistics
are based on this. 
For an example of the result, 6myo. From the validation report,Data and
refinement statistics:
Resolution (Å) 
47.64 – 2.20  Depositor
80.48 – 2.20   EDS

% Data completeness (in resolution range)
91.1 (47.64-2.20)   Depositor
86.2 (80.48-2.20)  EDS

There were a few very low resolution reflections (probably behind the
beamstop) 
in the .sca file, resulting  in the low resolution seen by EDS. Phenix
rejects those 
and reports the low-res cutoff 40 A   But there are not a lot 
of reflections between 40 and 80 A, so I think most of the difference is
due to F, 
which EDS apparently does not apply.

I think this low-res cutoff is also responsible for the absurdly good
RSR-Z score for 6myo
compared to the ridiculously bad RSR-Z for otherwise very similar 6myp. 
The ultra-low reflections aren't going to affect shape of the density
around individual residues. But RSR-Z is not a correlation, it depends
on the actual value of the (scaled) map, and that will be affected by
strong low-res reflections. So if RSR-Z is comparing a 2Fo-Fc map,
perhaps with fill-in, to an atom-map which effectively goes to
infinitely low resolution, data lacking those low-res reflections will
compare poorly, whereas data that is artificially extended to 80A with
fill-in will do much better.
(Still looking at this, may come back for advice later.)
Ed

>>> Ian Tickle  05/30/20 7:14 AM >>>
Hi Robbie

I don't see that anisotropic truncation has anything to do with the low
spherical completeness as compared with the info in the co-ordinate
file.  Yes the spherical completeness after anisotropic truncation will
be reduced, but why would it cause it to become inconsistent with that
reported (unless of course the completeness calculations were performed
on two different reflection files)?  Besides, the anisotropy is quite
low (Delta-B eigenvalues: 3.42  -1.95 -1.47) so
that couldn't explain it.

I do agree that something has clearly gone wrong with the reflection
deposition for 6RJY.  It could of course go right back to the collection
or processing, but I think it unlikely anyone could solve the structure
with data in this state!  Approximately alternate reflections are
missing, but the pattern of absences does not correspond with any space
group.  For example from MTZDUMP on the reflection file:

   3   1   00.00 21.21  0.22 
3   1   20.00 23.83  0.19 
3   1   40.00 34.71  0.26 
3   1   60.00  9.06  0.11 
3   1   80.00 31.64  0.24 
3   1  100.00 31.22  0.25 
3   1  120.00  1.28  0.39 
3   1  140.00  6.59  0.12 
3   1  160.00 17.58  0.15 
3   1  180.00  3.94  0.18 
3   1  200.00 11.05  0.12 
3   1  220.00 34.24  0.24 
3   1  240.00 12.39  0.14 
3   1  260.00 12.76  0.15 
3   1  280.00 20.80  0.18
   3   1  300.00 23.70  0.19 
3   1  320.00 23.47  0.20 
3   1  340.00 30.50  0.23 
3   1  360.00 10.93  0.22 
3   1  380.00 28.11  0.22 
3   1  400.00 24.41  0.21 
3   1  420.00   3   1  470.00 10.54  0.29 
3   1  490.00 10.54  0.23 
3   1  510.00  2.98  0.70 
3   1  530.00  5.84  0.39 
3   1  550.00  9.79  0.27 
3   1  570.00 11.33  0.26 
3   1  590.00  8.99  0.30
3   1  610.00  1.84  0.76 
3   1  630.00  2.63  0.78 
3   1  650.00  4.91  0.46 
3   1  670.00  3.50  0.64 
3   1  690.00  1.93  0.76 
3   1  710.00  4.57  0.52 
3   1  730.00  1.71  0.73
 

Note how the pattern switches between (3 1 44) and (3 1 47).


So sometimes k+l = 2n are absent and sometimes k+l = 2n+1 are: this
pattern pervades the whole dataset so the completeness (both spherical
and ellipsoidal) is reduced by a factor of about two.  This makes no
sense in terms of known systematic absences, and certainly not for the
reported space group P212121.  This 

Re: [ccp4bb] Completeness question

[Robbie Joosten]

Hi Ian,

> I don't see that anisotropic truncation has anything to do with the low
> spherical completeness as compared with the info in the co-ordinate file.
> Yes the spherical completeness after anisotropic truncation will be reduced,
> but why would it cause it to become inconsistent with that reported (unless
> of course the completeness calculations were performed on two different
> reflection files)?  Besides, the anisotropy is quite low (Delta-B eigenvalues:
> 3.42  -1.95 -1.47) so that couldn't explain it.
This is a general problem with anisotropic truncation. 6rjy is an example of 
something else going on. What happens is that people report the ellipsoidal 
completeness, but when you import the reflection data from the PDB and add the 
missing reflections (because those are typically not deposited) with the CCP4 
program complete you get a spherical dataset. If you now calculate the 
completeness it will be very low and inconsistent with the reported number. 
This is actually also a problem on the side of the PDB validation as EDS does a 
similar thing.

So what I want is a way to reproduce the reported completeness even when it is 
the ellipsoidal completeness. Not being able to do so would then be a clear 
indication that something else is going on like you see below. I suppose 
replacing complete with something that can add the missing ellipsoidal 
reflections would also solve the problem. If anything that would be even more 
elegant.

Cheers,
Robbie







> 
> I do agree that something has clearly gone wrong with the reflection
> deposition for 6RJY.  It could of course go right back to the collection or
> processing, but I think it unlikely anyone could solve the structure with data
> in this state!  Approximately alternate reflections are missing, but the
> pattern of absences does not correspond with any space group.  For example
> from MTZDUMP on the reflection file:
> 
>3   1   00.00 21.21  0.22
>3   1   20.00 23.83  0.19
>3   1   40.00 34.71  0.26
>3   1   60.00  9.06  0.11
>3   1   80.00 31.64  0.24
>3   1  100.00 31.22  0.25
>3   1  120.00  1.28  0.39
>3   1  140.00  6.59  0.12
>3   1  160.00 17.58  0.15
>3   1  180.00  3.94  0.18
>3   1  200.00 11.05  0.12
>3   1  220.00 34.24  0.24
>3   1  240.00 12.39  0.14
>3   1  260.00 12.76  0.15
>3   1  280.00 20.80  0.18
> 
>3   1  300.00 23.70  0.19
>3   1  320.00 23.47  0.20
>3   1  340.00 30.50  0.23
>3   1  360.00 10.93  0.22
>3   1  380.00 28.11  0.22
>3   1  400.00 24.41  0.21
>3   1  420.00 11.04  0.21
>3   1  440.00 12.58  0.28
>3   1  470.00 10.54  0.29
>3   1  490.00 10.54  0.23
>3   1  510.00  2.98  0.70
>3   1  530.00  5.84  0.39
>3   1  550.00  9.79  0.27
>3   1  570.00 11.33  0.26
>3   1  590.00  8.99  0.30
>3   1  610.00  1.84  0.76
>3   1  630.00  2.63  0.78
>3   1  650.00  4.91  0.46
>3   1  670.00  3.50  0.64
>3   1  690.00  1.93  0.76
>3   1  710.00  4.57  0.52
>3   1  730.00  1.71  0.73
> 
> 
> Note how the pattern switches between (3 1 44) and (3 1 47).
> 
> 
> So sometimes k+l = 2n are absent and sometimes k+l = 2n+1 are: this pattern
> pervades the whole dataset so the completeness (both spherical and
> ellipsoidal) is reduced by a factor of about two.  This makes no sense in
> terms of known systematic absences, and certainly not for the reported
> space group P212121.  This alternating pattern of absences is of course
> extremely unlikely in valid data: normally low completeness arises from
> whole missing wedges of data, or cusps, or to a smaller extent detector gaps,
> i.e. usually missing data are largely contiguous, not alternating as here.
> 
> 
> I think the only solution here is to get the authors to deposit the data
> correctly: is there any commonality of the authors for the structures where
> you have noted this problem?
> 
> 
> Cheers
> 
> 
> -- Ian
> 
> 
> 
> On Sat, 30 May 2020 at 09:36, Robbie Joosten
> mailto:robbie_joos...@hotmail.com> >
> wrote:
> 
> 
>   Hi Everyone,
> 
>   I've been looking at some recent PDB entries that have much lower
> spherical) completeness than reported in the coordinate file. One reason for
> this is that the data were anisotropicly truncated, another reason is some
> mess-up with the deposition of the reflection data. There is a lot 

Re: [ccp4bb] Completeness question

[Ian Tickle]

Hi Robbie

I don't see that anisotropic truncation has anything to do with the low
spherical completeness as compared with the info in the co-ordinate file.
Yes the spherical completeness after anisotropic truncation will be
reduced, but why would it cause it to become inconsistent with that
reported (unless of course the completeness calculations were performed on
two different reflection files)?  Besides, the anisotropy is quite low (Delta-B
eigenvalues: 3.42  -1.95 -1.47) so that couldn't explain it.

I do agree that something has clearly gone wrong with the reflection
deposition for 6RJY.  It could of course go right back to the collection or
processing, but I think it unlikely anyone could solve the structure with
data in this state!  Approximately alternate reflections are missing, but
the pattern of absences does not correspond with any space group.  For
example from MTZDUMP on the reflection file:

   3   1   00.00 21.21  0.22
   3   1   20.00 23.83  0.19
   3   1   40.00 34.71  0.26
   3   1   60.00  9.06  0.11
   3   1   80.00 31.64  0.24
   3   1  100.00 31.22  0.25
   3   1  120.00  1.28  0.39
   3   1  140.00  6.59  0.12
   3   1  160.00 17.58  0.15
   3   1  180.00  3.94  0.18
   3   1  200.00 11.05  0.12
   3   1  220.00 34.24  0.24
   3   1  240.00 12.39  0.14
   3   1  260.00 12.76  0.15
   3   1  280.00 20.80  0.18
   3   1  300.00 23.70  0.19
   3   1  320.00 23.47  0.20
   3   1  340.00 30.50  0.23
   3   1  360.00 10.93  0.22
   3   1  380.00 28.11  0.22
   3   1  400.00 24.41  0.21
   3   1  420.00 11.04  0.21
   3   1  440.00 12.58  0.28
   3   1  470.00 10.54  0.29
   3   1  490.00 10.54  0.23
   3   1  510.00  2.98  0.70
   3   1  530.00  5.84  0.39
   3   1  550.00  9.79  0.27
   3   1  570.00 11.33  0.26
   3   1  590.00  8.99  0.30
   3   1  610.00  1.84  0.76
   3   1  630.00  2.63  0.78
   3   1  650.00  4.91  0.46
   3   1  670.00  3.50  0.64
   3   1  690.00  1.93  0.76
   3   1  710.00  4.57  0.52
   3   1  730.00  1.71  0.73

Note how the pattern switches between (3 1 44) and (3 1 47).

So sometimes k+l = 2n are absent and sometimes k+l = 2n+1 are: this pattern
pervades the whole dataset so the completeness (both spherical and
ellipsoidal) is reduced by a factor of about two.  This makes no sense
in terms of known systematic absences, and certainly not for the reported
space group P212121.  This alternating pattern of absences is of course
extremely unlikely in valid data: normally low completeness arises from
whole missing wedges of data, or cusps, or to a smaller extent detector
gaps, i.e. usually missing data are largely contiguous, not alternating as
here.

I think the only solution here is to get the authors to deposit the data
correctly: is there any commonality of the authors for the structures where
you have noted this problem?

Cheers

-- Ian


On Sat, 30 May 2020 at 09:36, Robbie Joosten 
wrote:

> Hi Everyone,
>
> I've been looking at some recent PDB entries that have much lower
> spherical) completeness than reported in the coordinate file. One reason
> for this is that the data were anisotropicly truncated, another reason is
> some mess-up with the deposition of the reflection data. There is a lot of
> discussion about the former practice and I don't want to go in to that, but
> the second one is obviously an error. Now how do I distinguish these cases?
>
> Sometimes, you can look at the reported number of reflections and compare
> that to the deposited reflection file and you will find that something has
> clearly gone wrong. However, the reported number of reflections is not
> entirely reliable because of other issues so I'd rather not use it. If you
> use PDBpeep (e.g. for 6rjy) you can see something is wrong, but that is
> completely visual. Is there a tool in CCP4 that reports both spherical and
> ellipsoidal completeness (on merged reflection data)? That would make it
> easy to distinguish such cases.
>
> Cheers,
> Robbie
>
> 
>
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1
>
> This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a
> mailing list hosted by www.jiscmail.ac.uk, terms & conditions are
> available at https://www.jiscmail.ac.uk/policyandsecurity/
>

[ccp4bb] Completeness question

[Robbie Joosten]

Hi Everyone,

I've been looking at some recent PDB entries that have much lower spherical) 
completeness than reported in the coordinate file. One reason for this is that 
the data were anisotropicly truncated, another reason is some mess-up with the 
deposition of the reflection data. There is a lot of discussion about the 
former practice and I don't want to go in to that, but the second one is 
obviously an error. Now how do I distinguish these cases?

Sometimes, you can look at the reported number of reflections and compare that 
to the deposited reflection file and you will find that something has clearly 
gone wrong. However, the reported number of reflections is not entirely 
reliable because of other issues so I'd rather not use it. If you use PDBpeep 
(e.g. for 6rjy) you can see something is wrong, but that is completely visual. 
Is there a tool in CCP4 that reports both spherical and ellipsoidal 
completeness (on merged reflection data)? That would make it easy to 
distinguish such cases.

Cheers,
Robbie



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/