Re: [ccp4bb] Optimising data processing of a I432 dataset with 75% solvent content.

[Kay Diederichs]

Dear Michael,

as said in this thread, careful re-integration (or other programs) might give 
you better data. If using XDS, you could try RELRAD=7 (or even 10 if the 
background is very low; the default is 5) in INTEGRATE  (this as yet 
undocumented option is in the latest XDS and determines the radius of the 
background area relative to that of the peak; 
BEAM_DIVERGENCE/BEAM_DIVERGENCE_E.S.D.) Of course, all other optimiatzion 
options should be tried.. 

On the other hand (and again as others already said), some of the stats look 
strange - the CC1/2 of 53% asks for extending the resolution, whereas the low 
 of 0.1 says it won't help much. You can evaluate radiation damage 
with the R_d plot from XDSSTAT. This is most easily produced and interpreted in 
XDSGUI - if the red fit line intersects the green level then this means that 
after this many frames, radiation damage starts dominating all other sources of 
error.

Most importantly - which problem do you really need to solve? 3.5A structures 
are not unheard of, and can be published and deposited. Another 0.1A higher 
resolution may not help much. Is there a problem in refinement or map 
interpretation? What is current Rwork/Rfree overall and in the highest shell?

best,

Kay


On Thu, 18 May 2017 11:47:30 +, Michael Jarva  
wrote:

>Dear all,
>
>I have a dataset that have two very interesting properties: a) It's in I432, 
>and b) has a whooping 75% solvent content.
>You might think that the solvent content obviously is a big red flag, and so 
>did I, but I have phased this successfully with just one monomer, and the 
>packing result does makes a lot of sense. The resulting maps contain no extra 
>umodelled blobs, and trying to phase it with an additional molecules does not 
>give a good solution.
>
>The problem I have is that the diffraction intensity/Rmerge plummets/explodes 
>around the 3.5� mark (I assume because of the high solvent content) to such an 
>extent that even though I have little radiation damage, 100% completeness in 
>high resolution shells, and very high redundancy, any attempt to merge the 
>dataset at a higher resolution has so far given no improvement to the maps.
>
>I'm hoping that there might be a few tricks out there I can apply to the spot 
>finding/integration/scaling steps have it merge in a even slightly higher 
>resolution than I currently have been able to do.
>Although I have a feeling that the only thing I can do is to grow another, 
>much bigger, crystal�
>
>many thanks for any feedback
>/michael
>
>See below for sample outputs from aimless:
>
>   Overall  InnerShell  OuterShell
>Low resolution limit   43.50 43.50  3.32
>High resolution limit   3.10  8.78  3.10
>
>Rmerge  (within I+/I-) 0.079 0.01021.891
>Rmerge  (all I+ and I-)0.081 0.01122.502
>Rmeas (within I+/I-)   0.084 0.01123.102
>Rmeas (all I+ & I-)0.084 0.01123.169
>Rpim (within I+/I-)0.027 0.004 7.335
>Rpim (all I+ & I-) 0.020 0.003 5.450
>Rmerge in top intensity bin0.010- -
>Total number of observations   34917  1495  6448
>Total number unique 2057   112   362
>Mean((I)/sd(I)) 18.3 130.9   0.1
>Mn(I) half-set correlation CC(1/2) 1.000 1.000 0.533
>Completeness99.9  97.4 100.0
>Multiplicity17.0  13.3  17.8
>
>  Overall  InnerShell  OuterShell
>Low resolution limit   43.50 43.50  3.84
>High resolution limit   3.50  8.58  3.50
>
>Rmerge  (within I+/I-) 0.052 0.011 2.422
>Rmerge  (all I+ and I-)0.056 0.012 2.659
>Rmeas (within I+/I-)   0.055 0.011 2.553
>Rmeas (all I+ & I-)0.058 0.013 2.738
>Rpim (within I+/I-)0.017 0.004 0.804
>Rpim (all I+ & I-) 0.014 0.003 0.644
>Rmerge in top intensity bin0.010- -
>Total number of observations   24596  1690  6071
>Total number unique 1462   120   343
>Mean((I)/sd(I)) 25.8 132.0   1.0
>Mn(I) half-set correlation CC(1/2) 1.000 1.000 0.771
>Completeness99.8  97.6 100.0
>Multiplicity16.8  14.1  17.7
>
>

Re: [ccp4bb] Optimising data processing of a I432 dataset with 75% solvent content.

[Clemens Vonrhein]

Dear Michael,

throwing a few other suggestions into the mix (on top of the good
advice you got already from Graeme and Phil):

 * The low resolution I/sigI is quite large and although we have seen
   such values for extremely good crystals collected very carefully on
   very good instruments, it is rather uncommon for "standard"
   (whatever that means) protein crystals that diffract to lowish
   resolution.

   So Phil's suggestion about problems in your error model is quite
   likely - and since refinement programs usually take sigmas into
   account, this might explain why there is no effect when using
   higher-resolution data.

 * You have quite high multiplicity so that you can easily check for
   possible effects of radiation damage, which could also be at play
   here: is there an increase in cell dimensions, a "smiley" like
   shape to Rmerge-vs-Image number plots, visible radiation damage in
   F(early)-F(late) maps ... ?

 * Any ice-rings present? The 3.5A range is known for those ;-)

 * Did you mask your beamstop shadow best/correctly? This shouldn't
   impact the high-resolution limit, but sometimes things can still go
   wrong there.

 * Completeness as calculated in that table is only ever completeness
   of Miller indices and not necessarily observations
   (i.e. significant data) [1].

 * If there is a very large empty detector area (because the
   crystal-detector distance was over-optimistic) and
   integration/scaling is not restricted to something more sensible,
   things can sometimes go wrong in those steps: programs are very
   good in dealing with noise, but if nearly all incoming data is pure
   noise even they can sometimes go off into a wrong minimum (during
   scale or error model parameter refinement).

 * Have a close look at the detailed processing output from the
   program(s)/pipeline(s) you used for processing. Especially if you
   picked up your data directly from the automatic processing options
   available at a synchrotron, you have to be aware that sometimes
   sensible restrictions imposed by hardware, data policy or time
   requirements mean that some programs/pipelines can not necessarily
   be run in recommended default mode (assuming that the defaults are
   chosen by the developers for very good reasons).

   If you processed your data yourself: all programs and pipelines
   should give you plenty of clear warning messages early on that can
   highlight any possible problem mentioned above.

There are some more potential issues one could think of I guess
... see also [2] and [3].

Cheers

Clemens

[1] http://staraniso.globalphasing.org/
[2] http://www.globalphasing.com/autoproc/manual/autoPROC7.html
[3] http://www.globalphasing.com/autoproc/



On Thu, May 18, 2017 at 11:47:30AM +, Michael Jarva wrote:
> Dear all,
> 
> I have a dataset that have two very interesting properties: a) It's in I432, 
> and b) has a whooping 75% solvent content.
> You might think that the solvent content obviously is a big red flag, and so 
> did I, but I have phased this successfully with just one monomer, and the 
> packing result does makes a lot of sense. The resulting maps contain no extra 
> umodelled blobs, and trying to phase it with an additional molecules does not 
> give a good solution.
> 
> The problem I have is that the diffraction intensity/Rmerge plummets/explodes 
> around the 3.5Å mark (I assume because of the high solvent content) to such 
> an extent that even though I have little radiation damage, 100% completeness 
> in high resolution shells, and very high redundancy, any attempt to merge the 
> dataset at a higher resolution has so far given no improvement to the maps.
> 
> I'm hoping that there might be a few tricks out there I can apply to the spot 
> finding/integration/scaling steps have it merge in a even slightly higher 
> resolution than I currently have been able to do.
> Although I have a feeling that the only thing I can do is to grow another, 
> much bigger, crystal…
> 
> many thanks for any feedback
> /michael
> 
> See below for sample outputs from aimless:
> 
>Overall  InnerShell  OuterShell
> Low resolution limit   43.50 43.50  3.32
> High resolution limit   3.10  8.78  3.10
> 
> Rmerge  (within I+/I-) 0.079 0.01021.891
> Rmerge  (all I+ and I-)0.081 0.01122.502
> Rmeas (within I+/I-)   0.084 0.01123.102
> Rmeas (all I+ & I-)0.084 0.01123.169
> Rpim (within I+/I-)0.027 0.004 7.335
> Rpim (all I+ & I-) 0.020 0.003 5.450
> Rmerge in top intensity bin0.010- -
> Total number of observations   34917  1495  6448
> Total number unique 2057   112   362
> Mean((I)/sd(I))  

Re: [ccp4bb] Optimising data processing of a I432 dataset with 75% solvent content.

[Keller, Jacob]

It's radiation-damaged. Based on what did you say that it is not?

Also, do you have ice rings or diffuse scattering which might be present in the 
higher-res shells in certain rotation ranges but not others, such as from 
solvent in the loop?

JPK

From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Michael 
Jarva
Sent: Thursday, May 18, 2017 7:48 AM
To: CCP4BB@JISCMAIL.AC.UK
Subject: [ccp4bb] Optimising data processing of a I432 dataset with 75% solvent 
content.

Dear all,

I have a dataset that have two very interesting properties: a) It's in I432, 
and b) has a whooping 75% solvent content.
You might think that the solvent content obviously is a big red flag, and so 
did I, but I have phased this successfully with just one monomer, and the 
packing result does makes a lot of sense. The resulting maps contain no extra 
umodelled blobs, and trying to phase it with an additional molecules does not 
give a good solution.

The problem I have is that the diffraction intensity/Rmerge plummets/explodes 
around the 3.5Å mark (I assume because of the high solvent content) to such an 
extent that even though I have little radiation damage, 100% completeness in 
high resolution shells, and very high redundancy, any attempt to merge the 
dataset at a higher resolution has so far given no improvement to the maps.

I'm hoping that there might be a few tricks out there I can apply to the spot 
finding/integration/scaling steps have it merge in a even slightly higher 
resolution than I currently have been able to do.
Although I have a feeling that the only thing I can do is to grow another, much 
bigger, crystal...

many thanks for any feedback
/michael

See below for sample outputs from aimless:

   Overall  InnerShell  OuterShell
Low resolution limit   43.50 43.50  3.32
High resolution limit   3.10  8.78  3.10

Rmerge  (within I+/I-) 0.079 0.01021.891
Rmerge  (all I+ and I-)0.081 0.01122.502
Rmeas (within I+/I-)   0.084 0.01123.102
Rmeas (all I+ & I-)0.084 0.01123.169
Rpim (within I+/I-)0.027 0.004 7.335
Rpim (all I+ & I-) 0.020 0.003 5.450
Rmerge in top intensity bin0.010- -
Total number of observations   34917  1495  6448
Total number unique 2057   112   362
Mean((I)/sd(I)) 18.3 130.9   0.1
Mn(I) half-set correlation CC(1/2) 1.000 1.000 0.533
Completeness99.9  97.4 100.0
Multiplicity17.0  13.3  17.8

  Overall  InnerShell  OuterShell
Low resolution limit   43.50 43.50  3.84
High resolution limit   3.50  8.58  3.50

Rmerge  (within I+/I-) 0.052 0.011 2.422
Rmerge  (all I+ and I-)0.056 0.012 2.659
Rmeas (within I+/I-)   0.055 0.011 2.553
Rmeas (all I+ & I-)0.058 0.013 2.738
Rpim (within I+/I-)0.017 0.004 0.804
Rpim (all I+ & I-) 0.014 0.003 0.644
Rmerge in top intensity bin0.010- -
Total number of observations   24596  1690  6071
Total number unique 1462   120   343
Mean((I)/sd(I)) 25.8 132.0   1.0
Mn(I) half-set correlation CC(1/2) 1.000 1.000 0.771
Completeness99.8  97.6 100.0
Multiplicity16.8  14.1  17.7

Re: [ccp4bb] Optimising data processing of a I432 dataset with 75% solvent content.

[Phil Evans]

It is puzzling that in the high res shell (to 3.1A) CC(1/2) is 0.533 (OK), but 
I/sigI is 0.1 (v low). Is the SDcorrection (SDFAC, SDADD) sensible (and 
“Analysis of standard deviations vs. intensity”)? In Aimless the determination 
of SDFAC and SDADD sometimes go haywire - with high multiplicity you could try 
“SDCORRECTION SAMPLESD” to see what it does to mean I/sigI

 Is there a lot of radiation damage (you say not, but if so with the high 
multiplicity you could cut the end of the data)? 

At least the data can’t be anisotropic in the cubic space group!

Probably none of this helps in improving maps. It may be worth looking at 
sharpened maps (from refmac or in coot)

Phil


> On 18 May 2017, at 12:47, Michael Jarva 
> <1295eb3572d0-dmarc-requ...@jiscmail.ac.uk> wrote:
> 
> Dear all,
> 
> I have a dataset that have two very interesting properties: a) It's in I432, 
> and b) has a whooping 75% solvent content.  
> You might think that the solvent content obviously is a big red flag, and so 
> did I, but I have phased this successfully with just one monomer, and the 
> packing result does makes a lot of sense. The resulting maps contain no extra 
> umodelled blobs, and trying to phase it with an additional molecules does not 
> give a good solution.
> 
> The problem I have is that the diffraction intensity/Rmerge plummets/explodes 
> around the 3.5Å mark (I assume because of the high solvent content) to such 
> an extent that even though I have little radiation damage, 100% completeness 
> in high resolution shells, and very high redundancy, any attempt to merge the 
> dataset at a higher resolution has so far given no improvement to the maps.
> 
> I'm hoping that there might be a few tricks out there I can apply to the spot 
> finding/integration/scaling steps have it merge in a even slightly higher 
> resolution than I currently have been able to do.
> Although I have a feeling that the only thing I can do is to grow another, 
> much bigger, crystal…
> 
> many thanks for any feedback
> /michael
> 
> See below for sample outputs from aimless:
> 
>Overall  InnerShell  OuterShell
> Low resolution limit   43.50 43.50  3.32
> High resolution limit   3.10  8.78  3.10
> 
> Rmerge  (within I+/I-) 0.079 0.01021.891
> Rmerge  (all I+ and I-)0.081 0.01122.502
> Rmeas (within I+/I-)   0.084 0.01123.102
> Rmeas (all I+ & I-)0.084 0.01123.169
> Rpim (within I+/I-)0.027 0.004 7.335
> Rpim (all I+ & I-) 0.020 0.003 5.450
> Rmerge in top intensity bin0.010- - 
> Total number of observations   34917  1495  6448
> Total number unique 2057   112   362
> Mean((I)/sd(I)) 18.3 130.9   0.1
> Mn(I) half-set correlation CC(1/2) 1.000 1.000 0.533
> Completeness99.9  97.4 100.0
> Multiplicity17.0  13.3  17.8
> 
>   Overall  InnerShell  OuterShell
> Low resolution limit   43.50 43.50  3.84
> High resolution limit   3.50  8.58  3.50
> 
> Rmerge  (within I+/I-) 0.052 0.011 2.422
> Rmerge  (all I+ and I-)0.056 0.012 2.659
> Rmeas (within I+/I-)   0.055 0.011 2.553
> Rmeas (all I+ & I-)0.058 0.013 2.738
> Rpim (within I+/I-)0.017 0.004 0.804
> Rpim (all I+ & I-) 0.014 0.003 0.644
> Rmerge in top intensity bin0.010- - 
> Total number of observations   24596  1690  6071
> Total number unique 1462   120   343
> Mean((I)/sd(I)) 25.8 132.0   1.0
> Mn(I) half-set correlation CC(1/2) 1.000 1.000 0.771
> Completeness99.8  97.6 100.0
> Multiplicity16.8  14.1  17.7
> 
> 

Re: [ccp4bb] Optimising data processing of a I432 dataset with 75% solvent content.

[Graeme Winter]

Hi Michael

What integration program did you use? Different programs can sometimes give 
rather different results with very weak data. I'd try everything to see if one 
is better than the others.

It's also worth making sure you are up to date. Certainly there are recent 
changes in xds and dials that are relevant to this problem

Best wishes Graeme

On 18 May 2017, at 12:58, Michael Jarva 
<1295eb3572d0-dmarc-requ...@jiscmail.ac.uk>
 wrote:

Dear all,

I have a dataset that have two very interesting properties: a) It's in I432, 
and b) has a whooping 75% solvent content.
You might think that the solvent content obviously is a big red flag, and so 
did I, but I have phased this successfully with just one monomer, and the 
packing result does makes a lot of sense. The resulting maps contain no extra 
umodelled blobs, and trying to phase it with an additional molecules does not 
give a good solution.

The problem I have is that the diffraction intensity/Rmerge plummets/explodes 
around the 3.5Å mark (I assume because of the high solvent content) to such an 
extent that even though I have little radiation damage, 100% completeness in 
high resolution shells, and very high redundancy, any attempt to merge the 
dataset at a higher resolution has so far given no improvement to the maps.

I'm hoping that there might be a few tricks out there I can apply to the spot 
finding/integration/scaling steps have it merge in a even slightly higher 
resolution than I currently have been able to do.
Although I have a feeling that the only thing I can do is to grow another, much 
bigger, crystal…

many thanks for any feedback
/michael

See below for sample outputs from aimless:

   Overall  InnerShell  OuterShell
Low resolution limit   43.50 43.50  3.32
High resolution limit   3.10  8.78  3.10

Rmerge  (within I+/I-) 0.079 0.01021.891
Rmerge  (all I+ and I-)0.081 0.01122.502
Rmeas (within I+/I-)   0.084 0.01123.102
Rmeas (all I+ & I-)0.084 0.01123.169
Rpim (within I+/I-)0.027 0.004 7.335
Rpim (all I+ & I-) 0.020 0.003 5.450
Rmerge in top intensity bin0.010- -
Total number of observations   34917  1495  6448
Total number unique 2057   112   362
Mean((I)/sd(I)) 18.3 130.9   0.1
Mn(I) half-set correlation CC(1/2) 1.000 1.000 0.533
Completeness99.9  97.4 100.0
Multiplicity17.0  13.3  17.8

  Overall  InnerShell  OuterShell
Low resolution limit   43.50 43.50  3.84
High resolution limit   3.50  8.58  3.50

Rmerge  (within I+/I-) 0.052 0.011 2.422
Rmerge  (all I+ and I-)0.056 0.012 2.659
Rmeas (within I+/I-)   0.055 0.011 2.553
Rmeas (all I+ & I-)0.058 0.013 2.738
Rpim (within I+/I-)0.017 0.004 0.804
Rpim (all I+ & I-) 0.014 0.003 0.644
Rmerge in top intensity bin0.010- -
Total number of observations   24596  1690  6071
Total number unique 1462   120   343
Mean((I)/sd(I)) 25.8 132.0   1.0
Mn(I) half-set correlation CC(1/2) 1.000 1.000 0.771
Completeness99.8  97.6 100.0
Multiplicity16.8  14.1  17.7


-- 
This e-mail and any attachments may contain confidential, copyright and or 
privileged material, and are for the use of the intended addressee only. If you 
are not the intended addressee or an authorised recipient of the addressee 
please notify us of receipt by returning the e-mail and do not use, copy, 
retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not 
necessarily of Diamond Light Source Ltd. 
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments 
are free from viruses and we cannot accept liability for any damage which you 
may sustain as a result of software viruses which may be transmitted in or with 
the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and 
Wales with its registered office at Diamond House, Harwell Science and 
Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

[ccp4bb] Optimising data processing of a I432 dataset with 75% solvent content.

[Michael Jarva]

Dear all,

I have a dataset that have two very interesting properties: a) It's in I432, 
and b) has a whooping 75% solvent content.
You might think that the solvent content obviously is a big red flag, and so 
did I, but I have phased this successfully with just one monomer, and the 
packing result does makes a lot of sense. The resulting maps contain no extra 
umodelled blobs, and trying to phase it with an additional molecules does not 
give a good solution.

The problem I have is that the diffraction intensity/Rmerge plummets/explodes 
around the 3.5Å mark (I assume because of the high solvent content) to such an 
extent that even though I have little radiation damage, 100% completeness in 
high resolution shells, and very high redundancy, any attempt to merge the 
dataset at a higher resolution has so far given no improvement to the maps.

I'm hoping that there might be a few tricks out there I can apply to the spot 
finding/integration/scaling steps have it merge in a even slightly higher 
resolution than I currently have been able to do.
Although I have a feeling that the only thing I can do is to grow another, much 
bigger, crystal…

many thanks for any feedback
/michael

See below for sample outputs from aimless:

   Overall  InnerShell  OuterShell
Low resolution limit   43.50 43.50  3.32
High resolution limit   3.10  8.78  3.10

Rmerge  (within I+/I-) 0.079 0.01021.891
Rmerge  (all I+ and I-)0.081 0.01122.502
Rmeas (within I+/I-)   0.084 0.01123.102
Rmeas (all I+ & I-)0.084 0.01123.169
Rpim (within I+/I-)0.027 0.004 7.335
Rpim (all I+ & I-) 0.020 0.003 5.450
Rmerge in top intensity bin0.010- -
Total number of observations   34917  1495  6448
Total number unique 2057   112   362
Mean((I)/sd(I)) 18.3 130.9   0.1
Mn(I) half-set correlation CC(1/2) 1.000 1.000 0.533
Completeness99.9  97.4 100.0
Multiplicity17.0  13.3  17.8

  Overall  InnerShell  OuterShell
Low resolution limit   43.50 43.50  3.84
High resolution limit   3.50  8.58  3.50

Rmerge  (within I+/I-) 0.052 0.011 2.422
Rmerge  (all I+ and I-)0.056 0.012 2.659
Rmeas (within I+/I-)   0.055 0.011 2.553
Rmeas (all I+ & I-)0.058 0.013 2.738
Rpim (within I+/I-)0.017 0.004 0.804
Rpim (all I+ & I-) 0.014 0.003 0.644
Rmerge in top intensity bin0.010- -
Total number of observations   24596  1690  6071
Total number unique 1462   120   343
Mean((I)/sd(I)) 25.8 132.0   1.0
Mn(I) half-set correlation CC(1/2) 1.000 1.000 0.771
Completeness99.8  97.6 100.0
Multiplicity16.8  14.1  17.7