Re: [ccp4bb] Merging hi and low res data

2012-02-08 Thread Graeme Winter
Dear Nick,

If you're happy to keep on using xia2, you can just put both of the
data sets in a single directory and run

xia2 -3dii /heres/where/the/data/went

And wait a little while.

To comment on your analysis of the statistics: inside xia2 the scaling
is switched off as far as possible in the XDS CORRECT step and is
instead done with XSCALE, which will scale several data sets at the
same time. The data are then merged with Scala. I have found that data
scaled with XSCALE will always give lower residuals than the same data
(say from INTEGRATE.HKL) scaled with Scala or Aimless. I suspect that
this relates to the parametrisations used - Scala (and I assume
Aimless) use an analytical model with a small number of parameters,
XDS and XSCALE use a numerical approach with rather more parameters. I
will leave discussion of which way is best to the experts :o)

On the subject of overloads, the 2^20 limit is an absolute limit -
however there is a more real (and lower) limit which is written in the
image headers, which relates to the count rate dead time correction -
if the pixel count exceeds this threshold then the correction becomes
less reliable. Typically though this is quite a large number, and is
usually not met, though I should include this in the xia2 / XDS
running code.

If you'd like any help specifically with fiddling the xia2 run please
feel free to contact me off list - at Diamond we also have facilities
for remotely reprocessing the data which may help to make things a
little quicker and would save moving the data around.

Best wishes,

Graeme

On 7 February 2012 17:36, Nicholas Keep n.k...@mail.cryst.bbk.ac.uk wrote:
 We have a high and low res pass on a crystal that is going to around 1.2 A
 on the pilatus at Diamond.
 We tried to scale them together using the xds CORRECT files from the xia2
 -3dii runs (gives best results as far as we can see comparing xia2.html
 files)  through aimless via ccp4i interface.

 The combined dataset looks worse (based on merging statistics at lower
 resolutions) than the individual particular at low res.(Compared to scala
 xia2 runs).
 XDS does not record the pilatus as having overloads even on the high res
 pass using the correct detector parameters
 http://xds.mpimf-heidelberg.mpg.de/html_doc/detectors.html

 What I wanted to do was limit the resolution of the two runs ie to use low
 res between 100-2.5 beyond which completeness plummets and then the highres
 between 8 and 1.2.  Aimless does not seem to allow me to do that at least by
 the ccp4i interface- is this then bad practice?  I guess I could run a
 utility to remove the resolutions I don't want.

 The one effect that makes me wonder if the lowres is better data, and worth
 combining, ie there is some overload in the high res, is that it goes from
 30-8 mean I/SIGI whereas the high res has I/SIGI of 15 in most of the low
 res shells and then slowly falls off to 2.0 at 1.2 angstrom.

 Am intending to carry on just using the hires pass.

 Any comments?

 I think Phil is in NZ and not looking at email hence putting this to the
 whole community
 --
 Prof Nicholas H. Keep
 Executive Dean of School of Science
 Professor of Biomolecular Science
 Crystallography, Institute for Structural and Molecular Biology,
 Department of Biological Sciences
 Birkbeck,  University of London,
 Malet Street,
 Bloomsbury
 LONDON
 WC1E 7HX

 email     n.k...@mail.cryst.bbk.ac.uk
 Telephone 020-7631-6852  (Room G57 Office)
          020-7631-6800  (Department Office)
 Fax       020-7631-6803
 If you want to access me in person you have to come to the crystallography
 entrance
 and ring me or the department office from the internal phone by the door


[ccp4bb] Merging hi and low res data

2012-02-07 Thread Nicholas Keep

We have a high and low res pass on a crystal that is going to around 1.2 A on 
the pilatus at Diamond.
We tried to scale them together using the xds CORRECT files from the xia2 -3dii runs (gives best results as far as we 
can see comparing xia2.html files)  through aimless via ccp4i interface.


The combined dataset looks worse (based on merging statistics at lower resolutions) than the individual particular at 
low res.(Compared to scala xia2 runs).

XDS does not record the pilatus as having overloads even on the high res pass 
using the correct detector parameters
http://xds.mpimf-heidelberg.mpg.de/html_doc/detectors.html

What I wanted to do was limit the resolution of the two runs ie to use low res between 100-2.5 beyond which completeness 
plummets and then the highres between 8 and 1.2.  Aimless does not seem to allow me to do that at least by the ccp4i 
interface- is this then bad practice?  I guess I could run a utility to remove the resolutions I don't want.


The one effect that makes me wonder if the lowres is better data, and worth combining, ie there is some overload in the 
high res, is that it goes from 30-8 mean I/SIGI whereas the high res has I/SIGI of 15 in most of the low res shells and 
then slowly falls off to 2.0 at 1.2 angstrom.


Am intending to carry on just using the hires pass.

Any comments?

I think Phil is in NZ and not looking at email hence putting this to the whole 
community
--
Prof Nicholas H. Keep
Executive Dean of School of Science
Professor of Biomolecular Science
Crystallography, Institute for Structural and Molecular Biology,
Department of Biological Sciences
Birkbeck,  University of London,
Malet Street,
Bloomsbury
LONDON
WC1E 7HX

email n.k...@mail.cryst.bbk.ac.uk
Telephone 020-7631-6852  (Room G57 Office)
  020-7631-6800  (Department Office)
Fax   020-7631-6803
If you want to access me in person you have to come to the crystallography 
entrance
and ring me or the department office from the internal phone by the door


Re: [ccp4bb] Merging hi and low res data

2012-02-07 Thread Kay Diederichs

Hi,

from your description I do not quite understand a few points:
- what is the evidence that there are overloads? The Pilatus pixels have 
20bits (meaning max contents of a pixel = 1048575) , and the OVERLOAD 
parameter is set to 1048500 (slightly below that). Do you actually see 
pixels with a contents of 1048575, but INTEGRATE and CORRECT do not 
report any overloads? That could indicate a bug in XDS, and I'd 
appreciate to receive a frame with overloads.
- could it be that the hi and the low resolution run do not scale well 
because they correspond to alternative indexing modes? - this can happen 
in spacegroups P3x, P4x, P6x and a couple of others. In that case you 
can use the XDS_ASCII.HKL of one run as a REFERENCE_DATA_SET for the 
other run, and run the CORRECT job - it will pick a consistent indexing.
- did you collect the hi res first, and the low res after that? If so, 
bad scaling is to be expected, because the crystal has a lot of 
radiation damage in the second run, resulting in elevated R values. It 
is much better to do the low res run first, producing very little 
radiation damage due to the much lower dose required.


best,
Kay