Re: [ccp4bb] Issues with latest XDS (20171218)

[James Holton]

Thank you Kay!

Speaking as a beamline scientist I agree that the instrument should be 
set up right and things like the distance, beam center, and various 
angles should be known.


However, also speaking as a beamline scientist I feel I should point out 
that EVERY instrument goes through at least one period where this is NOT 
the case!  Those of us who are responsible for putting all those 
properly-calibrated numbers into the image headers before your beam time 
starts really do appreciate the "feature" of broad radius of convergence 
on camera parameters.  But if you can get better data by turning it off 
once the parameters are known, that is really interesting too!


-James Holton
MAD Scientist

On 1/31/2018 1:07 AM, Kay Diederichs wrote:

Dear all,

a new BUILT of XDS is available for academic users at 
http://xds.mpimf-heidelberg.mpg.de . This reverts to the old distance 
refinement behavior with its larger radius of convergence, and is relevant for 
processing data from beamlines where the header distance is not accurate.

Thanks again to Oliver and Keitaro!

Kay

On Fri, 26 Jan 2018 16:56:30 +0900, Keitaro Yamashita 
 wrote:


Dear Kay,

I also tested this using a publicly available data of thaumatin
https://zenodo.org/record/10271
(resolution ~1.4 Å, wavelength= 0.97625 Å).
The camera distance from header is 265.27 mm. I tested this original
distance and some shifts (+1, +2, +4, ..., +32 mm) with different
versions of XDS.

As a result, IDXREF of the old version (170615) successfully refined
camera distance to ~265.0 mm if shift was less than or equal to +16
mm. The statistics in CORRECT.LP was as good as that without shift.

If the camera distance was fixed in IDXREF, statistics was
compromised, but the refinement in CORRECT worked well and recycling
of GXPARM.XDS improved the data quality.

However, the new version (170720 or later) changed camera distance
very little. Even with +1 mm shift the statistics was deteriorated,
and CORRECT (GXPARM.XDS) also didn't change the distance a lot, which
means the recycling of optimized parameters do not help. The new
version seems to stick to the given camera distance too much as
pointed out by many people.

I agree that camera distance in the header should be accurate.
However, I also wish to revert to the former behavior of XDS that
expires very soon.

Actually in previous versions (170615 or former) fixing POSITION in
REFINE(IDXREF) was recommended, but the recycling could give an
accurate distance. The current version seems to virtually have no
option to refine the distance.

Best regards,
Keitaro

On 24 January 2018 at 05:16, Kay Diederichs
 wrote:

Dear Oliver,

sorry for the trouble!
A default should be correct in the majority of situations, but it's impossible 
to have it work for _all_ situations. The XDS default for REFINE(IDXREF) was 
changed (i.e. POSITION was removed) to improve the indexing for weak and lousy 
data, _and_ because the distance values from the header are nowadays almost 
always accurate. For data with very high resolution such as yours, and 
significantly wrong distance, this means that at high resolution in particular 
this default will lead to worse data. generate_XDS.INP (in XDSwiki) and (I 
think) autoPROC and xia2 explicitly include POSITION in the REFINE(IDXREF), 
i.e. override the default. Where did you get XDS.INP from?
Some comments:
a) this is why I recommend, for data sets that may be important, to always do one cycle of 
optimization, i.e. after CORRECT, "mv GXPARM.XDS XPARM.XDS", change XDS.INP to have the 
correct high-resolution cutoff (cutting after the last shell which still has a "*" in the 
CC1/2 column), adjust the FRIEDEL'S_LAW= setting, and run JOB=INTEGRATE CORRECT. In your case this 
would make the refined distance available to INTEGRATE, and should result in very good data.
b) at which beamline did you collect the data? Such a high difference between 
refined distance and distance from header is unusual, and should be reported to 
(and fixed by) the beamline staff.
c) for this beamline at least, one should use REFINE(IDXREF)=CELL BEAM 
ORIENTATION AXIS POSITION . I understand that you tried this and it didn't 
work? Maybe there is something else wrong then, e.g. another line later in 
XDS.INP resetting REFINE(IDXREF) to something else.

If you cannot find the reason why including POSITiON into REFINE(IDXREF) does 
not help, pls send me enough data to reproduce the problem.

best wishes,

Kay

On Tue, 23 Jan 2018 14:31:09 +, "Weiergräber, Oliver H." 
 wrote:


Dear all,

After upgrading XDS from build date 20170601 to 20171218, I am experiencing 
severe degradation of apparent data quality reported by CORRECT for certain 
data sets. Following first indications of issues with a slightly problematic 
candidate, I went back to a previously very well-behaved test case with 
diffraction extending beyond 1.1 A. 

Re: [ccp4bb] Issues with latest XDS (20171218)

[Kay Diederichs]

Dear all,

a new BUILT of XDS is available for academic users at 
http://xds.mpimf-heidelberg.mpg.de . This reverts to the old distance 
refinement behavior with its larger radius of convergence, and is relevant for 
processing data from beamlines where the header distance is not accurate.

Thanks again to Oliver and Keitaro!

Kay

On Fri, 26 Jan 2018 16:56:30 +0900, Keitaro Yamashita 
 wrote:

>Dear Kay,
>
>I also tested this using a publicly available data of thaumatin
>https://zenodo.org/record/10271
>(resolution ~1.4 Å, wavelength= 0.97625 Å).
>The camera distance from header is 265.27 mm. I tested this original
>distance and some shifts (+1, +2, +4, ..., +32 mm) with different
>versions of XDS.
>
>As a result, IDXREF of the old version (170615) successfully refined
>camera distance to ~265.0 mm if shift was less than or equal to +16
>mm. The statistics in CORRECT.LP was as good as that without shift.
>
>If the camera distance was fixed in IDXREF, statistics was
>compromised, but the refinement in CORRECT worked well and recycling
>of GXPARM.XDS improved the data quality.
>
>However, the new version (170720 or later) changed camera distance
>very little. Even with +1 mm shift the statistics was deteriorated,
>and CORRECT (GXPARM.XDS) also didn't change the distance a lot, which
>means the recycling of optimized parameters do not help. The new
>version seems to stick to the given camera distance too much as
>pointed out by many people.
>
>I agree that camera distance in the header should be accurate.
>However, I also wish to revert to the former behavior of XDS that
>expires very soon.
>
>Actually in previous versions (170615 or former) fixing POSITION in
>REFINE(IDXREF) was recommended, but the recycling could give an
>accurate distance. The current version seems to virtually have no
>option to refine the distance.
>
>Best regards,
>Keitaro
>
>On 24 January 2018 at 05:16, Kay Diederichs
> wrote:
>> Dear Oliver,
>>
>> sorry for the trouble!
>> A default should be correct in the majority of situations, but it's 
>> impossible to have it work for _all_ situations. The XDS default for 
>> REFINE(IDXREF) was changed (i.e. POSITION was removed) to improve the 
>> indexing for weak and lousy data, _and_ because the distance values from the 
>> header are nowadays almost always accurate. For data with very high 
>> resolution such as yours, and significantly wrong distance, this means that 
>> at high resolution in particular this default will lead to worse data. 
>> generate_XDS.INP (in XDSwiki) and (I think) autoPROC and xia2 explicitly 
>> include POSITION in the REFINE(IDXREF), i.e. override the default. Where did 
>> you get XDS.INP from?
>> Some comments:
>> a) this is why I recommend, for data sets that may be important, to always 
>> do one cycle of optimization, i.e. after CORRECT, "mv GXPARM.XDS XPARM.XDS", 
>> change XDS.INP to have the correct high-resolution cutoff (cutting after the 
>> last shell which still has a "*" in the CC1/2 column), adjust the 
>> FRIEDEL'S_LAW= setting, and run JOB=INTEGRATE CORRECT. In your case this 
>> would make the refined distance available to INTEGRATE, and should result in 
>> very good data.
>> b) at which beamline did you collect the data? Such a high difference 
>> between refined distance and distance from header is unusual, and should be 
>> reported to (and fixed by) the beamline staff.
>> c) for this beamline at least, one should use REFINE(IDXREF)=CELL BEAM 
>> ORIENTATION AXIS POSITION . I understand that you tried this and it didn't 
>> work? Maybe there is something else wrong then, e.g. another line later in 
>> XDS.INP resetting REFINE(IDXREF) to something else.
>>
>> If you cannot find the reason why including POSITiON into REFINE(IDXREF) 
>> does not help, pls send me enough data to reproduce the problem.
>>
>> best wishes,
>>
>> Kay
>>
>> On Tue, 23 Jan 2018 14:31:09 +, "Weiergräber, Oliver H." 
>>  wrote:
>>
>>>Dear all,
>>>
>>>After upgrading XDS from build date 20170601 to 20171218, I am experiencing 
>>>severe degradation of apparent data quality reported by CORRECT for certain 
>>>data sets. Following first indications of issues with a slightly problematic 
>>>candidate, I went back to a previously very well-behaved test case with 
>>>diffraction extending beyond 1.1 A. Using the same input with both program 
>>>versions, statistics are not too different out to approx. 1.6 A, but become 
>>>more and more divergent in outer shells. These are the values for the 
>>>highest-resolution shell (1.10--1.16 A):
>>>
>>>20170601:  I/sigI = 1.80; Rmeas = 59.9 %; CC(1/2) = 82.0 %
>>>20171218:  I/sigI = 0.14; Rmeas = 506.4 %; CC(1/2) = 5.8 %
>>>
>>>The refined cell constants are unreasonably different as well. Obviously, 
>>>something is going awfully wrong here, presumably related to errors in 
>>>geometry parameters (which are expected to become increasingly 

Re: [ccp4bb] Issues with latest XDS (20171218)

[Kay Diederichs]

Dear Keitaro,

I have come to the exact same conclusions, and the next version of XDS will 
revert to the old refinement behavior. 

As a preview, I've uploaded a fixed Linux binary (not the whole package, just 
xds_par)  to 
ftp://turn5.biologie.uni-konstanz.de/pub/xds/xds_par.refi_cell_position_together.bz2
 .

Thanks - also to Oliver for reporting the problem!

Kay 

On Fri, 26 Jan 2018 16:56:30 +0900, Keitaro Yamashita 
 wrote:

>Dear Kay,
>
>I also tested this using a publicly available data of thaumatin
>https://zenodo.org/record/10271
>(resolution ~1.4 Å, wavelength= 0.97625 Å).
>The camera distance from header is 265.27 mm. I tested this original
>distance and some shifts (+1, +2, +4, ..., +32 mm) with different
>versions of XDS.
>
>As a result, IDXREF of the old version (170615) successfully refined
>camera distance to ~265.0 mm if shift was less than or equal to +16
>mm. The statistics in CORRECT.LP was as good as that without shift.
>
>If the camera distance was fixed in IDXREF, statistics was
>compromised, but the refinement in CORRECT worked well and recycling
>of GXPARM.XDS improved the data quality.
>
>However, the new version (170720 or later) changed camera distance
>very little. Even with +1 mm shift the statistics was deteriorated,
>and CORRECT (GXPARM.XDS) also didn't change the distance a lot, which
>means the recycling of optimized parameters do not help. The new
>version seems to stick to the given camera distance too much as
>pointed out by many people.
>
>I agree that camera distance in the header should be accurate.
>However, I also wish to revert to the former behavior of XDS that
>expires very soon.
>
>Actually in previous versions (170615 or former) fixing POSITION in
>REFINE(IDXREF) was recommended, but the recycling could give an
>accurate distance. The current version seems to virtually have no
>option to refine the distance.
>
>Best regards,
>Keitaro
>
>On 24 January 2018 at 05:16, Kay Diederichs
> wrote:
>> Dear Oliver,
>>
>> sorry for the trouble!
>> A default should be correct in the majority of situations, but it's 
>> impossible to have it work for _all_ situations. The XDS default for 
>> REFINE(IDXREF) was changed (i.e. POSITION was removed) to improve the 
>> indexing for weak and lousy data, _and_ because the distance values from the 
>> header are nowadays almost always accurate. For data with very high 
>> resolution such as yours, and significantly wrong distance, this means that 
>> at high resolution in particular this default will lead to worse data. 
>> generate_XDS.INP (in XDSwiki) and (I think) autoPROC and xia2 explicitly 
>> include POSITION in the REFINE(IDXREF), i.e. override the default. Where did 
>> you get XDS.INP from?
>> Some comments:
>> a) this is why I recommend, for data sets that may be important, to always 
>> do one cycle of optimization, i.e. after CORRECT, "mv GXPARM.XDS XPARM.XDS", 
>> change XDS.INP to have the correct high-resolution cutoff (cutting after the 
>> last shell which still has a "*" in the CC1/2 column), adjust the 
>> FRIEDEL'S_LAW= setting, and run JOB=INTEGRATE CORRECT. In your case this 
>> would make the refined distance available to INTEGRATE, and should result in 
>> very good data.
>> b) at which beamline did you collect the data? Such a high difference 
>> between refined distance and distance from header is unusual, and should be 
>> reported to (and fixed by) the beamline staff.
>> c) for this beamline at least, one should use REFINE(IDXREF)=CELL BEAM 
>> ORIENTATION AXIS POSITION . I understand that you tried this and it didn't 
>> work? Maybe there is something else wrong then, e.g. another line later in 
>> XDS.INP resetting REFINE(IDXREF) to something else.
>>
>> If you cannot find the reason why including POSITiON into REFINE(IDXREF) 
>> does not help, pls send me enough data to reproduce the problem.
>>
>> best wishes,
>>
>> Kay
>>
>> On Tue, 23 Jan 2018 14:31:09 +, "Weiergräber, Oliver H." 
>>  wrote:
>>
>>>Dear all,
>>>
>>>After upgrading XDS from build date 20170601 to 20171218, I am experiencing 
>>>severe degradation of apparent data quality reported by CORRECT for certain 
>>>data sets. Following first indications of issues with a slightly problematic 
>>>candidate, I went back to a previously very well-behaved test case with 
>>>diffraction extending beyond 1.1 A. Using the same input with both program 
>>>versions, statistics are not too different out to approx. 1.6 A, but become 
>>>more and more divergent in outer shells. These are the values for the 
>>>highest-resolution shell (1.10--1.16 A):
>>>
>>>20170601:  I/sigI = 1.80; Rmeas = 59.9 %; CC(1/2) = 82.0 %
>>>20171218:  I/sigI = 0.14; Rmeas = 506.4 %; CC(1/2) = 5.8 %
>>>
>>>The refined cell constants are unreasonably different as well. Obviously, 
>>>something is going awfully wrong here, presumably related to errors in 
>>>geometry parameters 

Re: [ccp4bb] Issues with latest XDS (20171218)

[Keitaro Yamashita]

Dear Kay,

I also tested this using a publicly available data of thaumatin
https://zenodo.org/record/10271
(resolution ~1.4 Å, wavelength= 0.97625 Å).
The camera distance from header is 265.27 mm. I tested this original
distance and some shifts (+1, +2, +4, ..., +32 mm) with different
versions of XDS.

As a result, IDXREF of the old version (170615) successfully refined
camera distance to ~265.0 mm if shift was less than or equal to +16
mm. The statistics in CORRECT.LP was as good as that without shift.

If the camera distance was fixed in IDXREF, statistics was
compromised, but the refinement in CORRECT worked well and recycling
of GXPARM.XDS improved the data quality.

However, the new version (170720 or later) changed camera distance
very little. Even with +1 mm shift the statistics was deteriorated,
and CORRECT (GXPARM.XDS) also didn't change the distance a lot, which
means the recycling of optimized parameters do not help. The new
version seems to stick to the given camera distance too much as
pointed out by many people.

I agree that camera distance in the header should be accurate.
However, I also wish to revert to the former behavior of XDS that
expires very soon.

Actually in previous versions (170615 or former) fixing POSITION in
REFINE(IDXREF) was recommended, but the recycling could give an
accurate distance. The current version seems to virtually have no
option to refine the distance.

Best regards,
Keitaro

On 24 January 2018 at 05:16, Kay Diederichs
 wrote:
> Dear Oliver,
>
> sorry for the trouble!
> A default should be correct in the majority of situations, but it's 
> impossible to have it work for _all_ situations. The XDS default for 
> REFINE(IDXREF) was changed (i.e. POSITION was removed) to improve the 
> indexing for weak and lousy data, _and_ because the distance values from the 
> header are nowadays almost always accurate. For data with very high 
> resolution such as yours, and significantly wrong distance, this means that 
> at high resolution in particular this default will lead to worse data. 
> generate_XDS.INP (in XDSwiki) and (I think) autoPROC and xia2 explicitly 
> include POSITION in the REFINE(IDXREF), i.e. override the default. Where did 
> you get XDS.INP from?
> Some comments:
> a) this is why I recommend, for data sets that may be important, to always do 
> one cycle of optimization, i.e. after CORRECT, "mv GXPARM.XDS XPARM.XDS", 
> change XDS.INP to have the correct high-resolution cutoff (cutting after the 
> last shell which still has a "*" in the CC1/2 column), adjust the 
> FRIEDEL'S_LAW= setting, and run JOB=INTEGRATE CORRECT. In your case this 
> would make the refined distance available to INTEGRATE, and should result in 
> very good data.
> b) at which beamline did you collect the data? Such a high difference between 
> refined distance and distance from header is unusual, and should be reported 
> to (and fixed by) the beamline staff.
> c) for this beamline at least, one should use REFINE(IDXREF)=CELL BEAM 
> ORIENTATION AXIS POSITION . I understand that you tried this and it didn't 
> work? Maybe there is something else wrong then, e.g. another line later in 
> XDS.INP resetting REFINE(IDXREF) to something else.
>
> If you cannot find the reason why including POSITiON into REFINE(IDXREF) does 
> not help, pls send me enough data to reproduce the problem.
>
> best wishes,
>
> Kay
>
> On Tue, 23 Jan 2018 14:31:09 +, "Weiergräber, Oliver H." 
>  wrote:
>
>>Dear all,
>>
>>After upgrading XDS from build date 20170601 to 20171218, I am experiencing 
>>severe degradation of apparent data quality reported by CORRECT for certain 
>>data sets. Following first indications of issues with a slightly problematic 
>>candidate, I went back to a previously very well-behaved test case with 
>>diffraction extending beyond 1.1 A. Using the same input with both program 
>>versions, statistics are not too different out to approx. 1.6 A, but become 
>>more and more divergent in outer shells. These are the values for the 
>>highest-resolution shell (1.10--1.16 A):
>>
>>20170601:  I/sigI = 1.80; Rmeas = 59.9 %; CC(1/2) = 82.0 %
>>20171218:  I/sigI = 0.14; Rmeas = 506.4 %; CC(1/2) = 5.8 %
>>
>>The refined cell constants are unreasonably different as well. Obviously, 
>>something is going awfully wrong here, presumably related to errors in 
>>geometry parameters (which are expected to become increasingly detrimental 
>>with decreasing dmin). As it turns out, geometry refinement behaves 
>>differently in the latest version of XDS: most notably, IDXREF no longer 
>>refines the detector distance by default. This is significant in this case, 
>>as in version 20170601 the distance would shift by as much as 1.3 mm, 
>>resulting in successful integration and scaling. Unfortunately, re-adding 
>>POSITION refinement into REFINE(IDXREF) does not help, and even having it in 
>>all refinement scopes (IDXREF, INTEGRATE, 

Re: [ccp4bb] Issues with latest XDS (20171218)

[Keller, Jacob]

How about including all of this stuff in refinement? Only adds another ~10-100 
[restrained] parameters…

JPK

+
Jacob Pearson Keller
Research Scientist / Looger Lab
HHMI Janelia Research Campus
19700 Helix Dr, Ashburn, VA 20147
Desk: (571)209-4000 x3159
Cell: (301)592-7004
+

The content of this email is confidential and intended for the recipient 
specified in message only. It is strictly forbidden to share any part of this 
message with any third party, without a written consent of the sender. If you 
received this message by mistake, please reply to this message and follow with 
its deletion, so that we can ensure such a mistake does not occur in the future.

From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of wtempel
Sent: Wednesday, January 24, 2018 12:19 PM
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] Issues with latest XDS (20171218)

I agree that accurate and complete image header information is useful, but not 
universal. How about a routine similar to 
dials.discover_better_experimental_model?
W.

On Wed, Jan 24, 2018 at 10:12 AM, Engin Özkan 
<eoz...@uchicago.edu<mailto:eoz...@uchicago.edu>> wrote:
Dear all,

I have to agree with all sentiments stated, and I have definitely seen the 
value of restraining distance values in weak and anisotropic data, which is 
great for those of us who deal with such data on a regular basis. However, I 
also just went through a search of all refined DISTANCE values on all (~20) 
CORRECT.LP files on my hard drive. Mismatches between input and refined values 
of up to 0.25% (1 mm in 400 mm) is not uncommon for data collected over several 
different beam lines. Many of us collect data at synchrotrons remotely and/or 
without ever talking to a beamline scientists, and expecting users to end up 
with perfect distance values in their auto-scripted XDS.INP files might be 
expecting the perfect in an imperfect, fast-moving data-collection world. A 
solution that accommodates both viewpoints, if at all possible, would indeed be 
great.

Best,

Engin


On 1/24/18 8:06 AM, "Weiergräber, Oliver H." wrote:
Dear Clemens, dear Kay,

I absolutely agree with your statement regarding responsibilities. 
Unfortunately, for a user of a synchrotron beamline, it is hardly possible to 
_know_ that the distance (or any beam or camera parameter) provided by the 
beamline software is in error. In the end indications can only be derived from 
the behaviour of software used for processing. Of course developers are not 
responsible for hardware issues, but since such issues do occur in real life, 
the software may still help spotting them ;-)
Given that IDXREF runs very fast, why not have it probe the shifts of 
parameters for a full refinement scope and issue a warning if things look 
suspicious, giving the user options how to proceed (similar to the 
not-so-infrequent case of "insufficient percentage of indexed reflections").

Best regards
Oliver


   PD Dr. Oliver H. Weiergräber
   Institute of Complex Systems
   ICS-6: Structural Biochemistry
   Tel.: +49 2461 61-2028<tel:%2B49%202461%2061-2028>
   Fax: +49 2461 61-9540<tel:%2B49%202461%2061-9540>





From: CCP4 bulletin board [CCP4BB@JISCMAIL.AC.UK<mailto:CCP4BB@JISCMAIL.AC.UK>] 
on behalf of Clemens Vonrhein 
[vonrh...@globalphasing.com<mailto:vonrh...@globalphasing.com>]
Sent: Wednesday, January 24, 2018 2:39 PM
To: CCP4BB@JISCMAIL.AC.UK<mailto:CCP4BB@JISCMAIL.AC.UK>
Subject: Re: [ccp4bb] Issues with latest XDS (20171218)

Dear Oliver,

yes, there are other changes to the parameter refinement procedure
within XDS as far as I understand. These were introduced to robustify
XDS processing (and parameter refinement) when encountering very poor
datasets, cases where the crystal moves out of the beam in certain
orientations or empty loops. That works very well it seems - at least
for us (and we were involved in discussing those cases with both
Wolfgang Kabsch and Kay Diederichs).

I'm sure you agree that if the crystal to detector distance in the
header is wrong by such a large amount (over 1 mm), the (proper)
solution is to fix this at the point of collecting the data and
writing the header. As Kay says: there should be no reason for an
instrument/beamline to not know that distance very accurately and to
write the correct value in the header. It has the same importance as
e.g. wavelength/energy or oscillation range.

Of course, if a user already has images with an inaccurate value and
wants to process those, the old behaviour of XDS might be able to fix
that particular problem for you. But this does feel like patching up
simple-to-fix problems at the wrong stage, namely processing instead
of at the beamline side ... wit

Re: [ccp4bb] Issues with latest XDS (20171218)

[wtempel]

I agree that accurate and complete image header information is useful, but
not universal. How about a routine similar to dials.discover_better_
experimental_model?
W.

On Wed, Jan 24, 2018 at 10:12 AM, Engin Özkan <eoz...@uchicago.edu> wrote:

> Dear all,
>
> I have to agree with all sentiments stated, and I have definitely seen the
> value of restraining distance values in weak and anisotropic data, which is
> great for those of us who deal with such data on a regular basis. However,
> I also just went through a search of all refined DISTANCE values on all
> (~20) CORRECT.LP files on my hard drive. Mismatches between input and
> refined values of up to 0.25% (1 mm in 400 mm) is not uncommon for data
> collected over several different beam lines. Many of us collect data at
> synchrotrons remotely and/or without ever talking to a beamline scientists,
> and expecting users to end up with perfect distance values in their
> auto-scripted XDS.INP files might be expecting the perfect in an imperfect,
> fast-moving data-collection world. A solution that accommodates both
> viewpoints, if at all possible, would indeed be great.
>
> Best,
>
> Engin
>
>
> On 1/24/18 8:06 AM, "Weiergräber, Oliver H." wrote:
>
>> Dear Clemens, dear Kay,
>>
>> I absolutely agree with your statement regarding responsibilities.
>> Unfortunately, for a user of a synchrotron beamline, it is hardly possible
>> to _know_ that the distance (or any beam or camera parameter) provided by
>> the beamline software is in error. In the end indications can only be
>> derived from the behaviour of software used for processing. Of course
>> developers are not responsible for hardware issues, but since such issues
>> do occur in real life, the software may still help spotting them ;-)
>> Given that IDXREF runs very fast, why not have it probe the shifts of
>> parameters for a full refinement scope and issue a warning if things look
>> suspicious, giving the user options how to proceed (similar to the
>> not-so-infrequent case of "insufficient percentage of indexed reflections").
>>
>> Best regards
>> Oliver
>>
>> 
>>PD Dr. Oliver H. Weiergräber
>>Institute of Complex Systems
>>ICS-6: Structural Biochemistry
>>Tel.: +49 2461 61-2028
>>Fax: +49 2461 61-9540
>> 
>>
>>
>>
>> ____________
>> From: CCP4 bulletin board [CCP4BB@JISCMAIL.AC.UK] on behalf of Clemens
>> Vonrhein [vonrh...@globalphasing.com]
>> Sent: Wednesday, January 24, 2018 2:39 PM
>> To: CCP4BB@JISCMAIL.AC.UK
>> Subject: Re: [ccp4bb] Issues with latest XDS (20171218)
>>
>> Dear Oliver,
>>
>> yes, there are other changes to the parameter refinement procedure
>> within XDS as far as I understand. These were introduced to robustify
>> XDS processing (and parameter refinement) when encountering very poor
>> datasets, cases where the crystal moves out of the beam in certain
>> orientations or empty loops. That works very well it seems - at least
>> for us (and we were involved in discussing those cases with both
>> Wolfgang Kabsch and Kay Diederichs).
>>
>> I'm sure you agree that if the crystal to detector distance in the
>> header is wrong by such a large amount (over 1 mm), the (proper)
>> solution is to fix this at the point of collecting the data and
>> writing the header. As Kay says: there should be no reason for an
>> instrument/beamline to not know that distance very accurately and to
>> write the correct value in the header. It has the same importance as
>> e.g. wavelength/energy or oscillation range.
>>
>> Of course, if a user already has images with an inaccurate value and
>> wants to process those, the old behaviour of XDS might be able to fix
>> that particular problem for you. But this does feel like patching up
>> simple-to-fix problems at the wrong stage, namely processing instead
>> of at the beamline side ... with the "danger" that they will never get
>> fixed properly because processing software will "somehow patch things
>> up anyway".
>>
>> All those software packages do already a very large amount of
>> workarounds because of real life being what it is, but if we want to
>> achieve the highest quality of processed data I think we can expect
>> the same amount of accuracy when it comes to the description of the
>> experiment that goes into programs like XDS.
>>
>> I think the current XDS behaviour is much more 

Re: [ccp4bb] Issues with latest XDS (20171218)

[Engin Özkan]

Dear all,

I have to agree with all sentiments stated, and I have definitely seen 
the value of restraining distance values in weak and anisotropic data, 
which is great for those of us who deal with such data on a regular 
basis. However, I also just went through a search of all refined 
DISTANCE values on all (~20) CORRECT.LP files on my hard drive. 
Mismatches between input and refined values of up to 0.25% (1 mm in 400 
mm) is not uncommon for data collected over several different beam 
lines. Many of us collect data at synchrotrons remotely and/or without 
ever talking to a beamline scientists, and expecting users to end up 
with perfect distance values in their auto-scripted XDS.INP files might 
be expecting the perfect in an imperfect, fast-moving data-collection 
world. A solution that accommodates both viewpoints, if at all possible, 
would indeed be great.


Best,

Engin


On 1/24/18 8:06 AM, "Weiergräber, Oliver H." wrote:

Dear Clemens, dear Kay,

I absolutely agree with your statement regarding responsibilities. 
Unfortunately, for a user of a synchrotron beamline, it is hardly possible to 
_know_ that the distance (or any beam or camera parameter) provided by the 
beamline software is in error. In the end indications can only be derived from 
the behaviour of software used for processing. Of course developers are not 
responsible for hardware issues, but since such issues do occur in real life, 
the software may still help spotting them ;-)
Given that IDXREF runs very fast, why not have it probe the shifts of parameters for a 
full refinement scope and issue a warning if things look suspicious, giving the user 
options how to proceed (similar to the not-so-infrequent case of "insufficient 
percentage of indexed reflections").

Best regards
Oliver


   PD Dr. Oliver H. Weiergräber
   Institute of Complex Systems
   ICS-6: Structural Biochemistry
   Tel.: +49 2461 61-2028
   Fax: +49 2461 61-9540





From: CCP4 bulletin board [CCP4BB@JISCMAIL.AC.UK] on behalf of Clemens Vonrhein 
[vonrh...@globalphasing.com]
Sent: Wednesday, January 24, 2018 2:39 PM
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] Issues with latest XDS (20171218)

Dear Oliver,

yes, there are other changes to the parameter refinement procedure
within XDS as far as I understand. These were introduced to robustify
XDS processing (and parameter refinement) when encountering very poor
datasets, cases where the crystal moves out of the beam in certain
orientations or empty loops. That works very well it seems - at least
for us (and we were involved in discussing those cases with both
Wolfgang Kabsch and Kay Diederichs).

I'm sure you agree that if the crystal to detector distance in the
header is wrong by such a large amount (over 1 mm), the (proper)
solution is to fix this at the point of collecting the data and
writing the header. As Kay says: there should be no reason for an
instrument/beamline to not know that distance very accurately and to
write the correct value in the header. It has the same importance as
e.g. wavelength/energy or oscillation range.

Of course, if a user already has images with an inaccurate value and
wants to process those, the old behaviour of XDS might be able to fix
that particular problem for you. But this does feel like patching up
simple-to-fix problems at the wrong stage, namely processing instead
of at the beamline side ... with the "danger" that they will never get
fixed properly because processing software will "somehow patch things
up anyway".

All those software packages do already a very large amount of
workarounds because of real life being what it is, but if we want to
achieve the highest quality of processed data I think we can expect
the same amount of accuracy when it comes to the description of the
experiment that goes into programs like XDS.

I think the current XDS behaviour is much more robust and reliable
_if_ the experiment is described accurately. The latter is the
responsibility of the beamline (image headers etc) and the user. It
should not be XDS's task to calibrate the instrument parameters
(against a single sweep dataset from whatever crystal it sees), but
rather index, integrate and scale/merge the data collected and
described.

Anyway, just my 2 cents ;-)

Cheers

Clemens

On Wed, Jan 24, 2018 at 09:36:19AM +, "Weiergräber, Oliver H." wrote:

a) Recycling of geometry parameters is certainly worth trying, although in our 
experience it rarely yields large improvements. Obviously, XDS used to do a 
pretty good job in default mode. In the latest version, however, since 
refinement of the detector distance is disabled by default, recycling cannot be 
expected to (and indeed does not) help.
b) The data I mentioned have been collected at ESRF ID29 and processed using 
the XDS.INP file 

Re: [ccp4bb] Issues with latest XDS (20171218)

[Kay Diederichs]

Dear Georg,


On Wed, 24 Jan 2018 14:38:09 +0100, Georg Mlynek  
wrote:

>Dear Kay, thank you ver much for the (as always) detailed and nicely
>explained answer.
>
>However this brings up some questions for me:
>
>1. Could you please tell me how the "correct high-resolution cutoff"
>will effect the data processing in the INTEGRATE CORRECT step.
>
>In other words what will be the difference if I leave
>INCLUDE_RESOLUTION_RANGE=50.0 0.0

a) it will take a bit longer to process the data
b) the statistics, in particular the overall statistics, will not be meaningful 
since they will include shells that one is not interested in

I'm not saying that overall statistics are necessarily useful, but to get an 
idea about the data quality, it is better to have statistics for 9 resolution 
shells to look at, than just 3 or 4 or 5!
Concerning data quality, the results in the good resolution should not be 
different whether you also integrate higher shells without signal, or not.

Personally, I always use INCLUDE_RESOLUTION_RANGE=X 0.0 in the first run, where 
X depends on beamstop shadow size and position. And I always mask the beamstop 
unless the shadow is circular and centered.
In the optimization run, I adjust the high resolution cutoff, and FRIEDEL'S_LAW 
depending on what I find in the first run.

>
>2. Additionally if I leave FRIEDEL'S_LAW= TRUE in XDS.INP and choose
>FRIEDEL'S_LAW= FALSE in XDSCONV.INP will the results be different?
>

if there actually _is_ anomalous signal but you have FRIEDEL'S_LAW= TRUE in 
XDS.INP, then you lie to the program, and you risk rejection of strong 
anomalous differences as outliers. Conversely, if you have FRIEDEL'S_LAW= FALSE 
in XDS but there is no anomalous signal, the statistics are less useful, and 
the outlier rejection is less efficient. 

>3. With all the optimizations done in the last years are these tips and
>tricks still valid?
>
>FRIEDEL'S_LAW=FALSE ! This acts only on the CORRECT step ! If the anom
>signal turns out to be, or is known to be, very low or absent, ! use
>FRIEDEL'S_LAW=TRUE instead (or comment out the line); re-run CORRECT !
>remove the "!" in the following line: !

This should not be seen as a trick, rather as saying "just use what is adequate 
for the situation"

>STRICT_ABSORPTION_CORRECTION=TRUE ! if the anomalous signal is strong:
>in that case, in CORRECT.LP the three ! "CHI^2-VALUE OF FIT OF
>CORRECTION FACTORS" values are significantly> 1, e.g. 1.5 (from
>https://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/2QVO.xds)

this is what I do but it might not make a big difference in practice. But to 
see CHI^2 come down to 1 with STRICT_ABSORPTION_CORRECTION=TRUE when it is 2 
with STRICT_ABSORPTION_CORRECTION=FALSE tells me that it is actually the 
anomalous signal that was responsible for the elevated differences between 
sym-related observations.

I understand your questions in a context of streamlined data processing, and 
economizing efforts and time. On the other hand, questions such as 

 - why not try make the most of your data? Sometimes this might tip the scale 
in your favor, and allow to solve a difficult structure - but you will never 
know unless you try 
 - by _not_ adapting XDS.INP to the actual data, and running a second INTEGRATE 
CORRECT, can one really save time in the long run? After all, getting the 
crystals was worth your time, and that probably took much longer.

must then also be answered.

best wishes,

Kay




>
>Many thanks in advance, best regards, Georg.
>
>Am 23.01.2018 um 21:16 schrieb Kay Diederichs:
>> Dear Oliver,
>>
>> sorry for the trouble!
>> A default should be correct in the majority of situations, but it's 
>> impossible to have it work for _all_ situations. The XDS default for 
>> REFINE(IDXREF) was changed (i.e. POSITION was removed) to improve the 
>> indexing for weak and lousy data, _and_ because the distance values from the 
>> header are nowadays almost always accurate. For data with very high 
>> resolution such as yours, and significantly wrong distance, this means that 
>> at high resolution in particular this default will lead to worse data. 
>> generate_XDS.INP (in XDSwiki) and (I think) autoPROC and xia2 explicitly 
>> include POSITION in the REFINE(IDXREF), i.e. override the default. Where did 
>> you get XDS.INP from?
>> Some comments:
>> a) this is why I recommend, for data sets that may be important, to always 
>> do one cycle of optimization, i.e. after CORRECT, "mv GXPARM.XDS XPARM.XDS", 
>> change XDS.INP to have the correct high-resolution cutoff (cutting after the 
>> last shell which still has a "*" in the CC1/2 column), adjust the 
>> FRIEDEL'S_LAW= setting, and run JOB=INTEGRATE CORRECT. In your case this 
>> would make the refined distance available to INTEGRATE, and should result in 
>> very good data.
>> b) at which beamline did you collect the data? Such a high difference 
>> between refined distance and distance from header is unusual, and 

Re: [ccp4bb] Issues with latest XDS (20171218)

[Weiergräber, Oliver H.]

Dear Clemens, dear Kay,

I absolutely agree with your statement regarding responsibilities. 
Unfortunately, for a user of a synchrotron beamline, it is hardly possible to 
_know_ that the distance (or any beam or camera parameter) provided by the 
beamline software is in error. In the end indications can only be derived from 
the behaviour of software used for processing. Of course developers are not 
responsible for hardware issues, but since such issues do occur in real life, 
the software may still help spotting them ;-)
Given that IDXREF runs very fast, why not have it probe the shifts of 
parameters for a full refinement scope and issue a warning if things look 
suspicious, giving the user options how to proceed (similar to the 
not-so-infrequent case of "insufficient percentage of indexed reflections").

Best regards
Oliver


  PD Dr. Oliver H. Weiergräber
  Institute of Complex Systems
  ICS-6: Structural Biochemistry
  Tel.: +49 2461 61-2028
  Fax: +49 2461 61-9540





From: CCP4 bulletin board [CCP4BB@JISCMAIL.AC.UK] on behalf of Clemens Vonrhein 
[vonrh...@globalphasing.com]
Sent: Wednesday, January 24, 2018 2:39 PM
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] Issues with latest XDS (20171218)

Dear Oliver,

yes, there are other changes to the parameter refinement procedure
within XDS as far as I understand. These were introduced to robustify
XDS processing (and parameter refinement) when encountering very poor
datasets, cases where the crystal moves out of the beam in certain
orientations or empty loops. That works very well it seems - at least
for us (and we were involved in discussing those cases with both
Wolfgang Kabsch and Kay Diederichs).

I'm sure you agree that if the crystal to detector distance in the
header is wrong by such a large amount (over 1 mm), the (proper)
solution is to fix this at the point of collecting the data and
writing the header. As Kay says: there should be no reason for an
instrument/beamline to not know that distance very accurately and to
write the correct value in the header. It has the same importance as
e.g. wavelength/energy or oscillation range.

Of course, if a user already has images with an inaccurate value and
wants to process those, the old behaviour of XDS might be able to fix
that particular problem for you. But this does feel like patching up
simple-to-fix problems at the wrong stage, namely processing instead
of at the beamline side ... with the "danger" that they will never get
fixed properly because processing software will "somehow patch things
up anyway".

All those software packages do already a very large amount of
workarounds because of real life being what it is, but if we want to
achieve the highest quality of processed data I think we can expect
the same amount of accuracy when it comes to the description of the
experiment that goes into programs like XDS.

I think the current XDS behaviour is much more robust and reliable
_if_ the experiment is described accurately. The latter is the
responsibility of the beamline (image headers etc) and the user. It
should not be XDS's task to calibrate the instrument parameters
(against a single sweep dataset from whatever crystal it sees), but
rather index, integrate and scale/merge the data collected and
described.

Anyway, just my 2 cents ;-)

Cheers

Clemens

On Wed, Jan 24, 2018 at 09:36:19AM +, "Weiergräber, Oliver H." wrote:
> a) Recycling of geometry parameters is certainly worth trying, although in 
> our experience it rarely yields large improvements. Obviously, XDS used to do 
> a pretty good job in default mode. In the latest version, however, since 
> refinement of the detector distance is disabled by default, recycling cannot 
> be expected to (and indeed does not) help.
> b) The data I mentioned have been collected at ESRF ID29 and processed using 
> the XDS.INP file they provided, with slight modifications unrelated to 
> geometry refinement.
> c) Yes, defining REFINE(IDXREF) to include POSITION does not solve the 
> problem. It seems that refinement of the distance is still strongly 
> restrained, so there must have been changes to the code other than removal of 
> POSITION refinement from the REFINE(IDXREF) defaults. Consequently, parameter 
> recycling does not help much even with POSITION refinement re-enabled in 
> IDXREF.
>
> The bottom line is that there appears to be no obvious way to restore the 
> previous behaviour by just changing parameters (but of course I may have 
> missed something). I think such an option is urgently required. The most 
> worrisome aspect about this issue, in my opinion, is that it may go unnoticed 
> in many cases. The way it stands now, people affected by inaccuracies in 
> detector distance (or maybe 

Re: [ccp4bb] Issues with latest XDS (20171218)

[Clemens Vonrhein]

Dear Oliver,

yes, there are other changes to the parameter refinement procedure
within XDS as far as I understand. These were introduced to robustify
XDS processing (and parameter refinement) when encountering very poor
datasets, cases where the crystal moves out of the beam in certain
orientations or empty loops. That works very well it seems - at least
for us (and we were involved in discussing those cases with both
Wolfgang Kabsch and Kay Diederichs).

I'm sure you agree that if the crystal to detector distance in the
header is wrong by such a large amount (over 1 mm), the (proper)
solution is to fix this at the point of collecting the data and
writing the header. As Kay says: there should be no reason for an
instrument/beamline to not know that distance very accurately and to
write the correct value in the header. It has the same importance as
e.g. wavelength/energy or oscillation range.

Of course, if a user already has images with an inaccurate value and
wants to process those, the old behaviour of XDS might be able to fix
that particular problem for you. But this does feel like patching up
simple-to-fix problems at the wrong stage, namely processing instead
of at the beamline side ... with the "danger" that they will never get
fixed properly because processing software will "somehow patch things
up anyway".

All those software packages do already a very large amount of
workarounds because of real life being what it is, but if we want to
achieve the highest quality of processed data I think we can expect
the same amount of accuracy when it comes to the description of the
experiment that goes into programs like XDS.

I think the current XDS behaviour is much more robust and reliable
_if_ the experiment is described accurately. The latter is the
responsibility of the beamline (image headers etc) and the user. It
should not be XDS's task to calibrate the instrument parameters
(against a single sweep dataset from whatever crystal it sees), but
rather index, integrate and scale/merge the data collected and
described.

Anyway, just my 2 cents ;-)

Cheers

Clemens

On Wed, Jan 24, 2018 at 09:36:19AM +, "Weiergräber, Oliver H." wrote:
> a) Recycling of geometry parameters is certainly worth trying, although in 
> our experience it rarely yields large improvements. Obviously, XDS used to do 
> a pretty good job in default mode. In the latest version, however, since 
> refinement of the detector distance is disabled by default, recycling cannot 
> be expected to (and indeed does not) help.
> b) The data I mentioned have been collected at ESRF ID29 and processed using 
> the XDS.INP file they provided, with slight modifications unrelated to 
> geometry refinement.
> c) Yes, defining REFINE(IDXREF) to include POSITION does not solve the 
> problem. It seems that refinement of the distance is still strongly 
> restrained, so there must have been changes to the code other than removal of 
> POSITION refinement from the REFINE(IDXREF) defaults. Consequently, parameter 
> recycling does not help much even with POSITION refinement re-enabled in 
> IDXREF.
>
> The bottom line is that there appears to be no obvious way to restore the 
> previous behaviour by just changing parameters (but of course I may have 
> missed something). I think such an option is urgently required. The most 
> worrisome aspect about this issue, in my opinion, is that it may go unnoticed 
> in many cases. The way it stands now, people affected by inaccuracies in 
> detector distance (or maybe other parameters) may be misled to choose 
> cut-offs for integration at much too high dmin, discarding valuable data.
>
> I am happy to provide data needed for investigation. Let's discuss this part 
> off-list.
>
> Best regards
> Oliver
>
> 
>   PD Dr. Oliver H. Weiergräber
>   Institute of Complex Systems
>   ICS-6: Structural Biochemistry
>   Tel.: +49 2461 61-2028
>   Fax: +49 2461 61-9540
> 
>
>
>
> 
> From: CCP4 bulletin board [CCP4BB@JISCMAIL.AC.UK] on behalf of Kay Diederichs 
> [kay.diederi...@uni-konstanz.de]
> Sent: Tuesday, January 23, 2018 9:16 PM
> To: CCP4BB@JISCMAIL.AC.UK
> Subject: Re: [ccp4bb] Issues with latest XDS (20171218)
>
> Dear Oliver,
>
> sorry for the trouble!
> A default should be correct in the majority of situations, but it's 
> impossible to have it work for _all_ situations. The XDS default for 
> REFINE(IDXREF) was changed (i.e. POSITION was removed) to improve the 
> indexing for weak and lousy data, _and_ because the distance values from the 
> header are nowadays almost always accurate. For data with very high 
> resolution such as yours, and significantly wrong distance, this means 

Re: [ccp4bb] Issues with latest XDS (20171218)

[Georg Mlynek]

Dear Kay, thank you ver much for the (as always) detailed and nicely 
explained answer.


However this brings up some questions for me:

1. Could you please tell me how the "correct high-resolution cutoff" 
will effect the data processing in the INTEGRATE CORRECT step.


In other words what will be the difference if I leave 
INCLUDE_RESOLUTION_RANGE=50.0 0.0


2. Additionally if I leave FRIEDEL'S_LAW= TRUE in XDS.INP and choose 
FRIEDEL'S_LAW= FALSE in XDSCONV.INP will the results be different?


3. With all the optimizations done in the last years are these tips and 
tricks still valid?


FRIEDEL'S_LAW=FALSE ! This acts only on the CORRECT step ! If the anom 
signal turns out to be, or is known to be, very low or absent, ! use 
FRIEDEL'S_LAW=TRUE instead (or comment out the line); re-run CORRECT ! 
remove the "!" in the following line: ! 
STRICT_ABSORPTION_CORRECTION=TRUE ! if the anomalous signal is strong: 
in that case, in CORRECT.LP the three ! "CHI^2-VALUE OF FIT OF 
CORRECTION FACTORS" values are significantly> 1, e.g. 1.5 (from 
https://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/2QVO.xds)


Many thanks in advance, best regards, Georg.

Am 23.01.2018 um 21:16 schrieb Kay Diederichs:

Dear Oliver,

sorry for the trouble!
A default should be correct in the majority of situations, but it's impossible 
to have it work for _all_ situations. The XDS default for REFINE(IDXREF) was 
changed (i.e. POSITION was removed) to improve the indexing for weak and lousy 
data, _and_ because the distance values from the header are nowadays almost 
always accurate. For data with very high resolution such as yours, and 
significantly wrong distance, this means that at high resolution in particular 
this default will lead to worse data. generate_XDS.INP (in XDSwiki) and (I 
think) autoPROC and xia2 explicitly include POSITION in the REFINE(IDXREF), 
i.e. override the default. Where did you get XDS.INP from?
Some comments:
a) this is why I recommend, for data sets that may be important, to always do one cycle of 
optimization, i.e. after CORRECT, "mv GXPARM.XDS XPARM.XDS", change XDS.INP to have the 
correct high-resolution cutoff (cutting after the last shell which still has a "*" in the 
CC1/2 column), adjust the FRIEDEL'S_LAW= setting, and run JOB=INTEGRATE CORRECT. In your case this 
would make the refined distance available to INTEGRATE, and should result in very good data.
b) at which beamline did you collect the data? Such a high difference between 
refined distance and distance from header is unusual, and should be reported to 
(and fixed by) the beamline staff.
c) for this beamline at least, one should use REFINE(IDXREF)=CELL BEAM 
ORIENTATION AXIS POSITION . I understand that you tried this and it didn't 
work? Maybe there is something else wrong then, e.g. another line later in 
XDS.INP resetting REFINE(IDXREF) to something else.

If you cannot find the reason why including POSITiON into REFINE(IDXREF) does 
not help, pls send me enough data to reproduce the problem.

best wishes,

Kay

On Tue, 23 Jan 2018 14:31:09 +, "Weiergräber, Oliver H." 
 wrote:


Dear all,

After upgrading XDS from build date 20170601 to 20171218, I am experiencing 
severe degradation of apparent data quality reported by CORRECT for certain 
data sets. Following first indications of issues with a slightly problematic 
candidate, I went back to a previously very well-behaved test case with 
diffraction extending beyond 1.1 A. Using the same input with both program 
versions, statistics are not too different out to approx. 1.6 A, but become 
more and more divergent in outer shells. These are the values for the 
highest-resolution shell (1.10--1.16 A):

20170601:  I/sigI = 1.80; Rmeas = 59.9 %; CC(1/2) = 82.0 %
20171218:  I/sigI = 0.14; Rmeas = 506.4 %; CC(1/2) = 5.8 %

The refined cell constants are unreasonably different as well. Obviously, 
something is going awfully wrong here, presumably related to errors in geometry 
parameters (which are expected to become increasingly detrimental with 
decreasing dmin). As it turns out, geometry refinement behaves differently in 
the latest version of XDS: most notably, IDXREF no longer refines the detector 
distance by default. This is significant in this case, as in version 20170601 
the distance would shift by as much as 1.3 mm, resulting in successful 
integration and scaling. Unfortunately, re-adding POSITION refinement into 
REFINE(IDXREF) does not help, and even having it in all refinement scopes 
(IDXREF, INTEGRATE, CORRECT) only yields a limited improvement of CORRECT 
statistics.
It is worth noting that examination of a dataset from an unrelated crystal 
(dmin = 1.4 A) did not reveal such enormous differences -- in this case, 
however, the shift in detector distance applied in the old version of XDS was 
quite small (0.2 mm), which, together with the lower resolution, explains the 
absence of large discrepancies.

To sum up, I suspect that, due 

Re: [ccp4bb] Issues with latest XDS (20171218)

[Weiergräber, Oliver H.]

Dear Kay,

Thanks for the quick reply. Let me answer your questions:
a) Recycling of geometry parameters is certainly worth trying, although in our 
experience it rarely yields large improvements. Obviously, XDS used to do a 
pretty good job in default mode. In the latest version, however, since 
refinement of the detector distance is disabled by default, recycling cannot be 
expected to (and indeed does not) help.
b) The data I mentioned have been collected at ESRF ID29 and processed using 
the XDS.INP file they provided, with slight modifications unrelated to geometry 
refinement.
c) Yes, defining REFINE(IDXREF) to include POSITION does not solve the problem. 
It seems that refinement of the distance is still strongly restrained, so there 
must have been changes to the code other than removal of POSITION refinement 
from the REFINE(IDXREF) defaults. Consequently, parameter recycling does not 
help much even with POSITION refinement re-enabled in IDXREF.

The bottom line is that there appears to be no obvious way to restore the 
previous behaviour by just changing parameters (but of course I may have missed 
something). I think such an option is urgently required. The most worrisome 
aspect about this issue, in my opinion, is that it may go unnoticed in many 
cases. The way it stands now, people affected by inaccuracies in detector 
distance (or maybe other parameters) may be misled to choose cut-offs for 
integration at much too high dmin, discarding valuable data. 

I am happy to provide data needed for investigation. Let's discuss this part 
off-list.

Best regards
Oliver


  PD Dr. Oliver H. Weiergräber
  Institute of Complex Systems
  ICS-6: Structural Biochemistry
  Tel.: +49 2461 61-2028
  Fax: +49 2461 61-9540





From: CCP4 bulletin board [CCP4BB@JISCMAIL.AC.UK] on behalf of Kay Diederichs 
[kay.diederi...@uni-konstanz.de]
Sent: Tuesday, January 23, 2018 9:16 PM
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] Issues with latest XDS (20171218)

Dear Oliver,

sorry for the trouble!
A default should be correct in the majority of situations, but it's impossible 
to have it work for _all_ situations. The XDS default for REFINE(IDXREF) was 
changed (i.e. POSITION was removed) to improve the indexing for weak and lousy 
data, _and_ because the distance values from the header are nowadays almost 
always accurate. For data with very high resolution such as yours, and 
significantly wrong distance, this means that at high resolution in particular 
this default will lead to worse data. generate_XDS.INP (in XDSwiki) and (I 
think) autoPROC and xia2 explicitly include POSITION in the REFINE(IDXREF), 
i.e. override the default. Where did you get XDS.INP from?
Some comments:
a) this is why I recommend, for data sets that may be important, to always do 
one cycle of optimization, i.e. after CORRECT, "mv GXPARM.XDS XPARM.XDS", 
change XDS.INP to have the correct high-resolution cutoff (cutting after the 
last shell which still has a "*" in the CC1/2 column), adjust the 
FRIEDEL'S_LAW= setting, and run JOB=INTEGRATE CORRECT. In your case this would 
make the refined distance available to INTEGRATE, and should result in very 
good data.
b) at which beamline did you collect the data? Such a high difference between 
refined distance and distance from header is unusual, and should be reported to 
(and fixed by) the beamline staff.
c) for this beamline at least, one should use REFINE(IDXREF)=CELL BEAM 
ORIENTATION AXIS POSITION . I understand that you tried this and it didn't 
work? Maybe there is something else wrong then, e.g. another line later in 
XDS.INP resetting REFINE(IDXREF) to something else.

If you cannot find the reason why including POSITiON into REFINE(IDXREF) does 
not help, pls send me enough data to reproduce the problem.

best wishes,

Kay

On Tue, 23 Jan 2018 14:31:09 +, "Weiergräber, Oliver H." 
<o.h.weiergrae...@fz-juelich.de> wrote:

>Dear all,
>
>After upgrading XDS from build date 20170601 to 20171218, I am experiencing 
>severe degradation of apparent data quality reported by CORRECT for certain 
>data sets. Following first indications of issues with a slightly problematic 
>candidate, I went back to a previously very well-behaved test case with 
>diffraction extending beyond 1.1 A. Using the same input with both program 
>versions, statistics are not too different out to approx. 1.6 A, but become 
>more and more divergent in outer shells. These are the values for the 
>highest-resolution shell (1.10--1.16 A):
>
>20170601:  I/sigI = 1.80; Rmeas = 59.9 %; CC(1/2) = 82.0 %
>20171218:  I/sigI = 0.14; Rmeas = 506.4 %; CC(1/2) = 5.8 %
>
>The refined cell constants are unreasonably different as well. Obviously, 
>something is going awfully wrong here, presumably related to er

Re: [ccp4bb] Issues with latest XDS (20171218)

[Kay Diederichs]

Dear Oliver,

sorry for the trouble!
A default should be correct in the majority of situations, but it's impossible 
to have it work for _all_ situations. The XDS default for REFINE(IDXREF) was 
changed (i.e. POSITION was removed) to improve the indexing for weak and lousy 
data, _and_ because the distance values from the header are nowadays almost 
always accurate. For data with very high resolution such as yours, and 
significantly wrong distance, this means that at high resolution in particular 
this default will lead to worse data. generate_XDS.INP (in XDSwiki) and (I 
think) autoPROC and xia2 explicitly include POSITION in the REFINE(IDXREF), 
i.e. override the default. Where did you get XDS.INP from?
Some comments:
a) this is why I recommend, for data sets that may be important, to always do 
one cycle of optimization, i.e. after CORRECT, "mv GXPARM.XDS XPARM.XDS", 
change XDS.INP to have the correct high-resolution cutoff (cutting after the 
last shell which still has a "*" in the CC1/2 column), adjust the 
FRIEDEL'S_LAW= setting, and run JOB=INTEGRATE CORRECT. In your case this would 
make the refined distance available to INTEGRATE, and should result in very 
good data.
b) at which beamline did you collect the data? Such a high difference between 
refined distance and distance from header is unusual, and should be reported to 
(and fixed by) the beamline staff.
c) for this beamline at least, one should use REFINE(IDXREF)=CELL BEAM 
ORIENTATION AXIS POSITION . I understand that you tried this and it didn't 
work? Maybe there is something else wrong then, e.g. another line later in 
XDS.INP resetting REFINE(IDXREF) to something else. 

If you cannot find the reason why including POSITiON into REFINE(IDXREF) does 
not help, pls send me enough data to reproduce the problem.

best wishes,

Kay

On Tue, 23 Jan 2018 14:31:09 +, "Weiergräber, Oliver H." 
 wrote:

>Dear all,
>
>After upgrading XDS from build date 20170601 to 20171218, I am experiencing 
>severe degradation of apparent data quality reported by CORRECT for certain 
>data sets. Following first indications of issues with a slightly problematic 
>candidate, I went back to a previously very well-behaved test case with 
>diffraction extending beyond 1.1 A. Using the same input with both program 
>versions, statistics are not too different out to approx. 1.6 A, but become 
>more and more divergent in outer shells. These are the values for the 
>highest-resolution shell (1.10--1.16 A):
>
>20170601:  I/sigI = 1.80; Rmeas = 59.9 %; CC(1/2) = 82.0 %
>20171218:  I/sigI = 0.14; Rmeas = 506.4 %; CC(1/2) = 5.8 %
>
>The refined cell constants are unreasonably different as well. Obviously, 
>something is going awfully wrong here, presumably related to errors in 
>geometry parameters (which are expected to become increasingly detrimental 
>with decreasing dmin). As it turns out, geometry refinement behaves 
>differently in the latest version of XDS: most notably, IDXREF no longer 
>refines the detector distance by default. This is significant in this case, as 
>in version 20170601 the distance would shift by as much as 1.3 mm, resulting 
>in successful integration and scaling. Unfortunately, re-adding POSITION 
>refinement into REFINE(IDXREF) does not help, and even having it in all 
>refinement scopes (IDXREF, INTEGRATE, CORRECT) only yields a limited 
>improvement of CORRECT statistics.
>It is worth noting that examination of a dataset from an unrelated crystal 
>(dmin = 1.4 A) did not reveal such enormous differences -- in this case, 
>however, the shift in detector distance applied in the old version of XDS was 
>quite small (0.2 mm), which, together with the lower resolution, explains the 
>absence of large discrepancies.
>
>To sum up, I suspect that, due to recent changes to the XDS code, the 
>refinement of geometry parameters is now overly restrained, resulting in 
>drastic problems in cases where the detector distance is not as precise as 
>desirable (and even more so at very high resolution). For such datasets, the 
>new version appears to be essentially unusable (at least within the parameter 
>space I have tested), and even in modest cases may often be inferior to the 
>previous one. Is there an option to revert to the former behaviour?
>
>Best regards
>Oliver
>
>
>  PD Dr. Oliver H. Weiergr�ber
>  Institute of Complex Systems
>  ICS-6: Structural Biochemistry
>  Tel.: +49 2461 61-2028
>  Fax: +49 2461 61-9540
>
>
>
>
>
>
>
>Forschungszentrum Juelich GmbH
>52425 Juelich
>Sitz der Gesellschaft: Juelich
>Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
>Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen 

[ccp4bb] Issues with latest XDS (20171218)

[Weiergräber, Oliver H.]

Dear all,

After upgrading XDS from build date 20170601 to 20171218, I am experiencing 
severe degradation of apparent data quality reported by CORRECT for certain 
data sets. Following first indications of issues with a slightly problematic 
candidate, I went back to a previously very well-behaved test case with 
diffraction extending beyond 1.1 A. Using the same input with both program 
versions, statistics are not too different out to approx. 1.6 A, but become 
more and more divergent in outer shells. These are the values for the 
highest-resolution shell (1.10--1.16 A):

20170601:  I/sigI = 1.80; Rmeas = 59.9 %; CC(1/2) = 82.0 %
20171218:  I/sigI = 0.14; Rmeas = 506.4 %; CC(1/2) = 5.8 %

The refined cell constants are unreasonably different as well. Obviously, 
something is going awfully wrong here, presumably related to errors in geometry 
parameters (which are expected to become increasingly detrimental with 
decreasing dmin). As it turns out, geometry refinement behaves differently in 
the latest version of XDS: most notably, IDXREF no longer refines the detector 
distance by default. This is significant in this case, as in version 20170601 
the distance would shift by as much as 1.3 mm, resulting in successful 
integration and scaling. Unfortunately, re-adding POSITION refinement into 
REFINE(IDXREF) does not help, and even having it in all refinement scopes 
(IDXREF, INTEGRATE, CORRECT) only yields a limited improvement of CORRECT 
statistics.
It is worth noting that examination of a dataset from an unrelated crystal 
(dmin = 1.4 A) did not reveal such enormous differences -- in this case, 
however, the shift in detector distance applied in the old version of XDS was 
quite small (0.2 mm), which, together with the lower resolution, explains the 
absence of large discrepancies.

To sum up, I suspect that, due to recent changes to the XDS code, the 
refinement of geometry parameters is now overly restrained, resulting in 
drastic problems in cases where the detector distance is not as precise as 
desirable (and even more so at very high resolution). For such datasets, the 
new version appears to be essentially unusable (at least within the parameter 
space I have tested), and even in modest cases may often be inferior to the 
previous one. Is there an option to revert to the former behaviour?

Best regards
Oliver


  PD Dr. Oliver H. Weiergräber
  Institute of Complex Systems
  ICS-6: Structural Biochemistry
  Tel.: +49 2461 61-2028
  Fax: +49 2461 61-9540







Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt