To second the observation, we have also seen the extremely sigmoidal
cumulative intensity distributions from the affected versions of XDS,
which are accompanied by higher I/s estimations in the higher resolution
bins, odd-looking Wilson plot, and a bloated resolution estimation. In
our case, they were not affecting model refinement greatly, but we were
still worried.
These were observed with mtz files produced by automated beamline
pipelines, which were also reporting unrealistically high resolution
estimates, so people who depend on such pipelines (and like the higher
I/s values) should probably look at their data more critically and
re-process.
Engin
On 5/29/25 12:06 PM, Gerard Bricogne wrote:
Dear Phil,
I am not sure that I follow you there: perfectly twinned acentric
intensities will follow a Chi-squared distribution with four degrees
of freedom, while untwinned ones have only two degrees of freedom. The
higher the number of degrees of freedom, the more the Chi-squared
distribution will peak around its mean value (Central Limit Theorem).
This trend is clearly visible in going from the red to the blue curve.
The green curve shows an even stronger trend towards a peaked
distribution, so actually the data coming our of 20240712 are, so to
speak, "hyper-twinned". This shift away from low values is consistent
with Gleb Bourenkov's findings, obtained by integrating images with
very low signal created with our image simulator, that integrated
intensities produced by the six successive buggy versions of XDS since
July last year (20240712, 20240723, 20241002, 20250119, 20250224 and
20250320) were affected by biases, mostly positive and causing a
depletion of the weak intensity population (hence spurious twinning
diagnostics), plus an overestimation of I/sig(I) and even Wilson plots
curving upwards (!) at high resolution.
Please peruse the autoPROC Wiki page for which I sent a link
earlier and take a look at the plots it contains: it is all described
in there.
With best wishes,
Gerard.
--
On Thu, May 29, 2025 at 05:30:54PM +0100, Phil Evans wrote:
Dear Kay
This is a plot I sent to Eleanor yesterday, from her log file. The cumulative
intensity plot shows I think that there are too few weak intensities, the
opposite of twinning. I don;t know where it comes from but it’s not right!
Best
Phil
########################################################################
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list
hosted by www.jiscmail.ac.uk, terms & conditions are available at
https://www.jiscmail.ac.uk/policyandsecurity/
On 29 May 2025, at 12:03, Kay Diederichs <kay.diederi...@uni-konstanz.de> wrote:
CAUTION: This email originated from outside of the LMB:
.-owner-ccp...@jiscmail.ac.uk-.
Do not click links or open attachments unless you recognize the sender and know
the content is safe.
If you think this is a phishing email, please forward it to
phish...@mrc-lmb.cam.ac.uk
--
Dear Gerard,
part of the problem was probably indeed due to the use of XDS BUILT=20240712
(that version was used for CORRECT.LP that Janani Ganesh sent to me). I
obtained the raw data from Janani Ganesh and processed with both the current
XDS (BUILT=20250430), and the bad version from July last year. My findings:
- with the bad version the data appear twinned, whereas with the current
version the data appear untwinned
- Wilson B is 34 A^2 with the current version, and significantly (and
unrealistically) lower with the bad version
The structure can be satisfactorily solved and refined with data from the
current XDS version. With data from the bad version, average B goes down to 19
and the R-values stall at high values.
This does not explain the weird B-values that were reported, though. So I
suspect another problem.
Best,
Kay
On Thu, 29 May 2025 11:37:14 +0100, Gerard Bricogne <g...@globalphasing.com>
wrote:
Dear all,
I wrote yesterday a contribution to this thread, but mistakenly
used a simple Reply instead of a Group Reply, so that my message did
not reach the BB while it intended to, as shown by the introductory
"Dear all".
I am including it below, as it contains a link to some material
that is of general interest but that we had not yet found a way of
disseminating to the whole community. Users of XDS, or of XDS-based
pipelines, please take notice and do take a close look at this link,
and ultimately at the BUILT= information about the exact XDS version
used in producing your results - it can make a lot of difference.
With best wishes,
Gerard.
--
Date: Wed, 28 May 2025 16:18:25 +0100
From: Gerard Bricogne <gb10>
Subject: Re: [ccp4bb] Requesting help with twin refinement
To: Eleanor Dodson <eleanor.dod...@york.ac.uk>
Dear all,
These kinds of anomalies (overoptimistic estimates of resolution,
unphysical persistence of signal at high resolution, spurious twinning
diagnostics) are reminiscent of what we observed with some of the
recent versions of XDS, as documented at
https://www.globalphasing.com/autoproc/wiki/index.cgi?ComparisonProcessing202504
and three predecessor pages referenced from within this one.
Only one version is currently available from the XDS download
page (that we consider as being OK) but there was a succession of many
problematic ones going back to July 2024. Perhaps the version used to
process this dataset was one of the unlucky ones.
Apologies if this is a red herring, but that check is worth
making.
Best wishes,
Gerard
--
On Thu, May 29, 2025 at 11:05:52AM +0100, Eleanor Dodson wrote:
After inspecting the log files it is obvious the problem is in the data
integration. (Wilson B negative! highest resolution shell stronger than low
res, etc..)
I think you used XDS for that? Do you have an alternative set of integrated
unmerged data, eg done with DIALS?
If so I would use that, or download a new version of XDS and try again..
On Wed, 28 May 2025 at 07:56, Janani Ganesh <jganesh3...@gmail.com> wrote:
Dear Martin,
Thank you for your suggestion. I will try it out and let you know.
Regards,
Janani Ganesh
On Tue, 27 May 2025 09:01:06 +0100, Martin Mal� <martin.maly...@email.cz>
wrote:
Dear Janani,
There has been recently a significant development in twin refinement
algorithms in Servalcat. Could you try this program and let us know if
it helps?
You can find Servalcat in i2 in the most recent version of CCP4. You can
also run it in terminal/CCP4Console:
servalcat refine_xtal_norefmac --hklin data.mtz --model model.pdb -s
xray --labin I,IMEAN,FreeR_flag --twin
(Assuming your MTZ data file has columns I,IMEAN,FreeR_flag.)
Full list of options:
servalcat refine_xtal_norefmac -h
Please feel free to ask if anything was not clear.
Cheers,
Martin
On 27/05/2025 07:00, Janani Ganesh wrote:
Hello,
I recently collected diffraction data and I was able to do the data
processing upto 1.88 �. I observed the R-merge remained the same throughout
the resolution bins. I could solve the structure using Phaser in the space
group P32 2 1 (no. 154). But when I refined the structure the R-factors
remained ~ 35 % even after several cycles of refinement and model building.
Then I looked for twinning in my data using Xtriage, Detwin, Pointless,
etc which indicated twining. Subsequently, I started twin refinement with
Refmac using twin law (-h, -k, l). The R-factors came down to ~25%.
However, the B-factors refined to 0.5 for all the atoms. I tried redoing
the data processing at lower symmetry space groups, P3, P31, P32, followed
by twin refinement but the issue with the B-factors persisted.
I would appreciate any suggestions and advice towards resolving this
problem.
Regards,
Janani Ganesh
########################################################################
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a
mailing list hosted by www.jiscmail.ac.uk, terms & conditions are
available at https://www.jiscmail.ac.uk/policyandsecurity/
########################################################################
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a
mailing list hosted by www.jiscmail.ac.uk, terms & conditions are
available at https://www.jiscmail.ac.uk/policyandsecurity/
########################################################################
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a
mailing list hosted by www.jiscmail.ac.uk, terms & conditions are
available at https://www.jiscmail.ac.uk/policyandsecurity/
########################################################################
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list
hosted by www.jiscmail.ac.uk, terms & conditions are available at
https://www.jiscmail.ac.uk/policyandsecurity/
########################################################################
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list
hosted by www.jiscmail.ac.uk, terms & conditions are available at
https://www.jiscmail.ac.uk/policyandsecurity/
########################################################################
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list
hosted by www.jiscmail.ac.uk, terms & conditions are available at
https://www.jiscmail.ac.uk/policyandsecurity/
########################################################################
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list
hosted by www.jiscmail.ac.uk, terms & conditions are available at
https://www.jiscmail.ac.uk/policyandsecurity/
########################################################################
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list
hosted by www.jiscmail.ac.uk, terms & conditions are available at
https://www.jiscmail.ac.uk/policyandsecurity/
--
Engin Özkan, Ph.D.
Associate Professor
Dept of Biochemistry and Molecular Biology
University of Chicago
Phone: (773) 834-5498
http://ozkan.uchicago.edu
########################################################################
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list
hosted by www.jiscmail.ac.uk, terms & conditions are available at
https://www.jiscmail.ac.uk/policyandsecurity/
