Re: [ccp4bb] Experimental phasing Selenomethionine data collection etc. tips

[Briony Yorke]

Hi,


> What if instead of images we just collected a list of x-y coordinates of 
> photon hits vs time?

This is how the Tristan detector works, it is based on timepix3 chips and the 
data is a read as time stamped x-y co-ordinates + counts and then diffraction 
images are reconstructed from this information downstream. This means you have 
complete freedom about how you want to bin your data in terms of time/dose.

https://ieeexplore.ieee.org/document/9875340

We’ve used the detector to test Hadamard multiplexing of time/dose bins to 
boost the ‘weak images’ from small bins to help with indexing and integration 
prior to deconvolution back into time/dose resolved data. However, this is a 
work in process we are waiting for some software updates to help with the image 
reconstruction part of the process but the detector appears to work very well 
for this kind of application!

Briony

From: CCP4 bulletin board  on behalf of Guillaume 
Gaullier 
Date: Friday, 17 May 2024 at 17:07
To: CCP4BB@JISCMAIL.AC.UK 
Subject: Re: [ccp4bb] Experimental phasing Selenomethionine data collection 
etc. tips

CAUTION: External Message. Use caution opening links and attachments.

Hello,



> What if instead of images we just collected a list of x-y coordinates of 
> photon hits vs time?



This already exists, but for electrons: 
https://doi.org/10.1107/S205225252000929X

This is the technology in Falcon4 detectors. The advantage is that it allows 
you to decide how to slice the total dose into dose fractions *after* the data 
collection. So, when setting up the collection at the microscope, you only need 
to worry about how much total dose to use.

With the previous generation of detectors, during set up you have to decide how 
much total dose but also how many dose fractions, and if it's not optimal you 
can't change it once the data is collected.



I don't understand detector technology enough to tell whether this should be 
possible for X-rays. But out of curiosity, how would you use this detector if 
you had one? What would it enable that you can't do now?



Cheers,



Guillaume


From: CCP4 bulletin board  on behalf of James Holton 

Sent: Friday, May 17, 2024 5:27:02 PM
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] Experimental phasing Selenomethionine data collection 
etc. tips

A few follow-up questions I got out-of-band:

> how did you get to the 1:1 relationship between Bijvoet ratio and dose?
>
I got this from fitting a straight line to Table 1 of Banu's 2004 paper:
10.1107/S0907444904007917
Is this a rough estimate based on a singular result?  Of course it is!
This is how we roll in radiation damage research.
>
> Comment:  with the more modern pixel array detectors (e.g. Eiger), you
> can slice your dose even more finely than 0.1s, and not worry about
> the readout time.
yes.

With a bit of a caveat on how many photons/pixel you need for stable
background subtraction. XDS starts having issues around 1 photon/pixel
or less, and DIALS claims to be able to get to 0.01 photons/pixel, but I
have not personally pushed it that far.  Not yet.

I have a plan to try and push zero-dose extrapolation to the
one-photon-per-image level, but that is on another thread.
> is it better to collect 360 or 720º at half the dose
Nothing wrong with going longer than 360, especially if you want to do
zero-dose extrapolation, because it is only by repeating the same phi
range (and everything else) exactly that you get a genuinely "same"
increment in dose.

However, once you go past 360 the "multiplicity" you gain starts turning
into what you might call a "redundancy". What I mean by that is that in
the first 360 each spot and its symmetry mates generally show up on
different pixels.  Each pixel has about 1% to 3% calibration error
associated with it (depending on the detector). So, for the 2nd 360 you
will re-measure all the same spots with the same pixels again, repeating
a systematic error.  You will also have the same sample self-absorption,
etc. But, the pixel calibration error starts to really matter for
anomalous at high "redundancy". To put it another way, if a particular
pixel has 1% error, then counting more than 10,000 photons with it is a
waste, because the systematic error of 1% will start to dominate the
total error at higher photon counts. So, for anomalous especially, I
recommend moving the detector between 360s. Sliding it horizontally is
best. Or you can use 2theta.  But, a small change in detector distance
can usually do it and is almost always an available option.

The only problem with all this "dose slicing" is the images get very
very weak.

And that brings us back to the "weak image limit".  What if instead of
images we just collected a list of x-y coordinates of photon hits vs
time? Anyone have a suggestion for the name to give to the program that
can process such data?

-James Holton
MAD Scientist


On 5/15/2024 3:28 PM, James Holton wrote:
>
> Thank you to all who provided 

Re: [ccp4bb] Experimental phasing Selenomethionine data collection etc. tips

[Guillaume Gaullier]

Hello,


> What if instead of images we just collected a list of x-y coordinates of 
> photon hits vs time?


This already exists, but for electrons: 
https://doi.org/10.1107/S205225252000929X

This is the technology in Falcon4 detectors. The advantage is that it allows 
you to decide how to slice the total dose into dose fractions *after* the data 
collection. So, when setting up the collection at the microscope, you only need 
to worry about how much total dose to use.

With the previous generation of detectors, during set up you have to decide how 
much total dose but also how many dose fractions, and if it's not optimal you 
can't change it once the data is collected.


I don't understand detector technology enough to tell whether this should be 
possible for X-rays. But out of curiosity, how would you use this detector if 
you had one? What would it enable that you can't do now?


Cheers,


Guillaume


From: CCP4 bulletin board  on behalf of James Holton 

Sent: Friday, May 17, 2024 5:27:02 PM
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] Experimental phasing Selenomethionine data collection 
etc. tips

A few follow-up questions I got out-of-band:

> how did you get to the 1:1 relationship between Bijvoet ratio and dose?
>
I got this from fitting a straight line to Table 1 of Banu's 2004 paper:
10.1107/S0907444904007917
Is this a rough estimate based on a singular result?  Of course it is!
This is how we roll in radiation damage research.
>
> Comment:  with the more modern pixel array detectors (e.g. Eiger), you
> can slice your dose even more finely than 0.1s, and not worry about
> the readout time.
yes.

With a bit of a caveat on how many photons/pixel you need for stable
background subtraction. XDS starts having issues around 1 photon/pixel
or less, and DIALS claims to be able to get to 0.01 photons/pixel, but I
have not personally pushed it that far.  Not yet.

I have a plan to try and push zero-dose extrapolation to the
one-photon-per-image level, but that is on another thread.
> is it better to collect 360 or 720º at half the dose
Nothing wrong with going longer than 360, especially if you want to do
zero-dose extrapolation, because it is only by repeating the same phi
range (and everything else) exactly that you get a genuinely "same"
increment in dose.

However, once you go past 360 the "multiplicity" you gain starts turning
into what you might call a "redundancy". What I mean by that is that in
the first 360 each spot and its symmetry mates generally show up on
different pixels.  Each pixel has about 1% to 3% calibration error
associated with it (depending on the detector). So, for the 2nd 360 you
will re-measure all the same spots with the same pixels again, repeating
a systematic error.  You will also have the same sample self-absorption,
etc. But, the pixel calibration error starts to really matter for
anomalous at high "redundancy". To put it another way, if a particular
pixel has 1% error, then counting more than 10,000 photons with it is a
waste, because the systematic error of 1% will start to dominate the
total error at higher photon counts. So, for anomalous especially, I
recommend moving the detector between 360s. Sliding it horizontally is
best. Or you can use 2theta.  But, a small change in detector distance
can usually do it and is almost always an available option.

The only problem with all this "dose slicing" is the images get very
very weak.

And that brings us back to the "weak image limit".  What if instead of
images we just collected a list of x-y coordinates of photon hits vs
time? Anyone have a suggestion for the name to give to the program that
can process such data?

-James Holton
MAD Scientist


On 5/15/2024 3:28 PM, James Holton wrote:
>
> Thank you to all who provided helpful suggestions so far.
>
> A few things I'd recommend for this particular beamline (which I have
> been running for 20+ years)
>
> Do NOT collect one wavelength at a time. This was a good strategy on
> old beamlines with noisy detectors and slow, drifty monochromators.
> This is not the case at any of the ALS beamlines today. With modern
> zero-read-noise detectors there is no penalty to spreading your
> photons over a lot more images, and round-robin changes between at
> least two wavelengths will double your phasing power for the same
> dose. With 8.3.1's monochromator, wavelength changes take about 1
> second and are reproducible to well within the intrinsic width of the
> Se peak.  So you don't need to worry about missing or drifting off the
> peak or inflection. The only thing you need to worry about is
> over-cooking your crystal before you get all the data you need.
>
> No matter what beamline you use the number of photons your crystal
> will give off before it dies is a fixed number. All you get to do is
> decide how to spread them over the images. Doing two wavelengths
> within this photon budget doesn't hurt. You can always scale and merge
> them 

Re: [ccp4bb] Experimental phasing Selenomethionine data collection etc. tips

[James Holton]

A few follow-up questions I got out-of-band:


how did you get to the 1:1 relationship between Bijvoet ratio and dose?

I got this from fitting a straight line to Table 1 of Banu's 2004 paper: 
10.1107/S0907444904007917
Is this a rough estimate based on a singular result?  Of course it is!  
This is how we roll in radiation damage research.


Comment:  with the more modern pixel array detectors (e.g. Eiger), you 
can slice your dose even more finely than 0.1s, and not worry about 
the readout time.

yes.

With a bit of a caveat on how many photons/pixel you need for stable 
background subtraction. XDS starts having issues around 1 photon/pixel 
or less, and DIALS claims to be able to get to 0.01 photons/pixel, but I 
have not personally pushed it that far.  Not yet.


I have a plan to try and push zero-dose extrapolation to the 
one-photon-per-image level, but that is on another thread.

is it better to collect 360 or 720º at half the dose
Nothing wrong with going longer than 360, especially if you want to do 
zero-dose extrapolation, because it is only by repeating the same phi 
range (and everything else) exactly that you get a genuinely "same" 
increment in dose.


However, once you go past 360 the "multiplicity" you gain starts turning 
into what you might call a "redundancy". What I mean by that is that in 
the first 360 each spot and its symmetry mates generally show up on 
different pixels.  Each pixel has about 1% to 3% calibration error 
associated with it (depending on the detector). So, for the 2nd 360 you 
will re-measure all the same spots with the same pixels again, repeating 
a systematic error.  You will also have the same sample self-absorption, 
etc. But, the pixel calibration error starts to really matter for 
anomalous at high "redundancy". To put it another way, if a particular 
pixel has 1% error, then counting more than 10,000 photons with it is a 
waste, because the systematic error of 1% will start to dominate the 
total error at higher photon counts. So, for anomalous especially, I 
recommend moving the detector between 360s. Sliding it horizontally is 
best. Or you can use 2theta.  But, a small change in detector distance 
can usually do it and is almost always an available option.


The only problem with all this "dose slicing" is the images get very 
very weak.


And that brings us back to the "weak image limit".  What if instead of 
images we just collected a list of x-y coordinates of photon hits vs 
time? Anyone have a suggestion for the name to give to the program that 
can process such data?


-James Holton
MAD Scientist


On 5/15/2024 3:28 PM, James Holton wrote:


Thank you to all who provided helpful suggestions so far.

A few things I'd recommend for this particular beamline (which I have 
been running for 20+ years)


Do NOT collect one wavelength at a time. This was a good strategy on 
old beamlines with noisy detectors and slow, drifty monochromators. 
This is not the case at any of the ALS beamlines today. With modern 
zero-read-noise detectors there is no penalty to spreading your 
photons over a lot more images, and round-robin changes between at 
least two wavelengths will double your phasing power for the same 
dose. With 8.3.1's monochromator, wavelength changes take about 1 
second and are reproducible to well within the intrinsic width of the 
Se peak.  So you don't need to worry about missing or drifting off the 
peak or inflection. The only thing you need to worry about is 
over-cooking your crystal before you get all the data you need.


No matter what beamline you use the number of photons your crystal 
will give off before it dies is a fixed number. All you get to do is 
decide how to spread them over the images. Doing two wavelengths 
within this photon budget doesn't hurt. You can always scale and merge 
them together. But keeping them separate gives you both kinds of 
anomalous differences, which are 90 deg apart. So, when one zigs the 
other zags. It is like having twice as many sites without the extra 
damage you would get from them. Also, by taking shorter/weaker 
exposures you maximize your chances of winning over radiation damage 
"in-post" by cutting off images that degrade your signal.


And before anybody says it: NO! Collecting fainter images does NOT 
degrade your resolution. I don't know where this idea comes from, but 
it never seems to die. It was true with film and image plates, but 
with pixel arrays and modern CCDs there is no penalty to weak images.  
Don't believe me? Read the manual for your detector. Modern PADs 
actually sum a bunch of weak images internally before writing them to 
disk. You can do the same "in post" if you want to.


Yes, there are many cases where SAD is good enough, but my advice is 
never to tempt fate.


What I recommend is:
1) collect two wavelengths: remote, and halfway between the peak and 
inflection.

        this will maximize both kinds of anomalous differences
2) calculate your Bijvoet ratio here: 

Re: [ccp4bb] Unable to send a query email to supp...@proteindiffraction.org

[Martin Maly]

Dear all,
After half an year from the original Shivani's message, I am having the same 
problem. Please has anybody successfully deposited data there recently or has 
any idea how to reach the admins?
Cheers,
Martin

On 14/11/2023 00:16, Shivani Sharma  wrote:
I deposited some raw data on the protein diffraction server >2 months 
ago, but the status still shows unreleased, and DOI is TBA. I have tried 
multiple times now to send an email with my query to 
supp...@proteindiffraction.org, but it keeps failing (Microsoft 
undeliverable email address error). Does anyone know how or who else to 
contact regarding this issue?



Shivani Sharma
Biology Program Representative, Graduate Center-CUNY
Director, Community and Events, Nucleate NY
Student representative, Users Executive Committee, Brookhaven National Lab
LinkedIn-https://www.linkedin.com/in/shivani-sharma-women-in-crystallography/ 




To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 








To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] Request for help in optimizing Coot for AMD 5950x CPU and RX6600XT GPU hardware PC

[Otsile Mojanaga]

Dear All
Thank you for the detailed response. I will implement the suggested fixes and 
report back on a later date.

Kind regards,
Otsile

-Original Message-
From: CCP4 bulletin board  On Behalf Of Bernhard Lohkamp
Sent: Wednesday, May 15, 2024 9:11 PM
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] Request for help in optimizing Coot for AMD 5950x CPU and 
RX6600XT GPU hardware PC

[You don't often get email from b.lohk...@gmail.com. Learn why this is 
important at https://aka.ms/LearnAboutSenderIdentification ]

CAUTION:  This email came from outside of the University. To keep your account 
safe, only click on links and open attachments if you know the person who sent 
the email, or you expected to receive this communication.



On 15/05/2024 19:03, Paul Emsley wrote:
>
> On 13/05/2024 17:38, Otsile Mojanaga wrote:
>>
>> *From:*Otsile Mojanaga
>> *Sent:* Monday, May 13, 2024 3:29 PM
>> *To:* Paul Emsley 
>> *Subject:* RE: [ccp4bb] Request for help in optimizing Coot for AMD
>> 5950x CPU and RX6600XT GPU hardware PC
>>
>> Dear Paul/All
>>
>> Thank you for the response. I am on Windows 11 and I tried looking
>> around Coot to see where I could input the commands you have
>> suggested but I cannot find anything . I also cannot seem to type the
>> commands into the 'Command prompt' that opens alongside Coot. I also
>> tried inputting the commands in the 'Coot Python scripting' within
>> Coot but this did not work. Please advise on when you can input these 
>> commands.
>>
>>
>>
>>
>>
>>
>> On 13/05/2024 13:03, Otsile Mojanaga wrote:
>>
>> Dear All
>>
>> I have been using coot as part of my normal workflow but since
>> last week I have been having issues with Coot after I built my own
>> PC which contains the following Hardware:
>>
>> CPU - AMD Ryzen 9 5950X 16-Core Processor - 16 Cores, 32 Threads
>>
>> RAM - 64 GB
>>
>> GPU - AMD Radeon RX 6600 XT - Primary/Discrete
>>
>> GPU VRAM - 8176 MB - GDDR6 2000 MHz
>>
>> Despite the improvement in hardware, Coot is now having issues
>> refining molecules when using the 'real space refine' tool and
>> other related tools. When trying to refine, the program becomes
>> stuttering and slow such that it becomes unfeasible to use the
>> program in a reasonable amount of time. My current work around is
>> to run Coot through CCP4 cloud but this greatly slows down my
>> workflow. >From what I can see Coot is not be optimised for the
>> AMD CPU.
>>
>> You don't mention the operating system.
>>
>> Maybe you have too many threads for Coot? Try setting the
>> COOT_N_THREADS to various values (starting, say, with 2) and see what
>> happens.
>>
>> But maybe it's a graphics (driver) issue and not a CPU/threads issue.
>> glxgears is the classic test for that.
>>
>> Paul.
>>
>
> What I mean here is that COOT_N_THREADS is an environment variable. It
> needs to be set in the environment from which Coot is launched -
> before Coot is launched, just to be clear.
>
>  From a quick googling, I see that environment variables are a thing
> under windows, but it is not clear to me how they could actually be
> used in the context of Coot.
>
> Paul.
>

You can add the environment variable in the coot startup script, i.e.
edit wincoot.bat and insert the following line (*) before coot-bin.exe is 
called:

set COOT_N_THREADS=2

Having said this, for me this doesnt make any difference but more importantly I 
dont think this is the issue. Your issue is almost certainly the graphics card 
drivers. Please make sure you install the appropriate drivers directly from the 
AMD rather than relying on the ones provided by Windows. I am confident this 
will resolve your issue.
That is unless some old issue with AMD cards shows up again then you may have 
to manually control threading in the Control panel of the graphics card.

HTH,

B


(*) there is another way in Windows to restrict the number of threads used (in 
the example to 2). To do so, insert the following BEFORE coot-bin.exe in the 
wincoot.bat file (i.e. in the same line):

start /affinity 2

>
> --
> --
>
> To unsubscribe from the CCP4BB list, click the following link:
> https://www/.
> jiscmail.ac.uk%2Fcgi-bin%2FWA-JISC.exe%3FSUBED1%3DCCP4BB%26A%3D1=
> 05%7C02%7Coom21%40BATH.AC.UK%7Cf21ff70a72ef4f6c222a08dc751b37e2%7C377e
> 3d224ea1422db0ad8fcc89406b9e%7C0%7C0%7C638514007252821705%7CUnknown%7C
> TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC
> I6Mn0%3D%7C0%7C%7C%7C=0QXEpdsJXNyVqn%2FcIsASq7aDjdFFSgpSGZGclm9X
> VyE%3D=0
>  .jiscmail.ac.uk%2Fcgi-bin%2FWA-JISC.exe%3FSUBED1%3DCCP4BB%26A%3D1
> =05%7C02%7Coom21%40BATH.AC.UK%7Cf21ff70a72ef4f6c222a08dc751b37e2%7C377
> e3d224ea1422db0ad8fcc89406b9e%7C0%7C0%7C638514007252829951%7CUnknown%7
> CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXV
> CI6Mn0%3D%7C0%7C%7C%7C=QbTZWhB7tyWgKynYFb%2FzjMn9YyaXeI98ymnpdai