Re: [ccp4bb] happy/sad maps
Hello James, Thank you for sharing! As usual, these messages and the answers they attract are very educational, and I enjoyed reading. Regarding "all-or-nothing real space maps", in single-particle cryoEM the masks we use always have a soft edge (continuously and smoothly decaying from 1 to 0 over several voxels). Some programs like Relion even let you decide how wide you want to make this soft edge. I don’t know enough Fourier theory to understand whether this is a satisfactory solution in principle, but in practice it seems to be good enough. Cheers, Guillaume On 1 May 2023, at 01:38, James Holton mailto:jmhol...@lbl.gov>> wrote: Thanks to all who replied both on- and off-list. A few (such as Charles below) have reported that using general-purpose FFT algorithms, such as those available in Matlab or NumPy do, in fact, get the smiley face map back. This is expected because such algorithms do not have a resolution cutoff. The programs I have found do turn the smile into a frown are: phenix.fft and phenix.map_to_structure_factors in Phenix "fft", "sfall" and "refmac5" in the CCP4 Suite The reason why this happens is not that these programs are "buggy", but simply because they require the array of hkls in reciprocal space to be a sphere, not a cube. That is, if h,k,l = 0,0,15 is at the resolution limit, then h,k,l = 15,15,15 is going to be outside of it, and discarded. The smiley face consists entirely of these high-order structure factors. Rejecting these "corner" regions may seem like a good idea until you realize that they are the only aspect of the map that carries happiness. ;) But all joking aside, the reason why these programs reject the "corners" is because they "should" be zero. And as long as you only ever work with maps that are generated from structure factors they will be zero. If they are not zero, why? And what happens if you rotate the object before recording it? Those non-zero "corners" will then be out beyond the faces of the cube, and therefore folding back into the lower-resolution data, messing it up. This implies, of course, that the non-rotated object is also being messed up by noise beyond the Nyquist limit. What I am suggesting is that non-zero "corner" regions may be a canary in our coal mine. Paul introduced the interesting possibility of super-sampling, and what I'm really trying to get at here is: what is the best way to "feather" an all-or-nothing real-space map (such as a bulk solvent mask, or a single-pixel detector event), so that it does not spray noise all over reciprocal space? Many of you may think this is a trick question and that I already have a beautiful answer I'm waiting to reveal. That is not really the case. What I have is a handful of rather unsatisfactory solutions, and I'm wondering if I'm just not aware of an existing solution to this problem. Hope everyone had a lovely weekend, -James Holton MAD Scientist On 4/28/2023 1:35 PM, Sindelar, Charles wrote: Hi James! I’m not sure exactly what cpp4, coot and phenix do for their FFT’s, but I’m assuming they must mask off the ‘corners’ of the 3D FFT data cube transform (where the frequency is greater than the Nyquist) during the forwards transform. If you take your Cheshire map and do the FFT in matlab or octave, this is not done and I can confirm you get the smiley back. The cryo-EM software I am familiar with does filter away the FFT corners during the first 3D reconstruction step, so bypassing this issue. Fun demo- this would be a great exercise for students! Chuck From: CCP4 bulletin board <mailto:CCP4BB@JISCMAIL.AC.UK> on behalf of James Holton <mailto:jmhol...@lbl.gov> Date: Friday, April 28, 2023 at 11:49 AM To: CCP4BB@JISCMAIL.AC.UK<mailto:CCP4BB@JISCMAIL.AC.UK> <mailto:CCP4BB@JISCMAIL.AC.UK> Subject: [ccp4bb] happy/sad maps Its still April, but this one isn't a joke. The smiley-face electron density in the left panel of the attached image has the remarkable property that any attempt to sharpen or blur the map turns it into the frowny-face on the right. If you'd like to try this yourself, the hidden_frown.map file is available in this tarball: https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbl831.als.lbl.gov%2F~jamesh%2Fbugreports%2Ffft_042423.tgz=05%7C01%7Ccharles.sindelar%40YALE.EDU%7Cc059178a4b774611f6ec08db48001b68%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C638182937669477928%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=jl16hGjaqmChVOHWb%2FAvgtvPTVFOt06wBqBYoxPk%2FZw%3D=0<https://bl831.als.lbl.gov/~jamesh/bugreports/fft_042423.tgz> In fact, any use of an FFT, even with the sharpening B set to zero, turns the smiley into a frowny face. There is no way to get the smiley face back (except opening the file agai
Re: [ccp4bb] happy/sad maps
A simple way to visualize what is happening is with UCSF ChimeraX. You can use the gaussian filter function to select either the frown or the smile from the “hidden_frown.map”, and then you can visualize the FFT structure factor magnitudes for each one – here are the commands to do it - open hidden_frown.map vol gaussian #1 sdev 0.5 vol gaussian #1 sdev 0.5 inv true vol fourier #1 vol fourier #2 vol fourier #3 Fun stuff- your example is a great way of illuminating what is going on with filtering and fourier transforms! Cheers, Chuck From: CCP4 bulletin board on behalf of James Holton Date: Sunday, April 30, 2023 at 5:16 PM To: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] happy/sad maps Thank you Paul. This is interesting! I have not played with super-sampling yet. I am assuming you mean creating a new map 8x the size? If so, did you fill the interstitial grid with zeroes? Local maximum? Linear interpolation? Tricubic spline? And when you say "sharpen/blur" with a factor of 4. Is 4 a scale factor? or a B factor? Cheers, and I hope you had a pleasant weekend, -James On 4/28/2023 5:29 PM, Paul Emsley wrote: > What fun! > > super-sampled by a factor of 2 then sharpen/blur with a factor of 4 > gives a superposition > > Paul. > To unsubscribe from the CCP4BB list, click the following link: https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2FWA-JISC.exe%3FSUBED1%3DCCP4BB%26A%3D1=05%7C01%7Ccharles.sindelar%40YALE.EDU%7Cacedb11c93ac4900f10408db49c02e38%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C638184862139535122%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=lfAErsKbQtKLBx8Xmmn%2BC7zG%2Bt6N43ChGLcniZrSyps%3D=0<https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1> This message was issued to members of https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.jiscmail.ac.uk%2FCCP4BB=05%7C01%7Ccharles.sindelar%40YALE.EDU%7Cacedb11c93ac4900f10408db49c02e38%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C638184862139535122%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=Nn%2BNpEPMzHrExi7899MakiwAJ8IJ5ZdgMsRIbuXjfZg%3D=0<http://www.jiscmail.ac.uk/CCP4BB>, a mailing list hosted by https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.jiscmail.ac.uk%2F=05%7C01%7Ccharles.sindelar%40YALE.EDU%7Cacedb11c93ac4900f10408db49c02e38%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C638184862139535122%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=AMn8ylqVwAO0sg%2BjlTi07Q20HVTD5ogHy2rQk1BPqfY%3D=0<http://www.jiscmail.ac.uk/>, terms & conditions are available at https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fpolicyandsecurity%2F=05%7C01%7Ccharles.sindelar%40YALE.EDU%7Cacedb11c93ac4900f10408db49c02e38%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C638184862139535122%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=R1Iq6MMuDWk2p%2FHKtmr9lWF7Vuzk%2FrZLMn7IVuZA4tI%3D=0<https://www.jiscmail.ac.uk/policyandsecurity/> To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/
Re: [ccp4bb] happy/sad maps
Thanks to all who replied both on- and off-list. A few (such as Charles below) have reported that using general-purpose FFT algorithms, such as those available in Matlab or NumPy do, in fact, get the smiley face map back. This is expected because such algorithms do not have a resolution cutoff. The programs I have found do turn the smile into a frown are: phenix.fft and phenix.map_to_structure_factors in Phenix "fft", "sfall" and "refmac5" in the CCP4 Suite The reason why this happens is not that these programs are "buggy", but simply because they require the array of hkls in reciprocal space to be a sphere, not a cube. That is, if h,k,l = 0,0,15 is at the resolution limit, then h,k,l = 15,15,15 is going to be outside of it, and discarded. The smiley face consists entirely of these high-order structure factors. Rejecting these "corner" regions may seem like a good idea until you realize that they are the only aspect of the map that carries happiness. ;) But all joking aside, the reason why these programs reject the "corners" is because they "should" be zero. And as long as you only ever work with maps that are generated from structure factors they will be zero. If they are not zero, why? And what happens if you rotate the object before recording it? Those non-zero "corners" will then be out beyond the faces of the cube, and therefore folding back into the lower-resolution data, messing it up. This implies, of course, that the non-rotated object is also being messed up by noise beyond the Nyquist limit. What I am suggesting is that non-zero "corner" regions may be a canary in our coal mine. Paul introduced the interesting possibility of super-sampling, and what I'm really trying to get at here is: what is the best way to "feather" an all-or-nothing real-space map (such as a bulk solvent mask, or a single-pixel detector event), so that it does not spray noise all over reciprocal space? Many of you may think this is a trick question and that I already have a beautiful answer I'm waiting to reveal. That is not really the case. What I have is a handful of rather unsatisfactory solutions, and I'm wondering if I'm just not aware of an existing solution to this problem. Hope everyone had a lovely weekend, -James Holton MAD Scientist On 4/28/2023 1:35 PM, Sindelar, Charles wrote: Hi James! I’m not sure exactly what cpp4, coot and phenix do for their FFT’s, but I’m assuming they must mask off the ‘corners’ of the 3D FFT data cube transform (where the frequency is greater than the Nyquist) during the forwards transform. If you take your Cheshire map and do the FFT in matlab or octave, this is not done and I can confirm you get the smiley back. The cryo-EM software I am familiar with does filter away the FFT corners during the first 3D reconstruction step, so bypassing this issue. Fun demo- this would be a great exercise for students! Chuck *From: *CCP4 bulletin board on behalf of James Holton *Date: *Friday, April 28, 2023 at 11:49 AM *To: *CCP4BB@JISCMAIL.AC.UK *Subject: *[ccp4bb] happy/sad maps Its still April, but this one isn't a joke. The smiley-face electron density in the left panel of the attached image has the remarkable property that any attempt to sharpen or blur the map turns it into the frowny-face on the right. If you'd like to try this yourself, the hidden_frown.map file is available in this tarball: https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbl831.als.lbl.gov%2F~jamesh%2Fbugreports%2Ffft_042423.tgz=05%7C01%7Ccharles.sindelar%40YALE.EDU%7Cc059178a4b774611f6ec08db48001b68%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C638182937669477928%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=jl16hGjaqmChVOHWb%2FAvgtvPTVFOt06wBqBYoxPk%2FZw%3D=0 <https://bl831.als.lbl.gov/~jamesh/bugreports/fft_042423.tgz> In fact, any use of an FFT, even with the sharpening B set to zero, turns the smiley into a frowny face. There is no way to get the smiley face back (except opening the file again). Yes, that's right, even just a simple back-and-forth FFT: turning this hidden_frown.map into structure factors and then back into a map again, gives you a frowny face. This happens using coot, ccp4 and phenix. Wait, what!? Isn't a Fourier transform supposed to preserve information? As in: you can jump back and forth between real and reciprocal space with impunity? Without introducing error? Well, yes, it is SUPPOSED to work like that, but the 3D FFT algorithms of structural biology have a ... quirk. If you start with structure factors and make a map out of them, you can convert it back-and-forth as often as you want with 100% preservation of information. However, if you start with a real-space map (such as from cryoEM), a back-and-forth conversion gives you a dif
Re: [ccp4bb] happy/sad maps
Dear James,
The questions of optimally combining oversampling, blurring and
de-blurring to obtain accurate approximations of Fourier transforms by
discrete Fourier transform calculations are old hat: they go back to papers
by David Sayre in 1951 ('The calculation of structure factors by Fourier
summation', Acta Cryst. 4, 362-367) and were dealt with in a landmark paper
by Lynn Ten Eyck in 1977 ('Efficient Structure-Factor Calculation for Large
Molecules by the Fast Fourier Transform', Acta Cryst. A33, 486-492) with a
follow-up by Jorge Navaza in 2002 ('On the computation of structure factors
by FFT techniques', Acta Cryst. (2002). A58, 568-573). All the ideas needed
to understand the phenomenon you describe (i.e. aliasing errors) are already
presented with remarkable clarity in David Sayre's paper.
With best wishes,
Gerard.
--
On Sun, Apr 30, 2023 at 02:15:54PM -0700, James Holton wrote:
> Thank you Paul. This is interesting!
>
> I have not played with super-sampling yet. I am assuming you mean
> creating a new map 8x the size? If so, did you fill the interstitial
> grid with zeroes? Local maximum? Linear interpolation? Tricubic
> spline?
>
> And when you say "sharpen/blur" with a factor of 4. Is 4 a scale
> factor? or a B factor?
>
> Cheers, and I hope you had a pleasant weekend,
>
> -James
>
> On 4/28/2023 5:29 PM, Paul Emsley wrote:
> >What fun!
> >
> >super-sampled by a factor of 2 then sharpen/blur with a factor of
> >4 gives a superposition
> >
> >Paul.
> >
>
>
>
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
>
> This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing
> list hosted by www.jiscmail.ac.uk, terms & conditions are available at
> https://www.jiscmail.ac.uk/policyandsecurity/
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list
hosted by www.jiscmail.ac.uk, terms & conditions are available at
https://www.jiscmail.ac.uk/policyandsecurity/
Re: [ccp4bb] happy/sad maps
Thank you Paul. This is interesting! I have not played with super-sampling yet. I am assuming you mean creating a new map 8x the size? If so, did you fill the interstitial grid with zeroes? Local maximum? Linear interpolation? Tricubic spline? And when you say "sharpen/blur" with a factor of 4. Is 4 a scale factor? or a B factor? Cheers, and I hope you had a pleasant weekend, -James On 4/28/2023 5:29 PM, Paul Emsley wrote: What fun! super-sampled by a factor of 2 then sharpen/blur with a factor of 4 gives a superposition Paul. To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/
Re: [ccp4bb] happy/sad maps
Hi James! I’m not sure exactly what cpp4, coot and phenix do for their FFT’s, but I’m assuming they must mask off the ‘corners’ of the 3D FFT data cube transform (where the frequency is greater than the Nyquist) during the forwards transform. If you take your Cheshire map and do the FFT in matlab or octave, this is not done and I can confirm you get the smiley back. The cryo-EM software I am familiar with does filter away the FFT corners during the first 3D reconstruction step, so bypassing this issue. Fun demo- this would be a great exercise for students! Chuck From: CCP4 bulletin board on behalf of James Holton Date: Friday, April 28, 2023 at 11:49 AM To: CCP4BB@JISCMAIL.AC.UK Subject: [ccp4bb] happy/sad maps Its still April, but this one isn't a joke. The smiley-face electron density in the left panel of the attached image has the remarkable property that any attempt to sharpen or blur the map turns it into the frowny-face on the right. If you'd like to try this yourself, the hidden_frown.map file is available in this tarball: https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbl831.als.lbl.gov%2F~jamesh%2Fbugreports%2Ffft_042423.tgz=05%7C01%7Ccharles.sindelar%40YALE.EDU%7Cc059178a4b774611f6ec08db48001b68%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C638182937669477928%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=jl16hGjaqmChVOHWb%2FAvgtvPTVFOt06wBqBYoxPk%2FZw%3D=0<https://bl831.als.lbl.gov/~jamesh/bugreports/fft_042423.tgz> In fact, any use of an FFT, even with the sharpening B set to zero, turns the smiley into a frowny face. There is no way to get the smiley face back (except opening the file again). Yes, that's right, even just a simple back-and-forth FFT: turning this hidden_frown.map into structure factors and then back into a map again, gives you a frowny face. This happens using coot, ccp4 and phenix. Wait, what!? Isn't a Fourier transform supposed to preserve information? As in: you can jump back and forth between real and reciprocal space with impunity? Without introducing error? Well, yes, it is SUPPOSED to work like that, but the 3D FFT algorithms of structural biology have a ... quirk. If you start with structure factors and make a map out of them, you can convert it back-and-forth as often as you want with 100% preservation of information. However, if you start with a real-space map (such as from cryoEM), a back-and-forth conversion gives you a different map. This new map can then be transformed back-and-forth all you want and be 100% preserved. It has been "christened" by the FFT, but it is not the same as the starting map, which is impossible to recover from the FFT-transformed data. Information has been lost. It is fine for crystallography (which starts with structure factors), but for techniques such as cryoEM that start with maps, using an FFT changes the data. What information is being lost? Sharp edges. These turn into ripples covering all of real and reciprocal space. Do real-world data have sharp edges? Well, the all-or-nothing masks we use to model bulk solvent are one example. Also, if you "mask off" otherwise smooth density with an all-or-nothing mask, you will get similar ripples. Another example of a sharp edge might be the large changes between adjacent pixels you see when a single electron hits a detector. For example, if you make a map with just one non-zero voxel and run it back-and-forth through FFT you will find that voxel loses from 50% to 99% of its value (depending on the size of the map). How much does this actually impact cryo-EM data? That is my question. What evil magic did I wield to make this map? Well, I drew a smiley and frowny face by hand, converted them to maps, and then I generated random noise within the boundaries of the smiley face. I ran this noisy map back-and-forth through FFT, and then subtracted the map that survives the FFT from the pre-FFT map. This cheshire_smile.map has the interesting property that all of the structure factors calculated from it are zero. It has an RMSD of 1.4, but after a back-and-forth FFT this RMSD drops to 1e-7. I generated the hidden_frown.map by simply summing the frowny.map with cheshire_smile.map. But isn't this map getting less noisy? Yes it is, but the interpretation clearly changes as well. Why does this happen? It is because of a finite resolution cutoff. Oh! What a relief! You don't have super-high resolution, do you? Well, no, almost nobody has signal out beyond 1.0 A, but we do have noise. In diffraction data this noise is removed by simply not measuring it. For map data, however, the problem is that noise at very high frequencies (small-number resolutions) is hard to avoid. This is because of another phenomenon NMR spectroscopists are very familiar with: aliasing, or "folding". If any high-spatial frequency noise exists above half the sampling rate (o
Re: [ccp4bb] happy/sad maps
what about if you run a 2-D FFT rather than 2-D FFT, if your hand drawing map in on a 2-D paper? > On Apr 28, 2023, at 10:48 AM, James Holton wrote: > > Its still April, but this one isn't a joke. > > The smiley-face electron density in the left panel of the attached image has > the remarkable property that any attempt to sharpen or blur the map turns it > into the frowny-face on the right. If you'd like to try this yourself, the > hidden_frown.map file is available in this tarball: > https://bl831.als.lbl.gov/~jamesh/bugreports/fft_042423.tgz > > In fact, any use of an FFT, even with the sharpening B set to zero, turns the > smiley into a frowny face. There is no way to get the smiley face back > (except opening the file again). Yes, that's right, even just a simple > back-and-forth FFT: turning this hidden_frown.map into structure factors and > then back into a map again, gives you a frowny face. This happens using > coot, ccp4 and phenix. > > Wait, what!? Isn't a Fourier transform supposed to preserve information? As > in: you can jump back and forth between real and reciprocal space with > impunity? Without introducing error? Well, yes, it is SUPPOSED to work like > that, but the 3D FFT algorithms of structural biology have a ... quirk. If > you start with structure factors and make a map out of them, you can convert > it back-and-forth as often as you want with 100% preservation of information. > However, if you start with a real-space map (such as from cryoEM), a > back-and-forth conversion gives you a different map. This new map can then be > transformed back-and-forth all you want and be 100% preserved. It has been > "christened" by the FFT, but it is not the same as the starting map, which is > impossible to recover from the FFT-transformed data. Information has been > lost. It is fine for crystallography (which starts with structure factors), > but for techniques such as cryoEM that start with maps, using an FFT changes > the data. > > What information is being lost? Sharp edges. These turn into ripples > covering all of real and reciprocal space. Do real-world data have sharp > edges? Well, the all-or-nothing masks we use to model bulk solvent are one > example. Also, if you "mask off" otherwise smooth density with an > all-or-nothing mask, you will get similar ripples. Another example of a > sharp edge might be the large changes between adjacent pixels you see when a > single electron hits a detector. For example, if you make a map with just one > non-zero voxel and run it back-and-forth through FFT you will find that voxel > loses from 50% to 99% of its value (depending on the size of the map). How > much does this actually impact cryo-EM data? That is my question. > > What evil magic did I wield to make this map? Well, I drew a smiley and > frowny face by hand, converted them to maps, and then I generated random > noise within the boundaries of the smiley face. I ran this noisy map > back-and-forth through FFT, and then subtracted the map that survives the FFT > from the pre-FFT map. This cheshire_smile.map has the interesting property > that all of the structure factors calculated from it are zero. It has an RMSD > of 1.4, but after a back-and-forth FFT this RMSD drops to 1e-7. I generated > the hidden_frown.map by simply summing the frowny.map with cheshire_smile.map. > > But isn't this map getting less noisy? Yes it is, but the interpretation > clearly changes as well. > > Why does this happen? It is because of a finite resolution cutoff. Oh! What > a relief! You don't have super-high resolution, do you? Well, no, almost > nobody has signal out beyond 1.0 A, but we do have noise. In diffraction > data this noise is removed by simply not measuring it. For map data, > however, the problem is that noise at very high frequencies (small-number > resolutions) is hard to avoid. This is because of another phenomenon NMR > spectroscopists are very familiar with: aliasing, or "folding". If any > high-spatial frequency noise exists above half the sampling rate (or "Nyquist > frequency") it still gets recorded, but shows up in a lower-frequency Fourier > coefficient. It is not possible to remove such aliasing noise after > digitization. Upon discretization of the signal (FFT or no) all these high > frequencies join with lower frequency terms, and so survive any low-pass > filtering. Darn. > > Why am I bringing this up? Because if there is noise out beyond the FFT > resolution limit it implies there is also noise out beyond the > Nyquist-Shannon limit as well. If that is the case, direct-space imaging data > may be a lot noisier than it needs to be. In general, in digital signal > processing of things like sound an analog low-pass filter is always installed > at the input of any digitizer. Perhaps this is why de-focusing works better > than being at focus? > > What is the solution? Well, for things like the bulk
[ccp4bb] happy/sad maps
Its still April, but this one isn't a joke. The smiley-face electron density in the left panel of the attached image has the remarkable property that any attempt to sharpen or blur the map turns it into the frowny-face on the right. If you'd like to try this yourself, the hidden_frown.map file is available in this tarball: https://bl831.als.lbl.gov/~jamesh/bugreports/fft_042423.tgz In fact, any use of an FFT, even with the sharpening B set to zero, turns the smiley into a frowny face. There is no way to get the smiley face back (except opening the file again). Yes, that's right, even just a simple back-and-forth FFT: turning this hidden_frown.map into structure factors and then back into a map again, gives you a frowny face. This happens using coot, ccp4 and phenix. Wait, what!? Isn't a Fourier transform supposed to preserve information? As in: you can jump back and forth between real and reciprocal space with impunity? Without introducing error? Well, yes, it is SUPPOSED to work like that, but the 3D FFT algorithms of structural biology have a ... quirk. If you start with structure factors and make a map out of them, you can convert it back-and-forth as often as you want with 100% preservation of information. However, if you start with a real-space map (such as from cryoEM), a back-and-forth conversion gives you a different map. This new map can then be transformed back-and-forth all you want and be 100% preserved. It has been "christened" by the FFT, but it is not the same as the starting map, which is impossible to recover from the FFT-transformed data. Information has been lost. It is fine for crystallography (which starts with structure factors), but for techniques such as cryoEM that start with maps, using an FFT changes the data. What information is being lost? Sharp edges. These turn into ripples covering all of real and reciprocal space. Do real-world data have sharp edges? Well, the all-or-nothing masks we use to model bulk solvent are one example. Also, if you "mask off" otherwise smooth density with an all-or-nothing mask, you will get similar ripples. Another example of a sharp edge might be the large changes between adjacent pixels you see when a single electron hits a detector. For example, if you make a map with just one non-zero voxel and run it back-and-forth through FFT you will find that voxel loses from 50% to 99% of its value (depending on the size of the map). How much does this actually impact cryo-EM data? That is my question. What evil magic did I wield to make this map? Well, I drew a smiley and frowny face by hand, converted them to maps, and then I generated random noise within the boundaries of the smiley face. I ran this noisy map back-and-forth through FFT, and then subtracted the map that survives the FFT from the pre-FFT map. This cheshire_smile.map has the interesting property that all of the structure factors calculated from it are zero. It has an RMSD of 1.4, but after a back-and-forth FFT this RMSD drops to 1e-7. I generated the hidden_frown.map by simply summing the frowny.map with cheshire_smile.map. But isn't this map getting less noisy? Yes it is, but the interpretation clearly changes as well. Why does this happen? It is because of a finite resolution cutoff. Oh! What a relief! You don't have super-high resolution, do you? Well, no, almost nobody has signal out beyond 1.0 A, but we do have noise. In diffraction data this noise is removed by simply not measuring it. For map data, however, the problem is that noise at very high frequencies (small-number resolutions) is hard to avoid. This is because of another phenomenon NMR spectroscopists are very familiar with: aliasing, or "folding". If any high-spatial frequency noise exists above half the sampling rate (or "Nyquist frequency") it still gets recorded, but shows up in a lower-frequency Fourier coefficient. It is not possible to remove such aliasing noise after digitization. Upon discretization of the signal (FFT or no) all these high frequencies join with lower frequency terms, and so survive any low-pass filtering. Darn. Why am I bringing this up? Because if there is noise out beyond the FFT resolution limit it implies there is also noise out beyond the Nyquist-Shannon limit as well. If that is the case, direct-space imaging data may be a lot noisier than it needs to be. In general, in digital signal processing of things like sound an analog low-pass filter is always installed at the input of any digitizer. Perhaps this is why de-focusing works better than being at focus? What is the solution? Well, for things like the bulk solvent mask I'd say some real-space "feathering" is called for before performing FFTs. Same goes for masked density like that used to compute CCmask. It may also be worth looking into the digitization process of "image first" structural biology methods? My question for the BB: can
