Re: [ccp4bb] output individual redundancies
I suppose observations from the same unmerged hkl (also known as a relp) are independent in some ways, but not in others. For example, partials of the same relp are independent with respect to photon-counting noise, detector read-out noise, flicker in the source, and most types of shutter jitter and sample vibration, but not independent when it comes to other kinds of error, like detector pixel calibration and absorption errors. Depending on which of these sources of error dominates, a given pair of observations may or may not be independent. With CuKa radiation, absorption errors usually dominate, so I suppose it does make sense to treat partials of the same relp as non-independent observations. However, I think it is important that at ~1 A, absorption errors are much smaller. Perhaps a better name for absorption errors is differential attenuation, since they arise not from absorption itself, but rather the difference in Beer's-law attenuation along two different paths taken by the x-rays through the crystal and out to two different spots. For example, an extra 100 um of water or protein along one path vs another will create a 10% difference in spot intensities with a CuKa radiation, but less than 3% with 1 A x-rays. Scaling programs attempt to correct for this, but depending on how crazy the crystal/drop/loop/etc. shape is, they may or may not be able to do it perfectly. It is also hard to calibrate a detector to better than a few percent, but that is a very long story. Of course, errors of a few percent will never be important for refinement, where the model-data systematic error (Rcryst/Rfree 15%) dominates. But for anomalous differences a few percent is the magnitude of the signal you are trying to measure! So, it can be easy to convince SCALA (and indeed oneself) that you are getting good signal-to-noise (DANO/SIGDANO), even if you're not. This is especially true if you measure the same phi range over and over again. For example, if you average n=100 observations, each with sigma = 1.0, then you usually assign a sigma to the average value that is 1/sqrt(n) smaller, or 0.1. This is why the signal to noise ratio improves with averaging. Or, at least, we think it does. If the error in all 100 observations is a systematic shift in the same direction, then the true error of the average is not 0.1, but actually closer to 1.0. My point is, multiplicity is not the whole story. It really comes down to the sigmas and how realistic they actually are. You never know, the extra measurements in your high-multiplicity dataset might really be redundant. -James Holton MAD Scientist On Fri, Jul 15, 2011 at 11:37 AM, Phil Evans p...@mrc-lmb.cam.ac.uk wrote: Summed partials count as one. SCALA doesn't adjust for 360deg, maybe it should as they are not independent. What would you call them? I prefer multiplicity since Elspeth Garman commented if they are redundant why bother measuring them Phil Sent from my iPhone On 15 Jul 2011, at 18:24, James Holton jmhol...@lbl.gov wrote: At the risk of asking a question to which I should already know the answer: do partials count as redundancy? That is, in SCALA, is the number of observations the number of recorded spots? Or is it the number of recorded spots after adding partials? If it is the latter, what happens if you collect more than 360 degrees of data? Does the second pass through a given unmerged hkl index count as more partials or is it now somehow upgraded to an independent observation? Then again, in Eastern English the word redundancy has a negative connotation, and the output of SCALA actually uses the word multiplicity. I wonder if that makes unmerged partials redundant? -James Holton MAD Scientist On 7/15/2011 8:09 AM, Ed Pozharski wrote: On Fri, 2011-07-15 at 09:26 +0100, Phil Evans wrote: Ed. You could count them from the unmerged output as you say, or I could make you a special version of SCALA or Aimless maybe next week Phil, that would be fantastic! Hope there is broader interest in such option (beyond Robbie and myself). I'll try unmerged output in the meantime. Ed.
Re: [ccp4bb] output individual redundancies
No M/ISYM is different it's the symmetry number plus a full or partial flag. Ed. You could count them from the unmerged output as you say, or I could make you a special version of SCALA or Aimless maybe next week Phil Sent from my iPhone On 14 Jul 2011, at 23:15, Ethan Merritt merr...@u.washington.edu wrote: On Thursday, July 14, 2011 02:55:26 pm Ed Pozharski wrote: I am looking for a way to output redundancy per individual reflection, preferably for scala but if that is not possible then maybe for scalepack. If you read the unmerged file from scalepack into ccp4 using combat, it creates a data column with label M/ISYM that I think is what you are asking for. You can use the Import Unmerged Data (Combat) tab in the ccp4i GUI. Ethan From my (admittedly quick) look at the scala manual it seems that I can use something like UNMERGED output option to exclude outliers and then would need to write a bit of code to calculate the redundancies. But I hope that I missed something and there is a secret keyword that would add redundancies to the merged mtz file. Cheers, Ed. -- Ethan A Merritt Biomolecular Structure Center, K-428 Health Sciences Bldg University of Washington, Seattle 98195-7742
Re: [ccp4bb] output individual redundancies
Hi Ed, I was recently looking for that value myself, but couldn't find it. I suppose (at some point) it may be useful information to deposit. If something is a mean value, it is nice to know how many individual values were used to construct that mean. Unfortunately, there doesn't seem to be a cif token for that. Cheers, Robbie Date: Fri, 15 Jul 2011 09:26:39 +0100 From: p...@mrc-lmb.cam.ac.uk Subject: Re: [ccp4bb] output individual redundancies To: CCP4BB@JISCMAIL.AC.UK No M/ISYM is different it's the symmetry number plus a full or partial flag. Ed. You could count them from the unmerged output as you say, or I could make you a special version of SCALA or Aimless maybe next week Phil Sent from my iPhone On 14 Jul 2011, at 23:15, Ethan Merritt merr...@u.washington.edu wrote: On Thursday, July 14, 2011 02:55:26 pm Ed Pozharski wrote: I am looking for a way to output redundancy per individual reflection, preferably for scala but if that is not possible then maybe for scalepack. If you read the unmerged file from scalepack into ccp4 using combat, it creates a data column with label M/ISYM that I think is what you are asking for. You can use the Import Unmerged Data (Combat) tab in the ccp4i GUI. Ethan From my (admittedly quick) look at the scala manual it seems that I can use something like UNMERGED output option to exclude outliers and then would need to write a bit of code to calculate the redundancies. But I hope that I missed something and there is a secret keyword that would add redundancies to the merged mtz file. Cheers, Ed. -- Ethan A Merritt Biomolecular Structure Center, K-428 Health Sciences Bldg University of Washington, Seattle 98195-7742
Re: [ccp4bb] output individual redundancies
On Fri, 2011-07-15 at 09:26 +0100, Phil Evans wrote: Ed. You could count them from the unmerged output as you say, or I could make you a special version of SCALA or Aimless maybe next week Phil, that would be fantastic! Hope there is broader interest in such option (beyond Robbie and myself). I'll try unmerged output in the meantime. Ed. -- I'd jump in myself, if I weren't so good at whistling. Julian, King of Lemurs
Re: [ccp4bb] output individual redundancies
At the risk of asking a question to which I should already know the answer: do partials count as redundancy? That is, in SCALA, is the number of observations the number of recorded spots? Or is it the number of recorded spots after adding partials? If it is the latter, what happens if you collect more than 360 degrees of data? Does the second pass through a given unmerged hkl index count as more partials or is it now somehow upgraded to an independent observation? Then again, in Eastern English the word redundancy has a negative connotation, and the output of SCALA actually uses the word multiplicity. I wonder if that makes unmerged partials redundant? -James Holton MAD Scientist On 7/15/2011 8:09 AM, Ed Pozharski wrote: On Fri, 2011-07-15 at 09:26 +0100, Phil Evans wrote: Ed. You could count them from the unmerged output as you say, or I could make you a special version of SCALA or Aimless maybe next week Phil, that would be fantastic! Hope there is broader interest in such option (beyond Robbie and myself). I'll try unmerged output in the meantime. Ed.
Re: [ccp4bb] output individual redundancies
Summed partials count as one. SCALA doesn't adjust for 360deg, maybe it should as they are not independent. What would you call them? I prefer multiplicity since Elspeth Garman commented if they are redundant why bother measuring them Phil Sent from my iPhone On 15 Jul 2011, at 18:24, James Holton jmhol...@lbl.gov wrote: At the risk of asking a question to which I should already know the answer: do partials count as redundancy? That is, in SCALA, is the number of observations the number of recorded spots? Or is it the number of recorded spots after adding partials? If it is the latter, what happens if you collect more than 360 degrees of data? Does the second pass through a given unmerged hkl index count as more partials or is it now somehow upgraded to an independent observation? Then again, in Eastern English the word redundancy has a negative connotation, and the output of SCALA actually uses the word multiplicity. I wonder if that makes unmerged partials redundant? -James Holton MAD Scientist On 7/15/2011 8:09 AM, Ed Pozharski wrote: On Fri, 2011-07-15 at 09:26 +0100, Phil Evans wrote: Ed. You could count them from the unmerged output as you say, or I could make you a special version of SCALA or Aimless maybe next week Phil, that would be fantastic! Hope there is broader interest in such option (beyond Robbie and myself). I'll try unmerged output in the meantime. Ed.
Re: [ccp4bb] output individual redundancies
Hi, I was wondering if anyone knows what HKL 2000 does? Does it merge all partials and treat it as one, because often times I noticed with increase in partials the redundancy increases. Shya On Fri, Jul 15, 2011 at 1:24 PM, James Holton jmhol...@lbl.gov wrote: At the risk of asking a question to which I should already know the answer: do partials count as redundancy? That is, in SCALA, is the number of observations the number of recorded spots? Or is it the number of recorded spots after adding partials? If it is the latter, what happens if you collect more than 360 degrees of data? Does the second pass through a given unmerged hkl index count as more partials or is it now somehow upgraded to an independent observation? Then again, in Eastern English the word redundancy has a negative connotation, and the output of SCALA actually uses the word multiplicity. I wonder if that makes unmerged partials redundant? -James Holton MAD Scientist On 7/15/2011 8:09 AM, Ed Pozharski wrote: On Fri, 2011-07-15 at 09:26 +0100, Phil Evans wrote: Ed. You could count them from the unmerged output as you say, or I could make you a special version of SCALA or Aimless maybe next week Phil, that would be fantastic! Hope there is broader interest in such option (beyond Robbie and myself). I'll try unmerged output in the meantime. Ed.
Re: [ccp4bb] output individual redundancies
For each observation or measurement, HKL, d*TREK, and all other common programs to my knowledge add up the bits of each relfection from its parts. They do not add parts of of other reflections (even if symmetry-related) into a reflection. Reflections for which only part of the Bragg peak is measured are tossed out. For example, at the beginning and end of a scan there are partial reflections which cannot be made full. (I am aware that one can take a partial measurement and scale it up by the inverse of its so-called partiality to make it into a fake full reflection.) One can use the NO MERGE ORIGINAL INDEX macro of scalepack and output the individual measurements of denzo or HK that have had scale factors applied. This file can be a converted to a d*TREK reflectiion file with SCA2DTREK and then the individual measurements averaged with dtscaleaverage to get statistics such as Rmeas, completeness, reduced ChiSq, multiplicity, and a Table 1 suitable for framing in your Nature paper. The unique reflection list output could have the multiplicity of each unique reflection added pretty trivially if there is a demand for this. Jim _ From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Shya Biswas Sent: Friday, July 15, 2011 6:20 PM To: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] output individual redundancies Hi, I was wondering if anyone knows what HKL 2000 does? Does it merge all partials and treat it as one, because often times I noticed with increase in partials the redundancy increases. Shya On Fri, Jul 15, 2011 at 1:24 PM, James Holton jmhol...@lbl.gov wrote: At the risk of asking a question to which I should already know the answer: do partials count as redundancy? That is, in SCALA, is the number of observations the number of recorded spots? Or is it the number of recorded spots after adding partials? If it is the latter, what happens if you collect more than 360 degrees of data? Does the second pass through a given unmerged hkl index count as more partials or is it now somehow upgraded to an independent observation? Then again, in Eastern English the word redundancy has a negative connotation, and the output of SCALA actually uses the word multiplicity. I wonder if that makes unmerged partials redundant? -James Holton MAD Scientist On 7/15/2011 8:09 AM, Ed Pozharski wrote: On Fri, 2011-07-15 at 09:26 +0100, Phil Evans wrote: Ed. You could count them from the unmerged output as you say, or I could make you a special version of SCALA or Aimless maybe next week Phil, that would be fantastic! Hope there is broader interest in such option (beyond Robbie and myself). I'll try unmerged output in the meantime. Ed.
[ccp4bb] output individual redundancies
I am looking for a way to output redundancy per individual reflection, preferably for scala but if that is not possible then maybe for scalepack. From my (admittedly quick) look at the scala manual it seems that I can use something like UNMERGED output option to exclude outliers and then would need to write a bit of code to calculate the redundancies. But I hope that I missed something and there is a secret keyword that would add redundancies to the merged mtz file. Cheers, Ed. -- I'd jump in myself, if I weren't so good at whistling. Julian, King of Lemurs
Re: [ccp4bb] output individual redundancies
On Thursday, July 14, 2011 02:55:26 pm Ed Pozharski wrote: I am looking for a way to output redundancy per individual reflection, preferably for scala but if that is not possible then maybe for scalepack. If you read the unmerged file from scalepack into ccp4 using combat, it creates a data column with label M/ISYM that I think is what you are asking for. You can use the Import Unmerged Data (Combat) tab in the ccp4i GUI. Ethan From my (admittedly quick) look at the scala manual it seems that I can use something like UNMERGED output option to exclude outliers and then would need to write a bit of code to calculate the redundancies. But I hope that I missed something and there is a secret keyword that would add redundancies to the merged mtz file. Cheers, Ed. -- Ethan A Merritt Biomolecular Structure Center, K-428 Health Sciences Bldg University of Washington, Seattle 98195-7742
