Re: [ccp4bb] What really happens in XDSCONV?
Dear Kay, Concerning usage of programs, everybody has his/her preferences, but what could be simpler than a 2-liner XDSCONV.INP like INPUT_FILE=XDS_ASCII.HKL OUTPUT_FILE=temp.hkl CCP4 ! or CCP4_F or CCP4_I or SHELX or CNS and then running XDSCONV by running xdsconv? At least there's not much room for mistakes. Exactly! I beg to disagree, and this was the point of my question. The output of XDSCONV literally says that 190093 reflections are read, and [of those, my interpretation] 44047 are accepted. I may be a pedant but I can't read that output in any other way. To me it looks like it is reading only the reflections that already fall into the asymmetric unit and is ignoring all the others. So if XDSCONV is really doing what it is supposed to, I would suggest rephrasing that output line. good point about the rephrasing. I'll see to making the wording consistent between XDSCONV and XDS. Thanks for that. But irrespective of the wording, it does take all observations into account when calculating the intensity (and amplitude) of the unique reflections (as it should - ignoring reflections would not make sense, and would produce significantly worse data). Great, this was just the reassurance I was looking for. However, it does so not by calculating the geometric mean (which you seem to assume), but by calculating the weighted mean. Weighting is done with the variances, and here it also does not differ from (c)truncate or other programs. Sorry, that was a typo born of thinking I could quote the manual from memory. But yes, pedantry aside, I will start using pointless in my csh pipelines. please report back whether that changes (or even improves) your results! It is always very good when people compare programs in a meaningful way, but from my own experience I can say that meaningful comparisons are sometimes not entirely straightforward to get right. (for the German-speaking: wer misst, misst Mist!) Probably I will still be in too much of a hurry each time to make a meaningful and thorough comparison, but if time allows I will compare XDSCONV with pointless/scala. /Derek
Re: [ccp4bb] What really happens in XDSCONV?
Hi Derek I strongly recommend comparing with pointless/aimless, *not* pointless/ scala (if you have the time!) Scala is obsolete and was superseded by Aimless several years ago (I think 2010, without checking...). I find that Aimless is not only much faster than Scala, but also scales better. On 16 Feb 2014, at 19:26, Derek Logan wrote: Probably I will still be in too much of a hurry each time to make a meaningful and thorough comparison, but if time allows I will compare XDSCONV with pointless/scala. /Derek Harry -- ** note change of address ** Dr Harry Powell, MRC Laboratory of Molecular Biology, Francis Crick Avenue, Cambridge Biomedical Campus, Cambridge, CB2 0QH
Re: [ccp4bb] What really happens in XDSCONV?
On Fri, 14 Feb 2014 17:25:36 +, Derek Logan derek.lo...@biochemistry.lu.se wrote: Hi Tim, I would actually recommend using pointless (or xprep) instead of xdsconv. It is much easier to use and maybe even less error prone. I take the point about pointless but XPREP is commercial software sold by Bruker and costs �700 (as I remember), so is not really an option for everyone. pointless is a good program, and I also use it, but to call it a replacement for XDSCONV would be wrong. But yes, it can be used to convert XDS_ASCII.HKL to a multi-record MTZ file (this is just a format conversion!) which is then fed into CCP4 truncate (which calculates amplitudes from intensities). At that point, you have replaced one implementation of FrenchWilson 1978 by a different one. Which one is better is a possible question (and you _can_ find meaningful information about this in the CCP4BB archives), but the differences are probably not very significant for real data. I didn't even know that xprep calculates amplitudes from intensities, but I wonder why it should do it any better than other well-tested programs. Concerning usage of programs, everybody has his/her preferences, but what could be simpler than a 2-liner XDSCONV.INP like INPUT_FILE=XDS_ASCII.HKL OUTPUT_FILE=temp.hkl CCP4 ! or CCP4_F or CCP4_I or SHELX or CNS and then running XDSCONV by running xdsconv? At least there's not much room for mistakes. Within XDSGUI (http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/XDSGUI) , running XDSCONV is a matter of two mouse clicks, nothing else. All your quotes from the output are perfectly consistent. The first table tells you there are 190093 unmerged reflections in total, but only 44059 unique reflections. Hence if you ask xdsconv to merge the output (MERGE=TRUE), it will do so and only write 44047 (a few less than 44059 because it rejects those 19 unmerged reflections with I-3sigma). I beg to disagree, and this was the point of my question. The output of XDSCONV literally says that 190093 reflections are read, and [of those, my interpretation] 44047 are accepted. I may be a pedant but I can't read that output in any other way. To me it looks like it is reading only the reflections that already fall into the asymmetric unit and is ignoring all the others. So if XDSCONV is really doing what it is supposed to, I would suggest rephrasing that output line. good point about the rephrasing. I'll see to making the wording consistent between XDSCONV and XDS. But irrespective of the wording, it does take all observations into account when calculating the intensity (and amplitude) of the unique reflections (as it should - ignoring reflections would not make sense, and would produce significantly worse data). However, it does so not by calculating the geometric mean (which you seem to assume), but by calculating the weighted mean. Weighting is done with the variances, and here it also does not differ from (c)truncate or other programs. The documentation tells you: MERGE=TRUE means that the weighted mean of symmetry equivalent reflection intensities appearing in the input file will be determined and used in the output file. Yes, I paraphrased this in my original mail. I like to believe that the manual describes what the program is doing, but the output isn't consistent with this in my view. xscale does not scale data in a crystallographic sense since scaling is already done in the CORRECT step of XDS. It put the data on a common scale (in a sophisticated manner), i.e. if you only have one data set there is no need to run xscale except for the thinner shells for the statistics table compared to CORRECT. No need indeed, except that including XSCALE doesn't leave me sleepless about the data having been merged. And maybe correction for radiation damage when you have sufficient multiplicity? But yes, pedantry aside, I will start using pointless in my csh pipelines. please report back whether that changes (or even improves) your results! It is always very good when people compare programs in a meaningful way, but from my own experience I can say that meaningful comparisons are sometimes not entirely straightforward to get right. (for the German-speaking: wer misst, misst Mist!) good luck, Kay Best wishes Derek On 02/14/2014 03:57 PM, Derek Logan wrote: Hi, I am a long time user of XDS (20 years this year) but all the same I find that I have constant angst about losing observations because I don't understand what goes in in the conversion steps to get to CCP4 format. I used to believe that XSCALE was always necessary, and I always use it in my workflow even if there is only one dataset (after all, there's nothing to lose), but my Ph.D. students pointed out to me that XDSCONV could take output directly from CORRECT, and they often do it this way. The XDS wiki and XDSCONV docs seems to confirm this: using MERGE=TRUE in XDSCONV should output
Re: [ccp4bb] What really happens in XDSCONV?
On Sat, 15 Feb 2014 09:50:43 +, Kay Diederichs kay.diederi...@uni-konstanz.de wrote: pointless is a good program, and I also use it, to clarify: I use pointless for space group determination. I don't use it for producing multi-record MTZ files, since I use XDSCONV.
[ccp4bb] What really happens in XDSCONV?
Hi, I am a long time user of XDS (20 years this year) but all the same I find that I have constant angst about losing observations because I don't understand what goes in in the conversion steps to get to CCP4 format. I used to believe that XSCALE was always necessary, and I always use it in my workflow even if there is only one dataset (after all, there's nothing to lose), but my Ph.D. students pointed out to me that XDSCONV could take output directly from CORRECT, and they often do it this way. The XDS wiki and XDSCONV docs seems to confirm this: using MERGE=TRUE in XDSCONV should output the geometrical mean of the observations. However I am worried by what the XDSCONV output says. For today's example CORRECT gives me this: NUMBER OF REFLECTIONS IN SELECTED SUBSET OF IMAGES 190093 NUMBER OF REJECTED MISFITS3842 NUMBER OF SYSTEMATIC ABSENT REFLECTIONS 34 NUMBER OF ACCEPTED OBSERVATIONS 186217 NUMBER OF UNIQUE ACCEPTED REFLECTIONS44059 XDSCONV says the following: FRIEDEL'S_LAW=TRUE MERGE=TRUE NUMBER OF REFLECTION RECORDS ON INPUT FILE 190093 NUMBER OF IGNORED REFLECTIONS (I -3.0*SIGMA) 19 NUMBER OF REFLECTIONS ACCEPTED FROM INPUT FILE 44047 To my literal mind this says it is throwing away most of the observations. Now if I merge the reflections in XSCALE first it says this: FRIEDEL'S_LAW=TRUE MERGE=TRUE NUMBER OF REFLECTION RECORDS ON INPUT FILE 44046 NUMBER OF IGNORED REFLECTIONS (I -3.0*SIGMA)0 NUMBER OF REFLECTIONS ACCEPTED FROM INPUT FILE 44046 which makes more sense. If I compare the first few reflections of the output file with structure factor amplitudes from XDSCONV for each scenario they are different, but I believe that is because XSCALE has put the intensities on an absolute scale and CORRECT has not. Basically all I want to know is that the output from XDSCONV is misleadingly worded, i.e. that even if it appears to say that only the asymmetric unit has been accepted, actually all observations have gone in and the geometric mean is indeed output. That would put my mind to rest! /Derek Derek Logan tel: +46 46 222 1443 Associate Professor mob: +46 76 8585 707 Dept. of Biochemistry and Structural Biology www.cmps.lu.sehttp://www.cmps.lu.se Centre for Molecular Protein Science www.maxlab.lu.se/node/307http://www.maxlab.lu.se/node/307 Lund University, Box 124, 221 00 Lund, Sweden
Re: [ccp4bb] What really happens in XDSCONV?
Hi Tim, I would actually recommend using pointless (or xprep) instead of xdsconv. It is much easier to use and maybe even less error prone. I take the point about pointless but XPREP is commercial software sold by Bruker and costs €700 (as I remember), so is not really an option for everyone. All your quotes from the output are perfectly consistent. The first table tells you there are 190093 unmerged reflections in total, but only 44059 unique reflections. Hence if you ask xdsconv to merge the output (MERGE=TRUE), it will do so and only write 44047 (a few less than 44059 because it rejects those 19 unmerged reflections with I-3sigma). I beg to disagree, and this was the point of my question. The output of XDSCONV literally says that 190093 reflections are read, and [of those, my interpretation] 44047 are accepted. I may be a pedant but I can't read that output in any other way. To me it looks like it is reading only the reflections that already fall into the asymmetric unit and is ignoring all the others. So if XDSCONV is really doing what it is supposed to, I would suggest rephrasing that output line. The documentation tells you: MERGE=TRUE means that the weighted mean of symmetry equivalent reflection intensities appearing in the input file will be determined and used in the output file. Yes, I paraphrased this in my original mail. I like to believe that the manual describes what the program is doing, but the output isn't consistent with this in my view. xscale does not scale data in a crystallographic sense since scaling is already done in the CORRECT step of XDS. It put the data on a common scale (in a sophisticated manner), i.e. if you only have one data set there is no need to run xscale except for the thinner shells for the statistics table compared to CORRECT. No need indeed, except that including XSCALE doesn't leave me sleepless about the data having been merged. And maybe correction for radiation damage when you have sufficient multiplicity? But yes, pedantry aside, I will start using pointless in my csh pipelines. Best wishes Derek On 02/14/2014 03:57 PM, Derek Logan wrote: Hi, I am a long time user of XDS (20 years this year) but all the same I find that I have constant angst about losing observations because I don't understand what goes in in the conversion steps to get to CCP4 format. I used to believe that XSCALE was always necessary, and I always use it in my workflow even if there is only one dataset (after all, there's nothing to lose), but my Ph.D. students pointed out to me that XDSCONV could take output directly from CORRECT, and they often do it this way. The XDS wiki and XDSCONV docs seems to confirm this: using MERGE=TRUE in XDSCONV should output the geometrical mean of the observations. However I am worried by what the XDSCONV output says. For today's example CORRECT gives me this: NUMBER OF REFLECTIONS IN SELECTED SUBSET OF IMAGES 190093 NUMBER OF REJECTED MISFITS3842 NUMBER OF SYSTEMATIC ABSENT REFLECTIONS 34 NUMBER OF ACCEPTED OBSERVATIONS 186217 NUMBER OF UNIQUE ACCEPTED REFLECTIONS44059 XDSCONV says the following: FRIEDEL'S_LAW=TRUE MERGE=TRUE NUMBER OF REFLECTION RECORDS ON INPUT FILE 190093 NUMBER OF IGNORED REFLECTIONS (I -3.0*SIGMA) 19 NUMBER OF REFLECTIONS ACCEPTED FROM INPUT FILE 44047 To my literal mind this says it is throwing away most of the observations. Now if I merge the reflections in XSCALE first it says this: FRIEDEL'S_LAW=TRUE MERGE=TRUE NUMBER OF REFLECTION RECORDS ON INPUT FILE 44046 NUMBER OF IGNORED REFLECTIONS (I -3.0*SIGMA) 0 NUMBER OF REFLECTIONS ACCEPTED FROM INPUT FILE 44046 which makes more sense. If I compare the first few reflections of the output file with structure factor amplitudes from XDSCONV for each scenario they are different, but I believe that is because XSCALE has put the intensities on an absolute scale and CORRECT has not. Basically all I want to know is that the output from XDSCONV is misleadingly worded, i.e. that even if it appears to say that only the asymmetric unit has been accepted, actually all observations have gone in and the geometric mean is indeed output. That would put my mind to rest! /Derek Derek Logan tel: +46 46 222 1443 Associate Professor mob: +46 76 8585 707 Dept. of Biochemistry and Structural Biology www.cmps.lu.sehttp://www.cmps.lu.se Centre for Molecular Protein Science www.maxlab.lu.se/node/307http://www.maxlab.lu.se/node/307 Lund University, Box 124, 221 00 Lund, Sweden - -- - -- Dr Tim Gruene Institut fuer anorganische Chemie Tammannstr. 4 D-37077 Goettingen GPG Key ID = A46BEE1A -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12
