Re: [ccp4bb] how to ignore spot overlap in imosflm?
Hi, there are two points in Herman's post that I'd like to comment upon: 1) in case of XDS, there are two modes of reporting the completeness: one is triggered with FRIEDEL'S_LAW=TRUE, and the other with FRIEDEL'S_LAW=FALSE. Obviously the reported completeness will differ between these two modes, since the latter mode treats Friedel pairs as separate reflection, whereas the former doesn't. However, the XDS_ASCII.HKL file which has the reflections' intensities and standard deviations for downstream usage is almost the same (except for small differences in scaling) in both cases. This means that the actual completeness of the isomorphous signal (which implicitly does not care about whether an intensity is I+ or I-) is the same in both modes, which has the consequence that for molecular replacement and refinement calculations it does not matter which mode was used for producing XDS_ASCII.HKL . In other words, only concerning the numbers in the famous Table 1 you have to pay attention in which mode you produce the statistics reported in CORRECT.LP/XSCALE.LP, not for calculations which make no use of the anomalous signal. 2) to lie to the program by specifying a very low REFLECTING_RANGE_E.S.D. is _not_ the best way to make XDS produce a more complete dataset! The right way is to specify a lower MINPK than the default of 75 - see http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/INTEGRATE HTH, Kay
Re: [ccp4bb] how to ignore spot overlap in imosflm?
Hi Xinghua, Your completeness is very low and even with a perfect model the quality of your maps will suffer. Two things may help if you have not already done them: 1) Merge the Friedel pairs. If you have kept F+ and F- apart so far, you could get significantly more complete data by merging them. 2) As has been suggested before, try a three-dimensional profile fitting program (XDS) and give it a very low mosaicity (reflection_range in XDS) of say 0.1-0.2° and do not refine this value. In this way you will only use the central slice of the spots, which is not the optimal method but still much better than discarding the spots altogether. The much more complete data obtained this way should give much better maps. Best, Herman From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Xinghua Qin Sent: Tuesday, May 15, 2012 6:39 PM To: CCP4BB@JISCMAIL.AC.UK Subject: [ccp4bb] how to ignore spot overlap in imosflm? Dear all, Thanks for all of the suggestions, It helps me a lot as I am a newcomer to the structural world. Increasing the Profile Tolerance parameters as Dr Harry Powell has pointed out can increase the completeness by ten percent (from 50% to 60%). I will try other people's advice soon. Although the completeness is quite low (just 50%), now I have determined the structure (Rfree=0.31, Rfactor=0.24, resolution=2.6). I will look into the map with COOT, and try to get the structure more beautiful. Thanks again Best wishes Xinghua Qin -- Xinghua Qin State Key Laboratory of Plant Physiology and biochemistry College of Biological Sciences China Agricultural University No.2, Yuan Ming Yuan West Road Haidian District, Beijing, China 100193 Tel: +86-10-62732672 E-mail: xinghua...@126.com mailto:xhqin1...@gmail.com mailto:xhqin1...@gmail.com At 2012-05-14 11:48:58,Zhijie Li zhijie...@utoronto.ca wrote: Hi Xinghua, The total intensity of each reflection needs to be accurately quantitated in order to calculate the structure factors. Not only the dots need to be well separated in the 3D reciprocal space, but also a small area around the dots are often needed to calculate the background for subtraction. That is why when two dots are getting too close, the programs will reject both dots. The first thing you need to do is to inspect the images reported with large number of overlaps to see if the dots are really overlapping or just close to each other. If the dots are barely touching or just too close to each other, you can manipulate the SEPERATION parameter to force the program to take the closely spaced spots. But keep in mind that you may get less accurate integration by doing so. If many spots are really touching each other, normally we won't force the programs to use them. Then the proper ! remedy is to move the detector farther and collect the dataset again (also, try to optimize your freezing to get the mosaicity as low as possible). For how to play with the mosflm parameters, please read here: http://www.mrc-lmb.cam.ac.uk/harry/cgi-bin/keyword2.cgi?SEPARATION. What you need is probably CLOSE. The hazard of high percentage of overlaps: If the overlaps are only scattered in a whole dataset, it is OK, even if they make up 5-10% or even 20% of the whole dataset. It will only give you a lower completeness, which is not too detrimental to the structure solution. However, if large, continuous regions in the dataset are missing, that will cause you to have poorly defined regions in the calculated map, often seen as featureless stripes or layers in the map. Unfortunately, when you have closely spaced reflections, the latter is often the case. The proper solution is to collect the data at a greater detector distance to resolve the spots (after taking the test images, both imosflm and HKL2000 can simulate the collection run to help you to decide what distance you need). In cases that you have a long unit cell (200A), the first thing you need to do is to align the long edge of the Unix cell with the rotational axis of the! pin. In the difficult cases, you probably even need to shoot multiple crystals and combine the datasets to get enough completeness. Zhijie From: Xinghua Qin Sent: Sunday, May 13, 2012 10:22 PM To: CCP4BB@JISCMAIL.AC.UK Subject: [ccp4bb] how to ignore spot overlap in imosflm? Dear CCP4ers, We collected a diffraction
[ccp4bb] how to ignore spot overlap in imosflm?
Dear all, Thanks for all of the suggestions, It helps me a lot as I am a newcomer to the structural world. Increasing the Profile Tolerance parametersas Dr Harry Powell has pointed out can increase the completeness by ten percent (from 50% to 60%). I will try other people's advice soon. Although the completeness is quite low (just 50%), now I have determined the structure (Rfree=0.31, Rfactor=0.24, resolution=2.6). I will look into the map with COOT, and try to get the structure more beautiful. Thanks again Best wishes Xinghua Qin -- Xinghua Qin State Key Laboratory of Plant Physiology and biochemistry College of Biological Sciences China Agricultural University No.2, Yuan Ming Yuan West Road Haidian District, Beijing, China 100193 Tel: +86-10-62732672 E-mail: xinghua...@126.com At 2012-05-14 11:48:58,Zhijie Li zhijie...@utoronto.ca wrote: Hi Xinghua, The total intensity of each reflection needs to be accurately quantitated in order to calculate the structure factors. Not only the dots need to be well separated in the 3D reciprocal space, but also a small area around the dots are often needed to calculate the background for subtraction. That is why when two dots are getting too close, the programs will reject both dots. The first thing you need to do is to inspect the images reported with large number of overlaps to see if the dots are really overlapping or just close to each other. If the dots are barely touching or just too close to each other, you can manipulate the SEPERATION parameter to force the program to take the closely spaced spots. But keep in mind that you may get less accurate integration by doing so. If many spots are really touching each other, normally we won't force the programs to use them. Then the proper remedy is to move the detector farther and collect the dataset again (also, try to optimize your f! reezing to get the mosaicity as low as possible). For how to play with the mosflm parameters, please read here: http://www.mrc-lmb.cam.ac.uk/harry/cgi-bin/keyword2.cgi?SEPARATION. What you need is probably CLOSE. The hazard of high percentage of overlaps: If the overlaps are only scattered in a whole dataset, it is OK, even if they make up 5-10% or even 20% of the whole dataset. It will only give you a lower completeness, which is not too detrimental to the structure solution. However, if large, continuous regions in the dataset are missing, that will cause you to have poorly defined regions in the calculated map, often seen as featureless stripes or layers in the map. Unfortunately, when you have closely spaced reflections, the latter is often the case. The proper solution is to collect the data at a greater detector distance to resolve the spots (after taking the test images, both imosflm and HKL2000 can simulate the collection run to help you to decide what distance you need). In cases that you have a long unit cell (200A), the first thing you need to do is to align the long edge of the Unix cell with the rotational axis of the pin. In the difficult cases, you probably even need to shoot multiple crystals and combine the ! datasets to get enough completeness. Zhijie From:Xinghua Qin Sent: Sunday, May 13, 2012 10:22 PM To:CCP4BB@JISCMAIL.AC.UK Subject: [ccp4bb] how to ignore spot overlap in imosflm? Dear CCP4ers, We collected a diffraction dataset with high percentage of spot overlaps, It would be so kind to tell me how to ignore spot overlap in imosflm and explain the hazard of high percentage of spot overlaps. Thanks in advance. Best wishes Xinghua Qin -- Xinghua Qin State Key Laboratory of Plant Physiology and biochemistry College of Biological Sciences China Agricultural University No.2, Yuan Ming Yuan West Road Haidian District, Beijing, China 100193 Tel: +86-10-62732672 E-mail: xinghua...@126.com
Re: [ccp4bb] how to ignore spot overlap in imosflm?
Hi This is all good advice, but there is more that you could do if you're desperate to use these images. Having made sure that your spots on the images are not actually overlapping (i.e. by looking closely at the images), but just flagged as overlapping in iMosflm, you may be able (i.e. no guarantees) to persuade Mosflm to use them by - (i) go to Processing Options-Advanced Integration and increase the values for the Profiles-Tolerance. There's a tool-tip that becomes visible if you hover the mouse cursor over the entry boxes which gives more advice on values. (ii) if you're *really* desperate, and you have noticed that the overall box width and box height has increased to values much bigger than your spots (e.g. your spots are really only ~5-10 pixels across, but the box size has increased to ~30) you could try setting the overall box size to ~the separation distance in pixels and UNchecking the Optimise overall box size check box - this will fix the overall dimensions but allow the spot size within it to optimise. This might get you out of a hole... As Zhije says, the proper solution is to collect the data without overlaps, though. Either of the above steps will reduce the measurement quality, though (i) is better than (ii). A rule of thumb is to set the crystal to detector distance (in mm) to at least [maximum cell edge(Å)/wavelength(Å)] (pedants might want to multiply the RHS of that by 1mm. There are better methods than rules of thumb, though, e.g. the strategy options that are now widely available. On 14 May 2012, at 04:48, Zhijie Li wrote: Hi Xinghua, The total intensity of each reflection needs to be accurately quantitated in order to calculate the structure factors. Not only the dots need to be well separated in the 3D reciprocal space, but also a small area around the dots are often needed to calculate the background for subtraction. That is why when two dots are getting too close, the programs will reject both dots. The first thing you need to do is to inspect the images reported with large number of overlaps to see if the dots are really overlapping or just close to each other. If the dots are barely touching or just too close to each other, you can manipulate the SEPERATION parameter to force the program to take the closely spaced spots. But keep in mind that you may get less accurate integration by doing so. If many spots are really touching each other, normally we won't force the programs to use them. Then the proper remedy is to move the detector farther and collect the dataset again (also, try to optimize your freezing to get the mosaicity as low as possible). For how to play with the mosflm parameters, please read here: http://www.mrc-lmb.cam.ac.uk/harry/cgi-bin/keyword2.cgi?SEPARATION. What you need is probably CLOSE. The hazard of high percentage of overlaps: If the overlaps are only scattered in a whole dataset, it is OK, even if they make up 5-10% or even 20% of the whole dataset. It will only give you a lower completeness, which is not too detrimental to the structure solution. However, if large, continuous regions in the dataset are missing, that will cause you to have poorly defined regions in the calculated map, often seen as featureless stripes or layers in the map. Unfortunately, when you have closely spaced reflections, the latter is often the case. The proper solution is to collect the data at a greater detector distance to resolve the spots (after taking the test images, both imosflm and HKL2000 can simulate the collection run to help you to decide what distance you need). In cases that you have a long unit cell (200A), the first thing you need to do is to align the long edge of the Unix cell with the rotational axis of the pin. In the difficult cases, you probably even need to shoot multiple crystals and combine the datasets to get enough completeness. Zhijie From: Xinghua Qin Sent: Sunday, May 13, 2012 10:22 PM To: CCP4BB@JISCMAIL.AC.UK Subject: [ccp4bb] how to ignore spot overlap in imosflm? Dear CCP4ers, We collected a diffraction dataset with high percentage of spot overlaps, It would be so kind to tell me how to ignore spot overlap in imosflm and explain the hazard of high percentage of spot overlaps. Thanks in advance. Best wishes Xinghua Qin -- Xinghua Qin State Key Laboratory of Plant Physiology and biochemistry College of Biological Sciences China Agricultural University No.2, Yuan Ming Yuan West Road Haidian District, Beijing, China 100193 Tel: +86-10-62732672 E-mail: xinghua...@126.com Harry -- Dr Harry Powell, MRC Laboratory of Molecular Biology, MRC Centre, Hills Road, Cambridge, CB2 0QH
Re: [ccp4bb] how to ignore spot overlap in imosflm?
Although the question was asked for Mosflm I would like to briefly p[oint out that you might be able to also rescue your data by using a program that does 3D profile fitting e.g. d*trek and XDS. However as Andrew pointed out the shoot first try to fix it later mentality might have ruined the possibility of solving your crystal structure. Carefully deciding how to collect your data is worth the time. Jürgen On May 14, 2012, at 4:29 AM, A Leslie wrote: On 14 May 2012, at 03:22, Xinghua Qin wrote: Dear CCP4ers, We collected a diffraction dataset with high percentage of spot overlaps, It would be so kind to tell me how to ignore spot overlap in imosflm and explain the hazard of high percentage of spot overlaps. Thanks in advance. Best wishes Xinghua Qin -- Xinghua Qin State Key Laboratory of Plant Physiology and biochemistry College of Biological Sciences China Agricultural University No.2, Yuan Ming Yuan West Road Haidian District, Beijing, China 100193 Tel: +86-10-62732672 E-mail: xinghua...@126.commailto:xhqin1...@gmail.com mailto:xhqin1...@gmail.com .. Jürgen Bosch Johns Hopkins University Bloomberg School of Public Health Department of Biochemistry Molecular Biology Johns Hopkins Malaria Research Institute 615 North Wolfe Street, W8708 Baltimore, MD 21205 Office: +1-410-614-4742 Lab: +1-410-614-4894 Fax: +1-410-955-2926 http://web.mac.com/bosch_lab/
Re: [ccp4bb] how to ignore spot overlap in imosflm?
On Mon, 2012-05-14 at 13:01 -0400, Bosch, Juergen wrote: Although the question was asked for Mosflm I would like to briefly p[oint out that you might be able to also rescue your data by using a program that does 3D profile fitting e.g. d*trek and XDS. For the sake of completeness (and nothing more), denzo may be used to get around the overlap issue by using a smaller spot radius while keeping background radius large enough to prevent the spots from contaminating the background measurements. I recall this was also a useful trcik for really large unit cells (i.e. when you still have severe overlaps even after pushing detector all the way back). -- After much deep and profound brain things inside my head, I have decided to thank you for bringing peace to our home. Julian, King of Lemurs
Re: [ccp4bb] how to ignore spot overlap in imosflm?
Hi Xinghua, The total intensity of each reflection needs to be accurately quantitated in order to calculate the structure factors. Not only the dots need to be well separated in the 3D reciprocal space, but also a small area around the dots are often needed to calculate the background for subtraction. That is why when two dots are getting too close, the programs will reject both dots. The first thing you need to do is to inspect the images reported with large number of overlaps to see if the dots are really overlapping or just close to each other. If the dots are barely touching or just too close to each other, you can manipulate the SEPERATION parameter to force the program to take the closely spaced spots. But keep in mind that you may get less accurate integration by doing so. If many spots are really touching each other, normally we won't force the programs to use them. Then the proper remedy is to move the detector farther and collect the dataset again (also, try to optimize your freezing to get the mosaicity as low as possible). For how to play with the mosflm parameters, please read here: http://www.mrc-lmb.cam.ac.uk/harry/cgi-bin/keyword2.cgi?SEPARATION. What you need is probably CLOSE. The hazard of high percentage of overlaps: If the overlaps are only scattered in a whole dataset, it is OK, even if they make up 5-10% or even 20% of the whole dataset. It will only give you a lower completeness, which is not too detrimental to the structure solution. However, if large, continuous regions in the dataset are missing, that will cause you to have poorly defined regions in the calculated map, often seen as featureless stripes or layers in the map. Unfortunately, when you have closely spaced reflections, the latter is often the case. The proper solution is to collect the data at a greater detector distance to resolve the spots (after taking the test images, both imosflm and HKL2000 can simulate the collection run to help you to decide what distance you need). In cases that you have a long unit cell (200A), the first thing you need to do is to align the long edge of the Unix cell with the rotational axis of the pin. In the difficult cases, you probably even need to shoot multiple crystals and combine the datasets to get enough completeness. Zhijie From: Xinghua Qin Sent: Sunday, May 13, 2012 10:22 PM To: CCP4BB@JISCMAIL.AC.UK Subject: [ccp4bb] how to ignore spot overlap in imosflm? Dear CCP4ers, We collected a diffraction dataset with high percentage of spot overlaps, It would be so kind to tell me how to ignore spot overlap in imosflm and explain the hazard of high percentage of spot overlaps. Thanks in advance. Best wishes Xinghua Qin -- Xinghua Qin State Key Laboratory of Plant Physiology and biochemistry College of Biological Sciences China Agricultural University No.2, Yuan Ming Yuan West Road Haidian District, Beijing, China 100193 Tel: +86-10-62732672 E-mail: xinghua...@126.com
