Re: [ccp4bb] Process multiple data sets
Before any further attempts you must check that the crystals have the same unit cell volume. I usually do this using matthewscoeff from the GUI ( By the way why isn't the volume automatically written into the mtz header asap!!!) If the cell volumes differ by as much as 5% no reindexing or any other trick will be able to get those data sets to scale together. However if the cell volumes are similar then you may well be able to get a different indexing scheme which will match them. pointless will try to do this, or if all else fails you can run the old MR program almn, with two data sets as input and it will suggest how they mitt be related.. Eleanor PS Phil is too despondent about output from scale pack into aimless. (use the scale pack option NOMERGE to get an unmerged but scaled set of intensities.) SCALEPACK will already have done the internal scaling so you only need aimless to get scales between the different passes. Eleanor On 2 Aug 2012, at 15:53, Uma Ratu wrote: Hi, Micheal: Thank you for your comments. I am getting to know where is the problem. Sorry that I did not give the information of my datasets in details. Here are the answears of the some of the concerns: 1) putative space group? P2 or P21. 2) observed resolution 50 - 1.5 A 3) how big was the crystal and what was its shape? Was the crystal split? Crystals is abot 100 um x 20 um x 20 um. Thesy are thin rods. The crystal is not split. 4) were the data sets taken at different points on the crystal? Is radiation damage a factor? The data sets were taken at different points on the crystal. Radiation damage is not a facotr. 5) did you just rotate around phi (or omega) to collect the different data sets or did you change the other angular settings? Collections were taken at one direction along the long side of the crystal, at diffrent spots. 6) are all your data sets indexed in exactly the same way Yes, each can be index and intergrate individually without any probelm using HKL. But I can not combine them togehter using just one standard. I notice that the cell deminsions are diffrent from these sets. As you and other suggested, non-uniformity in a large crystal may be the cause. I will try to use other program as suggested by many of you. Thank you very much for all your inputs Uma On Thu, Aug 2, 2012 at 9:04 AM, R. M. Garavito rmgarav...@gmail.com wrote: Uma, Before this discussion goes much further, you need to provide more details: 1) putative space group? 2) observed resolution and diffraction anisotropy? 3) how big was the crystal and what was its shape? Was the crystal split? 4) were the data sets taken at different points on the crystal? Is radiation damage a factor? 5) did you just rotate around phi (or omega) to collect the different data sets or did you change the other angular settings? 6) are all your data sets indexed in exactly the same way (a tricky and non-obvious factor for a novice to appreciate). Using pointless on unmerged data sets helps with this. You have a number of unknowns here, and your problem in merging the data sets may be due to radiation damage, non-uniformity in a large crystal, index refinement problems due to diffraction anisotropy, etc. We routinely merge different data sets from a single crystal, which has been translated and rotated about one axis only. We try to index and process the data sets using a common setting matrix (which is easy with XDS). However, sometimes it just does work, but merging pairs of data sets often allowed us to discard the worst offender(s). Good luck, Michael R. Michael Garavito, Ph.D. Professor of Biochemistry Molecular Biology 603 Wilson Rd., Rm. 513 Michigan State University East Lansing, MI 48824-1319 Office: (517) 355-9724 Lab: (517) 353-9125 FAX: (517) 353-9334Email: rmgarav...@gmail.com On Aug 1, 2012, at 4:37 PM, Uma Ratu wrote: I notice one thing with my data sets. The unit cells is slightly different from each other. For example, one has a/b/c @ 79/126/83. The other has a/b/c @ 84/127/90. Although they are collected from the same crystal. Is this the reason that I can't index both with same parameter in HKL? And subsequently, can't integrate and scala together. If so, is there a way that I can fix it? Thank you for your advice Uma On Wed, Aug 1, 2012 at 8:50 AM, Uma Ratu rosiso2...@gmail.com wrote: Dear All: I collected 5 data sets from one crystal and would like to process them together. Here is how I did: In HKL2000, load the all data sets. Index each set. When I try Intergrate, the program automatically go through the whole data sets there, and do not go through. I then
Re: [ccp4bb] Process multiple data sets
An earlier post said the point-group is P2, and these reported cells do not quote the beta angles: what are these angles?. In the monoclinic system it is possible to have two closely-similar alternative cells in certain special cases, and if the crystals have been indexed differently this could prevent their merging. The program Pointless would sort that out for you. Phil On 2 Aug 2012, at 00:45, Edwin Pozharski wrote: The unit cells is slightly different from each other. For example, one has a/b/c @ 79/126/83. The other has a/b/c @ 84/127/90. Although they are collected from the same crystal. This is very substantial difference and with unit cell expanding by ~20% one would expect scaling problems. Try using the same unit cell/orientation on all datasets, it's easy to do if you abandon gui and feed input file directly to denzo. -- Edwin Pozharski, PhD University of Maryland, Baltimore
Re: [ccp4bb] Process multiple data sets
Dear All: Thank you very for your comments and advices. ' I am getting to know why are the problems. And will try again. I appreciate you all for your inputs regards Uma On Thu, Aug 2, 2012 at 4:11 AM, Phil Evans p...@mrc-lmb.cam.ac.uk wrote: An earlier post said the point-group is P2, and these reported cells do not quote the beta angles: what are these angles?. In the monoclinic system it is possible to have two closely-similar alternative cells in certain special cases, and if the crystals have been indexed differently this could prevent their merging. The program Pointless would sort that out for you. Phil On 2 Aug 2012, at 00:45, Edwin Pozharski wrote: The unit cells is slightly different from each other. For example, one has a/b/c @ 79/126/83. The other has a/b/c @ 84/127/90. Although they are collected from the same crystal. This is very substantial difference and with unit cell expanding by ~20% one would expect scaling problems. Try using the same unit cell/orientation on all datasets, it's easy to do if you abandon gui and feed input file directly to denzo. -- Edwin Pozharski, PhD University of Maryland, Baltimore
Re: [ccp4bb] Process multiple data sets
Uma, Before this discussion goes much further, you need to provide more details: 1) putative space group? 2) observed resolution and diffraction anisotropy? 3) how big was the crystal and what was its shape? Was the crystal split? 4) were the data sets taken at different points on the crystal? Is radiation damage a factor? 5) did you just rotate around phi (or omega) to collect the different data sets or did you change the other angular settings? 6) are all your data sets indexed in exactly the same way (a tricky and non-obvious factor for a novice to appreciate). Using pointless on unmerged data sets helps with this. You have a number of unknowns here, and your problem in merging the data sets may be due to radiation damage, non-uniformity in a large crystal, index refinement problems due to diffraction anisotropy, etc. We routinely merge different data sets from a single crystal, which has been translated and rotated about one axis only. We try to index and process the data sets using a common setting matrix (which is easy with XDS). However, sometimes it just does work, but merging pairs of data sets often allowed us to discard the worst offender(s). Good luck, Michael R. Michael Garavito, Ph.D. Professor of Biochemistry Molecular Biology 603 Wilson Rd., Rm. 513 Michigan State University East Lansing, MI 48824-1319 Office: (517) 355-9724 Lab: (517) 353-9125 FAX: (517) 353-9334Email: rmgarav...@gmail.com On Aug 1, 2012, at 4:37 PM, Uma Ratu wrote: I notice one thing with my data sets. The unit cells is slightly different from each other. For example, one has a/b/c @ 79/126/83. The other has a/b/c @ 84/127/90. Although they are collected from the same crystal. Is this the reason that I can't index both with same parameter in HKL? And subsequently, can't integrate and scala together. If so, is there a way that I can fix it? Thank you for your advice Uma On Wed, Aug 1, 2012 at 8:50 AM, Uma Ratu rosiso2...@gmail.com wrote: Dear All: I collected 5 data sets from one crystal and would like to process them together. Here is how I did: In HKL2000, load the all data sets. Index each set. When I try Intergrate, the program automatically go through the whole data sets there, and do not go through. I then process data sets by loading one at each time. Index, intergrate and scale all go through very smoothly. But when I put them together, the program just goes crazy. Thank you for advice Uma
[ccp4bb] Process multiple data sets
Dear All: I collected 5 data sets from one crystal and would like to process them together. Here is how I did: In HKL2000, load the all data sets. Index each set. When I try Intergrate, the program automatically go through the whole data sets there, and do not go through. I then process data sets by loading one at each time. Index, intergrate and scale all go through very smoothly. But when I put them together, the program just goes crazy. Thank you for advice Uma
Re: [ccp4bb] Process multiple data sets
The data sets were collected from the same crystal by scan collecting 40 frames from each section. The space group of this crystal is P2. My guess that I may have to index and integrate each set indivadually, and then scale them together. Thanks Uma On Wed, Aug 1, 2012 at 9:01 AM, Machius, Mischa Christian mischa_mach...@med.unc.edu wrote: Not much info to go by... Anyway, if the program 'goes crazy' you either have very different exposure levels, radiation damage leading to non-isomorphism, or you have a trigonal space group (P3xxx or P6xxx) and forgot to make sure all batches are indexed the same way. If you have translated your crystal between batches, HKL2000 won't of course be able to process all batches in one go. If you haven't touched the crystals at all, and alln-in-one processing doesn't work, the parameters at the end of one batch may not be accurate to start off a new batch, which is mostly due to inaccurate goniostats. In that case, you will need to process the batches individually and them combine them during scaling. Hope that helps. MM On Aug 1, 2012, at 8:50 AM, Uma Ratu wrote: Dear All: I collected 5 data sets from one crystal and would like to process them together. Here is how I did: In HKL2000, load the all data sets. Index each set. When I try Intergrate, the program automatically go through the whole data sets there, and do not go through. I then process data sets by loading one at each time. Index, intergrate and scale all go through very smoothly. But when I put them together, the program just goes crazy. Thank you for advice Uma
Re: [ccp4bb] Process multiple data sets
Hi I'd process (i.e. index, refine, integrate) each data set individually, check that (at least) they all have the same crystal system, then combine the datasets using Pointless. Then scale with Aimless. Of course, I'd use Mosflm/Pointless/Aimless rather than HKL, but that's another question (pace, ZO, WM, MM!) On 1 Aug 2012, at 14:08, Uma Ratu wrote: The data sets were collected from the same crystal by scan collecting 40 frames from each section. The space group of this crystal is P2. My guess that I may have to index and integrate each set indivadually, and then scale them together. Thanks Uma On Wed, Aug 1, 2012 at 9:01 AM, Machius, Mischa Christian mischa_mach...@med.unc.edu wrote: Not much info to go by... Anyway, if the program 'goes crazy' you either have very different exposure levels, radiation damage leading to non-isomorphism, or you have a trigonal space group (P3xxx or P6xxx) and forgot to make sure all batches are indexed the same way. If you have translated your crystal between batches, HKL2000 won't of course be able to process all batches in one go. If you haven't touched the crystals at all, and alln-in-one processing doesn't work, the parameters at the end of one batch may not be accurate to start off a new batch, which is mostly due to inaccurate goniostats. In that case, you will need to process the batches individually and them combine them during scaling. Hope that helps. MM On Aug 1, 2012, at 8:50 AM, Uma Ratu wrote: Dear All: I collected 5 data sets from one crystal and would like to process them together. Here is how I did: In HKL2000, load the all data sets. Index each set. When I try Intergrate, the program automatically go through the whole data sets there, and do not go through. I then process data sets by loading one at each time. Index, intergrate and scale all go through very smoothly. But when I put them together, the program just goes crazy. Thank you for advice Uma Harry -- Dr Harry Powell, MRC Laboratory of Molecular Biology, MRC Centre, Hills Road, Cambridge, CB2 0QH
Re: [ccp4bb] Process multiple data sets
Note that neither Aimless nor Scala will do a particularly good job at scaling data from Denzo or Scalepack, since the output files from Scalepack are missing essential geometrical information. They work well with data from Mosflm or XDS (or Saint) (although AFAIK the XDS Saint scaling programs work perfectly well) However, Pointless may still be useful to check that you have indexed them consistently Phil On 1 Aug 2012, at 14:36, Harry Powell wrote: Hi I'd process (i.e. index, refine, integrate) each data set individually, check that (at least) they all have the same crystal system, then combine the datasets using Pointless. Then scale with Aimless. Of course, I'd use Mosflm/Pointless/Aimless rather than HKL, but that's another question (pace, ZO, WM, MM!) On 1 Aug 2012, at 14:08, Uma Ratu wrote: The data sets were collected from the same crystal by scan collecting 40 frames from each section. The space group of this crystal is P2. My guess that I may have to index and integrate each set indivadually, and then scale them together. Thanks Uma On Wed, Aug 1, 2012 at 9:01 AM, Machius, Mischa Christian mischa_mach...@med.unc.edu wrote: Not much info to go by... Anyway, if the program 'goes crazy' you either have very different exposure levels, radiation damage leading to non-isomorphism, or you have a trigonal space group (P3xxx or P6xxx) and forgot to make sure all batches are indexed the same way. If you have translated your crystal between batches, HKL2000 won't of course be able to process all batches in one go. If you haven't touched the crystals at all, and alln-in-one processing doesn't work, the parameters at the end of one batch may not be accurate to start off a new batch, which is mostly due to inaccurate goniostats. In that case, you will need to process the batches individually and them combine them during scaling. Hope that helps. MM On Aug 1, 2012, at 8:50 AM, Uma Ratu wrote: Dear All: I collected 5 data sets from one crystal and would like to process them together. Here is how I did: In HKL2000, load the all data sets. Index each set. When I try Intergrate, the program automatically go through the whole data sets there, and do not go through. I then process data sets by loading one at each time. Index, intergrate and scale all go through very smoothly. But when I put them together, the program just goes crazy. Thank you for advice Uma Harry -- Dr Harry Powell, MRC Laboratory of Molecular Biology, MRC Centre, Hills Road, Cambridge, CB2 0QH
Re: [ccp4bb] Process multiple data sets
Please correct me if I am wrong: The HKL is not good to combine multiple data sets, even they are come from the same crystal? With HKL, I also tried this way: Index, integrate each data set individually, they all have the same space group. Then scale them together. Still, the graph from scale of the whole set look very wired compared with those of individuals. Uma On Wed, Aug 1, 2012 at 9:55 AM, Phil Evans p...@mrc-lmb.cam.ac.uk wrote: Note that neither Aimless nor Scala will do a particularly good job at scaling data from Denzo or Scalepack, since the output files from Scalepack are missing essential geometrical information. They work well with data from Mosflm or XDS (or Saint) (although AFAIK the XDS Saint scaling programs work perfectly well) However, Pointless may still be useful to check that you have indexed them consistently Phil On 1 Aug 2012, at 14:36, Harry Powell wrote: Hi I'd process (i.e. index, refine, integrate) each data set individually, check that (at least) they all have the same crystal system, then combine the datasets using Pointless. Then scale with Aimless. Of course, I'd use Mosflm/Pointless/Aimless rather than HKL, but that's another question (pace, ZO, WM, MM!) On 1 Aug 2012, at 14:08, Uma Ratu wrote: The data sets were collected from the same crystal by scan collecting 40 frames from each section. The space group of this crystal is P2. My guess that I may have to index and integrate each set indivadually, and then scale them together. Thanks Uma On Wed, Aug 1, 2012 at 9:01 AM, Machius, Mischa Christian mischa_mach...@med.unc.edu wrote: Not much info to go by... Anyway, if the program 'goes crazy' you either have very different exposure levels, radiation damage leading to non-isomorphism, or you have a trigonal space group (P3xxx or P6xxx) and forgot to make sure all batches are indexed the same way. If you have translated your crystal between batches, HKL2000 won't of course be able to process all batches in one go. If you haven't touched the crystals at all, and alln-in-one processing doesn't work, the parameters at the end of one batch may not be accurate to start off a new batch, which is mostly due to inaccurate goniostats. In that case, you will need to process the batches individually and them combine them during scaling. Hope that helps. MM On Aug 1, 2012, at 8:50 AM, Uma Ratu wrote: Dear All: I collected 5 data sets from one crystal and would like to process them together. Here is how I did: In HKL2000, load the all data sets. Index each set. When I try Intergrate, the program automatically go through the whole data sets there, and do not go through. I then process data sets by loading one at each time. Index, intergrate and scale all go through very smoothly. But when I put them together, the program just goes crazy. Thank you for advice Uma Harry -- Dr Harry Powell, MRC Laboratory of Molecular Biology, MRC Centre, Hills Road, Cambridge, CB2 0QH
Re: [ccp4bb] Process multiple data sets
Hi I don't think Phil or I are saying that HKL is not a good tool to use in this case - but (*) I develop Mosflm, so I have a certain bias. (*) As far as Pointless and Aimless are concerned, to get the best out of them you need to have some geometrical information that the authors of HKL have not included in the reflection output files. Mosflm (and XDS Saint...) do give this information. So if you want to use Pointless and Aimless at any point, it may be better to use a different integration package than HKL (bearing in mind Phil's point about checking your symmetry with Pointless)... On 1 Aug 2012, at 16:16, Uma Ratu wrote: Please correct me if I am wrong: The HKL is not good to combine multiple data sets, even they are come from the same crystal? With HKL, I also tried this way: Index, integrate each data set individually, they all have the same space group. Then scale them together. Still, the graph from scale of the whole set look very wired compared with those of individuals. Uma On Wed, Aug 1, 2012 at 9:55 AM, Phil Evans p...@mrc-lmb.cam.ac.uk wrote: Note that neither Aimless nor Scala will do a particularly good job at scaling data from Denzo or Scalepack, since the output files from Scalepack are missing essential geometrical information. They work well with data from Mosflm or XDS (or Saint) (although AFAIK the XDS Saint scaling programs work perfectly well) However, Pointless may still be useful to check that you have indexed them consistently Phil On 1 Aug 2012, at 14:36, Harry Powell wrote: Hi I'd process (i.e. index, refine, integrate) each data set individually, check that (at least) they all have the same crystal system, then combine the datasets using Pointless. Then scale with Aimless. Of course, I'd use Mosflm/Pointless/Aimless rather than HKL, but that's another question (pace, ZO, WM, MM!) On 1 Aug 2012, at 14:08, Uma Ratu wrote: The data sets were collected from the same crystal by scan collecting 40 frames from each section. The space group of this crystal is P2. My guess that I may have to index and integrate each set indivadually, and then scale them together. Thanks Uma On Wed, Aug 1, 2012 at 9:01 AM, Machius, Mischa Christian mischa_mach...@med.unc.edu wrote: Not much info to go by... Anyway, if the program 'goes crazy' you either have very different exposure levels, radiation damage leading to non- isomorphism, or you have a trigonal space group (P3xxx or P6xxx) and forgot to make sure all batches are indexed the same way. If you have translated your crystal between batches, HKL2000 won't of course be able to process all batches in one go. If you haven't touched the crystals at all, and alln-in-one processing doesn't work, the parameters at the end of one batch may not be accurate to start off a new batch, which is mostly due to inaccurate goniostats. In that case, you will need to process the batches individually and them combine them during scaling. Hope that helps. MM On Aug 1, 2012, at 8:50 AM, Uma Ratu wrote: Dear All: I collected 5 data sets from one crystal and would like to process them together. Here is how I did: In HKL2000, load the all data sets. Index each set. When I try Intergrate, the program automatically go through the whole data sets there, and do not go through. I then process data sets by loading one at each time. Index, intergrate and scale all go through very smoothly. But when I put them together, the program just goes crazy. Thank you for advice Uma Harry -- Dr Harry Powell, MRC Laboratory of Molecular Biology, MRC Centre, Hills Road, Cambridge, CB2 0QH Harry -- Dr Harry Powell, MRC Laboratory of Molecular Biology, MRC Centre, Hills Road, Cambridge, CB2 0QH
Re: [ccp4bb] Process multiple data sets
I notice one thing with my data sets. The unit cells is slightly different from each other. For example, one has a/b/c @ 79/126/83. The other has a/b/c @ 84/127/90. Although they are collected from the same crystal. Is this the reason that I can't index both with same parameter in HKL? And subsequently, can't integrate and scala together. If so, is there a way that I can fix it? Thank you for your advice Uma On Wed, Aug 1, 2012 at 8:50 AM, Uma Ratu rosiso2...@gmail.com wrote: Dear All: I collected 5 data sets from one crystal and would like to process them together. Here is how I did: In HKL2000, load the all data sets. Index each set. When I try Intergrate, the program automatically go through the whole data sets there, and do not go through. I then process data sets by loading one at each time. Index, intergrate and scale all go through very smoothly. But when I put them together, the program just goes crazy. Thank you for advice Uma
Re: [ccp4bb] Process multiple data sets
Hi Uma, I've used HKL2000 to combine datasets from different crystals, so it's definitely possible to do so (although depending on the data volume it may be better to deal with scalepack directly). There are two things that you don't mention about your data - the (approximate) resolution, and the relationship of your datasets (different angular ranges, different exposure times, or something else?). Unit cell parameters are not always well determined for low resolution datasets, so this could be one cause of your cell variability. If not, check that the usual suspects (wavelength and detector to crystal distance) are correct, especially if they may have changed during data collection. This can be exacerbated if the datasets cover different angular ranges. From your initial description: In HKL2000, load the all data sets. Index each set. When I try Intergrate, the program automatically go through the whole data sets there, and do not go through. It's possible that there's an issue with discontinuities in the orientation matrices for the different datasets. In my hands, mosflm is easier for dealing with integration when you have to explicitly specify the orientation matrix for a dataset/wedge/batch (for example: integrate 20 images, switch the orientation matrix back to the beginning, and integrate another 20 images). Good luck, Pete Uma Ratu wrote: I notice one thing with my data sets. The unit cells is slightly different from each other. For example, one has a/b/c @ 79/126/83. The other has a/b/c @ 84/127/90. Although they are collected from the same crystal. Is this the reason that I can't index both with same parameter in HKL? And subsequently, can't integrate and scala together. If so, is there a way that I can fix it? Thank you for your advice Uma On Wed, Aug 1, 2012 at 8:50 AM, Uma Ratu rosiso2...@gmail.com wrote: Dear All: I collected 5 data sets from one crystal and would like to process them together. Here is how I did: I then process data sets by loading one at each time. Index, intergrate and scale all go through very smoothly. But when I put them together, the program just goes crazy. Thank you for advice Uma
