Re: [ccp4bb] Rescale merged data?
Dear all, Just to reiterate that wwPDB can accept multiple data blocks of data before and after process, and the first block should be the one that used for the refinement of the deposited coordinates. Kind regards, Deborah On 18/04/2024 13:37, Randy John Read wrote: I haven’t deposited any PDB entries for a while. The last time I did, I remember it not being completely trivial to add these loops. However, I was hoping that someone from wwPDB or CCP4 would weigh in with advice on how it can be done! For those who use StarAniso, the current version makes a CIF file with the required loops, and they now have advice on their website about this: https://staraniso.globalphasing.org/deposition_about.html. Randy -- --- Deborah Harrus, Ph.D. PDBe Archive Project Leader, Biocuration Lead PDBe - Protein Data Bank in Europe European Bioinformatics Institute (EMBL-EBI) European Molecular Biology Laboratory Wellcome Trust Genome Campus Hinxton Cambridge CB10 1SD UK http://www.PDBe.org --- To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/
Re: [ccp4bb] Rescale merged data?
Hi Clemens I’m “quite”* surprised. Do the authors of these programs offer any explanation as to why the scaling programs (given that these are the ones that usually provide F/I information for subsequent programs) bother to calculate sigmas if they’re going to be ignored? Harry * for “quite” surprised, read “you could have knocked me down with a feather”. > On Thu, Apr 18, 2024 at 09:22:04AM +0100, Harry Powell wrote: >> Only comment is that (surely) any decent refinement program these days >> would down-weight any reflections with negligible I/sig(I) (for example, >> those in the “unobserved” high resolution regions) so that they do not >> contribute (significantly) to the refinement. > > Not all refinement programs use sigmas as far as we know. To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/
Re: [ccp4bb] Rescale merged data?
Dear Randy, Harry et al, sorry for replying to several emails in one message - but as always, everything is connected ;-) On Thu, Apr 18, 2024 at 10:02:40AM +, Randy John Read wrote: > I’d like to add my strong agreement to what Robbie said, but also point > out a wrinkle. When the PDB runs validation, it just takes the data that > are in the first reflection loop of the reflections.cif file. Just a small adjustment of naming convention here (as I understand it): you are referring to so-called "data blocks" within an mmCIF file (delimited by a "data_XYZ" token) while a "loop" is just a CIF format construct. So a single datablock can have multiple loops - as visible in a model mmCIF file. Right? > So if you want the validation statistics to match your reported > refinement statistics, that loop should contain the set of data you gave > the refinement program, Couldn't agree more - but there are a few additional creases (see below) to try and stay in your picture ... > especially if you’ve done something like apply an elliptical truncation, > correct for anisotropy, or convert intensities to amplitudes, all of > which change the data in ways that can’t be reversed later. ... some of those are actually applied by the refinement program itself (rejection of outlier reflections or scaling of the observed amplitudes against the model using anisotropic B-factors). That immediately creates a problem: we require the output reflection data (containing map coefficients for e.g. 2mFo-DFc and mFo-DFc maps) but can't rely on the observed intensities/amplitudes in that file to accurately represent the input data. Therefore, we might also need to collect the observed data as-is from the input reflection file - and combine those two files into a single datablock of a reflection mmCIF file intended for deposition. There are a few points we could make about (a) ellipsoidal truncation and its relation to isotropic and anisotropic truncation, and (b) correction for anisotropy, but we placed them onto a separate page at [1] to keep this email thread reasonably short. Your remark about the conversion from intensities to amplitudes in the context of multiple datablocks in mmCIF files is probably out of a concern that one might end up just with amplitudes for a given PDB deposition - which indeed would not be ideal. That danger lies mostly in the way we now seem to split, shuffle, reassemble and push our original reflection data files around different programs, online services and GUIs. Originally, any program that we know of doing that I->F conversion would always carry over the input intensities into the output (together with the newly created amplitudes and other items like anomalous data etc), so there shouldn't be a reason for losing the intensities at this point ... ideally. The beauty of the good ol' MTZ files was that all those items were always kept together and - once a test-set flag was added - one only ever needed to refer to that "master" reflection file for any subsequent steps to keep all those relations and the content intact ... no daisy-chaining of reflection data input/output channels with the subsequent loss of information, provenance and meta-data. > Whenever you’ve done any of this (and many people are using StarAniso > these days, which does all of those things), Just as a clarification for the less experienced users: there are lots of other programs that do exactly the same thing - or at least variants that follow the same underlying idea. Most (all?) data-processing programs/packages/pipelines will apply a data truncation step and several refinement programs will reject observations or apply anisotropic corrections to the observed data. STARANISO is not really unique in those underlying concepts as far as we can see. > please put in a second reflection loop containing the whole set of > intensities to the highest resolution you used, without any anisotropic > scaling or elliptical cutoffs. Then anyone wanting to re-refine your > structure or check your data for artefacts will have more information > available. Absolutely: we couldn't agree more and are trying to provide as simple to use tools as possible to users (not only of our software - see [2], since October 2020). The hope is that the deposition-preparation step becomes as painless as possible, even if it can't (yet) be fully automatic because (a) the processed diffraction data usually comes from a synchrotron system (provided as mmCIF for deposition, but usually the MTZ file is picked up for downstream work), and (b) the refinement is done in a separate system resulting in a separate, related but not directly linked reflection file after model refinement. > Of course, I hope we’re moving to a world in which we all also deposit > the intensities before merging, which in principle allows even more > quality control to be done. For the last several years we are providing that feature as part of our own
Re: [ccp4bb] Rescale merged data?
I haven’t deposited any PDB entries for a while. The last time I did, I remember it not being completely trivial to add these loops. However, I was hoping that someone from wwPDB or CCP4 would weigh in with advice on how it can be done! For those who use StarAniso, the current version makes a CIF file with the required loops, and they now have advice on their website about this: https://staraniso.globalphasing.org/deposition_about.html. Randy > On 18 Apr 2024, at 12:25, Frank von Delft wrote: > > Is it easy/non-arcane or indeed automatic for non-experts to add these loops? > Because that's the only way this will be achieved systemically. > > Presumably ccp4i2 can be wrangled into making it happen magically. (Apologies > if it does already, if so, a comment here would help the discussion...) > > Frank > > > > On 18/04/2024 11:02, Randy John Read wrote: >> I’d like to add my strong agreement to what Robbie said, but also point out >> a wrinkle. When the PDB runs validation, it just takes the data that are in >> the first reflection loop of the reflections.cif file. So if you want the >> validation statistics to match your reported refinement statistics, that >> loop should contain the set of data you gave the refinement program, >> especially if you’ve done something like apply an elliptical truncation, >> correct for anisotropy, or convert intensities to amplitudes, all of which >> change the data in ways that can’t be reversed later. Whenever you’ve done >> any of this (and many people are using StarAniso these days, which does all >> of those things), please put in a second reflection loop containing the >> whole set of intensities to the highest resolution you used, without any >> anisotropic scaling or elliptical cutoffs. Then anyone wanting to re-refine >> your structure or check your data for artefacts will have more information >> available. >> >> Of course, I hope we’re moving to a world in which we all also deposit the >> intensities before merging, which in principle allows even more quality >> control to be done. >> >> Best wishes, >> >> Randy Read >> >>> On 18 Apr 2024, at 05:04, Robbie Joosten wrote: >>> >>> If I may add to that: Please deposit the full dataset, not just the set of >>> reflections you end up using. This allows people to use all the data if >>> they are interested. >>> >>> Cheers, >>> Robbie >>> >>> On 17 Apr 2024 22:35, "Hekstra, Doeke Romke" >>> wrote: >>> Hi Matt, >>> I appreciate disagreement and comments from colleagues. My two cents are >>> that it seems unnecessary to repeat scaling and merging, or any earlier >>> step. If you want to remove structure factor amplitudes or merged >>> intensities from the MTZ file you can do so using MTZUTILS or similar >>> functionality in CCP4 >>> (https://www.ccp4.ac.uk/html/mtzutils.html#generalresolution). For >>> refinement, you can specify the desired resolution range in your favorite >>> refinement program. >>> My personal convention is to use CC1/2 = 0.30 as the point to which retain >>> data and = 2 as the nominal resolution of the dataset. If you have >>> the HKL2000 scaling log, you should be able to retrieve this information. I >>> frankly wish we’d just deposit all data in the PDB rather than truncate >>> based on some criterion or another. >>> Best, Doeke >>> From: Matt Mcleod >>> Sent: Wednesday, April 17, 2024 4:12 PM >>> To: Hekstra, Doeke Romke >>> Cc: CCP4BB@JISCMAIL.AC.UK >>> Subject: Re: [ccp4bb] Rescale merged data? >>> Sure thing. >>> A former student left somewhere between 30-50 datasets but they scaled the >>> data to the detector corners (or maybe edge) in HKL2000. There are many of >>> the high-resolution bins with no reflections in them. He then went forward >>> and merged this data, presumably in HKL2000 again and did his model >>> building/refinement. We now need to re-refine the models against this >>> data for publication but we need a more suitable resolution cutoff for the >>> data. >>> Rather than go back and index/integrate all the data and then rescale the >>> data to a more appropriate place (then merge), I was wondering if there was >>> a way to take the merged reflections as either .sca or .mtz (from >>> scalepacktomtz output) and then rescale to a more appropriate resolution. >>> It doesn't seem like the student left unmerged data. >>> So, nothing f
Re: [ccp4bb] Rescale merged data?
Is it easy/non-arcane or indeed automatic for non-experts to add these loops? Because that's the only way this will be achieved systemically. Presumably ccp4i2 can be wrangled into making it happen magically. (Apologies if it does already, if so, a comment here would help the discussion...) Frank On 18/04/2024 11:02, Randy John Read wrote: I’d like to add my strong agreement to what Robbie said, but also point out a wrinkle. When the PDB runs validation, it just takes the data that are in the first reflection loop of the reflections.cif file. So if you want the validation statistics to match your reported refinement statistics, that loop should contain the set of data you gave the refinement program, especially if you’ve done something like apply an elliptical truncation, correct for anisotropy, or convert intensities to amplitudes, all of which change the data in ways that can’t be reversed later. Whenever you’ve done any of this (and many people are using StarAniso these days, which does all of those things), please put in a second reflection loop containing the whole set of intensities to the highest resolution you used, without any anisotropic scaling or elliptical cutoffs. Then anyone wanting to re-refine your structure or check your data for artefacts will have more information available. Of course, I hope we’re moving to a world in which we all also deposit the intensities before merging, which in principle allows even more quality control to be done. Best wishes, Randy Read On 18 Apr 2024, at 05:04, Robbie Joosten wrote: If I may add to that: Please deposit the full dataset, not just the set of reflections you end up using. This allows people to use all the data if they are interested. Cheers, Robbie On 17 Apr 2024 22:35, "Hekstra, Doeke Romke" wrote: Hi Matt, I appreciate disagreement and comments from colleagues. My two cents are that it seems unnecessary to repeat scaling and merging, or any earlier step. If you want to remove structure factor amplitudes or merged intensities from the MTZ file you can do so using MTZUTILS or similar functionality in CCP4 (https://www.ccp4.ac.uk/html/mtzutils.html#generalresolution). For refinement, you can specify the desired resolution range in your favorite refinement program. My personal convention is to use CC1/2 = 0.30 as the point to which retain data and = 2 as the nominal resolution of the dataset. If you have the HKL2000 scaling log, you should be able to retrieve this information. I frankly wish we’d just deposit all data in the PDB rather than truncate based on some criterion or another. Best, Doeke From: Matt Mcleod Sent: Wednesday, April 17, 2024 4:12 PM To: Hekstra, Doeke Romke Cc: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] Rescale merged data? Sure thing. A former student left somewhere between 30-50 datasets but they scaled the data to the detector corners (or maybe edge) in HKL2000. There are many of the high-resolution bins with no reflections in them. He then went forward and merged this data, presumably in HKL2000 again and did his model building/refinement. We now need to re-refine the models against this data for publication but we need a more suitable resolution cutoff for the data. Rather than go back and index/integrate all the data and then rescale the data to a more appropriate place (then merge), I was wondering if there was a way to take the merged reflections as either .sca or .mtz (from scalepacktomtz output) and then rescale to a more appropriate resolution. It doesn't seem like the student left unmerged data. So, nothing fancy (aniostropy etc), there is just a lot of data that needs to be adjusted and I am trying to avoid reprocessing all the frames again. Matt On Wed, 17 Apr 2024 at 15:59, Hekstra, Doeke Romke wrote: Hi Matt, It would be helpful if you could describe your case in more detail. Do you want to change the resolution cutoff after scaling? Do you want to keep more data? Fewer? Or do you mean something different such as truncation to generate amplitudes, application of anisotropic resolution cutoffs, or outlier rejection? Are you referring to data that were scaled in HKL2000? Best, Doeke -Original Message- From: CCP4 bulletin board On Behalf Of Matt McLeod Sent: Wednesday, April 17, 2024 3:04 PM To: CCP4BB@JISCMAIL.AC.UK Subject: [ccp4bb] Rescale merged data? Hi all, I am looking at a old students data and it looks like they didn't properly cut off the data during scaling. All of the files I have appear to be the merged .sca (or mtz after converting with scalepacktomtz) - is there a way to retruncate the data after merging or do I have to reprocess the data? Thanks, To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 This message was issued
Re: [ccp4bb] Rescale merged data?
If you are using CCP4I2 and use the task "Import merged data" that sort of reprocesses that data through AIMLESS. You get all the usual graphsv resolution - completeness, I/SigI , Wilson plot, Second moments etc and from those can make a sensible decision on where the resolution limit should be set.. In my experience rescaling isnt usually necessary - the scales are weighted by SigI and very weak data has little effect.. Eleanor On Thu, 18 Apr 2024 at 09:22, Harry Powell < 193323b1e616-dmarc-requ...@jiscmail.ac.uk> wrote: > Hi > > Only comment is that (surely) any decent refinement program these days > would down-weight any reflections with negligible I/sig(I) (for example, > those in the “unobserved” high resolution regions) so that they do not > contribute (significantly) to the refinement. Or am I wrong (I don’t mind > being wrong, and since I am a little rusty in these mattrers I would be > very happy to be educated)? > > Doesn’t Aimless produce a table with the cumulative statistics at various > resolution limits? So you don’t even need to re-scale & merge to get the > stats to whatever your chosen high resolution limit is (I’d choose CC -1/2 > = 0.30, as Doeke suggests)? Do HKL or XSCALE do the same (sorry, I haven’t > looked at their output for quite some time)? > > Just my two ha’porth > > Harry > > > On 17 Apr 2024, at 21:35, Hekstra, Doeke Romke < > doeke_heks...@harvard.edu> wrote: > > > > Hi Matt, > > > > I appreciate disagreement and comments from colleagues. My two cents are > that it seems unnecessary to repeat scaling and merging, or any earlier > step. If you want to remove structure factor amplitudes or merged > intensities from the MTZ file you can do so using MTZUTILS or similar > functionality in CCP4 ( > https://www.ccp4.ac.uk/html/mtzutils.html#generalresolution). For > refinement, you can specify the desired resolution range in your favorite > refinement program. > > > > My personal convention is to use CC1/2 = 0.30 as the point to which > retain data and = 2 as the nominal resolution of the dataset. If > you have the HKL2000 scaling log, you should be able to retrieve this > information. I frankly wish we’d just deposit all data in the PDB rather > than truncate based on some criterion or another. > > > > Best, Doeke > > > > From: Matt Mcleod > > Sent: Wednesday, April 17, 2024 4:12 PM > > To: Hekstra, Doeke Romke > > Cc: CCP4BB@JISCMAIL.AC.UK > > Subject: Re: [ccp4bb] Rescale merged data? > > > > Sure thing. > > > > A former student left somewhere between 30-50 datasets but they scaled > the data to the detector corners (or maybe edge) in HKL2000. There are > many of the high-resolution bins with no reflections in them. He then went > forward and merged this data, presumably in HKL2000 again and did his model > building/refinement. We now need to re-refine the models against this > data for publication but we need a more suitable resolution cutoff for the > data. > > > > Rather than go back and index/integrate all the data and then rescale > the data to a more appropriate place (then merge), I was wondering if there > was a way to take the merged reflections as either .sca or .mtz (from > scalepacktomtz output) and then rescale to a more appropriate resolution. > It doesn't seem like the student left unmerged data. > > > > So, nothing fancy (aniostropy etc), there is just a lot of data that > needs to be adjusted and I am trying to avoid reprocessing all the frames > again. > > > > Matt > > > > On Wed, 17 Apr 2024 at 15:59, Hekstra, Doeke Romke < > doeke_heks...@harvard.edu> wrote: > > Hi Matt, > > > > It would be helpful if you could describe your case in more detail. Do > you want to change the resolution cutoff after scaling? Do you want to keep > more data? Fewer? Or do you mean something different such as truncation to > generate amplitudes, application of anisotropic resolution cutoffs, or > outlier rejection? Are you referring to data that were scaled in HKL2000? > > > > Best, Doeke > > > > -Original Message- > > From: CCP4 bulletin board On Behalf Of Matt > McLeod > > Sent: Wednesday, April 17, 2024 3:04 PM > > To: CCP4BB@JISCMAIL.AC.UK > > Subject: [ccp4bb] Rescale merged data? > > > > Hi all, > > > > I am looking at a old students data and it looks like they didn't > properly cut off the data during scaling. All of the files I have appear > to be the merged .sca (or mtz after converting with scalepacktomtz) - is > there a way to retruncate the data after merging or do I have to reprocess > the
Re: [ccp4bb] Rescale merged data?
I’d like to add my strong agreement to what Robbie said, but also point out a wrinkle. When the PDB runs validation, it just takes the data that are in the first reflection loop of the reflections.cif file. So if you want the validation statistics to match your reported refinement statistics, that loop should contain the set of data you gave the refinement program, especially if you’ve done something like apply an elliptical truncation, correct for anisotropy, or convert intensities to amplitudes, all of which change the data in ways that can’t be reversed later. Whenever you’ve done any of this (and many people are using StarAniso these days, which does all of those things), please put in a second reflection loop containing the whole set of intensities to the highest resolution you used, without any anisotropic scaling or elliptical cutoffs. Then anyone wanting to re-refine your structure or check your data for artefacts will have more information available. Of course, I hope we’re moving to a world in which we all also deposit the intensities before merging, which in principle allows even more quality control to be done. Best wishes, Randy Read > On 18 Apr 2024, at 05:04, Robbie Joosten wrote: > > If I may add to that: Please deposit the full dataset, not just the set of > reflections you end up using. This allows people to use all the data if they > are interested. > > Cheers, > Robbie > > On 17 Apr 2024 22:35, "Hekstra, Doeke Romke" > wrote: > Hi Matt, > > I appreciate disagreement and comments from colleagues. My two cents are that > it seems unnecessary to repeat scaling and merging, or any earlier step. If > you want to remove structure factor amplitudes or merged intensities from the > MTZ file you can do so using MTZUTILS or similar functionality in CCP4 > (https://www.ccp4.ac.uk/html/mtzutils.html#generalresolution). For > refinement, you can specify the desired resolution range in your favorite > refinement program. > > My personal convention is to use CC1/2 = 0.30 as the point to which retain > data and = 2 as the nominal resolution of the dataset. If you have > the HKL2000 scaling log, you should be able to retrieve this information. I > frankly wish we’d just deposit all data in the PDB rather than truncate based > on some criterion or another. > > Best, Doeke > > From: Matt Mcleod > Sent: Wednesday, April 17, 2024 4:12 PM > To: Hekstra, Doeke Romke > Cc: CCP4BB@JISCMAIL.AC.UK > Subject: Re: [ccp4bb] Rescale merged data? > > Sure thing. > > A former student left somewhere between 30-50 datasets but they scaled the > data to the detector corners (or maybe edge) in HKL2000. There are many of > the high-resolution bins with no reflections in them. He then went forward > and merged this data, presumably in HKL2000 again and did his model > building/refinement. We now need to re-refine the models against this data > for publication but we need a more suitable resolution cutoff for the data. > > Rather than go back and index/integrate all the data and then rescale the > data to a more appropriate place (then merge), I was wondering if there was a > way to take the merged reflections as either .sca or .mtz (from > scalepacktomtz output) and then rescale to a more appropriate resolution. It > doesn't seem like the student left unmerged data. > > So, nothing fancy (aniostropy etc), there is just a lot of data that needs to > be adjusted and I am trying to avoid reprocessing all the frames again. > > Matt > > On Wed, 17 Apr 2024 at 15:59, Hekstra, Doeke Romke > wrote: > Hi Matt, > > It would be helpful if you could describe your case in more detail. Do you > want to change the resolution cutoff after scaling? Do you want to keep more > data? Fewer? Or do you mean something different such as truncation to > generate amplitudes, application of anisotropic resolution cutoffs, or > outlier rejection? Are you referring to data that were scaled in HKL2000? > > Best, Doeke > > -Original Message- > From: CCP4 bulletin board On Behalf Of Matt McLeod > Sent: Wednesday, April 17, 2024 3:04 PM > To: CCP4BB@JISCMAIL.AC.UK > Subject: [ccp4bb] Rescale merged data? > > Hi all, > > I am looking at a old students data and it looks like they didn't properly > cut off the data during scaling. All of the files I have appear to be the > merged .sca (or mtz after converting with scalepacktomtz) - is there a way to > retruncate the data after merging or do I have to reprocess the data? > > Thanks, > > > > To unsubscribe from the CCP4BB list, click the following link: &
Re: [ccp4bb] Rescale merged data?
Hi Only comment is that (surely) any decent refinement program these days would down-weight any reflections with negligible I/sig(I) (for example, those in the “unobserved” high resolution regions) so that they do not contribute (significantly) to the refinement. Or am I wrong (I don’t mind being wrong, and since I am a little rusty in these mattrers I would be very happy to be educated)? Doesn’t Aimless produce a table with the cumulative statistics at various resolution limits? So you don’t even need to re-scale & merge to get the stats to whatever your chosen high resolution limit is (I’d choose CC -1/2 = 0.30, as Doeke suggests)? Do HKL or XSCALE do the same (sorry, I haven’t looked at their output for quite some time)? Just my two ha’porth Harry > On 17 Apr 2024, at 21:35, Hekstra, Doeke Romke > wrote: > > Hi Matt, > > I appreciate disagreement and comments from colleagues. My two cents are that > it seems unnecessary to repeat scaling and merging, or any earlier step. If > you want to remove structure factor amplitudes or merged intensities from the > MTZ file you can do so using MTZUTILS or similar functionality in CCP4 > (https://www.ccp4.ac.uk/html/mtzutils.html#generalresolution). For > refinement, you can specify the desired resolution range in your favorite > refinement program. > > My personal convention is to use CC1/2 = 0.30 as the point to which retain > data and = 2 as the nominal resolution of the dataset. If you have > the HKL2000 scaling log, you should be able to retrieve this information. I > frankly wish we’d just deposit all data in the PDB rather than truncate based > on some criterion or another. > > Best, Doeke > > From: Matt Mcleod > Sent: Wednesday, April 17, 2024 4:12 PM > To: Hekstra, Doeke Romke > Cc: CCP4BB@JISCMAIL.AC.UK > Subject: Re: [ccp4bb] Rescale merged data? > > Sure thing. > > A former student left somewhere between 30-50 datasets but they scaled the > data to the detector corners (or maybe edge) in HKL2000. There are many of > the high-resolution bins with no reflections in them. He then went forward > and merged this data, presumably in HKL2000 again and did his model > building/refinement. We now need to re-refine the models against this data > for publication but we need a more suitable resolution cutoff for the data. > > Rather than go back and index/integrate all the data and then rescale the > data to a more appropriate place (then merge), I was wondering if there was a > way to take the merged reflections as either .sca or .mtz (from > scalepacktomtz output) and then rescale to a more appropriate resolution. It > doesn't seem like the student left unmerged data. > > So, nothing fancy (aniostropy etc), there is just a lot of data that needs to > be adjusted and I am trying to avoid reprocessing all the frames again. > > Matt > > On Wed, 17 Apr 2024 at 15:59, Hekstra, Doeke Romke > wrote: > Hi Matt, > > It would be helpful if you could describe your case in more detail. Do you > want to change the resolution cutoff after scaling? Do you want to keep more > data? Fewer? Or do you mean something different such as truncation to > generate amplitudes, application of anisotropic resolution cutoffs, or > outlier rejection? Are you referring to data that were scaled in HKL2000? > > Best, Doeke > > -----Original Message- > From: CCP4 bulletin board On Behalf Of Matt McLeod > Sent: Wednesday, April 17, 2024 3:04 PM > To: CCP4BB@JISCMAIL.AC.UK > Subject: [ccp4bb] Rescale merged data? > > Hi all, > > I am looking at a old students data and it looks like they didn't properly > cut off the data during scaling. All of the files I have appear to be the > merged .sca (or mtz after converting with scalepacktomtz) - is there a way to > retruncate the data after merging or do I have to reprocess the data? > > Thanks, > > > > To unsubscribe from the CCP4BB list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 > > This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing > list hosted by www.jiscmail.ac.uk, terms & conditions are available at > https://www.jiscmail.ac.uk/policyandsecurity/ > > > -- > Matthew Jordan McLeod, PhD > Post-Doctoral Fellow - Cornell University > > > > > To unsubscribe from the CCP4BB list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 > To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/
Re: [ccp4bb] Rescale merged data?
If I may add to that: Please deposit the full dataset, not just the set of reflections you end up using. This allows people to use all the data if they are interested.Cheers,RobbieOn 17 Apr 2024 22:35, "Hekstra, Doeke Romke" wrote: Hi Matt, I appreciate disagreement and comments from colleagues. My two cents are that it seems unnecessary to repeat scaling and merging, or any earlier step. If you want to remove structure factor amplitudes or merged intensities from the MTZ file you can do so using MTZUTILS or similar functionality in CCP4 (https://www.ccp4.ac.uk/html/mtzutils.html#generalresolution). For refinement, you can specify the desired resolution range in your favorite refinement program. My personal convention is to use CC1/2 = 0.30 as the point to which retain data and = 2 as the nominal resolution of the dataset. If you have the HKL2000 scaling log, you should be able to retrieve this information. I frankly wish we’d just deposit all data in the PDB rather than truncate based on some criterion or another. Best, Doeke From: Matt Mcleod Sent: Wednesday, April 17, 2024 4:12 PM To: Hekstra, Doeke Romke Cc: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] Rescale merged data? Sure thing. A former student left somewhere between 30-50 datasets but they scaled the data to the detector corners (or maybe edge) in HKL2000. There are many of the high-resolution bins with no reflections in them. He then went forward and merged this data, presumably in HKL2000 again and did his model building/refinement. We now need to re-refine the models against this data for publication but we need a more suitable resolution cutoff for the data. Rather than go back and index/integrate all the data and then rescale the data to a more appropriate place (then merge), I was wondering if there was a way to take the merged reflections as either .sca or .mtz (from scalepacktomtz output) and then rescale to a more appropriate resolution. It doesn't seem like the student left unmerged data. So, nothing fancy (aniostropy etc), there is just a lot of data that needs to be adjusted and I am trying to avoid reprocessing all the frames again. Matt On Wed, 17 Apr 2024 at 15:59, Hekstra, Doeke Romke <doeke_heks...@harvard.edu> wrote: Hi Matt, It would be helpful if you could describe your case in more detail. Do you want to change the resolution cutoff after scaling? Do you want to keep more data? Fewer? Or do you mean something different such as truncation to generate amplitudes, application of anisotropic resolution cutoffs, or outlier rejection? Are you referring to data that were scaled in HKL2000? Best, Doeke -Original Message- From: CCP4 bulletin board <CCP4BB@JISCMAIL.AC.UK> On Behalf Of Matt McLeod Sent: Wednesday, April 17, 2024 3:04 PM To: CCP4BB@JISCMAIL.AC.UK Subject: [ccp4bb] Rescale merged data? Hi all, I am looking at a old students data and it looks like they didn't properly cut off the data during scaling. All of the files I have appear to be the merged .sca (or mtz after converting with scalepacktomtz) - is there a way to retruncate the data after merging or do I have to reprocess the data? Thanks, To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/ -- Matthew Jordan McLeod, PhD Post-Doctoral Fellow - Cornell University To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 On 17 Apr 2024 22:35, "Hekstra, Doeke Romke" wrote: Hi Matt, I appreciate disagreement and comments from colleagues. My two cents are that it seems unnecessary to repeat scaling and merging, or any earlier step. If you want to remove structure factor amplitudes or merged intensities from the MTZ file you can do so using MTZUTILS or similar functionality in CCP4 (https://www.ccp4.ac.uk/html/mtzutils.html#generalresolution). For refinement, you can specify the desired resolution range in your favorite refinement program. My personal convention is to use CC1/2 = 0.30 as the point to which retain data and = 2 as the nominal resolution of the dataset. If you have the HKL2000 scaling log, you should be able to retrieve this information. I frankly wish we’d just deposit all data in the PDB rather than truncate based on some criterion or another. Best, Doeke From: Matt Mcleod Sent: Wednesday, April 17, 2024 4:12 PM To: Hekstra, Doeke Romke Cc: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] Rescale merged data? Sure thing. A former student left somewhere between 30-50 datasets but
Re: [ccp4bb] Rescale merged data?
Thanks Doeke (and others who have replied directly). I suppose my big concern with just truncating the data after merging would be the bin size/data processing statistics. If the edge/corner is binned 10x and I end up removing 5 of the bins by truncating the resolution post-merging, the Table 1 reported merged statistics will be based on the entire detector face being scaled rather than if I reprocessed the truncated dataset into 10 bins. I thought maybe that phenix.table_one would fix this by just using the 10% high-resolution reflections and recalculating the merging statistics, but this only does so if you supply the unmerged data (which is missing). I guess the transparent thing to do if I do not reprocess the data is to report the number of bins used and manually curate the high-resolution statistics in Table 1 based on the last bin (at the resolution limit used in refinement or applied with CAD) in scale.log file from HKL2000. Just want to say thanks again to this community, it's great to get insight in these problems so easily and quickly. Matt On Wed, 17 Apr 2024 at 16:35, Hekstra, Doeke Romke < doeke_heks...@harvard.edu> wrote: > Hi Matt, > > > > I appreciate disagreement and comments from colleagues. My two cents are > that it seems unnecessary to repeat scaling and merging, or any earlier > step. If you want to remove structure factor amplitudes or merged > intensities from the MTZ file you can do so using MTZUTILS or similar > functionality in CCP4 ( > https://www.ccp4.ac.uk/html/mtzutils.html#generalresolution). For > refinement, you can specify the desired resolution range in your favorite > refinement program. > > > > My personal convention is to use CC1/2 = 0.30 as the point to which retain > data and = 2 as the nominal resolution of the dataset. If you have > the HKL2000 scaling log, you should be able to retrieve this information. I > frankly wish we’d just deposit all data in the PDB rather than truncate > based on some criterion or another. > > > > Best, Doeke > > > > *From:* Matt Mcleod > *Sent:* Wednesday, April 17, 2024 4:12 PM > *To:* Hekstra, Doeke Romke > *Cc:* CCP4BB@JISCMAIL.AC.UK > *Subject:* Re: [ccp4bb] Rescale merged data? > > > > Sure thing. > > > > A former student left somewhere between 30-50 datasets but they scaled the > data to the detector corners (or maybe edge) in HKL2000. There are many of > the high-resolution bins with no reflections in them. He then went forward > and merged this data, presumably in HKL2000 again and did his model > building/refinement. We now need to re-refine the models against this > data for publication but we need a more suitable resolution cutoff for > the data. > > > > Rather than go back and index/integrate all the data and then rescale the > data to a more appropriate place (then merge), I was wondering if there was > a way to take the merged reflections as either .sca or .mtz (from > scalepacktomtz output) and then rescale to a more appropriate resolution. > It doesn't seem like the student left unmerged data. > > > > So, nothing fancy (aniostropy etc), there is just a lot of data that needs > to be adjusted and I am trying to avoid reprocessing all the frames again. > > > > Matt > > > > On Wed, 17 Apr 2024 at 15:59, Hekstra, Doeke Romke < > doeke_heks...@harvard.edu> wrote: > > Hi Matt, > > It would be helpful if you could describe your case in more detail. Do you > want to change the resolution cutoff after scaling? Do you want to keep > more data? Fewer? Or do you mean something different such as truncation to > generate amplitudes, application of anisotropic resolution cutoffs, or > outlier rejection? Are you referring to data that were scaled in HKL2000? > > Best, Doeke > > -Original Message- > From: CCP4 bulletin board On Behalf Of Matt McLeod > Sent: Wednesday, April 17, 2024 3:04 PM > To: CCP4BB@JISCMAIL.AC.UK > Subject: [ccp4bb] Rescale merged data? > > Hi all, > > I am looking at a old students data and it looks like they didn't properly > cut off the data during scaling. All of the files I have appear to be the > merged .sca (or mtz after converting with scalepacktomtz) - is there a way > to retruncate the data after merging or do I have to reprocess the data? > > Thanks, > > > > To unsubscribe from the CCP4BB list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.jiscmail.ac.uk_cgi-2Dbin_WA-2DJISC.exe-3FSUBED1-3DCCP4BB-26A-3D1=DwMFaQ=WO-RGvefibhHBZq3fL85hQ=DPfbtiuJFZDsaMqRC3wABRuP0iJZMKOIsdAAHocfxcg=hJXGz_3_uI3SAKeW8GEqS
Re: [ccp4bb] Rescale merged data?
Hi Matt, I appreciate disagreement and comments from colleagues. My two cents are that it seems unnecessary to repeat scaling and merging, or any earlier step. If you want to remove structure factor amplitudes or merged intensities from the MTZ file you can do so using MTZUTILS or similar functionality in CCP4 (https://www.ccp4.ac.uk/html/mtzutils.html#generalresolution). For refinement, you can specify the desired resolution range in your favorite refinement program. My personal convention is to use CC1/2 = 0.30 as the point to which retain data and = 2 as the nominal resolution of the dataset. If you have the HKL2000 scaling log, you should be able to retrieve this information. I frankly wish we’d just deposit all data in the PDB rather than truncate based on some criterion or another. Best, Doeke From: Matt Mcleod Sent: Wednesday, April 17, 2024 4:12 PM To: Hekstra, Doeke Romke Cc: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] Rescale merged data? Sure thing. A former student left somewhere between 30-50 datasets but they scaled the data to the detector corners (or maybe edge) in HKL2000. There are many of the high-resolution bins with no reflections in them. He then went forward and merged this data, presumably in HKL2000 again and did his model building/refinement. We now need to re-refine the models against this data for publication but we need a more suitable resolution cutoff for the data. Rather than go back and index/integrate all the data and then rescale the data to a more appropriate place (then merge), I was wondering if there was a way to take the merged reflections as either .sca or .mtz (from scalepacktomtz output) and then rescale to a more appropriate resolution. It doesn't seem like the student left unmerged data. So, nothing fancy (aniostropy etc), there is just a lot of data that needs to be adjusted and I am trying to avoid reprocessing all the frames again. Matt On Wed, 17 Apr 2024 at 15:59, Hekstra, Doeke Romke mailto:doeke_heks...@harvard.edu>> wrote: Hi Matt, It would be helpful if you could describe your case in more detail. Do you want to change the resolution cutoff after scaling? Do you want to keep more data? Fewer? Or do you mean something different such as truncation to generate amplitudes, application of anisotropic resolution cutoffs, or outlier rejection? Are you referring to data that were scaled in HKL2000? Best, Doeke -Original Message- From: CCP4 bulletin board mailto:CCP4BB@JISCMAIL.AC.UK>> On Behalf Of Matt McLeod Sent: Wednesday, April 17, 2024 3:04 PM To: CCP4BB@JISCMAIL.AC.UK<mailto:CCP4BB@JISCMAIL.AC.UK> Subject: [ccp4bb] Rescale merged data? Hi all, I am looking at a old students data and it looks like they didn't properly cut off the data during scaling. All of the files I have appear to be the merged .sca (or mtz after converting with scalepacktomtz) - is there a way to retruncate the data after merging or do I have to reprocess the data? Thanks, To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.jiscmail.ac.uk_cgi-2Dbin_WA-2DJISC.exe-3FSUBED1-3DCCP4BB-26A-3D1=DwMFaQ=WO-RGvefibhHBZq3fL85hQ=DPfbtiuJFZDsaMqRC3wABRuP0iJZMKOIsdAAHocfxcg=hJXGz_3_uI3SAKeW8GEqSK_1Qr5wKgBiSnbDjVjBNhSVakQOsdVMVRRe_MBbhJ4L=4rZTwcJhEjd1bJl5IPP_Hx0Y_vlFww2ZSsUH0nDMTQE=> This message was issued to members of www.jiscmail.ac.uk/CCP4BB<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.jiscmail.ac.uk_CCP4BB=DwMFaQ=WO-RGvefibhHBZq3fL85hQ=DPfbtiuJFZDsaMqRC3wABRuP0iJZMKOIsdAAHocfxcg=hJXGz_3_uI3SAKeW8GEqSK_1Qr5wKgBiSnbDjVjBNhSVakQOsdVMVRRe_MBbhJ4L=Q9SS6-TevYcWhuxShDbUJofSUXOmTisCs4Ri8YccEHw=>, a mailing list hosted by www.jiscmail.ac.uk<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.jiscmail.ac.uk=DwMFaQ=WO-RGvefibhHBZq3fL85hQ=DPfbtiuJFZDsaMqRC3wABRuP0iJZMKOIsdAAHocfxcg=hJXGz_3_uI3SAKeW8GEqSK_1Qr5wKgBiSnbDjVjBNhSVakQOsdVMVRRe_MBbhJ4L=9Jze4xWSveGay7htjcwemmwFSwvtjCqm1SvXSmmr3g4=>, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.jiscmail.ac.uk_policyandsecurity_=DwMFaQ=WO-RGvefibhHBZq3fL85hQ=DPfbtiuJFZDsaMqRC3wABRuP0iJZMKOIsdAAHocfxcg=hJXGz_3_uI3SAKeW8GEqSK_1Qr5wKgBiSnbDjVjBNhSVakQOsdVMVRRe_MBbhJ4L=SlvCI5t8-1t-4jbVfCyG2ENCo3VDDqOQbYElsa7eHKc=> -- Matthew Jordan McLeod, PhD Post-Doctoral Fellow - Cornell University To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/
Re: [ccp4bb] Rescale merged data?
Sure thing. A former student left somewhere between 30-50 datasets but they scaled the data to the detector corners (or maybe edge) in HKL2000. There are many of the high-resolution bins with no reflections in them. He then went forward and merged this data, presumably in HKL2000 again and did his model building/refinement. We now need to re-refine the models against this data for publication but we need a more suitable resolution cutoff for the data. Rather than go back and index/integrate all the data and then rescale the data to a more appropriate place (then merge), I was wondering if there was a way to take the merged reflections as either .sca or .mtz (from scalepacktomtz output) and then rescale to a more appropriate resolution. It doesn't seem like the student left unmerged data. So, nothing fancy (aniostropy etc), there is just a lot of data that needs to be adjusted and I am trying to avoid reprocessing all the frames again. Matt On Wed, 17 Apr 2024 at 15:59, Hekstra, Doeke Romke < doeke_heks...@harvard.edu> wrote: > Hi Matt, > > It would be helpful if you could describe your case in more detail. Do you > want to change the resolution cutoff after scaling? Do you want to keep > more data? Fewer? Or do you mean something different such as truncation to > generate amplitudes, application of anisotropic resolution cutoffs, or > outlier rejection? Are you referring to data that were scaled in HKL2000? > > Best, Doeke > > -Original Message- > From: CCP4 bulletin board On Behalf Of Matt McLeod > Sent: Wednesday, April 17, 2024 3:04 PM > To: CCP4BB@JISCMAIL.AC.UK > Subject: [ccp4bb] Rescale merged data? > > Hi all, > > I am looking at a old students data and it looks like they didn't properly > cut off the data during scaling. All of the files I have appear to be the > merged .sca (or mtz after converting with scalepacktomtz) - is there a way > to retruncate the data after merging or do I have to reprocess the data? > > Thanks, > > > > To unsubscribe from the CCP4BB list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 > > This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a > mailing list hosted by www.jiscmail.ac.uk, terms & conditions are > available at https://www.jiscmail.ac.uk/policyandsecurity/ > -- *Matthew Jordan McLeod, PhD* *Post-Doctoral Fellow - Cornell University* To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/
Re: [ccp4bb] Rescale merged data?
L pm xzx Have a grea lol On Wednesday, April 17, 2024, 1:59 PM, Hekstra, Doeke Romke wrote: Hi Matt, It would be helpful if you could describe your case in more detail. Do you want to change the resolution cutoff after scaling? Do you want to keep more data? Fewer? Or do you mean something different such as truncation to generate amplitudes, application of anisotropic resolution cutoffs, or outlier rejection? Are you referring to data that were scaled in HKL2000? Best, Doeke -Original Message- From: CCP4 bulletin board On Behalf Of Matt McLeod Sent: Wednesday, April 17, 2024 3:04 PM To: CCP4BB@JISCMAIL.AC.UK Subject: [ccp4bb] Rescale merged data? Hi all, I am looking at a old students data and it looks like they didn't properly cut off the data during scaling. All of the files I have appear to be the merged .sca (or mtz after converting with scalepacktomtz) - is there a way to retruncate the data after merging or do I have to reprocess the data? Thanks, To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/ To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/ To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/
Re: [ccp4bb] Rescale merged data?
Hi Matt, It would be helpful if you could describe your case in more detail. Do you want to change the resolution cutoff after scaling? Do you want to keep more data? Fewer? Or do you mean something different such as truncation to generate amplitudes, application of anisotropic resolution cutoffs, or outlier rejection? Are you referring to data that were scaled in HKL2000? Best, Doeke -Original Message- From: CCP4 bulletin board On Behalf Of Matt McLeod Sent: Wednesday, April 17, 2024 3:04 PM To: CCP4BB@JISCMAIL.AC.UK Subject: [ccp4bb] Rescale merged data? Hi all, I am looking at a old students data and it looks like they didn't properly cut off the data during scaling. All of the files I have appear to be the merged .sca (or mtz after converting with scalepacktomtz) - is there a way to retruncate the data after merging or do I have to reprocess the data? Thanks, To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/ To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/
[ccp4bb] Rescale merged data?
Hi all, I am looking at a old students data and it looks like they didn't properly cut off the data during scaling. All of the files I have appear to be the merged .sca (or mtz after converting with scalepacktomtz) - is there a way to retruncate the data after merging or do I have to reprocess the data? Thanks, To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/
