Re: [ccp4bb] About solvent flattening
On Fri, Jun 4, 2010 at 2:28 AM, zhan...@umbc.edu wrote: I was using ccp4-6.0.2. I was using the following scripts: sigmaa HKLIN test.mtz HKLOUT test-sigmaa.mtz eof title tt labin FP=FP SIGFP=SIGFP FC=FP_atm PHIC=PHIC labout DELFWT=DELFWT FWT=FWT WCMB=WCMB symmetry P1211 END eof I get the same results, i.e. some FOMs (about 6%) have gone negative, and about 75% are exactly zero, whether I use the 6.0.2 version or the current one. This message in the output is pertinent: Correlation between E**2's is non-positive -- SIGMAA set to 0.05 Also this table (final iteration of optimization of sigma-A) is telling: Iteration Number 10 on SIGMAA OLD NEW MEAN D Limits SIGMAA ShiftSIGMAA W 39.89 -- 8.74 0.0 0.0 0.0 0.0 8.74 -- 6.25 -0.59308 -0.00638 -0.59946 -0.45201 6.25 -- 5.13 0.66855 0.0 0.66855 0.54835 5.13 -- 4.45 -0.61159 -0.00813 -0.61972 -0.49740 4.45 -- 3.99 0.71063 0.0 0.71063 0.58634 3.99 -- 3.64 0.55934 0.0 0.55934 0.43656 3.64 -- 3.37 0.0 0.0 0.0 0.0 3.37 -- 3.16 0.42551 0.0 0.42551 0.32230 3.16 -- 2.98 0.0 0.0 0.0 0.0 2.98 -- 2.83 0.0 0.0 0.0 0.0 2.83 -- 2.69 0.0 0.0 0.0 0.0 2.69 -- 2.58 0.0 0.0 0.0 0.0 2.58 -- 2.48 0.0 0.0 0.0 0.0 2.48 -- 2.39 0.0 0.0 0.0 0.0 2.39 -- 2.31 0.0 0.0 0.0 0.0 2.31 -- 2.24 0.0 0.0 0.0 0.0 2.24 -- 2.17 0.0 0.0 0.0 0.0 2.17 -- 2.11 0.0 0.0 0.0 0.0 2.11 -- 2.05 0.0 0.0 0.0 0.0 2.05 -- 2.00 0.0 0.0 0.0 0.0 Refinement of SIGMAA has not converged -- using current values Overall MEAN W is0.05260 DATA for SIGMAA PLOT STHOL**2 LN(SIGMAA) 0.008000 -0.402640 0.014211 -0.341601 0.017326 -0.580991 0.023544 -0.854478 ln(SigmaA) vs sthol**2 should give straight line with slope = -26.3189*(mean square coordinate error) and intercept = 0.5*ln(sigmap/sigman). Is that possible that the negative FOM is due to the large discrepancy between FP and FC? Thanks for your kindly help! Yes I think that's it: the Fo/Fc agreement is essentially random at high resolution, so the sigma-A plot data only goes up to ~ 6.5 Ang and is meaningless, i.e. this is a case of the method being inappropriate rather than a program bug. Perhaps others who have experience of using this program in this situation can comment, my experience is only cases where agreement is good. Cheers -- Ian
Re: [ccp4bb] About solvent flattening
Hi, I believe there used to be a constraint stopping refined SigmaA values from going negative, so I'm not sure how this is happening. The basic problem here is that the correlation between Fo and Fc must be very close to zero, so that the refined SigmaA values go to zero (or should, in the case of the ones that go negative). It turns out that the likelihood is very insensitive to the exact value of SigmaA when the true value is close to zero, so it can become unstable. Something odd must be happening in the density modification, which tends to increase the correlation between Fo and Fc even if the phases aren't improving (which is one of the reasons that the use of SIGMAA for back-transformed density-modified structure factors isn't really theoretically justified). If the algorithm had some kind of smoothness restraint (like the algorithms used in modern refinement programs to determine SigmaA values from cross-validation data), it would probably behave better, but I suspect something more basic is going wrong. Regards, Randy Read On 4 Jun 2010, at 11:09, Ian Tickle wrote: On Fri, Jun 4, 2010 at 2:28 AM, zhan...@umbc.edu wrote: I was using ccp4-6.0.2. I was using the following scripts: sigmaa HKLIN test.mtz HKLOUT test-sigmaa.mtz eof title tt labin FP=FP SIGFP=SIGFP FC=FP_atm PHIC=PHIC labout DELFWT=DELFWT FWT=FWT WCMB=WCMB symmetry P1211 END eof I get the same results, i.e. some FOMs (about 6%) have gone negative, and about 75% are exactly zero, whether I use the 6.0.2 version or the current one. This message in the output is pertinent: Correlation between E**2's is non-positive -- SIGMAA set to 0.05 Also this table (final iteration of optimization of sigma-A) is telling: Iteration Number 10 on SIGMAA OLD NEW MEAN D Limits SIGMAA ShiftSIGMAA W 39.89 -- 8.74 0.0 0.0 0.0 0.0 8.74 -- 6.25 -0.59308 -0.00638 -0.59946 -0.45201 6.25 -- 5.13 0.66855 0.0 0.66855 0.54835 5.13 -- 4.45 -0.61159 -0.00813 -0.61972 -0.49740 4.45 -- 3.99 0.71063 0.0 0.71063 0.58634 3.99 -- 3.64 0.55934 0.0 0.55934 0.43656 3.64 -- 3.37 0.0 0.0 0.0 0.0 3.37 -- 3.16 0.42551 0.0 0.42551 0.32230 3.16 -- 2.98 0.0 0.0 0.0 0.0 2.98 -- 2.83 0.0 0.0 0.0 0.0 2.83 -- 2.69 0.0 0.0 0.0 0.0 2.69 -- 2.58 0.0 0.0 0.0 0.0 2.58 -- 2.48 0.0 0.0 0.0 0.0 2.48 -- 2.39 0.0 0.0 0.0 0.0 2.39 -- 2.31 0.0 0.0 0.0 0.0 2.31 -- 2.24 0.0 0.0 0.0 0.0 2.24 -- 2.17 0.0 0.0 0.0 0.0 2.17 -- 2.11 0.0 0.0 0.0 0.0 2.11 -- 2.05 0.0 0.0 0.0 0.0 2.05 -- 2.00 0.0 0.0 0.0 0.0 Refinement of SIGMAA has not converged -- using current values Overall MEAN W is0.05260 DATA for SIGMAA PLOT STHOL**2 LN(SIGMAA) 0.008000 -0.402640 0.014211 -0.341601 0.017326 -0.580991 0.023544 -0.854478 ln(SigmaA) vs sthol**2 should give straight line with slope = -26.3189*(mean square coordinate error) and intercept = 0.5*ln(sigmap/sigman). Is that possible that the negative FOM is due to the large discrepancy between FP and FC? Thanks for your kindly help! Yes I think that's it: the Fo/Fc agreement is essentially random at high resolution, so the sigma-A plot data only goes up to ~ 6.5 Ang and is meaningless, i.e. this is a case of the method being inappropriate rather than a program bug. Perhaps others who have experience of using this program in this situation can comment, my experience is only cases where agreement is good. Cheers -- Ian -- Randy J. Read Department of Haematology, University of Cambridge Cambridge Institute for Medical Research Tel: + 44 1223 336500 Wellcome Trust/MRC Building Fax: + 44 1223 336827 Hills RoadE-mail: rj...@cam.ac.uk Cambridge CB2 0XY, U.K. www-structmed.cimr.cam.ac.uk
Re: [ccp4bb] About solvent flattening
Hi Hailiang, My original ISAS program has been incorporated in the SGXPro program suite (Fu, Rose and Wang, 2005, Acta Cryst. D61, 951) and is currently used with other phasing programs in SGXPro at SER-CAT. You may find the description on the use of SGXPro by accessing the SER-CAT website at www.ser-cat.org, then click Beamline User's Guide == Novel Structure Solution. For questions on how to access SGXPro, you may contact Zheng-Qing (Albert) Fu by clicking About SER-CAT == Staff == Albert. SGXPro is a parallel workflow engine that was designed to automatically manage communication between the different processes and build systematic searches of algorithm/program/parameter space to generate the best possible result for a given data set. There were a number of cases when we were happily surprised that this 25-year old ISAS program was actually selected as the best choice by SGXPro for phasing some new data sets. We do not have an explanation on why the ISAS program works well on these particular data sets, but we are documenting the cases closely. Albert also routinely uses the ISAS program for determining the handedness of the heavy atom (anomalous scatterer) site as it works quite well automatically. With best wishes, B.C. On Jun 2, 2010, at 3:04 PM, Hailiang Zhang wrote: Hi, I wanted to do solvent flattening for my map using Wang's method. I used CCP4-DM, and now have several questions: 1. DM seems requiring the FOM, so I generated FOM using SIGMAA by providing FP, FC and SIFFP using the following: sigmaa HKLIN in.mtz HKLOUT out-sigmaa.mtz eof title tt labin FP=FP SIGFP=SIGFP FC=FC PHIC=PHIC labout DELFWT=DELFWT FWT=FWT WCMB=WCMB symmetry $spcgrp END eof # I think the output FOM should be in range between 0 to 1; however, it produced FOM between -1 to 1 based on my in.mtz. This leads to complaints by the following DM calculation, and I am not sure whether I could avoid this. 2. My DM script is as follows: # dm HKLIN ./1KP8-NewSharpRescaleB0-sigmaa-oriB.mtz HKLOUT ./1KP8-NewSharpRescaleB0-sigmaa-oriB_dm.mtzdmtest mode - SOLV - NOHIST combine PERT scheme ALL ncycles - 1 solc 0.6 solmask - frac 0.6 - 0.4 - radius 3.0 2 ncsmask LABIN FP = FWT SIGFP = SIGFP PHIO = PHIC FOMO = WCMB LABOUT FDM=FDM PHIDM=PHIDM FOMDM=FOMDM FCDM=FCDM PHICDM=PHICDM END dmtest ## I am not sure whether there the above is ok for the purpose of a simple real-space solvent flattening using Wang's method. By the way, my map is at resolution 2.0, and I am not sure what is the best radius for this resolution. 2. Based on Wang's paper (Wang, B. C. (1985) Methods in Enzymology 115, 90-112), the solvent flattening is carried out in real space, and since my goal it simply modify my map, and I don't think I need FOM etc. So, can CCP4 (or anyother packages like Phenix, CNS, UPPSALA...,) provide a simple real-space solvent flattening without too much complications? Thanks a lot for any hints. Best Regards, Hailiang
Re: [ccp4bb] About solvent flattening
Oh dear - I havent used SIGMAA for a long time . If you use REFMAC to calculate your SFs then it generates FC PHIC and FOM in the output. That is the easiset approach to getting started.. Isnt there a GUI task to do just this? Eleanor Hailiang Zhang wrote: Hi, I wanted to do solvent flattening for my map using Wang's method. I used CCP4-DM, and now have several questions: 1. DM seems requiring the FOM, so I generated FOM using SIGMAA by providing FP, FC and SIFFP using the following: sigmaa HKLIN in.mtz HKLOUT out-sigmaa.mtz eof title tt labin FP=FP SIGFP=SIGFP FC=FC PHIC=PHIC labout DELFWT=DELFWT FWT=FWT WCMB=WCMB symmetry $spcgrp END eof # I think the output FOM should be in range between 0 to 1; however, it produced FOM between -1 to 1 based on my in.mtz. This leads to complaints by the following DM calculation, and I am not sure whether I could avoid this. 2. My DM script is as follows: # dm HKLIN ./1KP8-NewSharpRescaleB0-sigmaa-oriB.mtz HKLOUT ./1KP8-NewSharpRescaleB0-sigmaa-oriB_dm.mtzdmtest mode - SOLV - NOHIST combine PERT scheme ALL ncycles - 1 solc 0.6 solmask - frac 0.6 - 0.4 - radius 3.0 2 ncsmask LABIN FP = FWT SIGFP = SIGFP PHIO = PHIC FOMO = WCMB LABOUT FDM=FDM PHIDM=PHIDM FOMDM=FOMDM FCDM=FCDM PHICDM=PHICDM END dmtest ## I am not sure whether there the above is ok for the purpose of a simple real-space solvent flattening using Wang's method. By the way, my map is at resolution 2.0, and I am not sure what is the best radius for this resolution. 2. Based on Wang's paper (Wang, B. C. (1985) Methods in Enzymology 115, 90-112), the solvent flattening is carried out in real space, and since my goal it simply modify my map, and I don't think I need FOM etc. So, can CCP4 (or anyother packages like Phenix, CNS, UPPSALA...,) provide a simple real-space solvent flattening without too much complications? Thanks a lot for any hints. Best Regards, Hailiang
Re: [ccp4bb] About solvent flattening
One or two more comments. There is a later paper by Andrew Leslie showing that the WANG real space averaging can be carried out very simply and much faster in reciprocal space - and this is the method used by all subsequent programs. density modification using calculated phases is somewhat fraught - of course the map will have strong density where there are atoms in the model, regardless of whether they are in the right place or not. Using FOM helps reduce this bias but it will still be there. It is most useful when you only have a partial model.. Eleanor there Hailiang Zhang wrote: Hi, I wanted to do solvent flattening for my map using Wang's method. I used CCP4-DM, and now have several questions: 1. DM seems requiring the FOM, so I generated FOM using SIGMAA by providing FP, FC and SIFFP using the following: sigmaa HKLIN in.mtz HKLOUT out-sigmaa.mtz eof title tt labin FP=FP SIGFP=SIGFP FC=FC PHIC=PHIC labout DELFWT=DELFWT FWT=FWT WCMB=WCMB symmetry $spcgrp END eof # I think the output FOM should be in range between 0 to 1; however, it produced FOM between -1 to 1 based on my in.mtz. This leads to complaints by the following DM calculation, and I am not sure whether I could avoid this. 2. My DM script is as follows: # dm HKLIN ./1KP8-NewSharpRescaleB0-sigmaa-oriB.mtz HKLOUT ./1KP8-NewSharpRescaleB0-sigmaa-oriB_dm.mtzdmtest mode - SOLV - NOHIST combine PERT scheme ALL ncycles - 1 solc 0.6 solmask - frac 0.6 - 0.4 - radius 3.0 2 ncsmask LABIN FP = FWT SIGFP = SIGFP PHIO = PHIC FOMO = WCMB LABOUT FDM=FDM PHIDM=PHIDM FOMDM=FOMDM FCDM=FCDM PHICDM=PHICDM END dmtest ## I am not sure whether there the above is ok for the purpose of a simple real-space solvent flattening using Wang's method. By the way, my map is at resolution 2.0, and I am not sure what is the best radius for this resolution. 2. Based on Wang's paper (Wang, B. C. (1985) Methods in Enzymology 115, 90-112), the solvent flattening is carried out in real space, and since my goal it simply modify my map, and I don't think I need FOM etc. So, can CCP4 (or anyother packages like Phenix, CNS, UPPSALA...,) provide a simple real-space solvent flattening without too much complications? Thanks a lot for any hints. Best Regards, Hailiang
Re: [ccp4bb] About solvent flattening
I'm not quite clear what your aim is here, but if you want to reproduce Wan'g method exactly, DM can't do it as it has a whole lot of more recent stuff built in (gamma correction, masking algorithms etc). If you really want to implement Wang's method (and I'm not sure quite why you would, because density modification has come a long way since then), then I think you're going to have to do it stepwise. - fft or cfft can calculate the map. - sfall or cinvfft can calculate structure factors from the modified map. - sigmaa or csigmaa can calculate weights from the modified structure factors (although if I remember correctly Wang used Sim rather than Sigmaa weighting - not sure if there is a program that will still do that). - modifying the map can be done with mapmask. - calculating the mask - tricky. Multiple runs of mapmask, fft and sfall will do it I think. Or you could use solomon, but that uses a different masking algorithm. Hailiang Zhang wrote: Hi, I wanted to do solvent flattening for my map using Wang's method. I used CCP4-DM, and now have several questions: 1. DM seems requiring the FOM, so I generated FOM using SIGMAA by providing FP, FC and SIFFP using the following: sigmaa HKLIN in.mtz HKLOUT out-sigmaa.mtz eof title tt labin FP=FP SIGFP=SIGFP FC=FC PHIC=PHIC labout DELFWT=DELFWT FWT=FWT WCMB=WCMB symmetry $spcgrp END eof # I think the output FOM should be in range between 0 to 1; however, it produced FOM between -1 to 1 based on my in.mtz. This leads to complaints by the following DM calculation, and I am not sure whether I could avoid this. 2. My DM script is as follows: # dm HKLIN ./1KP8-NewSharpRescaleB0-sigmaa-oriB.mtz HKLOUT ./1KP8-NewSharpRescaleB0-sigmaa-oriB_dm.mtzdmtest mode - SOLV - NOHIST combine PERT scheme ALL ncycles - 1 solc 0.6 solmask - frac 0.6 - 0.4 - radius 3.0 2 ncsmask LABIN FP = FWT SIGFP = SIGFP PHIO = PHIC FOMO = WCMB LABOUT FDM=FDM PHIDM=PHIDM FOMDM=FOMDM FCDM=FCDM PHICDM=PHICDM END dmtest ## I am not sure whether there the above is ok for the purpose of a simple real-space solvent flattening using Wang's method. By the way, my map is at resolution 2.0, and I am not sure what is the best radius for this resolution. 2. Based on Wang's paper (Wang, B. C. (1985) Methods in Enzymology 115, 90-112), the solvent flattening is carried out in real space, and since my goal it simply modify my map, and I don't think I need FOM etc. So, can CCP4 (or anyother packages like Phenix, CNS, UPPSALA...,) provide a simple real-space solvent flattening without too much complications? Thanks a lot for any hints. Best Regards, Hailiang
Re: [ccp4bb] About solvent flattening
On Wed, Jun 2, 2010 at 8:04 PM, Hailiang Zhang zhan...@umbc.edu wrote: sigmaa HKLIN in.mtz HKLOUT out-sigmaa.mtz eof title tt labin FP=FP SIGFP=SIGFP FC=FC PHIC=PHIC labout DELFWT=DELFWT FWT=FWT WCMB=WCMB symmetry $spcgrp END eof # I think the output FOM should be in range between 0 to 1; however, it produced FOM between -1 to 1 based on my in.mtz. Yes that shouldn't happen (and indeed it doesn't when I run it with my own data). Which version are you using? I'm probably the last person to have significantly hacked the SIGMAA source: is it possible for you to send the data and the log file (for testing purposes only)? Cheers -- Ian
[ccp4bb] About solvent flattening
Hi, I wanted to do solvent flattening for my map using Wang's method. I used CCP4-DM, and now have several questions: 1. DM seems requiring the FOM, so I generated FOM using SIGMAA by providing FP, FC and SIFFP using the following: sigmaa HKLIN in.mtz HKLOUT out-sigmaa.mtz eof title tt labin FP=FP SIGFP=SIGFP FC=FC PHIC=PHIC labout DELFWT=DELFWT FWT=FWT WCMB=WCMB symmetry $spcgrp END eof # I think the output FOM should be in range between 0 to 1; however, it produced FOM between -1 to 1 based on my in.mtz. This leads to complaints by the following DM calculation, and I am not sure whether I could avoid this. 2. My DM script is as follows: # dm HKLIN ./1KP8-NewSharpRescaleB0-sigmaa-oriB.mtz HKLOUT ./1KP8-NewSharpRescaleB0-sigmaa-oriB_dm.mtzdmtest mode - SOLV - NOHIST combine PERT scheme ALL ncycles - 1 solc 0.6 solmask - frac 0.6 - 0.4 - radius 3.0 2 ncsmask LABIN FP = FWT SIGFP = SIGFP PHIO = PHIC FOMO = WCMB LABOUT FDM=FDM PHIDM=PHIDM FOMDM=FOMDM FCDM=FCDM PHICDM=PHICDM END dmtest ## I am not sure whether there the above is ok for the purpose of a simple real-space solvent flattening using Wang's method. By the way, my map is at resolution 2.0, and I am not sure what is the best radius for this resolution. 2. Based on Wang's paper (Wang, B. C. (1985) Methods in Enzymology 115, 90-112), the solvent flattening is carried out in real space, and since my goal it simply modify my map, and I don't think I need FOM etc. So, can CCP4 (or anyother packages like Phenix, CNS, UPPSALA...,) provide a simple real-space solvent flattening without too much complications? Thanks a lot for any hints. Best Regards, Hailiang
Re: [ccp4bb] About solvent flattening
Hi Hailiang, Be sure to include PARTIAL WCMB in your sigmaa script. This option outputs something more like a conventional FOM for use in DM. Also, I would allow DM to run more cycles and suggest starting with default parameters for solvent masking and density levels and then looking at the resulting maps. You have also specified ncsmask parameters without specifying ncs or asking the program to utilize ncs. Try the CCP4i GUI, it avoids many of these technical problems. Best, --Paul --- On Wed, 6/2/10, Hailiang Zhang zhan...@umbc.edu wrote: From: Hailiang Zhang zhan...@umbc.edu Subject: [ccp4bb] About solvent flattening To: CCP4BB@JISCMAIL.AC.UK Date: Wednesday, June 2, 2010, 3:04 PM Hi, I wanted to do solvent flattening for my map using Wang's method. I used CCP4-DM, and now have several questions: 1. DM seems requiring the FOM, so I generated FOM using SIGMAA by providing FP, FC and SIFFP using the following: sigmaa HKLIN in.mtz HKLOUT out-sigmaa.mtz eof title tt labin FP=FP SIGFP=SIGFP FC=FC PHIC=PHIC labout DELFWT=DELFWT FWT=FWT WCMB=WCMB symmetry $spcgrp END eof # I think the output FOM should be in range between 0 to 1; however, it produced FOM between -1 to 1 based on my in.mtz. This leads to complaints by the following DM calculation, and I am not sure whether I could avoid this. 2. My DM script is as follows: # dm HKLIN ./1KP8-NewSharpRescaleB0-sigmaa-oriB.mtz HKLOUT ./1KP8-NewSharpRescaleB0-sigmaa-oriB_dm.mtzdmtest mode - SOLV - NOHIST combine PERT scheme ALL ncycles - 1 solc 0.6 solmask - frac 0.6 - 0.4 - radius 3.0 2 ncsmask LABIN FP = FWT SIGFP = SIGFP PHIO = PHIC FOMO = WCMB LABOUT FDM=FDM PHIDM=PHIDM FOMDM=FOMDM FCDM=FCDM PHICDM=PHICDM END dmtest ## I am not sure whether there the above is ok for the purpose of a simple real-space solvent flattening using Wang's method. By the way, my map is at resolution 2.0, and I am not sure what is the best radius for this resolution. 2. Based on Wang's paper (Wang, B. C. (1985) Methods in Enzymology 115, 90-112), the solvent flattening is carried out in real space, and since my goal it simply modify my map, and I don't think I need FOM etc. So, can CCP4 (or anyother packages like Phenix, CNS, UPPSALA...,) provide a simple real-space solvent flattening without too much complications? Thanks a lot for any hints. Best Regards, Hailiang
