Re: [ccp4bb] 8/ 2263 spots indexed XDS
Dear all, after a week looking more closely into the datasets, also with the help of Manfred Weiss and Kay Diederichs, who kindly offered to look at some of them, I come back to you to report on our findings (should this info be useful in the future to any of you). Thank you very much everybody for your replies. :) I have 11 datasets, collected from 3 very small crystals (30-40 um aprox). While the highest resolution for the "best datasets" extended to 4 or 4.5 Angstroms, two aspects that contributed to the difficulty in processing were: radiation damage and also the presence of multiple lattices. It was necessary to look through the images and select the wedges where there were (single) spots at "high resolution" (often after only 180 degrees of data collection the crystals were considerably damaged and attempts to process the 360 degrees that were collected didn't work or resulted in extremely poor overall statistics). For the indexing of multiple lattices I have had the opportunity to try programs like CrysalisPro from Rigaku and Proteum from Bruker (sorry I didn't try DIALS yet, although I was advised to do it too!) and I can tell that they do a fine job in finding the crystal lattice despite the ugliness of the spots. However, since only small wedges of the data were useful and I have a low symmetry space group, I couldn't take great advantage of their capabilities, though. In some of the datasets, we could also see that indexing of few frames gave ok statistics, but as more and more frames were added to the processing (also as the presence of the second lattice became more evident) the statistics became worse. With the exception of 3 datasets (belonging to the same crystal), were the spots were streaky/multiple/splitted, the others could be treated to a greater or a lesser extent and processed with the XDS.INP file resulting from generate_XDS.INP. Also, the fact that many spots were selected by COLSPOT but very few were used in the IDXREF step was due to the fact that I had some ice rings (how could I miss that, right?) and many were picked from there. So, once again, carefully selecting the SPOT_RANGE or defining it more than once, as well as using the keyword EXCLUDE_RESOLUTION_RANGE where the ice rings were, or simply cutting the resolution at 4 Angstroms, alleviated the problem in most cases. Kay also advised me to use DELPHI=20, which is used during the INTEGRATE step, and although its default is 5, it's good to increase its value in instances where there are few strong reflections that can be used for learning spot profiles (avoids geometry refinement with too few reflections in INTEGRATE). With the data that I have, very weak, I was also advised by Kay not to lower the MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT below 3 - but rather to increase it to 6 if my reflections were big. And if the latter was done (MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=6) lower STRONG_PIXEL to 3. Although the first instinct from many of you was to think that there was something wrong with the input, like detector distance, origin, oscillation range, etc., all these parameters were recorded correctly. I think that the fair way to summarize this all is "it's crappy data" (for a number of reasons), because the programs and the synchrotron software did their job fine. So thank you once again everyone for helping me and participating in this discussion, I will now go back to the lab and try to grow nicer, better ordered, bigger crystals ;) All the best, Almu El vie., 29 nov. 2019 a las 14:43, Tim Gruene () escribió: > Dear Almudena, > > SPOT_MAXIMUM-CENTROID is probably the keyword in XDS.INP that you want to > increase in your situation. The default is 3. Try e.g. 5, and increase > slowly > to make sure that the suggested solution makes sense > > Best regards, > Tim > > On Friday, November 29, 2019 12:48:11 PM CET Almudena Ponce Salvatierra > wrote: > > Dear all, > > > > I have some data sets that don't want to be processed :p > > > > In one of them, when I look at IDXREF.LP I see virtually none of the > found > > spots were indexed and the reason is that they are "too far from the > > expected position". The spots are smeary and elongated, so not the > > prettiest. > > > > I have managed to process so far only one data set with decent statistics > > from another crystal harvested from the same drop, where the diffraction > > spots look better. > > > > I am trying to find in the xds wiki the keyword I should fine tune in > order > > to make those spots indexable. > > > > Could you help me please? > > > > Thank you very much in advance. > > > > Best wishes, > > > > Almu > > > > > > > > To unsubscribe from the CCP4BB list, click the following link: > > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1 > > -- > -- > Tim Gruene > Head of the Centre for X-ray Structure Analysis > Faculty of Chemistry > University of Vienna > > Phone:
Re: [ccp4bb] 8/ 2263 spots indexed XDS
Hi Almu, some other pointers that might be useful in your case: * Check the list of known beamline configurations (and particulars) at [1] and [2] to see if there is something you need to tell the processing software about. * If you have multiple lattices, split spots or ice rings (or combinations of these), something like "iterative indexing" can ensure that subsequent integration starts from a good, initial orientation matrix [3,4]. * Finally a note about damaged/unmasked pixels: these can greatly confuse COLSPOT when it comes to classifying so-called "weak spots" with odd results [5,6,7]. As Kay said, there are very few datasets where XDS indexing is genuinely stuck - as long as (a) there are spots with a hint of a spotlike shape and (b) the processing is told about beamline and detector specifics. Cheers Clemens [1] https://www.globalphasing.com/autoproc/wiki/index.cgi?BeamlineSettings [2] https://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/Generate_XDS.INP [3] https://www.globalphasing.com/autoproc/manual/autoPROC7.html#step1_IterativeIndexing [4] https://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/Indexing [5] https://www.globalphasing.com/autoproc/wiki/index.cgi?BeamlineSettings#hotpixels [6] https://www.globalphasing.com/autoproc/wiki/index.cgi?AnalyseDataForDamagedPixels [7] https://www.globalphasing.com/autoproc/wiki/index.cgi?DataProcessingHdf5Eiger16bit201910 On Sat, Nov 30, 2019 at 11:06:42AM +0100, Almudena Ponce Salvatierra wrote: > Hi Raphael, > > Thank you very much. I tried yesterday with different origins and I have > changed the REFINE(IDXREF) keyword several times as to refine BEAM, > POSITION, DISTANCE... I have to keep trying because fo far I didn't succeed. > > I will get back to you once I find what the problem is. > > Thank you once again! > > Best, > > Almudena > > El vie., 29 nov. 2019 a las 12:56, R. Gasper-Schönenbrücher (< > raphael.gas...@mpi-dortmund.mpg.de>) escribió: > > > Hi Almu, > > > > this is most often the case, when your beam center is not assigned > > correctly. Try to find out the beam center: > > > > > > https://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/Finding_out_ORGX_ORGY > > Another possibilty is that the distance that you get from the > > header/machine is not very accurate. You can try to slightly adjust your > > distance and see, whether this improves your indexing (e.g. instead of 50 > > start from 52). > > > > In XDS.INP you can enable distance refinement (as well as beam center > > refinement) throughout all steps. Just remove the "!" in "Refine" > > > > Hope this helps, > > > > Best wishes, > > > > Raphael > > > > > > On 29.11.19 12:48, Almudena Ponce Salvatierra wrote: > > > > Dear all, > > > > I have some data sets that don't want to be processed :p > > > > In one of them, when I look at IDXREF.LP I see virtually none of the found > > spots were indexed and the reason is that they are "too far from the > > expected position". The spots are smeary and elongated, so not the > > prettiest. > > > > I have managed to process so far only one data set with decent statistics > > from another crystal harvested from the same drop, where the diffraction > > spots look better. > > > > I am trying to find in the xds wiki the keyword I should fine tune in > > order to make those spots indexable. > > > > Could you help me please? > > > > Thank you very much in advance. > > > > Best wishes, > > > > Almu > > > > -- > > > > To unsubscribe from the CCP4BB list, click the following link: > > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1 > > > > -- > > Dr. Raphael Gasper-Schönenbrücher > > Crystallography and Biophysics Facility > > Max-Planck-Institute of Molecular Physiology > > Room F0.92 > > Otto-Hahn-Str. 11 > > 44227 Dortmund, Germany > > Tel +49 (0)231-133 2729 > > > > > > > > To unsubscribe from the CCP4BB list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1 -- *-- * Clemens Vonrhein, Ph.D. vonrhein AT GlobalPhasing DOT com * Global Phasing Ltd., Sheraton House, Castle Park * Cambridge CB3 0AX, UK www.globalphasing.com *-- To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1
Re: [ccp4bb] 8/ 2263 spots indexed XDS
Hi Raphael, Thank you very much. I tried yesterday with different origins and I have changed the REFINE(IDXREF) keyword several times as to refine BEAM, POSITION, DISTANCE... I have to keep trying because fo far I didn't succeed. I will get back to you once I find what the problem is. Thank you once again! Best, Almudena El vie., 29 nov. 2019 a las 12:56, R. Gasper-Schönenbrücher (< raphael.gas...@mpi-dortmund.mpg.de>) escribió: > Hi Almu, > > this is most often the case, when your beam center is not assigned > correctly. Try to find out the beam center: > > > https://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/Finding_out_ORGX_ORGY > Another possibilty is that the distance that you get from the > header/machine is not very accurate. You can try to slightly adjust your > distance and see, whether this improves your indexing (e.g. instead of 50 > start from 52). > > In XDS.INP you can enable distance refinement (as well as beam center > refinement) throughout all steps. Just remove the "!" in "Refine" > > Hope this helps, > > Best wishes, > > Raphael > > > On 29.11.19 12:48, Almudena Ponce Salvatierra wrote: > > Dear all, > > I have some data sets that don't want to be processed :p > > In one of them, when I look at IDXREF.LP I see virtually none of the found > spots were indexed and the reason is that they are "too far from the > expected position". The spots are smeary and elongated, so not the > prettiest. > > I have managed to process so far only one data set with decent statistics > from another crystal harvested from the same drop, where the diffraction > spots look better. > > I am trying to find in the xds wiki the keyword I should fine tune in > order to make those spots indexable. > > Could you help me please? > > Thank you very much in advance. > > Best wishes, > > Almu > > -- > > To unsubscribe from the CCP4BB list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1 > > -- > Dr. Raphael Gasper-Schönenbrücher > Crystallography and Biophysics Facility > Max-Planck-Institute of Molecular Physiology > Room F0.92 > Otto-Hahn-Str. 11 > 44227 Dortmund, Germany > Tel +49 (0)231-133 2729 > > To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1
Re: [ccp4bb] 8/ 2263 spots indexed XDS
Hi Andy, thank you so much for your comprehensive answer. I have played with all those parameters, following the instructions I found in the XDS wiki, as they seem to be those having an influence in spot picking. However, I have then realized it is not a matter of the number of spots, but rather a matter of how many of them are indexed. I will get back to you and to everybody once I find out what is going wrong. Thank you very much and thank you very much everyone for all your replies :) Best, Almu El vie., 29 nov. 2019 a las 13:34, THOMPSON Andrew (< andrew.thomp...@synchrotron-soleil.fr>) escribió: > Hi Almudena > I wouldn't claim to be a great expert on XDS, but, based on the experience > of analyzing a load of data on the beamline, there are many things you can > try - > The most important ones are to use lots (all?) your images for the spot > search (SPOT_RANGE) , and not to hesitate cutting the resolution limit for > the search to something very low where the spots are not streaky (3, 4, 5 > Angstrom?). There may be "good regions" and "bad regions", so you can get > the integration "going" from a good region. Once things are indexed and > analyse OK you can add the other data back. > Other interesting keywords will be > STRONG_PIXEL (might need to be bigger or smaller!), > MAXIMUM_NUMBER_OF_PIXELS_IN_A_SPOT (same) and DELPHI (once you have indexed) > Since you have processed one data set, you can give XDS the correct unit > cell, that always helps. > Once you have got the integration going, *save* the processing directory > as, with terrible looking spots, you can get "driven away" from the right > indexing by the post refinement. > I always add data in "block by block". > I have successfully processed some pretty awful looking data with XDS, but > it needs a bit of determination and quite a lot of time (hours, not > minutes), so only invest if the data set is just unique > Good luck > Andy > > > De : CCP4 bulletin board [CCP4BB@JISCMAIL.AC.UK] de la part de Almudena > Ponce Salvatierra [maps.fa...@gmail.com] > Envoyé : vendredi 29 novembre 2019 12:48 > À : CCP4BB@JISCMAIL.AC.UK > Objet : [ccp4bb] 8/ 2263 spots indexed XDS > > Dear all, > > I have some data sets that don't want to be processed :p > > In one of them, when I look at IDXREF.LP I see virtually none of the found > spots were indexed and the reason is that they are "too far from the > expected position". The spots are smeary and elongated, so not the > prettiest. > > I have managed to process so far only one data set with decent statistics > from another crystal harvested from the same drop, where the diffraction > spots look better. > > I am trying to find in the xds wiki the keyword I should fine tune in > order to make those spots indexable. > > Could you help me please? > > Thank you very much in advance. > > Best wishes, > > Almu > > > > To unsubscribe from the CCP4BB list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1 > > To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1
Re: [ccp4bb] 8/ 2263 spots indexed XDS
Hi Tim, Thank you very much for replying. I thought this was the keyword I was looking for indeed, however, it has not worked so far... :( I will write back once I find out what the problem was. Best, Almu El vie., 29 nov. 2019 a las 14:43, Tim Gruene () escribió: > Dear Almudena, > > SPOT_MAXIMUM-CENTROID is probably the keyword in XDS.INP that you want to > increase in your situation. The default is 3. Try e.g. 5, and increase > slowly > to make sure that the suggested solution makes sense > > Best regards, > Tim > > On Friday, November 29, 2019 12:48:11 PM CET Almudena Ponce Salvatierra > wrote: > > Dear all, > > > > I have some data sets that don't want to be processed :p > > > > In one of them, when I look at IDXREF.LP I see virtually none of the > found > > spots were indexed and the reason is that they are "too far from the > > expected position". The spots are smeary and elongated, so not the > > prettiest. > > > > I have managed to process so far only one data set with decent statistics > > from another crystal harvested from the same drop, where the diffraction > > spots look better. > > > > I am trying to find in the xds wiki the keyword I should fine tune in > order > > to make those spots indexable. > > > > Could you help me please? > > > > Thank you very much in advance. > > > > Best wishes, > > > > Almu > > > > > > > > To unsubscribe from the CCP4BB list, click the following link: > > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1 > > -- > -- > Tim Gruene > Head of the Centre for X-ray Structure Analysis > Faculty of Chemistry > University of Vienna > > Phone: +43-1-4277-70202 > > GPG Key ID = A46BEE1A > To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1
Re: [ccp4bb] 8/ 2263 spots indexed XDS
Hi Almu, I have yet to see a data set with clear reflections that cannot be indexed with XDS. The most often reason for failure is some wrong input, most often ORGX/ORGY and ROTATION_AXIS, but of course OSCILLATION_RANGE, X-RAY_WAVELENGTH, DETECTOR_DISTANCE must also be correct. If you cannot solve the problem, then I offer my help - make the raw data available to me, using some cloud service, and I'll take a look. I'll keep your data confidential of course. best, Kay On Fri, 29 Nov 2019 16:06:53 +0100, Almudena Ponce Salvatierra wrote: >Thank you Tim, > >I will try and let you know how it goes. > >All the best, > >Almudena > >El vie., 29 nov. 2019 a las 14:43, Tim Gruene () >escribió: > >> Dear Almudena, >> >> SPOT_MAXIMUM-CENTROID is probably the keyword in XDS.INP that you want to >> increase in your situation. The default is 3. Try e.g. 5, and increase >> slowly >> to make sure that the suggested solution makes sense >> >> Best regards, >> Tim >> >> On Friday, November 29, 2019 12:48:11 PM CET Almudena Ponce Salvatierra >> wrote: >> > Dear all, >> > >> > I have some data sets that don't want to be processed :p >> > >> > In one of them, when I look at IDXREF.LP I see virtually none of the >> found >> > spots were indexed and the reason is that they are "too far from the >> > expected position". The spots are smeary and elongated, so not the >> > prettiest. >> > >> > I have managed to process so far only one data set with decent statistics >> > from another crystal harvested from the same drop, where the diffraction >> > spots look better. >> > >> > I am trying to find in the xds wiki the keyword I should fine tune in >> order >> > to make those spots indexable. >> > >> > Could you help me please? >> > >> > Thank you very much in advance. >> > >> > Best wishes, >> > >> > Almu >> > >> > >> > >> > To unsubscribe from the CCP4BB list, click the following link: >> > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1 >> >> -- >> -- >> Tim Gruene >> Head of the Centre for X-ray Structure Analysis >> Faculty of Chemistry >> University of Vienna >> >> Phone: +43-1-4277-70202 >> >> GPG Key ID = A46BEE1A >> > > > >To unsubscribe from the CCP4BB list, click the following link: >https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1 > To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1
Re: [ccp4bb] 8/ 2263 spots indexed XDS
Thank you Tim, I will try and let you know how it goes. All the best, Almudena El vie., 29 nov. 2019 a las 14:43, Tim Gruene () escribió: > Dear Almudena, > > SPOT_MAXIMUM-CENTROID is probably the keyword in XDS.INP that you want to > increase in your situation. The default is 3. Try e.g. 5, and increase > slowly > to make sure that the suggested solution makes sense > > Best regards, > Tim > > On Friday, November 29, 2019 12:48:11 PM CET Almudena Ponce Salvatierra > wrote: > > Dear all, > > > > I have some data sets that don't want to be processed :p > > > > In one of them, when I look at IDXREF.LP I see virtually none of the > found > > spots were indexed and the reason is that they are "too far from the > > expected position". The spots are smeary and elongated, so not the > > prettiest. > > > > I have managed to process so far only one data set with decent statistics > > from another crystal harvested from the same drop, where the diffraction > > spots look better. > > > > I am trying to find in the xds wiki the keyword I should fine tune in > order > > to make those spots indexable. > > > > Could you help me please? > > > > Thank you very much in advance. > > > > Best wishes, > > > > Almu > > > > > > > > To unsubscribe from the CCP4BB list, click the following link: > > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1 > > -- > -- > Tim Gruene > Head of the Centre for X-ray Structure Analysis > Faculty of Chemistry > University of Vienna > > Phone: +43-1-4277-70202 > > GPG Key ID = A46BEE1A > To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1
Re: [ccp4bb] 8/ 2263 spots indexed XDS
Dear Almudena, SPOT_MAXIMUM-CENTROID is probably the keyword in XDS.INP that you want to increase in your situation. The default is 3. Try e.g. 5, and increase slowly to make sure that the suggested solution makes sense Best regards, Tim On Friday, November 29, 2019 12:48:11 PM CET Almudena Ponce Salvatierra wrote: > Dear all, > > I have some data sets that don't want to be processed :p > > In one of them, when I look at IDXREF.LP I see virtually none of the found > spots were indexed and the reason is that they are "too far from the > expected position". The spots are smeary and elongated, so not the > prettiest. > > I have managed to process so far only one data set with decent statistics > from another crystal harvested from the same drop, where the diffraction > spots look better. > > I am trying to find in the xds wiki the keyword I should fine tune in order > to make those spots indexable. > > Could you help me please? > > Thank you very much in advance. > > Best wishes, > > Almu > > > > To unsubscribe from the CCP4BB list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1 -- -- Tim Gruene Head of the Centre for X-ray Structure Analysis Faculty of Chemistry University of Vienna Phone: +43-1-4277-70202 GPG Key ID = A46BEE1A To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1 signature.asc Description: This is a digitally signed message part.
Re: [ccp4bb] 8/ 2263 spots indexed XDS
Hi Almudena I wouldn't claim to be a great expert on XDS, but, based on the experience of analyzing a load of data on the beamline, there are many things you can try - The most important ones are to use lots (all?) your images for the spot search (SPOT_RANGE) , and not to hesitate cutting the resolution limit for the search to something very low where the spots are not streaky (3, 4, 5 Angstrom?). There may be "good regions" and "bad regions", so you can get the integration "going" from a good region. Once things are indexed and analyse OK you can add the other data back. Other interesting keywords will be STRONG_PIXEL (might need to be bigger or smaller!), MAXIMUM_NUMBER_OF_PIXELS_IN_A_SPOT (same) and DELPHI (once you have indexed) Since you have processed one data set, you can give XDS the correct unit cell, that always helps. Once you have got the integration going, *save* the processing directory as, with terrible looking spots, you can get "driven away" from the right indexing by the post refinement. I always add data in "block by block". I have successfully processed some pretty awful looking data with XDS, but it needs a bit of determination and quite a lot of time (hours, not minutes), so only invest if the data set is just unique Good luck Andy De : CCP4 bulletin board [CCP4BB@JISCMAIL.AC.UK] de la part de Almudena Ponce Salvatierra [maps.fa...@gmail.com] Envoyé : vendredi 29 novembre 2019 12:48 À : CCP4BB@JISCMAIL.AC.UK Objet : [ccp4bb] 8/ 2263 spots indexed XDS Dear all, I have some data sets that don't want to be processed :p In one of them, when I look at IDXREF.LP I see virtually none of the found spots were indexed and the reason is that they are "too far from the expected position". The spots are smeary and elongated, so not the prettiest. I have managed to process so far only one data set with decent statistics from another crystal harvested from the same drop, where the diffraction spots look better. I am trying to find in the xds wiki the keyword I should fine tune in order to make those spots indexable. Could you help me please? Thank you very much in advance. Best wishes, Almu To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1 To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1
