-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello James and all other contributors,
I admit not having read all contributions to this thread. I understand the "John Henry Challenge" as whether there is an 'automated way of producing a model from impossible.mtz'. From looking at it and without having gone all the way to a PDB-file my feeling is one could without too much effort from the baton mode in e.g. coot. I guess this is not what you (and this thread) mean by 'automated' which leaves the impression that crystallographers have become quite spoiled children for this notion undermines how much effort and ingenuity the authors of programs like coot, O, mifit, frodo, etc, etc, pp - compared how models were prepared before this algorithms had been implemented, there is a lot of automation even in looking at the skeleton of a map! Cheers, Tim On 01/14/2013 10:12 AM, James Holton wrote: > I am absolutely delighted at the response I have gotten to my > little "John Henry Challenge"! Three people already have managed > to do the "impossible". Congratulations to George Sheldrick, Pavol > Skubak and Raj Pannu for finding ways to improve the phases over > the ones I originally obtained (using the default settings of > mlphare and dm) and build their way out of it. This is quite > useful information! At least it is to me. > > Nevertheless, I do think Frances Reyes has a point. This was meant > to be a map interpretation challenge, and not a SAD-phasing > challenge. I appreciate that the two are linked, but the reason I > did not initially provide the anomalous data is because I thought > it would be too much to ask people to re-do all the phasing, etc. > Yes, there do appear to be ways to improve the maps beyond the > particular way I phased them, but no matter how good your phasing > program is, there will always be a level of anomalous signal that > will lead to phases that are "off" enough to make building the > model "impossible". Basically, once the map gets bad enough that > just as many "wrong" atoms get built in as "right" atoms, then > there is no escape. However, I think human beings should still > have an advantage when it comes to pattern recognition, and I > remain curious to see if an insightful crystallographer can tip > that balance in the right direction. I am also still curious to > see if tweaking some setting on some automated building program > will do that too. So, my original question remains: are automated > building programs better than humans? Any human? > > I therefore declare the John Henry Challenge still open. > > > But yes, improving the phases can tip the balance too, and the > accuracy of the anomalous differences will ultimately affect the > accuracy of the phases, and so on. This is a much broader > challenge. And I think the best way to frame it is with the > question: "How low can the anomalous signal be before any > conceivable approach fails?" and perhaps: "What is the best > procedure to use for weak anomalous signal?" > > For those who are interested in joining George, Pavol, Raj and > others in this new challenge, the full spectrum of "difficulty" > from trivial (100% Se incorporation) to a complete waste of time > (0% Se, 100% S) is here: > http://bl831.als.lbl.gov/~jamesh/challenge/occ_scan/ > > The "impossible.mtz" for the John Henry Map Interpretation > Challenge was derived from "frac0.79.mtz" and "possible.mtz" from > "frac0.78.mtz". These simulated 31% and 32% Se incorporation into > Met side chains (respectively). It has now been shown that both of > these can be solved automatically if you do the phasing right. But > what about frac0.80.mtz? Or frac0.90.mtz ? At least on this one > "coordinate" of Se incorporation, the prowess of a particular > approach can be given a "score". For example, a "score" of 0.78 > means that the indicated procedure could solve the frac0.78.mtz > dataset, but not the frac0.79.mtz dataset. > > Based on the reports I have gotten back so far, the "difficulty > score" lineup is: > > score method 0.86 xds, xscale, right sites, crank2 (Pavol > Skubak) 0.78 xds, xscale, right sites, mlphare, dm, > phenix.autobuild using 20 models (James Holton) 0.75 xds, xscale, > right sites, mlphare, dm, buccaneer/refmac/dm (James Holton) 0.71 > xds, xscale, right sites, mlphare, dm, ARP/wARP 7.3 (James Holton) > 0.51 xds, xscale, right sites, mlphare, dm, ARP/wARP 6.1.1 (James > Holton) > > Note that all of these attempts "cheated" on the sites. Finding > the sites seems to be harder than solving the structure once you've > got them. That lineup is: > > score method 0.82 cheating: xds, xscale, right phases, anomalous > difference Fourier (James Holton) 0.79 xds, xscale, shelxc/d/e > 3.5A NTRY=10000 (George Sheldrick) 0.74 xds, autorickshaw > (Santosh Panjikar) 0.65 xds, xscale, phenix.hyss --search=full > (James Holton) 0.60 xds, xscale, shelxc/d with NTRY=100 (James > Holton) > > Where again the "score" is the dataset where the heavy atom site > constellation found is close enough to the "right" one to move > forward. This transition, like the model-building one, is > remarkably sharp, particularly if you let each step run for a lot > of cycles. The graph for model-building is here: > http://bl831.als.lbl.gov/~jamesh/challenge/build_CC_vs_frac.png > Note how the final map quality is pretty much independent of the > initial map quality, up to the point where it all goes wrong. I > think this again is an example of the solution needing to be at > least "half right" before it can be improved. But perhaps someone > can prove me wrong on that one? > > For those who want the unmerged data, I have all the XDS_ASCII.HKL > files here: > http://bl831.als.lbl.gov/~jamesh/challenge/occ_scan/XDS_ASCII.tgz > > If you'd like to go all the way back to the images, you can get > them from here: http://bl831.als.lbl.gov/~jamesh/workshop2/ the > "badsignal" dataset is what produced frac1.00.mtz, and > "goodsignal" produced frac0.00.mtz. You can generate anything in > between using the provided img_mix.com script. > > > Oh, and when it comes to how useful it is to spend "weeks" > building manually into a bad map, well I suppose that does indeed > depend on what alternatives you have and on the science you are > trying to do. I agree that it is always better to have better > data, but if you spend too much time trying to improve your > crystals and waiting for your next beam time, then somebody else > who IS willing to build into dodgy maps will probably do the > science for you and publish it first. So, I think in a world of > competitive grant renewals it really is critically important to > know at what point it actually is "impossible" to solve the > structure, as opposed to a situation where trying some new > procedure (or collaborator) might be the way to go. I'd like to be > able to answer that question for my users, and that's why I'm doing > this. > > -James Holton MAD Scientist > > > On 1/13/2013 12:11 PM, jens Preben Morth wrote: >> I agree with Tassos, and btw think that this crystallographer, >> should be able to go back into the lab and optimize the present >> crystal conditions to get better crystals. In particularly, when >> he or she realize that the scientific question they set out to >> investigate cannot be answered, by analyzing the final structure, >> with the available data quality. Preben >> >> >> On 1/13/13 8:52 PM, Anastassis Perrakis wrote: >>>> I think the real challenge (and one that makes for an >>>> excellent macromolecular crystallographer) is how well one >>>> can interpret a map with poor phases. >>> Let me disagree ... An excellent macromolecular >>> crystallographer, is one that given some crystals can derive >>> the best strategy to collect data, process the data optimally, >>> derive phases using all available information, build a model >>> and refine it in such a way that it best explains both data and >>> geometrical expectations, and do these as efficiently as >>> possible. >>> >>> Efficiency may suggest using one automated suite or another - >>> or indeed may best be achieved by manual labor - be it in the >>> map or in data collection strategy or refinement or another >>> step: and here I am ignoring the art of transforming >>> hair-needle-crystalline-like-dingbits to a diffracting >>> crystal. >>> >>> One that can interpret a map with poor phases can be either a >>> genius in 3d orientation - or a not necessarily too intelligent >>> nor experienced but determined student that can drink and >>> breathe this map for a few weeks in a row until a solution is >>> in place. Neither would make an excellent macromolecular >>> crystallographer by necessity. >>> >>> Tassos > - -- - -- Dr Tim Gruene Institut fuer anorganische Chemie Tammannstr. 4 D-37077 Goettingen GPG Key ID = A46BEE1A -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iD8DBQFQ9FnyUxlJ7aRr7hoRAgpNAKCTOQm4AXLMlIhOybjNKcuob3lHUACglqW1 CWMPjLdYZ7/cVgxd5ig21r8= =gojG -----END PGP SIGNATURE-----
