The question is how significant is this bias, and is the cure (i.e. leaving out more reflections from the working set) worse than the disease? For refinements at 'medium' typical resolution (around 2.5 to 2 Ang) we are working with an observation/parameter count ratio of say < 3 (naturally I'm counting the geometric restraints with the X-ray observations). The amount of bias in Rwork and other statistics derived from the working set depends critically on how close the obs/param ratio is to 1. The Rfree optimisation is used only to determine weighting parameters (including sigma-A) and it's unlikely there will be more than say 20 of these. Typically there are at least 1000 refls in the test set, so for the Rfree optimisation the obs/param ratio will be around 50. This is much larger than the obs/param ratio for Rwork and may well mean that the biasing effect on Rfree is negligible. It should be easy to do some tests comparing Rfree with Rsleep to check the bias (taking into account errors to limited sample sizes of course), and also to see what are the effects of leaving out the sleeping set on the refinement and the maps. I don't think it would be wise to rush into this until we have done proper evaluations.
-- Ian > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Mark J. van Raaij > Sent: 01 October 2007 14:58 > To: CCP4BB@JISCMAIL.AC.UK > Subject: R-sleep > > Dear All, > > the short paper by Gerard Kleywegt (ActaD 63, 939-940) treats > an interesting subject (at least I think so...). I agree that > what we are now doing in many cases is effectively refining > against Rfree. For example, the standard CNS torsion angle > refinement does n refinement trials with randomised starting > points. If you then take the one with lowest Rfree (or let a > script do this for you), you are biasing Rfree! > Therefore, his proposal to put an extra set of reflections in > a dormant "vault" (R-sleep) sounds like a good idea to me. > However, how would the "vault" be implemented to be > effective? If left to the experimenter, it would be very > tempting to check R-sleep once in a while (or often) during > refinement, rendering it useless as an unbiased validator. > > or am I being paranoid and too pessimistic? > > Mark J. van Raaij > Unidad de Bioquímica Estructural > Dpto de Bioquímica, Facultad de Farmacia > and > Unidad de Rayos X, Edificio CACTUS > Universidad de Santiago > 15782 Santiago de Compostela > Spain > http://web.usc.es/~vanraaij/ > > > Disclaimer This communication is confidential and may contain privileged information intended solely for the named addressee(s). It may not be used or disclosed except for the purpose for which it has been sent. If you are not the intended recipient you must not review, use, disclose, copy, distribute or take any action in reliance upon it. If you have received this communication in error, please notify Astex Therapeutics Ltd by emailing [EMAIL PROTECTED] and destroy all copies of the message and any attached documents. Astex Therapeutics Ltd monitors, controls and protects all its messaging traffic in compliance with its corporate email policy. The Company accepts no liability or responsibility for any onward transmission or use of emails and attachments having left the Astex Therapeutics domain. Unless expressly stated, opinions in this message are those of the individual sender and not of Astex Therapeutics Ltd. The recipient should check this email and any attachments for the presence of computer viruses. Astex Therapeutics Ltd accepts no liability for damage caused by any virus transmitted by this email. E-mail is susceptible to data corruption, interception, unauthorized amendment, and tampering, Astex Therapeutics Ltd only send and receive e-mails on the basis that the Company is not liable for any such alteration or any consequences thereof. Astex Therapeutics Ltd., Registered in England at 436 Cambridge Science Park, Cambridge CB4 0QA under number 3751674
