The reason for not being able to use BIC in this new protocol is simple. The protocol is designed around a new concept - the iterative procedure is searching for the universal solution as I defined in equation 14 of my 2007 paper http://dx.doi.org/10.1039/b702202f. The use of AIC here is not for standard model-free model selection, that is only one part. The more important use if for heading towards the simultaneous optimisation and model selection minimum.
The Bayesian origin of BIC is incompatible with equation 14. BIC's purpose, design and derivation is different. The many other frequentest (http://en.wikipedia.org/wiki/Frequentist) model selection methods are compatible, but relax does not support these yet. For example the far more advanced ICOMP (Information COMPlexity) where model flexibility is taken into account. It is quite fundamental that BIC cannot be used for the purpose of solving eq. 14, and the result of using it could be quite unpredictable. It might work, but it could also result in heading off in the wrong direction and end up far from the real solution with artificial motions, hidden motions, incorrect tensors, etc. This is totally untested! Note that a redesign of the protocol and equation 14 around BIC will be required. So while BIC works very well for model-free model selection, it is totally incompatible with the hunt for the universal solution as currently defined. Regards, Edward On 14 April 2010 00:20, Michael Bieri <[email protected]> wrote: > This all sounds good. I also like the idea that users can add more > iterations. I will recode this. Why do you want to get rid of the BIC > selection mode? Even recent papers are still using it and as relax can > handle it, I would suggest not to force users too much to use AIC. What > do you think? > > > Edward d'Auvergne wrote: >> The local tm model for a peptide is highly probable because of high >> internal mobility being coupled to the external diffusion. I would >> guess that in this case the protocol was circling endlessly around the >> solution. We should store all the information (already converted for >> comparison to save a little computation time) for each iteration >> within lists from the convergence() method. For a maximum number, I >> would suggest 30. Each iteration is much quicker to complete than the >> previous, so 20 vs. 30 is much quicker than the first or second 10. >> It would be better to be on the safe side and have a max of 30 - it is >> too difficult for us to predict how a difficult, yet solvable case >> will look like. In this case, I think we should add an new list >> called self.max_iterations which starts as false for all. If 30 >> iterations is hit, then the position for that model should be set to >> True. I would also recommend removing the unused AIC vs. BIC GUI >> element and maybe replacing it with a text box where the user can >> change this number (and we only allow it to be bigger than 30 ;). >> >> The self.max_iterations list is then used in the model_selection() >> method to eliminate that global model prior to model selection. For >> the progress bar, it would be good to have the iteration number also >> displayed. That way it is better communicated that the bar is >> counting iterations and not a percentage of being complete. We >> shouldn't give to the user the impression that they have to wait until >> the progress bar hits the end. What kindm of granularity do you have >> here? Can we have a movement for the completion of each iteration? >> >> Regards, >> >> Edward >> >> >> >> >> On 13 April 2010 00:20, Michael Bieri <[email protected]> wrote: >> >>> It would be the best to store results of each iterations in >>> convergence(). I already had a couple of cases / models that never >>> converged. Predominately, these were molecules with a high degree o >>> motion. For example there was a peptide (40 residues), where none of the >>> models converged after 20 iterations, so I aborted them. The diffusion >>> tensor fits well so I thought that's alright. I heard from other people >>> that they also stop their calculation after 20 iterations. Nevertheless >>> and surprisingly, the model wich was selected for S2 (And the rest) was >>> local tm - possible? >>> >>> But in general I like the idea to compare the minimization of all the >>> iterations, as you suggested. I still would set a maximum of 20 (or 30) >>> iterations to avoid calculations over several days. Another benefit is >>> that we are able to update the progress bar. >>> >>> >>> Edward d'Auvergne wrote: >>> >>>> Hi Michael, >>>> >>>> We could add an arg to limit the number of iterations, but I think >>>> there is a much better way. The problem of iterating forever is >>>> because of circling around the minimum (not the optimisation minimum >>>> but Occam's razor, as described in equation (14) of my 2007 paper >>>> http://www.nmr-relax.com/refs.html). Currently only the current to >>>> previous iterations are compared, but some people have encountered a 3 >>>> iteration cycle. Even larger looping might occur. Note that the >>>> repeated position in this cycle is identical to one of the previous. >>>> So we should really store all of the required info in the >>>> convergence() method of auto_analyses/dauvergne_protocol.py and check >>>> the current against all iterations. >>>> >>>> If one of these converged loops has not been reached after 20 >>>> iterations, then either the system is complex but will still soon >>>> converge, or something is seriously, very seriously wrong. The model >>>> should be deselected and the user told of its total failure. My >>>> experience is that after 20, convergence has been reached. But I >>>> think we should allow more before termination, as the termination is a >>>> sign that something is sick. >>>> >>>> If we catch the large iteration circling about Occam's razor and >>>> terminate, do you think we would need a maximum iteration termination >>>> point and subsequent global model elimination? Have you encountered >>>> such a non-cycling, infinite looping? >>>> >>>> Regards, >>>> >>>> Edward >>>> >>>> >>>> >>>> >>>> On 12 April 2010 03:02, <[email protected]> wrote: >>>> >>>> >>>>> Author: michaelbieri >>>>> Date: Mon Apr 12 03:02:19 2010 >>>>> New Revision: 11061 >>>>> >>>>> URL: http://svn.gna.org/viewcvs/relax?rev=11061&view=rev >>>>> Log: >>>>> The relaxGUI controller updates its progress bar using informations from >>>>> the status singleton during model-free analysis. >>>>> >>>>> Currently, there is a maximum of iterations set for models 1 - 5. This >>>>> also has to be set in the dauvergne_protocol.py script. Edward, what do >>>>> you think about limiting to 20 iterations? >>>>> >>>>> Modified: >>>>> branches/bieri_gui/gui_bieri/controller.py >>>>> >>>>> Modified: branches/bieri_gui/gui_bieri/controller.py >>>>> URL: >>>>> http://svn.gna.org/viewcvs/relax/branches/bieri_gui/gui_bieri/controller.py?rev=11061&r1=11060&r2=11061&view=diff >>>>> ============================================================================== >>>>> --- branches/bieri_gui/gui_bieri/controller.py (original) >>>>> +++ branches/bieri_gui/gui_bieri/controller.py Mon Apr 12 03:02:19 2010 >>>>> @@ -142,13 +142,50 @@ >>>>> >>>>> def __init__(self,aWxTextCtrl): >>>>> self.out=aWxTextCtrl >>>>> + self.status = Status() >>>>> >>>>> + def limit_entries(self): >>>>> + """ Function to overcome feedback problem of wx.CallAfter() >>>>> command""" >>>>> + >>>>> + # Maximum allowed number of lines in log window. >>>>> + max_entries = 10000 >>>>> + >>>>> + # read number of lines in log window. >>>>> + total_entries = self.out.log_panel.GetNumberOfLines() >>>>> + >>>>> + # Shift entries backwards if maximum of line exeeded. >>>>> + if total_entries > max_entries: >>>>> + # Reset log window entries >>>>> + new_entries = 'Refreshing log window...\n\n' >>>>> + self.out.log_panel.SetValue(new_entries) >>>>> >>>>> def write(self,string): >>>>> - global progress >>>>> >>>>> # Limit panle entries to max_entries Lines. >>>>> wx.CallAfter(self.limit_entries) >>>>> + >>>>> + # Update Gauge (Progress bar). >>>>> + # Local tm model: >>>>> + if self.status.dAuvergne_protocol.diff_model == 'local_tm': >>>>> + # Current model. >>>>> + no = self.status.dAuvergne_protocol.current_model[2:] >>>>> + no = int(no) >>>>> + >>>>> + # Total selected models. >>>>> + total_models = >>>>> len(self.status.dAuvergne_protocol.local_mf_models) >>>>> + >>>>> + # update Progress bar. >>>>> + wx.CallAfter(self.out.progress_bar.SetValue, >>>>> (100*no/total_models)) >>>>> + >>>>> + # Sphere to Ellipsoid Models. >>>>> + if self.status.dAuvergne_protocol.diff_model in ['sphere', >>>>> 'prolate', 'oblate', 'ellipsoid']: >>>>> + # Determine actual round (maximum is 20). >>>>> + wx.CallAfter(self.out.progress_bar.SetValue, >>>>> (100*(self.status.dAuvergne_protocol.round-1)/20)) >>>>> + >>>>> + # Final analysis. >>>>> + if self.status.dAuvergne_protocol.diff_model == 'final': >>>>> + mc_simulation = self.status.mc_number >>>>> + >>>>> >>>>> # Add new output. >>>>> wx.CallAfter(self.out.log_panel.AppendText, string) >>>>> >>>>> >>>>> _______________________________________________ >>>>> relax (http://nmr-relax.com) >>>>> >>>>> This is the relax-commits mailing list >>>>> [email protected] >>>>> >>>>> To unsubscribe from this list, get a password >>>>> reminder, or change your subscription options, >>>>> visit the list information page at >>>>> https://mail.gna.org/listinfo/relax-commits >>>>> >>>>> >>>>> >>>> _______________________________________________ >>>> relax (http://nmr-relax.com) >>>> >>>> This is the relax-devel mailing list >>>> [email protected] >>>> >>>> To unsubscribe from this list, get a password >>>> reminder, or change your subscription options, >>>> visit the list information page at >>>> https://mail.gna.org/listinfo/relax-devel >>>> >>>> >>>> >>>> >>> _______________________________________________ >>> relax (http://nmr-relax.com) >>> >>> This is the relax-devel mailing list >>> [email protected] >>> >>> To unsubscribe from this list, get a password >>> reminder, or change your subscription options, >>> visit the list information page at >>> https://mail.gna.org/listinfo/relax-devel >>> >>> >> >> >> > > _______________________________________________ > relax (http://nmr-relax.com) > > This is the relax-devel mailing list > [email protected] > > To unsubscribe from this list, get a password > reminder, or change your subscription options, > visit the list information page at > https://mail.gna.org/listinfo/relax-devel > _______________________________________________ relax (http://nmr-relax.com) This is the relax-devel mailing list [email protected] To unsubscribe from this list, get a password reminder, or change your subscription options, visit the list information page at https://mail.gna.org/listinfo/relax-devel
