Dietterich had quite a comprehensive paper on this issue: [1] T. G. Dietterich. Approximate statistical tests for comparing supervised classification learning algorithms. Neural computation, 10(7):1895–1923, 1998. I am not sure if it applies to other error metrics rather than "misclassification error" though.
Best, Sebastian > On May 15, 2015, at 3:57 PM, Jack Alan <j.o.alan2...@gmail.com> wrote: > > Hi folks, > > I've a question in my mind I could not find a proper answer for it. Suppose I > have two different systems A and B applied on the same dataset and using > different algorithm. Each system scores a specific F-measure(F1) such as: > System A: 88% F1 > System B: 89.6% F1 > > I want to see if the difference of the F1 between two systems is > statistically significant? How to do so? > > > p.s. I did not apply k-fold evaluation. I just divided the dataset into > training and tests set. > > > Best Regards, > Jack > ------------------------------------------------------------------------------ > One dashboard for servers and applications across Physical-Virtual-Cloud > Widest out-of-the-box monitoring support with 50+ applications > Performance metrics, stats and reports that give you Actionable Insights > Deep dive visibility with transaction tracing using APM Insight. > http://ad.doubleclick.net/ddm/clk/290420510;117567292;y_______________________________________________ > Scikit-learn-general mailing list > Scikit-learn-general@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general ------------------------------------------------------------------------------ One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y _______________________________________________ Scikit-learn-general mailing list Scikit-learn-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
