On Sat, 2010-02-06 at 21:44 -0800, Li An wrote: > Dear Ecologers, > > In testing ecological models, we often use t-test as a way to compare > our model results with observed data. If they are close enough, we > obtain more confidence about our model. However, in most traditional > situations, we put "no difference" as the null and regarded it as the > default. This means that unless we find substantial evidence, we would > retain the null hypothesis. For instance, we can use this type of test > to examine if a drug has a noticeable effect. > > In our model performance situation (testing observed data = predicted > numbers from a model, assuming data independence), I argue that we > should keep the alternative hypothesis as the default, making every > effort to find substantial evidence to support the null hypothesis (if > unable, we retain the alternative hypothesis related to inequality > between the model predictions and the data). In this case, we can still > use the traditional test statistic such as z or p values, but interpret > the results differently. Rather than using the criterion of p > 0.05 (or > Z<1.96 or t < a big number) to retain the null hypothesis, we should use > a more strict standard--e.g., p > a much larger number (e.g., 0.9) or z > < a much smaller number (e.g.,0.125), to retain the null hypothesis > about equality between the model predictions and the data. This seems > mofrea philosophical issue. Does this make sense? > > Li
You might like to look at the field of equivalency testing. Some references cited in the 'equivalence' package by Andrew Robinson for R are: Robinson, A.P., and R.E. Froese. 2004. Model validation using equivalence tests. Ecological Modelling 176, 349–358. Wellek, S. 2003. Testing statistical hypotheses of equivalence. Chapman and Hall/CRC. 284 pp. Westlake, W.J. 1981. Response to T.B.L. Kirkwood: bioequivalence testing - a need to rethink. Biometrics 37, 589-594. HTH G -- %~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~% Dr. Gavin Simpson [t] +44 (0)20 7679 0522 ECRC, UCL Geography, [f] +44 (0)20 7679 0565 Pearson Building, [e] gavin.simpsonATNOSPAMucl.ac.uk Gower Street, London [w] http://www.ucl.ac.uk/~ucfagls/ UK. WC1E 6BT. [w] http://www.freshwaters.org.uk %~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%
