Dear Denis, It's not multcomp which is too conservative, it is the pairwise t-test which is too liberal. The pairwise t-test doesn't take the random effect of Case into account.
Best regards, ir. Thierry Onkelinx Instituut voor natuur- en bosonderzoek / Research Institute for Nature and Forest team Biometrie & Kwaliteitszorg / team Biometrics & Quality Assurance Kliniekstraat 25 1070 Anderlecht Belgium To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of. ~ Sir Ronald Aylmer Fisher The plural of anecdote is not data. ~ Roger Brinner The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data. ~ John Tukey 2015-06-23 5:17 GMT+02:00 Denis Chabot <[email protected]>: > Hi, > > I am working on a problem which I think can be handled as a repeated measures > analysis, and I have read many tutorials about how to do this with R. This > part goes well, but I get stuck with the multiple comparisons I'd like to run > afterward. I tried two methods that I have seen in my readings, but their > results are quite different and I don't know which one to trust. > > The two approaches are pairwise.t.test() and multcomp, although the latter is > not available after a repeated-measures aov model, but it is after a lme. > > I have a physiological variable measured frequently on each of 67 animals. > These are then summarized with a quantile for each animal. To check the > effect of experiment duration, I recalculated the quantile for each animal 4 > times, using different subset of the data (so the shortest subset is part of > all other subsets, the second subset is included in the 2 others, etc.). I > handle this as 4 repeated (non-independent) measurements for each animal, and > want to see if the average value (for 67 animals) differs for the 4 different > durations. > > Because animals with high values for this physiological trait have larger > differences between the 4 durations than animals with low values, the > observations were log transformed. > > I attach the small data set (Rda format) here, but it can be obtained here if > the attachment gets stripped: > <https://dl.dropboxusercontent.com/u/612902/RepMeasData.Rda> > > The data.frame is simply called Data. > My code is > > load("RepMeasData.Rda") > Data_Long = melt(Data, id="Case") > names(Data_Long) = c("Case","Duration", "SMR") > Data_Long$SMR = log10(Data_Long$SMR) > > # I only show essential code to reproduce my opposing results > mixmod = lme(SMR ~ Duration, data = Data_Long, random = ~ 1 | Case) > anova(mixmod) > posthoc <- glht(mixmod, linfct = mcp(Duration = "Tukey")) > summary(posthoc) > Simultaneous Tests for General Linear Hypotheses > > Multiple Comparisons of Means: Tukey Contrasts > > > Fit: lme.formula(fixed = SMR ~ Duration, data = Data_Long, random = ~1 | > Case) > > Linear Hypotheses: > Estimate Std. Error z value Pr(>|z|) > Set2 - Set1 == 0 -0.006135 0.003375 -1.818 0.265 > Set3 - Set1 == 0 -0.002871 0.003375 -0.851 0.830 > Set4 - Set1 == 0 0.015395 0.003375 4.561 <1e-04 *** > Set3 - Set2 == 0 0.003264 0.003375 0.967 0.768 > Set4 - Set2 == 0 0.021530 0.003375 6.379 <1e-04 *** > Set4 - Set3 == 0 0.018266 0.003375 5.412 <1e-04 *** > --- > Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 > (Adjusted p values reported -- single-step method) > > with(Data_Long, pairwise.t.test(SMR, Duration, p.adjust.method="holm", > paired=T)) > Pairwise comparisons using paired t tests > > data: SMR and Duration > > Set1 Set2 Set3 > Set2 < 2e-16 - - > Set3 0.11118 0.10648 - > Set4 0.00475 7.9e-05 0.00034 > > P value adjustment method: holm > > So the difference between sets 1 and 2 goes from non significant to very > significant, depending on method. > > I have other examples with essentially the same type of data and sometimes > the two approches differ in the opposing way. In the example shown here, > multcomp was more conservative, in some others it yielded a larger number of > significant differences. > > I admit not mastering all the intricacies of multcomp, but I have used > multcomp and other methods of doing multiple comparisons many times before > (but never with a repeated measures design), and always found the results > very similar. When there were small differences, I trusted multcomp. This > time, I get rather large differences and I am worried that I am doing > something wrong. > > Thanks in advance, > > Denis Chabot > Fisheries & Oceans Canada > > sessionInfo() > R version 3.2.0 (2015-04-16) > Platform: x86_64-apple-darwin13.4.0 (64-bit) > Running under: OS X 10.10.3 (Yosemite) > > locale: > [1] fr_CA.UTF-8/fr_CA.UTF-8/fr_CA.UTF-8/C/fr_CA.UTF-8/fr_CA.UTF-8 > > attached base packages: > [1] stats graphics grDevices utils datasets methods base > > other attached packages: > [1] multcomp_1.4-0 TH.data_1.0-6 survival_2.38-1 mvtnorm_1.0-2 > nlme_3.1-120 car_2.0-25 reshape2_1.4.1 > > loaded via a namespace (and not attached): > [1] Rcpp_0.11.5 magrittr_1.5 splines_3.2.0 MASS_7.3-40 > lattice_0.20-31 minqa_1.2.4 stringr_1.0.0 > [8] plyr_1.8.2 tools_3.2.0 nnet_7.3-9 pbkrtest_0.4-2 > parallel_3.2.0 grid_3.2.0 mgcv_1.8-6 > [15] quantreg_5.11 lme4_1.1-7 Matrix_1.2-0 nloptr_1.0.4 > codetools_0.2-11 sandwich_2.3-3 stringi_0.4-1 > [22] SparseM_1.6 zoo_1.7-12 > ______________________________________________ > [email protected] mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. ______________________________________________ [email protected] mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
