Yours is (primarily) a statistical question, not a question about R, and so off topic here. Post on a statistics list, like stats.stackexchange.com instead. Better yet, consult a local statistician. This is a thorny and difficult matter and, as you have already discovered, is "full of sound and fury, signifying nothing." (or, at least, what's being signaled depends exactly on what question is being asked, which is why this is a mess).
Cheers, Bert Bert Gunter "Data is not information. Information is not knowledge. And knowledge is certainly not wisdom." -- Clifford Stoll On Mon, Jun 22, 2015 at 8:17 PM, Denis Chabot <[email protected]> wrote: > Hi, > > I am working on a problem which I think can be handled as a repeated measures > analysis, and I have read many tutorials about how to do this with R. This > part goes well, but I get stuck with the multiple comparisons I'd like to run > afterward. I tried two methods that I have seen in my readings, but their > results are quite different and I don't know which one to trust. > > The two approaches are pairwise.t.test() and multcomp, although the latter is > not available after a repeated-measures aov model, but it is after a lme. > > I have a physiological variable measured frequently on each of 67 animals. > These are then summarized with a quantile for each animal. To check the > effect of experiment duration, I recalculated the quantile for each animal 4 > times, using different subset of the data (so the shortest subset is part of > all other subsets, the second subset is included in the 2 others, etc.). I > handle this as 4 repeated (non-independent) measurements for each animal, and > want to see if the average value (for 67 animals) differs for the 4 different > durations. > > Because animals with high values for this physiological trait have larger > differences between the 4 durations than animals with low values, the > observations were log transformed. > > I attach the small data set (Rda format) here, but it can be obtained here if > the attachment gets stripped: > <https://dl.dropboxusercontent.com/u/612902/RepMeasData.Rda> > > The data.frame is simply called Data. > My code is > > load("RepMeasData.Rda") > Data_Long = melt(Data, id="Case") > names(Data_Long) = c("Case","Duration", "SMR") > Data_Long$SMR = log10(Data_Long$SMR) > > # I only show essential code to reproduce my opposing results > mixmod = lme(SMR ~ Duration, data = Data_Long, random = ~ 1 | Case) > anova(mixmod) > posthoc <- glht(mixmod, linfct = mcp(Duration = "Tukey")) > summary(posthoc) > Simultaneous Tests for General Linear Hypotheses > > Multiple Comparisons of Means: Tukey Contrasts > > > Fit: lme.formula(fixed = SMR ~ Duration, data = Data_Long, random = ~1 | > Case) > > Linear Hypotheses: > Estimate Std. Error z value Pr(>|z|) > Set2 - Set1 == 0 -0.006135 0.003375 -1.818 0.265 > Set3 - Set1 == 0 -0.002871 0.003375 -0.851 0.830 > Set4 - Set1 == 0 0.015395 0.003375 4.561 <1e-04 *** > Set3 - Set2 == 0 0.003264 0.003375 0.967 0.768 > Set4 - Set2 == 0 0.021530 0.003375 6.379 <1e-04 *** > Set4 - Set3 == 0 0.018266 0.003375 5.412 <1e-04 *** > --- > Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 > (Adjusted p values reported -- single-step method) > > with(Data_Long, pairwise.t.test(SMR, Duration, p.adjust.method="holm", > paired=T)) > Pairwise comparisons using paired t tests > > data: SMR and Duration > > Set1 Set2 Set3 > Set2 < 2e-16 - - > Set3 0.11118 0.10648 - > Set4 0.00475 7.9e-05 0.00034 > > P value adjustment method: holm > > So the difference between sets 1 and 2 goes from non significant to very > significant, depending on method. > > I have other examples with essentially the same type of data and sometimes > the two approches differ in the opposing way. In the example shown here, > multcomp was more conservative, in some others it yielded a larger number of > significant differences. > > I admit not mastering all the intricacies of multcomp, but I have used > multcomp and other methods of doing multiple comparisons many times before > (but never with a repeated measures design), and always found the results > very similar. When there were small differences, I trusted multcomp. This > time, I get rather large differences and I am worried that I am doing > something wrong. > > Thanks in advance, > > Denis Chabot > Fisheries & Oceans Canada > > sessionInfo() > R version 3.2.0 (2015-04-16) > Platform: x86_64-apple-darwin13.4.0 (64-bit) > Running under: OS X 10.10.3 (Yosemite) > > locale: > [1] fr_CA.UTF-8/fr_CA.UTF-8/fr_CA.UTF-8/C/fr_CA.UTF-8/fr_CA.UTF-8 > > attached base packages: > [1] stats graphics grDevices utils datasets methods base > > other attached packages: > [1] multcomp_1.4-0 TH.data_1.0-6 survival_2.38-1 mvtnorm_1.0-2 > nlme_3.1-120 car_2.0-25 reshape2_1.4.1 > > loaded via a namespace (and not attached): > [1] Rcpp_0.11.5 magrittr_1.5 splines_3.2.0 MASS_7.3-40 > lattice_0.20-31 minqa_1.2.4 stringr_1.0.0 > [8] plyr_1.8.2 tools_3.2.0 nnet_7.3-9 pbkrtest_0.4-2 > parallel_3.2.0 grid_3.2.0 mgcv_1.8-6 > [15] quantreg_5.11 lme4_1.1-7 Matrix_1.2-0 nloptr_1.0.4 > codetools_0.2-11 sandwich_2.3-3 stringi_0.4-1 > [22] SparseM_1.6 zoo_1.7-12 > ______________________________________________ > [email protected] mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. ______________________________________________ [email protected] mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
