Dear Vera, Actually, summary.pairwise provides an option for adjusting the confidence level, if one wishes to use a different pairwise alpha than the experiment-wise alpha. For example, one can use confidence = 0.99, if the pairwise alpha is deemed to be 0.01. The upper confidence limits will adjust accordingly.
Regarding whether one *should*, we leave the opinion to the user rather than impose it, but there is one thing I wish to point out, with respect to permutational software. Some software will subset data to include only observations associated with two groups and perform multiple two-sample tests, perhaps with different random permutations in each test. RRPP does not do that. It fits, say 10,000, linear models with10,000 sets of coefficients (rather than 10,000 * g * (g - 1) / 2 sets of coefficients that would be calculated for the all pairwise two-sample tests of g groups). From the 10,000 sets of coefficients, 10,000 matrices of pairwise distances among group means can be estimated. So, for example, the 1,427th permutation of the pairwise distance between the 3rd and 4th group means uses the exact same randomization of residuals as the 1,427th permutation of the pairwise distance between the 1st and 10th means (because both are estimated form the 1,427th estimation of coefficients). If there were some borderline cases that I worry could by type I errors, I might consider the following, in addition to pairwise alpha: 1. The seed used (try the test changing the seed a few times to see how stable the P-values are). 2. The number of permutations (more is better). 3. Sample size (I might be more worried about a type I error with smaller samples). 4. Most importantly, effect sizes. (If a pairwise distance is “significant” but the effect size is small, does its significance mean as much, biologically?) As a final comment, the P-values in RRPP::pairwise are limited by the number of permutations. If 1,000 permutations are used, 0.001 is a lower limit. Therefore, a pairwise alpha that is really small because there are many groups to compare might also require many permutations to get P-values that are closer to exact values. Failing to reject the null hypothesis that a pairwise distance equals 0 because a P-value is 0.001 would be perhaps unfortunate and using 500,000 random permutations perhaps excessive, where common sense (evaluation of effect sizes) would probably be more useful. Hope that helps! Mike > On May 25, 2026, at 9:23 AM, Ian Dworkin <[email protected]> wrote: > > HI Vera, > > pairwise() in geomorph/RRPP does not do any adjustments. The last I spoke to > Mike and Dean about (when I prepped an online seminar about it during the > pandemic... so a while ago), it was in part because it provides ALL pairwise > comparisons. One of the things I point out to people using it, is that you > need to a priori plan which you are going to use, and do your adjustments > based upon that. > > Cheers > Ian > > On Mon, 25 May 2026 at 06:33, [email protected] > <mailto:[email protected]> <[email protected] > <mailto:[email protected]>> wrote: >> Hi everyone, >> >> I was recently asked if I implement any multiple comparison adjustments >> (like Holm) when using the pairwise() function in geomorph on RRPP-based >> procD.lm/procD.pgls fits involving a factor with more than two levels. I >> admit that this never crossed my mind, but I can’t see why there should not >> be a higher risk of getting false H0 rejections with increasing comparisons, >> as it would in other multiple comparison contexts. >> >> It is easy to do an adjustment manually, but I do not think I have seen it >> done anywhere – it is not part of the pairwise implementation as far as I >> can see, and I did not spot a comment on it in the Collyer & Adams 2018 >> paper on RRPP. I was wondering if there is a reason for this, perhaps >> related to the residual randomisation procedure or building of confidence >> intervals? >> >> Thanks a lot for any suggestions! >> >> Vera >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Morphmet" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] >> <mailto:[email protected]>. >> To view this discussion visit >> https://groups.google.com/d/msgid/morphmet2/cdfd532c-084e-4dbd-bc7a-190f6808ed2cn%40googlegroups.com >> >> <https://groups.google.com/d/msgid/morphmet2/cdfd532c-084e-4dbd-bc7a-190f6808ed2cn%40googlegroups.com?utm_medium=email&utm_source=footer>. > > > > -- > Ian Dworkin > Department of Biology > McMaster University > Office phone 905 525 9140 ext. 21775 > Lab phone 905 525 9140 ext. 20076 > [email protected] <mailto:[email protected]> > dworkinlab.github.io <http://dworkinlab.github.io/> > > -- > You received this message because you are subscribed to the Google Groups > "Morphmet" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion visit > https://groups.google.com/d/msgid/morphmet2/CAGudrj%3DZLnucD8G3k5qDFXzKjWBZrdWg_bX5z9Q90Pt90-hNCQ%40mail.gmail.com > > <https://groups.google.com/d/msgid/morphmet2/CAGudrj%3DZLnucD8G3k5qDFXzKjWBZrdWg_bX5z9Q90Pt90-hNCQ%40mail.gmail.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "Morphmet" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/morphmet2/4E1E1FC1-6AE3-4891-B870-CBE4B5C832A0%40gmail.com.
