-------- Original Message -------- Subject: Re: Tribulations with permutations Date: Tue, 12 Jul 2011 17:29:55 -0400 From: Dean Adams <[email protected]> To: [email protected] Hans, How to best implement a resampling procedure depends very heavily on the question one wishes to ask. The scenario you proposed was one of testing 2 groups for shape differences. Using parametric statistical procedures, this is often examined using a multivariate t-test (Hotelling's T), which is simply the squared Euclidean distance between group means standardized by pooled-within group covariance (and some sample size adjustments). For testing such hypotheses, I prefer using permutation (resampling without replacement), rather than bootstrap. As such I would compare the observed Euclidean distance between groups to a distribution of values obtained from data sets where specimens are re-assigned to groups via permutation. For 3 or more groups, permutational MANOVA follows this same general logic. Note that here no additional GPA is necessary for each permuted data set. One reason is that all specimens are found in each iteration of a permutation, and thus the each resampled dataset is already optimally aligned to the same reference. An alternative that appears logical, but that I would not recommend, is to perform the permutation test on Hotelling's T-square or Mahalanobis distance (which can be mathematically derived from T-square). The reason I would not recommend this is that in each iteration of this procedure, both the Euclidean distance between groups and the pooled-within group covariance matrix are altered in each iteration. Thus, if a significant result is obtained, one does not know whether the groups are farther apart than expected, or if the within-group covariance is smaller than expected. Better to just test the 'numerator' of the T-square, which is simply the Euclidean distance between group means. There are quite a few morphometric-oriented papers where such permutation procedures are described; and even more where they are implemented. A few of my own describing resampling procedures for testing various hypotheses include: Adams (2004: Ecology), Collyer and Adams (2007: Ecology), Adams and Collyer (2009: Evolution), Adams and Nistri (2010: BMC Evol. Biol.). These can be found on my web page. Note that there are MANY other MorphMet participants who have proposed and implemented various permutation methods for testing patterns and hypotheses of shape variation. I'm certain others will chime in with additional thoughts and references, and probably alternative perspectives. Hope this helps and all the best, Dean -- Dr. Dean C. Adams Associate Professor Department of Ecology, Evolution, and Organismal Biology Department of Statistics Iowa State University Ames, Iowa 50011 www.public.iastate.edu/~dcadams/ phone: 515-294-3834 On 7/12/2011 10:50 AM, morphmet wrote:
-------- Original Message -------- Subject: Tribulations with permutations Date: Mon, 11 Jul 2011 17:07:54 -0400 From: Hans Wellens<[email protected]> To: <[email protected]> CC: <[email protected]> Dear All, Various packages are available to perform permutation tests for shape differences between groups: for instance the IMP package (Twogroup), or the Shapes package (resampletest) (amongst others). The methodologies employed by the various available programs seem to differ somewhat. To gain some additional insight, I decided to program a permutation test in R. The aim of this permutation test would be to determine whether there is a difference in craniofacial shape between a male and female patient sample. A few different scenarios could be envisioned. My hope is that the formidable combined intelligence which is the morphmet list would be so kind to enlighten me on the correct path to choose… 1.Perform a pooled generalized Procrustes analysis, taking care of location, scale and rotation. 2.Calculate the male and female mean configurations 3.Then two options seem to be available: a.Calculate the Procrustes distance (PD) between the male and female mean configurations. b.Calculate the squared PD between the male configurations and male mean configuration and between the female configurations and female mean configuration. The first would directly generate the test statistic for comparison to the permutation results, while the latter would need the calculation of the F-ratio to compare the variances, in order to do the same. 4.Reshuffle the groups, generating pairs of groups of the same size as the original ones. Let`s say one thousand groups are generated, and let`s suppose a bootstrap is performed (permutation with replacement). Again 2 scenarios present themselves: a.Some people propose resampling from the pooled sample (males and females combined). b.In the Shapes package, resampling with replacement is performed WITHIN groups. Which one is preferable when resampling with replacement is performed?. Also, is a renewed Procrustes superimposition required of each permutation group? A very bright colleague of mine is probably right when pointing out this would not be necessary since scale and location and have been dealt with in the original Procrustes analysis. So this would mean that the Procrustes coordinates are reshuffled, and not the original ones. 5.Recalculate the male and female mean configurations, and the test statistic from point 3 for each permutation group. 6.Compare the resulting values to the original one. The number of values equaling or exceeding the original one serves as the p-value under the null hypothesis. Additional questions: should tangent space projection be considered in the permutation test? Would there be additional value in performing this test in form space? Best regards, Hans Wellens Orthodontist Groene-Poortdreef 16 8200 Sint-Michiels, Brugge [email protected]
