Katrien, My quick and dirty simulation used 3 modules of 3, then 10, then 20, then 30, and then 40 landmarks. (I also varied sample size but this did not seem to matter.) I saw results that varied as much as yours around 10 landmarks that were less distressing around 30 landmarks.
The modularity results are the same (I think) because columns of data are shuffled in the permutation procedure. For an integration test, the rows of one module are shuffled. When the order of the modules is changed, the one that has rows shuffled changes. This is what leads to different outcomes, simply by switching the order. My simulated data were just noise so I would have to think about actual integrated effects. For example, if modules 1 and 2 are integrated, but 3, 4, 5 are more or less “independent”, a different outcome — including conclusion about the hypothesis test — might be influenced by the order of modules and which one has values shuffled. I’ll confer with others on the geomorph team and maybe we will have to reconsider how the permutation procedure is implemented, especially if r-pls is averaged across r-pls pairs. Performing analyses by pairs and using compare.pls is different in philosophy but will not be impacted from any permutational considerations like when there are 3+ modules, as randomizing rows of one matrix for a pair of matrices is sufficient for creating random pairs of vectors in each permutation. I might have more to say sometime soon. Mike > On Oct 19, 2020, at 2:52 PM, Katrien Janin <[email protected]> wrote: > > Hello Mike, > > Thank you for your helpful reply. Interesting as indeed the landmark density > within the blocks of my posted test examples are not huge: ranging from 29 to > 13. I had kept these low being cautious about obtaining inflated r-pls values > due to sample size/number of landmarks. Out of curiosity, how few where the > few in your test when you saw proper numeric divergence in the effect values? > > Maybe helpful to know: I have run the same test with the modularity.test > function, and both sets obtained the exact same results: CR value of 0.7357 > (p=0.001), with effect size of -8.4022. So it does seem to be, or at least in > my case, confined to multi-block PLS effects. > > With best wishes and many thanks, > K. > > >> On 19 Oct 2020, at 17:57, Mike Collyer <[email protected] >> <mailto:[email protected]>> wrote: >> >> Katrien, >> >> Changing the order of factor levels can certainly change Z-scores and >> P-values, but I was surprised that it changed as much as you observed. The >> reason it is not constant is because the permutation schedule (random >> assignment of rows of values) will change with rearrangement of the module >> blocks. However, I would expect the differences to be subtle. >> >> I was able to recreate something like your scenario (using integration.test) >> with simulated data but found it only occurred if I had few landmarks per >> module, suggesting this could be a landmark density issue. Does that seem >> consistent with your data (are there only a few landmarks in each of the >> five modules)? When I increased the number of landmarks, I got what I >> expected: different but qualitatively similar Z-scores and P-values. >> >> Regarding the comapre.pls approach, the pooled standard error is calculated >> differently, so a >> change in this case should be expected. >> >> Hope that is helpful. >> >> Mike >> >> >>> On Oct 19, 2020, at 9:55 AM, Katrien Janin <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> Hey all, >>> >>> I hope somebody can help me out to make sense of the following: >>> Example #1 and #2 draw upon the same data set and same partition, just that >>> the blocks are in reverse order. >>> >>> #1 > L.sym.nh.int5.pairwise >>> A B C D E >>> A 0.000 0.782 0.688 0.756 0.791 >>> B 0.782 0.000 0.750 0.879 0.679 >>> C 0.688 0.750 0.000 0.723 0.815 >>> D 0.756 0.879 0.723 0.000 0.632 >>> E 0.791 0.679 0.815 0.632 0.000 >>> >>> #2 > L.sym.nh.int5.pairwise >>> E D C B A >>> E 0.000 0.632 0.815 0.679 0.791 >>> D 0.632 0.000 0.723 0.879 0.756 >>> C 0.815 0.723 0.000 0.750 0.688 >>> B 0.679 0.879 0.750 0.000 0.782 >>> A 0.791 0.756 0.688 0.782 0.000 >>> >>> both obtain an overall r-PLS: 0.750 (p= 0.001), and as you can see in the >>> tables the individual pairs obtain the exact same r-pls (as expected). >>> >>> the odd thing is that their effect size differ: example #1 returns an >>> effect size of 2.6798 whilst #2 has a effect size of 4.640. >>> >>> any ideas what may be causing this? what can I do to find out what creates >>> this difference in effect size, and which one I can report along the r-pls >>> values? >>> >>> >>> K. >>> >>> PS. I also computed the pls and effect for each individual pair with >>> two.block: I yet again obtain the same r-pls values as in tables above (as >>> expected) and when I average the effects of the pairs as obtained with the >>> compare function I arrive at 5.626 ... >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Morphmet" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected] >>> <mailto:[email protected]>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/morphmet2/4bb1e5d3-541e-4ce6-a9b0-aefd357d4a7fn%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/morphmet2/4bb1e5d3-541e-4ce6-a9b0-aefd357d4a7fn%40googlegroups.com?utm_medium=email&utm_source=footer>. >> > -- You received this message because you are subscribed to the Google Groups "Morphmet" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/morphmet2/69CB0825-9D46-491B-8640-9B9E4180495E%40gmail.com.
