Hi Tom actually the MegaTrawl latest version is here: https://db.humanconnectome.org/megatrawl/index.html <https://db.humanconnectome.org/megatrawl/index.html>
But yes indeed - early on within the HCP we discussed options for trying to deal with this large-scale-multiple-comparisons-problem, but quickly agreed that it really wasn't our role (and wasn't practical) for us to try to "police" on this issue. Another possible form of "policing" would be to keep-back a truly-left-out sample of subjects, but there were too many ethical, practical and statistical problems with that. We also (light-heartedly!) discussed having some kind of "multiple-comparison sin counter" that ticks up every time someone does a new analysis.... ;-) but as you say there was no straightforward solution presenting itself... Cheers. > On 14 Sep 2016, at 13:52, Thomas Nichols <t.e.nich...@warwick.ac.uk> wrote: > > Dear Don, (Simon?) > > I see two classes of use for the HCP data sets. > > (1) The HCP participant results may be used as norms for comparison with > matched participants from whom we capture measures which may be compared. > > (2) The HCP participant results may be used exclusively. > > > > I think it is only the latter, (2), for which there is a problem although I > certainly could be wrong. > > > I agree, the first doesn't have multiplicity problems (though accuracy with > which you can match subjects & scanner data is another concern). > > > Tom, you used the scenario of a bunch of labs using the data to do one test > each and stated: “…I would say that requires a 'science-wide' correction > applied by the reader of the 250 papers. …” > > That gets at what I’m asking. > > If I’m the author of one of those papers, I don’t want to be fooled or to > fool any of my readers with the results from my laboratory by failing to > correct for all the other comparisons which have been run on the same data. > > > Yes, but the basic problem you, the individual author, face is what sort of > correction should you apply. You only studied variable #132; should you do a > correction just for the 20 others in that domain, or all 250? That's why I > think all you can do is be open, and honestly report the scope of variables > you considered (and if you did, e.g., search over 20 variables in a domain, > correct over those), and report your result. If the reader collects your > result with 50 other papers they can use the appropriate level of criticism > for that collection, which will be different from a reader that collects 250 > papers measures for consideration. > > If I do that now, perhaps it’s workable to take account of all the work > which has appeared to date to do the correction for multiple comparisons. > > But what about a laboratory which runs some other test 5 years from now? > > They must use a more stringent criterion given all the additional results > which have since been published. > > At some point, it will become impossible to find a reliable result. > > > Exactly. > > > Of course, these notions apply to reviewers and other readers too which > places a new level of responsibility on them compared with reading papers > today. > > For editors and reviewers, the problem is particularly acute. > > If the authors of a paper used the correction criterion suggested by their > isolated analysis but a ‘science-wide’ reading calls for a more stringent > criterion, do they bounce the paper back or accept it? > > > > As you point out, Tom, there’s no simple answers to the base question, and > there are lots of scenarios which would be worth understanding in this > context. > > I wonder if there are those lurking on the list who would consider thinking > this through and if they deem it valuable, lay it out formally as a letter or > a paper for all of us. > > Those who are most directly involved with the HCP likely have thought about > it already and perhaps have something. > > > I hope others in the HCP team will chime in, but in our internal discussions > we could never arrive at an conclusive action. That is, the decision was > made early on that this is an *open* project; hypotheses will not be recorded > and registered, and data kept in a lock-box, only made available to those who > agree to study some particular hypothesis (though note, some large scale > projects are run exactly like that). > > Rather, it is left up to authors to honestly report to readers the scope of > the variables considered. Steve Smith's Mega Trawl > <http://humanconnectome.org/about/pressroom/netmats-megatrawl-released/> > openly acknowledges the consideration of nearly every behavioral and > demographic measure in the HCP. See also the OHBM COBIDAS report > <http://www.humanbrainmapping.org/cobidas/>, which implores authors to be > completely transparent in variables and hypotheses considered but not > necessarily highlighted in a publication. > > -Tom > > > > > Best - Don > > > > From: ten.pho...@gmail.com <mailto:ten.pho...@gmail.com> > [mailto:ten.pho...@gmail.com <mailto:ten.pho...@gmail.com>] On Behalf Of > Thomas Nichols > Sent: Tuesday, September 13, 2016 10:53 AM > To: Krieger, Donald N. > Cc: hcp-users@humanconnectome.org <mailto:hcp-users@humanconnectome.org> > Subject: Re: [HCP-Users] Same data / Multiple comparisons ? > > > > Dear Don, > > > > There are no simple answers to this question. Firstly, always be totally > transparent about the set of questions/contrasts you're investigating when > you write up your results. But, when it comes to decide over what set of > results to control multiple testing, I don't think you need to naively > correct for every question in a paper. For example, if you look at sex > differences, and then you look at age effects, I won't correct as there is a > literature on sex differences and a separate one on ageing. But, if there is > a natural set of questions that you are implicit or explicitly looking at > together, then you should correct. For example if you did a ICA dual > regression to get (say) 8 spatial maps of the main RSNs, and then test for > sex differences over those 8 and report all of them, you probalby should do > a correction for those 8 comparisons. > > > > About different labs, if each lab is working independently, they're surely > going to make slightly different choices about the analysis, and then it will > be a confidence building result if they all get the same/similar results. > But, if you're considering the thought experiment where 250 labs each publish > one paper on 1 variable in the 250+ behavioral/demographic meaures in the HCP > data, I would say that requires a 'science-wide' correction applied by the > reader of the 250 papers. > > > > You can use Bonferroni, changing a 0.05 threshold to 0.05/8=0.00625, but > alternatively you can use PALM, which can use a sharper (less conservative) > correction using "Tippets method" to correct for the 8 tests. > > > > Hope this helps. > > > > -Tom > > > > > > On Tue, Sep 13, 2016 at 2:00 PM, Krieger, Donald N. <krieg...@upmc.edu > <mailto:krieg...@upmc.edu>> wrote: > > Dear List, > > > > When a lab analyzes their own data, they control for the degradation in > confidence due to multiple comparisons. > > But how does that work when you have many labs analyzing the same data? > > > > At the one end, several labs could do exactly the same analysis and get the > same results. > > At the other end, several labs could run entirely different tests, each > controlling for the comparisons they do, and reporting their results with the > confidence levels they compute under the assumption that those are the only > tests. > > But since the total number of tests under these circumstances is the sum for > all the labs, isn’t that the number of comparisons for which each lab must > control? > > > > I hope I’ve expressed this clearly enough. > > I admit to being confused by the question. > > What do you think? > > > > Best - Don > > > > _______________________________________________ > HCP-Users mailing list > HCP-Users@humanconnectome.org <mailto:HCP-Users@humanconnectome.org> > http://lists.humanconnectome.org/mailman/listinfo/hcp-users > <http://lists.humanconnectome.org/mailman/listinfo/hcp-users> > > > > > -- > > __________________________________________________________ > Thomas Nichols, PhD > Professor, Head of Neuroimaging Statistics > Department of Statistics & Warwick Manufacturing Group > University of Warwick, Coventry CV4 7AL, United Kingdom > > Web: http://warwick.ac.uk/tenichols <http://warwick.ac.uk/tenichols> > Email: t.e.nich...@warwick.ac.uk <mailto:t.e.nich...@warwick.ac.uk> > Tel, Stats: +44 24761 51086, WMG: +44 24761 50752 > Fx, +44 24 7652 4532 <tel:%2B44%2024%207652%204532> > > > _______________________________________________ > HCP-Users mailing list > HCP-Users@humanconnectome.org <mailto:HCP-Users@humanconnectome.org> > http://lists.humanconnectome.org/mailman/listinfo/hcp-users > <http://lists.humanconnectome.org/mailman/listinfo/hcp-users> > > > -- > __________________________________________________________ > Thomas Nichols, PhD > Professor, Head of Neuroimaging Statistics > Department of Statistics & Warwick Manufacturing Group > University of Warwick, Coventry CV4 7AL, United Kingdom > > Web: http://warwick.ac.uk/tenichols <http://warwick.ac.uk/tenichols> > Email: t.e.nich...@warwick.ac.uk <mailto:t.e.nich...@warwick.ac.uk> > Tel, Stats: +44 24761 51086, WMG: +44 24761 50752 > Fx, +44 24 7652 4532 > > _______________________________________________ > HCP-Users mailing list > HCP-Users@humanconnectome.org > http://lists.humanconnectome.org/mailman/listinfo/hcp-users > --------------------------------------------------------------------------- Stephen M. Smith, Professor of Biomedical Engineering Head of Analysis, Oxford University FMRIB Centre FMRIB, JR Hospital, Headington, Oxford OX3 9DU, UK +44 (0) 1865 222726 (fax 222717) st...@fmrib.ox.ac.uk http://www.fmrib.ox.ac.uk/~steve <http://www.fmrib.ox.ac.uk/~steve> --------------------------------------------------------------------------- Stop the cultural destruction of Tibet <http://smithinks.net/> _______________________________________________ HCP-Users mailing list HCP-Users@humanconnectome.org http://lists.humanconnectome.org/mailman/listinfo/hcp-users
