Re: [HCP-Users] netmats prediction of fluid intelligence
Germane to this discussion is that using the same methodology, but a different sample of subjects, the same Yale group has recently reported that the correlation of predicted gF (from netmats) and observed gF was r=0.22. https://www.ncbi.nlm.nih.gov/pubmed/28968754 cheers, -MH -- Michael Harms, Ph.D. --- Conte Center for the Neuroscience of Mental Disorders Washington University School of Medicine Department of Psychiatry, Box 8134 660 South Euclid Ave.Tel: 314-747-6173 St. Louis, MO 63110Email: mha...@wustl.edu On 10/7/17, 2:43 PM, "hcp-users-boun...@humanconnectome.org on behalf of Nina de Lacy"wrote: This is a very interesting thread and discussion and many of the observations conform with ongoing work I'm doing in children/adolescents which generally suggests that predicting intelligence measures is very challenging using connectivity measures, after including confounders within multivariate frameworks. I personally wonder not only about confounding effects, but also the difficulty of working with neuropsychological 'intelligence' measures designed for other purposes than perhaps some of what we are trying to get at. As well, I would raise the question of our samples, which most/much of the time in neuroimaging rarely include individuals with lower IQs, therefore distorting the distribution. All that said, what I really joined in for was to ask Julien if he could comment more on what he meant by highlighting that part of the effect obtained in the FInn study was due to the "specific subject sample" used. Was this due to certain characteristics of the smaller subject sample? I of course respect this may be content germane to an as yet unpublished paper you may not want to share in detail :) Nina On Sat, 7 Oct 2017, Julien Dubois wrote: > Julien, when you say the method still has predictive value in the large > sample 'without confounds', do you mean without removing confounds or after > deconfounding? It's also not > clear to me whether the scores the Ma study reported are deconfounded > or not, but I guess they are not. If one is interested in the added value of > fMRI predicting cognition (my > case), it makes sense to be conservative, so I would be interested in > knowing whether there's something left in the deconfounded space. > > > Sorry, my phrasing wasn't clear. I mean that I obtain similar results to the > Megatrawl and to the Ma poster, WITHOUT deconfounding as performed in the > Megatrawl. I will let you know how it > looks once I use the same deconfounding as in the Megatrawl, i.e.: > "Prediction takes place after removing sex, age, age^2 , sex*age, sex*age^2 , > brain & head size (as estimated by > FreeSurfer), overall head motion (a summation over all timepoints of > timepoint-to-timepoint relative head motion) and acquisition date as > confounds (the last of these is actually the > “acquisition quarter”, which is useful to include because there was a slight > change in rfMRI reconstruction code during the third acquisition > year-quarter; in future we will instead use > the actual reconstruction code version as the confound)." > - Julien > > ___HCP-Users mailing > listHCP-Users@humanconnectome.orghttp://lists.humanconnectome.org/mailman/listinfo/hcp-users > > > This message and any attached files might contain confidential information protected by federal and state law. The information is intended only for the use of the individual(s) or entities originally named as addressees. The improper disclosure of such information may be subject to civil or criminal penalties. If this message reached you in error, please contact the sender and destroy this message. Disclosing, copying, forwarding, or distributing the information by unauthorized individuals or entities is strictly prohibited by law. ___ HCP-Users mailing list HCP-Users@humanconnectome.org http://lists.humanconnectome.org/mailman/listinfo/hcp-users The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. ___ HCP-Users mailing list HCP-Users@humanconnectome.org http://lists.humanconnectome.org/mailman/listinfo/hcp-users
Re: [HCP-Users] netmats prediction of fluid intelligence
> > Julien, when you say the method still has predictive value in the large > sample 'without confounds', do you mean without removing confounds or after > deconfounding? It's also not clear to me whether the scores the Ma study > reported are deconfounded or not, but I guess they are not. If one is > interested in the added value of fMRI predicting cognition (my case), it > makes sense to be conservative, so I would be interested in knowing whether > there's something left in the deconfounded space. > Sorry, my phrasing wasn't clear. I mean that I obtain similar results to the Megatrawl and to the Ma poster, WITHOUT deconfounding as performed in the Megatrawl. I will let you know how it looks once I use the same deconfounding as in the Megatrawl, i.e.: "Prediction takes place after removing sex, age, age^2 , sex*age, sex*age^2 , brain & head size (as estimated by FreeSurfer), overall head motion (a summation over all timepoints of timepoint-to-timepoint relative head motion) and acquisition date as confounds (the last of these is actually the “acquisition quarter”, which is useful to include because there was a slight change in rfMRI reconstruction code during the third acquisition year-quarter; in future we will instead use the actual reconstruction code version as the confound)." - Julien ___ HCP-Users mailing list HCP-Users@humanconnectome.org http://lists.humanconnectome.org/mailman/listinfo/hcp-users
Re: [HCP-Users] netmats prediction of fluid intelligence
Hi all, Thanks for your comments. Reassuring although somewhat disappointing at the same time. As as far as I know there's no external validation of their fluid intelligence model. They replicated the prediction for their sustained attention model (Rosenberg 2016), but there may be quite different processes at play there. I think it's a great study in any case. Julien, when you say the method still has predictive value in the large sample 'without confounds', do you mean without removing confounds or after deconfounding? It's also not clear to me whether the scores the Ma study reported are deconfounded or not, but I guess they are not. If one is interested in the added value of fMRI predicting cognition (my case), it makes sense to be conservative, so I would be interested in knowing whether there's something left in the deconfounded space. Best regards, Benjamín Garzón, PhD Department of Neurobiology, Care Sciences and Society Aging Research Center | 113 30 Stockholm | Gävlegatan 16 benjamin.gar...@ki.se<mailto:benjamin.gar...@ki.se> | www.ki-su-arc.se<https://email.ki.se/owa/redir.aspx?C=LDNa9T7Nak68Br6ZyIC_J4KUwCiWMdEIQwVElfLYlCPLbdpUruOe0XhySwY-dNAYT9JyRT4AtFo.=http%3a%2f%2fwww.ki-su-arc.se%2f> __ Karolinska Institutet – a medical university From: hcp-users-boun...@humanconnectome.org [hcp-users-boun...@humanconnectome.org] on behalf of Julien Dubois [jcrdub...@gmail.com] Sent: Friday, October 06, 2017 5:33 PM To: hcp-users@humanconnectome.org Subject: Re: [HCP-Users] netmats prediction of fluid intelligence Hi all to chime in: we have done extensive work over the past year to replicate and understand the prediction of IQ obtained in the Finn study. Our manuscript is about to be submitted. Take away points: -- the high effect size they find is partly due to small sample size (118 subjects) and to the specific subject sample -- their method still has predictive value in the larger sample of subjects, though the effect size is much reduced (similar to Megatrawl without confounds) -- the specifics of preprocessing/denoising and predictive model don't have a huge effect on the final result (when enough subjects are included) See also the work presented by Feilong Ma at OHBM this year, which took great care in aligning subjects (MSMall + whole-brain hyperalignment) in a much larger sample than the Finn study: https://files.aievolution.com/hbm1701/abstracts/37710/3928_Ma.pdf Happy to discuss further if someone is interested. - Julien Postdoc | Cedars-Sinai Medical Center // Caltech | +1 (310)423-8377 | nigiri.caltech.edu/~jdubois<http://nigiri.caltech.edu/~jdubois> ___ HCP-Users mailing list HCP-Users@humanconnectome.org http://lists.humanconnectome.org/mailman/listinfo/hcp-users ___ HCP-Users mailing list HCP-Users@humanconnectome.org http://lists.humanconnectome.org/mailman/listinfo/hcp-users
Re: [HCP-Users] netmats prediction of fluid intelligence
Hi all to chime in: we have done extensive work over the past year to replicate and understand the prediction of IQ obtained in the Finn study. Our manuscript is about to be submitted. Take away points: -- the high effect size they find is partly due to small sample size (118 subjects) and to the specific subject sample -- their method still has predictive value in the larger sample of subjects, though the effect size is much reduced (similar to Megatrawl without confounds) -- the specifics of preprocessing/denoising and predictive model don't have a huge effect on the final result (when enough subjects are included) See also the work presented by Feilong Ma at OHBM this year, which took great care in aligning subjects (MSMall + whole-brain hyperalignment) in a much larger sample than the Finn study: https://files.aievolution.com/hbm1701/abstracts/37710/3928_Ma.pdf Happy to discuss further if someone is interested. - Julien *Postdoc | *Cedars-Sinai Medical Center // Caltech | +1 (310)423-8377 | nigiri.caltech.edu/~jdubois ___ HCP-Users mailing list HCP-Users@humanconnectome.org http://lists.humanconnectome.org/mailman/listinfo/hcp-users
Re: [HCP-Users] netmats prediction of fluid intelligence
Hi all Yes - I've discussed this with Todd and it's not immediately clear whether the difference is due to: - they used full correlation not partial - they used fewer confound regressors (IIRC) - their prediction method is *very* different (pooling across all relevant features rather than keeping them separate in the multivariate elastic net regression prediction). Or some combination of all of this. I don't have a strong gut feeling which of these might be the biggest factor, but we should note that the Finn paper took a lot more care over many aspects of their analysis than many studies do, and in particular it was impressive how they got replication of the prediction between completelty separate studies. But yes I would be interested to see this resolved more. With respect to our CCA-based population mode, which covaried more highly with the intelligence measure as you mentioned - I think maybe this points at the main issue possibly being the noisiness of the individual features (netmat edges) and also of the intelligence feature (when all combined together within the elastic net prediction framework). Cheers. > On 6 Oct 2017, at 04:11, Harms, Michael <mha...@wustl.edu> wrote: > > > In the context of the long resting state runs that we have available, I would > argue that throwing in additional possible confounds is the appropriate thing > to do. Are you suggesting that sex, age, age^2, sex*age, sex*age^2, brain > size, head size, and average motion shouldn’t all be included? > > Regardless, r = 0.21 (without confounds in the MegaTrawl) is a long way from > the r = 0.5 prediction in Finn et al. > > Cheers, > -MH > > -- > Michael Harms, Ph.D. > --- > Conte Center for the Neuroscience of Mental Disorders > Washington University School of Medicine > Department of Psychiatry, Box 8134 > 660 South Euclid Ave.Tel: 314-747-6173 > St. Louis, MO 63110 Email: > mha...@wustl.edu <mailto:mha...@wustl.edu> > > From: <hcp-users-boun...@humanconnectome.org > <mailto:hcp-users-boun...@humanconnectome.org>> on behalf of Thomas Yeo > <ytho...@csail.mit.edu <mailto:ytho...@csail.mit.edu>> > Date: Thursday, October 5, 2017 at 10:01 PM > To: "Glasser, Matthew" <glass...@wustl.edu <mailto:glass...@wustl.edu>> > Cc: "hcp-users@humanconnectome.org <mailto:hcp-users@humanconnectome.org>" > <hcp-users@humanconnectome.org <mailto:hcp-users@humanconnectome.org>> > Subject: Re: [HCP-Users] netmats prediction of fluid intelligence > > Certainly one difference is that HCP (i.e., Steve) tends to take the more > conservative approach of regressing a *lot* of potential confounds, which > tends to result in a lower prediction values. You can see that without > confound regression, Steve's prediction is 0.21 versus 0.06. > > Regards, > Thomas > > On Fri, Oct 6, 2017 at 1:44 AM, Glasser, Matthew <glass...@wustl.edu > <mailto:glass...@wustl.edu>> wrote: >> Perhaps there is an issue related to data clean up or alignment of brain >> areas across subjects. The Finn study does not appear to have followed the >> recommended approach to either. >> >> Peace, >> >> Matt. >> >> From: <hcp-users-boun...@humanconnectome.org >> <mailto:hcp-users-boun...@humanconnectome.org>> on behalf of Benjamin Garzon >> <benjamin.gar...@ki.se <mailto:benjamin.gar...@ki.se>> >> Date: Thursday, October 5, 2017 at 1:39 PM >> To: "hcp-users@humanconnectome.org <mailto:hcp-users@humanconnectome.org>" >> <hcp-users@humanconnectome.org <mailto:hcp-users@humanconnectome.org>> >> Subject: [HCP-Users] netmats prediction of fluid intelligence >> >> Dear HCP experts, >> >> I'm trying to reconcile the MegaTrawl prediction of fluid intelligence >> (PMAT24_A_CR) >> >> https://db.humanconnectome.org/megatrawl/3T_HCP820_MSMAll_d200_ts2/megatrawl_1/sm203/index.html >> >> <https://db.humanconnectome.org/megatrawl/3T_HCP820_MSMAll_d200_ts2/megatrawl_1/sm203/index.html> >> >> (which shows r = 0.06 between predicted and measured scores) >> >> with the Finn 2015 study >> >> https://www.nature.com/neuro/journal/v18/n11/full/nn.4135.html >> <https://www.nature.com/neuro/journal/v18/n11/full/nn.4135.html> >> >> claiming an r = 0.5 correlation between predicted and measured scores. In >> the article they used a subset of the HCP data (126 subjects), but the >> measure of fluid intelligence
Re: [HCP-Users] netmats prediction of fluid intelligence
In the context of the long resting state runs that we have available, I would argue that throwing in additional possible confounds is the appropriate thing to do. Are you suggesting that sex, age, age^2, sex*age, sex*age^2, brain size, head size, and average motion shouldn’t all be included? Regardless, r = 0.21 (without confounds in the MegaTrawl) is a long way from the r = 0.5 prediction in Finn et al. Cheers, -MH -- Michael Harms, Ph.D. --- Conte Center for the Neuroscience of Mental Disorders Washington University School of Medicine Department of Psychiatry, Box 8134 660 South Euclid Ave.Tel: 314-747-6173 St. Louis, MO 63110 Email: mha...@wustl.edu From: <hcp-users-boun...@humanconnectome.org> on behalf of Thomas Yeo <ytho...@csail.mit.edu> Date: Thursday, October 5, 2017 at 10:01 PM To: "Glasser, Matthew" <glass...@wustl.edu> Cc: "hcp-users@humanconnectome.org" <hcp-users@humanconnectome.org> Subject: Re: [HCP-Users] netmats prediction of fluid intelligence Certainly one difference is that HCP (i.e., Steve) tends to take the more conservative approach of regressing a *lot* of potential confounds, which tends to result in a lower prediction values. You can see that without confound regression, Steve's prediction is 0.21 versus 0.06. Regards, Thomas On Fri, Oct 6, 2017 at 1:44 AM, Glasser, Matthew <glass...@wustl.edu<mailto:glass...@wustl.edu>> wrote: Perhaps there is an issue related to data clean up or alignment of brain areas across subjects. The Finn study does not appear to have followed the recommended approach to either. Peace, Matt. From: <hcp-users-boun...@humanconnectome.org<mailto:hcp-users-boun...@humanconnectome.org>> on behalf of Benjamin Garzon <benjamin.gar...@ki.se<mailto:benjamin.gar...@ki.se>> Date: Thursday, October 5, 2017 at 1:39 PM To: "hcp-users@humanconnectome.org<mailto:hcp-users@humanconnectome.org>" <hcp-users@humanconnectome.org<mailto:hcp-users@humanconnectome.org>> Subject: [HCP-Users] netmats prediction of fluid intelligence Dear HCP experts, I'm trying to reconcile the MegaTrawl prediction of fluid intelligence (PMAT24_A_CR) https://db.humanconnectome.org/megatrawl/3T_HCP820_MSMAll_d200_ts2/megatrawl_1/sm203/index.html (which shows r = 0.06 between predicted and measured scores) with the Finn 2015 study https://www.nature.com/neuro/journal/v18/n11/full/nn.4135.html claiming an r = 0.5 correlation between predicted and measured scores. In the article they used a subset of the HCP data (126 subjects), but the measure of fluid intelligence is the same one. What can explain the considerable difference? As far as I can see the article did not address confounding, but even in that case r = 0.21 for MegaTrawl, which is still far from 0.5. And this considering that the model used in the article is a much simpler one than the MegaTrawl elastic net regressor. I've been trying to predict fluid intelligence in an independent sample with 300 subjects and a netmats + confounds model does not perform better than a confounds-only model, more in agreement with the MegaTrawl results. In the Smith 2015 paper http://www.nature.com/neuro/journal/v18/n11/full/nn.4125.html the found mode of covariation with the netmats data correlates with fluid intelligence with r = 0.38. Should I conclude from the Megatrawl analysis (as well as from my own) that the single measure of fluid intelligence is not reliable enough to be predicted based on connectome data, or am I missing something from the Finn paper? I would be happy to read people 's thoughts about this topic, in view of the disparate results in the literature. Best regards, Benjamín Garzón, PhD Department of Neurobiology, Care Sciences and Society Aging Research Center | 113<https://maps.google.com/?q=113%C2%A030+Stockholm+%7C+G%C3%A4vlegatan+16=gmail=g> 30 Stockholm | Gävlegatan 16<https://maps.google.com/?q=113%C2%A030+Stockholm+%7C+G%C3%A4vlegatan+16=gmail=g> benjamin.gar...@ki.se<mailto:benjamin.gar...@ki.se> | www.ki-su-arc.se<https://email.ki.se/owa/redir.aspx?C=LDNa9T7Nak68Br6ZyIC_J4KUwCiWMdEIQwVElfLYlCPLbdpUruOe0XhySwY-dNAYT9JyRT4AtFo.=http%3a%2f%2fwww.ki-su-arc.se%2f> __ Karolinska Institutet – a medical university ___ HCP-Users mailing list HCP-Users@humanconnectome.org<mailto:HCP-Users@humanconnectome.org> http://lists.humanconnectome.org/mailman/listinfo/hcp-users ___ HCP-Users mailing list HCP-Users@humanconnectome.org<mailto:HCP-Users@humanconnectome.org> http://lists.humanconnectome.org/mailman/listinfo/hcp-users ___ HCP-Users mailin
Re: [HCP-Users] netmats prediction of fluid intelligence
Certainly one difference is that HCP (i.e., Steve) tends to take the more conservative approach of regressing a *lot* of potential confounds, which tends to result in a lower prediction values. You can see that without confound regression, Steve's prediction is 0.21 versus 0.06. Regards, Thomas On Fri, Oct 6, 2017 at 1:44 AM, Glasser, Matthew <glass...@wustl.edu> wrote: > Perhaps there is an issue related to data clean up or alignment of brain > areas across subjects. The Finn study does not appear to have followed the > recommended approach to either. > > Peace, > > Matt. > > From: <hcp-users-boun...@humanconnectome.org> on behalf of Benjamin > Garzon <benjamin.gar...@ki.se> > Date: Thursday, October 5, 2017 at 1:39 PM > To: "hcp-users@humanconnectome.org" <hcp-users@humanconnectome.org> > Subject: [HCP-Users] netmats prediction of fluid intelligence > > Dear HCP experts, > > I'm trying to reconcile the MegaTrawl prediction of fluid intelligence > (PMAT24_A_CR) > > https://db.humanconnectome.org/megatrawl/3T_HCP820_ > MSMAll_d200_ts2/megatrawl_1/sm203/index.html > > (which shows r = 0.06 between predicted and measured scores) > > with the Finn 2015 study > > https://www.nature.com/neuro/journal/v18/n11/full/nn.4135.html > > claiming an r = 0.5 correlation between predicted and measured scores. In > the article they used a subset of the HCP data (126 subjects), but the > measure of fluid intelligence is the same one. What can explain the > considerable difference? As far as I can see the article did not address > confounding, but even in that case r = 0.21 for MegaTrawl, which is still > far from 0.5. And this considering that the model used in the article is a > much simpler one than the MegaTrawl elastic net regressor. > > I've been trying to predict fluid intelligence in an independent sample > with 300 subjects and a netmats + confounds model does not perform better > than a confounds-only model, more in agreement with the MegaTrawl results. > > In the Smith 2015 paper > > http://www.nature.com/neuro/journal/v18/n11/full/nn.4125.html > > the found mode of covariation with the netmats data correlates with fluid > intelligence with r = 0.38. > > Should I conclude from the Megatrawl analysis (as well as from my own) > that the single measure of fluid intelligence is not reliable enough to be > predicted based on connectome data, or am I missing something from the Finn > paper? > > I would be happy to read people 's thoughts about this topic, in view of > the disparate results in the literature. > > Best regards, > > Benjamín Garzón, PhD > Department of Neurobiology, Care Sciences and Society > Aging Research Center | 113 > <https://maps.google.com/?q=113%C2%A030+Stockholm+%7C+G%C3%A4vlegatan+16=gmail=g> > 30 Stockholm | Gävlegatan 16 > <https://maps.google.com/?q=113%C2%A030+Stockholm+%7C+G%C3%A4vlegatan+16=gmail=g> > benjamin.gar...@ki.se | www.ki-su-arc.se > <https://email.ki.se/owa/redir.aspx?C=LDNa9T7Nak68Br6ZyIC_J4KUwCiWMdEIQwVElfLYlCPLbdpUruOe0XhySwY-dNAYT9JyRT4AtFo.=http%3a%2f%2fwww.ki-su-arc.se%2f> > __ > Karolinska Institutet – a medical university > > > ___ > HCP-Users mailing list > HCP-Users@humanconnectome.org > http://lists.humanconnectome.org/mailman/listinfo/hcp-users > > ___ > HCP-Users mailing list > HCP-Users@humanconnectome.org > http://lists.humanconnectome.org/mailman/listinfo/hcp-users > ___ HCP-Users mailing list HCP-Users@humanconnectome.org http://lists.humanconnectome.org/mailman/listinfo/hcp-users