Re: [HCP-Users] netmats prediction of fluid intelligence

[Harms, Michael]

Germane to this discussion is that using the same methodology, but a different 
sample of subjects, the same Yale group has recently reported that the 
correlation of predicted gF (from netmats) and observed gF was r=0.22.

https://www.ncbi.nlm.nih.gov/pubmed/28968754

cheers,
-MH

--
Michael Harms, Ph.D.

---
Conte Center for the Neuroscience of Mental Disorders
Washington University School of Medicine
Department of Psychiatry, Box 8134
660 South Euclid Ave.Tel: 314-747-6173
St. Louis, MO  63110Email: mha...@wustl.edu

On 10/7/17, 2:43 PM, "hcp-users-boun...@humanconnectome.org on behalf of Nina 
de Lacy"  wrote:

This is a very interesting thread and discussion and many of the observations 
conform with ongoing work I'm doing in children/adolescents which generally 
suggests that predicting intelligence measures is very challenging using 
connectivity measures, after including confounders within multivariate 
frameworks. I personally wonder not only about confounding effects, but also 
the difficulty of working with neuropsychological 'intelligence' measures 
designed for other purposes than perhaps some of what we are trying to get at. 
As well, I would raise the question of our samples, which most/much of the time 
in neuroimaging rarely include individuals with lower IQs, therefore distorting 
the distribution.

All that said, what I really joined in for was to ask Julien if he could 
comment more on what he meant by highlighting that part of the effect obtained 
in the FInn study was due to the "specific subject sample" used. Was this due 
to certain characteristics of the smaller subject sample? I of course respect 
this may be content germane to an as yet unpublished paper you may not want to 
share in detail :)

Nina


On Sat, 7 Oct 2017, Julien Dubois wrote:

>   Julien, when you say the method still has predictive value in the large 
> sample 'without confounds', do you mean without removing confounds or after 
> deconfounding? It's also not
>   clear to me whether the scores the Ma study reported are deconfounded 
> or not, but I guess they are not. If one is interested in the added value of 
> fMRI predicting cognition (my
>   case), it makes sense to be conservative, so I would be interested in 
> knowing whether there's something left in the deconfounded space.
>
>
> Sorry, my phrasing wasn't clear. I mean that I obtain similar results to the 
> Megatrawl and to the Ma poster, WITHOUT deconfounding as performed in the 
> Megatrawl. I will let you know how it
> looks once I use the same deconfounding as in the Megatrawl, i.e.: 
> "Prediction takes place after removing sex, age, age^2 , sex*age, sex*age^2 , 
> brain & head size (as estimated by
> FreeSurfer), overall head motion (a summation over all timepoints of 
> timepoint-to-timepoint relative head motion) and acquisition date as 
> confounds (the last of these is actually the
> “acquisition quarter”, which is useful to include because there was a slight 
> change in rfMRI reconstruction code during the third acquisition 
> year-quarter; in future we will instead use
> the actual reconstruction code version as the confound)."
> - Julien
>
> ___HCP-Users mailing 
> listHCP-Users@humanconnectome.orghttp://lists.humanconnectome.org/mailman/listinfo/hcp-users
>
>
>

This message and any attached files might contain confidential information 
protected by federal and state law. The information is intended only for the 
use of the individual(s) or entities originally named as addressees. The 
improper disclosure of such information may be subject to civil or criminal 
penalties. If this message reached you in error, please contact the sender and 
destroy this message. Disclosing, copying, forwarding, or distributing the 
information by unauthorized individuals or entities is strictly prohibited by 
law.

___
HCP-Users mailing list
HCP-Users@humanconnectome.org
http://lists.humanconnectome.org/mailman/listinfo/hcp-users




The materials in this message are private and may contain Protected Healthcare 
Information or other information of a sensitive nature. If you are not the 
intended recipient, be advised that any unauthorized use, disclosure, copying 
or the taking of any action in reliance on the contents of this information is 
strictly prohibited. If you have received this email in error, please 
immediately notify the sender via telephone or return mail.

___
HCP-Users mailing list
HCP-Users@humanconnectome.org
http://lists.humanconnectome.org/mailman/listinfo/hcp-users

Re: [HCP-Users] netmats prediction of fluid intelligence

[Julien Dubois]

>
> Julien, when you say the method still has predictive value in the large
> sample 'without confounds', do you mean without removing confounds or after
> deconfounding? It's also not clear to me whether the scores the Ma study
> reported are deconfounded or not, but I guess they are not. If one is
> interested in the added value of fMRI predicting cognition (my case), it
> makes sense to be conservative, so I would be interested in knowing whether
> there's something left in the deconfounded space.
>

Sorry, my phrasing wasn't clear. I mean that I obtain similar results to
the Megatrawl and to the Ma poster, WITHOUT deconfounding as performed in
the Megatrawl. I will let you know how it looks once I use the same
deconfounding as in the Megatrawl, i.e.: "Prediction takes place after
removing sex, age, age^2 , sex*age, sex*age^2 , brain & head size (as
estimated by FreeSurfer), overall head motion (a summation over all
timepoints of timepoint-to-timepoint relative head motion) and acquisition
date as confounds (the last of these is actually the “acquisition quarter”,
which is useful to include because there was a slight change in rfMRI
reconstruction code during the third acquisition year-quarter; in future we
will instead use the actual reconstruction code version as the confound)."
- Julien

___
HCP-Users mailing list
HCP-Users@humanconnectome.org
http://lists.humanconnectome.org/mailman/listinfo/hcp-users

Re: [HCP-Users] netmats prediction of fluid intelligence

[Benjamin Garzon]

Hi all,

Thanks for your comments. Reassuring although somewhat disappointing at the 
same time. As as far as I know there's no external validation of their fluid 
intelligence model. They replicated the prediction for their sustained 
attention model (Rosenberg 2016), but there may be quite different processes at 
play there. I think it's a great study in any case.

Julien, when you say the method still has predictive value in the large sample 
'without confounds', do you mean without removing confounds or after 
deconfounding? It's also not clear to me whether the scores the Ma study 
reported are deconfounded or not, but I guess they are not. If one is 
interested in the added value of fMRI predicting cognition (my case), it makes 
sense to be conservative, so I would be interested in knowing whether there's 
something left in the deconfounded space.

Best regards,

Benjamín Garzón, PhD
Department of Neurobiology, Care Sciences and Society
Aging Research Center | 113 30 Stockholm | Gävlegatan 16
benjamin.gar...@ki.se<mailto:benjamin.gar...@ki.se> | 
www.ki-su-arc.se<https://email.ki.se/owa/redir.aspx?C=LDNa9T7Nak68Br6ZyIC_J4KUwCiWMdEIQwVElfLYlCPLbdpUruOe0XhySwY-dNAYT9JyRT4AtFo.=http%3a%2f%2fwww.ki-su-arc.se%2f>
__
Karolinska Institutet – a medical university


From: hcp-users-boun...@humanconnectome.org 
[hcp-users-boun...@humanconnectome.org] on behalf of Julien Dubois 
[jcrdub...@gmail.com]
Sent: Friday, October 06, 2017 5:33 PM
To: hcp-users@humanconnectome.org
Subject: Re: [HCP-Users] netmats prediction of fluid intelligence

Hi all

to chime in: we have done extensive work over the past year to replicate and 
understand the prediction of IQ obtained in the Finn study. Our manuscript is 
about to be submitted. Take away points:
-- the high effect size they find is partly due to small sample size (118 
subjects) and to the specific subject sample
-- their method still has predictive value in the larger sample of subjects, 
though the effect size is much reduced (similar to Megatrawl without confounds)
-- the specifics of preprocessing/denoising and predictive model don't have a 
huge effect on the final result (when enough subjects are included)

See also the work presented by Feilong Ma at OHBM this year, which took great 
care in aligning subjects (MSMall + whole-brain hyperalignment) in a much 
larger sample than the Finn study: 
https://files.aievolution.com/hbm1701/abstracts/37710/3928_Ma.pdf

Happy to discuss further if someone is interested.
- Julien


Postdoc | Cedars-Sinai Medical Center // Caltech | +1 (310)423-8377 | 
nigiri.caltech.edu/~jdubois<http://nigiri.caltech.edu/~jdubois>




___
HCP-Users mailing list
HCP-Users@humanconnectome.org
http://lists.humanconnectome.org/mailman/listinfo/hcp-users

___
HCP-Users mailing list
HCP-Users@humanconnectome.org
http://lists.humanconnectome.org/mailman/listinfo/hcp-users

Re: [HCP-Users] netmats prediction of fluid intelligence

[Julien Dubois]

Hi all

to chime in: we have done extensive work over the past year to replicate
and understand the prediction of IQ obtained in the Finn study. Our
manuscript is about to be submitted. Take away points:
-- the high effect size they find is partly due to small sample size (118
subjects) and to the specific subject sample
-- their method still has predictive value in the larger sample of
subjects, though the effect size is much reduced (similar to Megatrawl
without confounds)
-- the specifics of preprocessing/denoising and predictive model don't have
a huge effect on the final result (when enough subjects are included)

See also the work presented by Feilong Ma at OHBM this year, which took
great care in aligning subjects (MSMall + whole-brain hyperalignment) in a
much larger sample than the Finn study:
https://files.aievolution.com/hbm1701/abstracts/37710/3928_Ma.pdf

Happy to discuss further if someone is interested.
- Julien


*Postdoc | *Cedars-Sinai Medical Center // Caltech | +1 (310)423-8377 |
nigiri.caltech.edu/~jdubois

___
HCP-Users mailing list
HCP-Users@humanconnectome.org
http://lists.humanconnectome.org/mailman/listinfo/hcp-users

Re: [HCP-Users] netmats prediction of fluid intelligence

[Stephen Smith]

Hi all

Yes - I've discussed this with Todd and it's not immediately clear whether the 
difference is due to:
 - they used full correlation not partial
 - they used fewer confound regressors (IIRC)
 - their prediction method is *very* different (pooling across all relevant 
features rather than keeping them separate in the multivariate elastic net 
regression prediction).

Or some combination of all of this.  I don't have a strong gut feeling which of 
these might be the biggest factor, but we should note that the Finn paper took 
a lot more care over many aspects of their analysis than many studies do, and 
in particular it was impressive how they got replication of the prediction 
between completelty separate studies. But yes I would be interested to see this 
resolved more.

With respect to our CCA-based population mode, which covaried more highly with 
the intelligence measure as you mentioned - I think maybe this points at the 
main issue possibly being the noisiness of the individual features (netmat 
edges) and also of the intelligence feature (when all combined together within 
the elastic net prediction framework).

Cheers.




> On 6 Oct 2017, at 04:11, Harms, Michael <mha...@wustl.edu> wrote:
> 
>  
> In the context of the long resting state runs that we have available, I would 
> argue that throwing in additional possible confounds is the appropriate thing 
> to do.  Are you suggesting that sex, age, age^2, sex*age, sex*age^2, brain 
> size, head size, and average motion shouldn’t all be included?
>  
> Regardless, r = 0.21 (without confounds in the MegaTrawl) is a long way from 
> the r = 0.5 prediction in Finn et al.
>  
> Cheers,
> -MH
>  
> -- 
> Michael Harms, Ph.D.
> ---
> Conte Center for the Neuroscience of Mental Disorders
> Washington University School of Medicine
> Department of Psychiatry, Box 8134
> 660 South Euclid Ave.Tel: 314-747-6173
> St. Louis, MO  63110  Email: 
> mha...@wustl.edu <mailto:mha...@wustl.edu>
>  
> From: <hcp-users-boun...@humanconnectome.org 
> <mailto:hcp-users-boun...@humanconnectome.org>> on behalf of Thomas Yeo 
> <ytho...@csail.mit.edu <mailto:ytho...@csail.mit.edu>>
> Date: Thursday, October 5, 2017 at 10:01 PM
> To: "Glasser, Matthew" <glass...@wustl.edu <mailto:glass...@wustl.edu>>
> Cc: "hcp-users@humanconnectome.org <mailto:hcp-users@humanconnectome.org>" 
> <hcp-users@humanconnectome.org <mailto:hcp-users@humanconnectome.org>>
> Subject: Re: [HCP-Users] netmats prediction of fluid intelligence
>  
> Certainly one difference is that HCP (i.e., Steve) tends to take the more 
> conservative approach of regressing a *lot* of potential confounds, which 
> tends to result in a lower prediction values. You can see that without 
> confound regression, Steve's prediction is 0.21 versus 0.06. 
>  
> Regards,
> Thomas
>  
> On Fri, Oct 6, 2017 at 1:44 AM, Glasser, Matthew <glass...@wustl.edu 
> <mailto:glass...@wustl.edu>> wrote:
>> Perhaps there is an issue related to data clean up or alignment of brain 
>> areas across subjects.  The Finn study does not appear to have followed the 
>> recommended approach to either.
>>  
>> Peace,
>>  
>> Matt.
>>  
>> From: <hcp-users-boun...@humanconnectome.org 
>> <mailto:hcp-users-boun...@humanconnectome.org>> on behalf of Benjamin Garzon 
>> <benjamin.gar...@ki.se <mailto:benjamin.gar...@ki.se>>
>> Date: Thursday, October 5, 2017 at 1:39 PM
>> To: "hcp-users@humanconnectome.org <mailto:hcp-users@humanconnectome.org>" 
>> <hcp-users@humanconnectome.org <mailto:hcp-users@humanconnectome.org>>
>> Subject: [HCP-Users] netmats prediction of fluid intelligence
>>  
>> Dear HCP experts, 
>>  
>> I'm trying to reconcile the MegaTrawl prediction of fluid intelligence 
>> (PMAT24_A_CR)
>>  
>> https://db.humanconnectome.org/megatrawl/3T_HCP820_MSMAll_d200_ts2/megatrawl_1/sm203/index.html
>>  
>> <https://db.humanconnectome.org/megatrawl/3T_HCP820_MSMAll_d200_ts2/megatrawl_1/sm203/index.html>
>>  
>> (which shows r = 0.06 between predicted and measured scores)
>>  
>> with the Finn 2015 study 
>>  
>> https://www.nature.com/neuro/journal/v18/n11/full/nn.4135.html 
>> <https://www.nature.com/neuro/journal/v18/n11/full/nn.4135.html>
>>  
>> claiming an r = 0.5 correlation between predicted and measured scores. In 
>> the article they used a subset of the HCP data (126 subjects), but the 
>> measure of fluid intelligence

Re: [HCP-Users] netmats prediction of fluid intelligence

[Harms, Michael]

In the context of the long resting state runs that we have available, I would 
argue that throwing in additional possible confounds is the appropriate thing 
to do.  Are you suggesting that sex, age, age^2, sex*age, sex*age^2, brain 
size, head size, and average motion shouldn’t all be included?

Regardless, r = 0.21 (without confounds in the MegaTrawl) is a long way from 
the r = 0.5 prediction in Finn et al.

Cheers,
-MH

--
Michael Harms, Ph.D.
---
Conte Center for the Neuroscience of Mental Disorders
Washington University School of Medicine
Department of Psychiatry, Box 8134
660 South Euclid Ave.Tel: 314-747-6173
St. Louis, MO  63110  Email: 
mha...@wustl.edu

From: <hcp-users-boun...@humanconnectome.org> on behalf of Thomas Yeo 
<ytho...@csail.mit.edu>
Date: Thursday, October 5, 2017 at 10:01 PM
To: "Glasser, Matthew" <glass...@wustl.edu>
Cc: "hcp-users@humanconnectome.org" <hcp-users@humanconnectome.org>
Subject: Re: [HCP-Users] netmats prediction of fluid intelligence

Certainly one difference is that HCP (i.e., Steve) tends to take the more 
conservative approach of regressing a *lot* of potential confounds, which tends 
to result in a lower prediction values. You can see that without confound 
regression, Steve's prediction is 0.21 versus 0.06.

Regards,
Thomas

On Fri, Oct 6, 2017 at 1:44 AM, Glasser, Matthew 
<glass...@wustl.edu<mailto:glass...@wustl.edu>> wrote:
Perhaps there is an issue related to data clean up or alignment of brain areas 
across subjects.  The Finn study does not appear to have followed the 
recommended approach to either.

Peace,

Matt.

From: 
<hcp-users-boun...@humanconnectome.org<mailto:hcp-users-boun...@humanconnectome.org>>
 on behalf of Benjamin Garzon 
<benjamin.gar...@ki.se<mailto:benjamin.gar...@ki.se>>
Date: Thursday, October 5, 2017 at 1:39 PM
To: "hcp-users@humanconnectome.org<mailto:hcp-users@humanconnectome.org>" 
<hcp-users@humanconnectome.org<mailto:hcp-users@humanconnectome.org>>
Subject: [HCP-Users] netmats prediction of fluid intelligence

Dear HCP experts,

I'm trying to reconcile the MegaTrawl prediction of fluid intelligence 
(PMAT24_A_CR)

https://db.humanconnectome.org/megatrawl/3T_HCP820_MSMAll_d200_ts2/megatrawl_1/sm203/index.html

(which shows r = 0.06 between predicted and measured scores)

with the Finn 2015 study

https://www.nature.com/neuro/journal/v18/n11/full/nn.4135.html

claiming an r = 0.5 correlation between predicted and measured scores. In the 
article they used a subset of the HCP data (126 subjects), but the measure of 
fluid intelligence is the same one. What can explain the considerable 
difference? As far as I can see the article did not address confounding, but 
even in that case r = 0.21 for MegaTrawl, which is still far from 0.5. And this 
considering that the model used in the article is a much simpler one than the 
MegaTrawl elastic net regressor.

I've been trying to predict fluid intelligence in an independent sample with 
300 subjects and a netmats + confounds model does not perform better than a 
confounds-only model, more in agreement with the MegaTrawl results.

In the Smith 2015 paper

http://www.nature.com/neuro/journal/v18/n11/full/nn.4125.html

the found mode of covariation with the netmats data correlates with fluid 
intelligence with r = 0.38.

Should I conclude from the Megatrawl analysis (as well as from my own) that the 
single measure of fluid intelligence is not reliable enough to be predicted 
based on connectome data, or am I missing something from the Finn paper?

I would be happy to read people 's thoughts about this topic, in view of the 
disparate results in the literature.

Best regards,

Benjamín Garzón, PhD
Department of Neurobiology, Care Sciences and Society
Aging Research Center | 
113<https://maps.google.com/?q=113%C2%A030+Stockholm+%7C+G%C3%A4vlegatan+16=gmail=g>
 30 Stockholm | Gävlegatan 
16<https://maps.google.com/?q=113%C2%A030+Stockholm+%7C+G%C3%A4vlegatan+16=gmail=g>
benjamin.gar...@ki.se<mailto:benjamin.gar...@ki.se> | 
www.ki-su-arc.se<https://email.ki.se/owa/redir.aspx?C=LDNa9T7Nak68Br6ZyIC_J4KUwCiWMdEIQwVElfLYlCPLbdpUruOe0XhySwY-dNAYT9JyRT4AtFo.=http%3a%2f%2fwww.ki-su-arc.se%2f>
__
Karolinska Institutet – a medical university



___
HCP-Users mailing list
HCP-Users@humanconnectome.org<mailto:HCP-Users@humanconnectome.org>
http://lists.humanconnectome.org/mailman/listinfo/hcp-users

___
HCP-Users mailing list
HCP-Users@humanconnectome.org<mailto:HCP-Users@humanconnectome.org>
http://lists.humanconnectome.org/mailman/listinfo/hcp-users


___
HCP-Users mailin

Re: [HCP-Users] netmats prediction of fluid intelligence

[Thomas Yeo]

Certainly one difference is that HCP (i.e., Steve) tends to take the more
conservative approach of regressing a *lot* of potential confounds, which
tends to result in a lower prediction values. You can see that without
confound regression, Steve's prediction is 0.21 versus 0.06.

Regards,
Thomas

On Fri, Oct 6, 2017 at 1:44 AM, Glasser, Matthew  wrote:

> Perhaps there is an issue related to data clean up or alignment of brain
> areas across subjects.  The Finn study does not appear to have followed the
> recommended approach to either.
>
> Peace,
>
> Matt.
>
> From:  on behalf of Benjamin
> Garzon 
> Date: Thursday, October 5, 2017 at 1:39 PM
> To: "hcp-users@humanconnectome.org" 
> Subject: [HCP-Users] netmats prediction of fluid intelligence
>
> Dear HCP experts,
>
> I'm trying to reconcile the MegaTrawl prediction of fluid intelligence
> (PMAT24_A_CR)
>
> https://db.humanconnectome.org/megatrawl/3T_HCP820_
> MSMAll_d200_ts2/megatrawl_1/sm203/index.html
>
> (which shows r = 0.06 between predicted and measured scores)
>
> with the Finn 2015 study
>
> https://www.nature.com/neuro/journal/v18/n11/full/nn.4135.html
>
> claiming an r = 0.5 correlation between predicted and measured scores. In
> the article they used a subset of the HCP data (126 subjects), but the
> measure of fluid intelligence is the same one. What can explain the
> considerable difference? As far as I can see the article did not address
> confounding, but even in that case r = 0.21 for MegaTrawl, which is still
> far from 0.5. And this considering that the model used in the article is a
> much simpler one than the MegaTrawl elastic net regressor.
>
> I've been trying to predict fluid intelligence in an independent sample
> with 300 subjects and a netmats + confounds model does not perform better
> than a confounds-only model, more in agreement with the MegaTrawl results.
>
> In the Smith 2015 paper
>
> http://www.nature.com/neuro/journal/v18/n11/full/nn.4125.html
>
> the found mode of covariation with the netmats data correlates with fluid
> intelligence with r = 0.38.
>
> Should I conclude from the Megatrawl analysis (as well as from my own)
> that the single measure of fluid intelligence is not reliable enough to be
> predicted based on connectome data, or am I missing something from the Finn
> paper?
>
> I would be happy to read people 's thoughts about this topic, in view of
> the disparate results in the literature.
>
> Best regards,
>
> Benjamín Garzón, PhD
> Department of Neurobiology, Care Sciences and Society
> Aging Research Center | 113
> 
>  30 Stockholm | Gävlegatan 16
> 
> benjamin.gar...@ki.se | www.ki-su-arc.se
> 
> __
> Karolinska Institutet – a medical university
>
>
> ___
> HCP-Users mailing list
> HCP-Users@humanconnectome.org
> http://lists.humanconnectome.org/mailman/listinfo/hcp-users
>
> ___
> HCP-Users mailing list
> HCP-Users@humanconnectome.org
> http://lists.humanconnectome.org/mailman/listinfo/hcp-users
>

___
HCP-Users mailing list
HCP-Users@humanconnectome.org
http://lists.humanconnectome.org/mailman/listinfo/hcp-users