Re: [HCP-Users] AWS sync error

[Timothy Coalson]

It looks like there is an alternate explanation: attempting to sync the
entire bucket fails, but syncing one folder in it works:

$ aws s3 --profile tempbucket sync --region us-east-1
's3://hcp-openaccess-temp' /mnt/myelin/tim/synctest
A client error (AccessDenied) occurred when calling the ListObjects
operation: Access Denied
Completed 1 part(s) with ... file(s) remaining
$ aws s3 --profile tempbucket sync --region us-east-1
's3://hcp-openaccess-temp/HCP_1200' /mnt/myelin/tim/synctest
download:
s3://hcp-openaccess-temp/HCP_1200/100206/.xdlm/100206_3T_Diffusion_unproc.json
to ../../synctest/100206/.xdlm/100206_3T_Diffusion_unproc.json
download:
s3://hcp-openaccess-temp/HCP_1200/100206/.xdlm/100206_3T_rfMRI_REST_fix.json
to ../../synctest/100206/.xdlm/100206_3T_rfMRI_REST_fix.json
...

So, this may be similar to the fact that you can't mount the bucket's root
with s3fs, but you can mount its folders.  This may have been by design on
our part, setting specific permissions on the bucket to give an error when
someone tries something that they probably don't want, but it doesn't seem
to be documented.

Tim


On Thu, Aug 9, 2018 at 2:22 PM, Hodge, Michael  wrote:

>
>
> Hi Xavier,
>
>
>
> That message indicates that the access key in your credentials isn’t a
> currently valid access key registered with AWS.  I’m thinking that would
> most likely either be a cut-paste issue or an issue or an issue where the
> access and secret keys are switched or something similar.  One thing that
> does happen is that requesting a new access key invalidates or removes any
> prior keys for the account.  It seems unlikely, but it’s possible that a
> double click on the key generation button could present you with an access
> key that’s already been invalidated.
>
>
>
> I’d suggest trying again with a fresh set of credentials.  If you’re still
> seeing this issue, feel free to contact me directly.  I’d be able to look
> up your account in AWS and get information on what key’s active, the age of
> the key, etc.  I don’t have access to the secret key value, but the issue
> here is that the key value stored in the credentials isn’t a currently
> valid one.
>
>
>
> Regards,
>
>
>
> Mike
>
>
>
> *From:* hcp-users-boun...@humanconnectome.org  humanconnectome.org> *On Behalf Of *Xavier Guell Paradis
> *Sent:* Thursday, August 9, 2018 11:43 AM
> *To:* hcp-users@humanconnectome.org
> *Subject:* [HCP-Users] AWS sync error
>
>
>
> Hello HCP,
>
> We are a team of researchers designing some deep learning tools we have
> created for modality conversions, and wanting to test our library on the
> HCP dataset (structural T1 and T2). We are having issues copying from the
> s3 bucket. Currently after creating our credentials, running aws configure,
> and attempting to sync via:
>
>
>
> $aws configure
>
> [entering access credentials]
>
> $aws s3 sync s3://hcp-openaccess-temp .
>
> Ïnvalid Access Key ID¨AWS key does not exist in the records.
>
>
>
> We appreciate any recommendations on how to proceed.
>
>
>
> Thank you,
>
> Patrick, Xavier, TJ, Anita, Shreyas, Saige.
>
> ___
> HCP-Users mailing list
> HCP-Users@humanconnectome.org
> http://lists.humanconnectome.org/mailman/listinfo/hcp-users
>
>
> --
>
> The materials in this message are private and may contain Protected
> Healthcare Information or other information of a sensitive nature. If you
> are not the intended recipient, be advised that any unauthorized use,
> disclosure, copying or the taking of any action in reliance on the contents
> of this information is strictly prohibited. If you have received this email
> in error, please immediately notify the sender via telephone or return mail.
>
> ___
> HCP-Users mailing list
> HCP-Users@humanconnectome.org
> http://lists.humanconnectome.org/mailman/listinfo/hcp-users
>

___
HCP-Users mailing list
HCP-Users@humanconnectome.org
http://lists.humanconnectome.org/mailman/listinfo/hcp-users

Re: [HCP-Users] AWS sync error

[Timothy Coalson]

I would suggest caution as to syncing the data, mounting may suit your
purposes better (can look around and fetch file contents as-needed rather
than getting everything even if it isn't used).  The different releases are
in different folders, and basically all of the earlier release subjects are
also included in every later release, so there is a lot of logical
redundancy (however, the processing was changed between releases, so the
"redundant" files aren't identical).  So, for starters, you may be better
off sticking to just the HCP_1200 folder.

However, be aware that each subjects' folder comes out to around 80GB, so
you are looking at transferring about 100TB of data for just the HCP_1200
folder - make sure you actually need everything you are downloading, and
have the space available for it.

Tim


On Thu, Aug 9, 2018 at 11:42 AM, Xavier Guell Paradis 
wrote:

> Hello HCP,
> We are a team of researchers designing some deep learning tools we have
> created for modality conversions, and wanting to test our library on the
> HCP dataset (structural T1 and T2). We are having issues copying from the
> s3 bucket. Currently after creating our credentials, running aws configure,
> and attempting to sync via:
>
> $aws configure
> [entering access credentials]
> $aws s3 sync s3://hcp-openaccess-temp .
> Ïnvalid Access Key ID¨AWS key does not exist in the records.
>
> We appreciate any recommendations on how to proceed.
>
> Thank you,
> Patrick, Xavier, TJ, Anita, Shreyas, Saige.
>
> ___
> HCP-Users mailing list
> HCP-Users@humanconnectome.org
> http://lists.humanconnectome.org/mailman/listinfo/hcp-users
>

___
HCP-Users mailing list
HCP-Users@humanconnectome.org
http://lists.humanconnectome.org/mailman/listinfo/hcp-users

Re: [HCP-Users] AWS sync error

[Hodge, Michael]

Hi Xavier,

That message indicates that the access key in your credentials isn't a 
currently valid access key registered with AWS.  I'm thinking that would most 
likely either be a cut-paste issue or an issue or an issue where the access and 
secret keys are switched or something similar.  One thing that does happen is 
that requesting a new access key invalidates or removes any prior keys for the 
account.  It seems unlikely, but it's possible that a double click on the key 
generation button could present you with an access key that's already been 
invalidated.

I'd suggest trying again with a fresh set of credentials.  If you're still 
seeing this issue, feel free to contact me directly.  I'd be able to look up 
your account in AWS and get information on what key's active, the age of the 
key, etc.  I don't have access to the secret key value, but the issue here is 
that the key value stored in the credentials isn't a currently valid one.

Regards,

Mike

From: hcp-users-boun...@humanconnectome.org 
 On Behalf Of Xavier Guell Paradis
Sent: Thursday, August 9, 2018 11:43 AM
To: hcp-users@humanconnectome.org
Subject: [HCP-Users] AWS sync error

Hello HCP,
We are a team of researchers designing some deep learning tools we have created 
for modality conversions, and wanting to test our library on the HCP dataset 
(structural T1 and T2). We are having issues copying from the s3 bucket. 
Currently after creating our credentials, running aws configure, and attempting 
to sync via:

$aws configure
[entering access credentials]
$aws s3 sync s3://hcp-openaccess-temp .
Ïnvalid Access Key ID¨AWS key does not exist in the records.

We appreciate any recommendations on how to proceed.

Thank you,
Patrick, Xavier, TJ, Anita, Shreyas, Saige.

___
HCP-Users mailing list
HCP-Users@humanconnectome.org
http://lists.humanconnectome.org/mailman/listinfo/hcp-users


The materials in this message are private and may contain Protected Healthcare 
Information or other information of a sensitive nature. If you are not the 
intended recipient, be advised that any unauthorized use, disclosure, copying 
or the taking of any action in reliance on the contents of this information is 
strictly prohibited. If you have received this email in error, please 
immediately notify the sender via telephone or return mail.

___
HCP-Users mailing list
HCP-Users@humanconnectome.org
http://lists.humanconnectome.org/mailman/listinfo/hcp-users

[HCP-Users] AWS sync error

[Xavier Guell Paradis]

Hello HCP,
We are a team of researchers designing some deep learning tools we have created 
for modality conversions, and wanting to test our library on the HCP dataset 
(structural T1 and T2). We are having issues copying from the s3 bucket. 
Currently after creating our credentials, running aws configure, and attempting 
to sync via:

$aws configure
[entering access credentials]
$aws s3 sync s3://hcp-openaccess-temp .
Ïnvalid Access Key ID¨AWS key does not exist in the records.

We appreciate any recommendations on how to proceed.

Thank you,
Patrick, Xavier, TJ, Anita, Shreyas, Saige.

___
HCP-Users mailing list
HCP-Users@humanconnectome.org
http://lists.humanconnectome.org/mailman/listinfo/hcp-users

Re: [HCP-Users] HCP lifespan diffusion gradient table

[Elam, Jennifer]

Hi Harith,

You can download the CCF protocol template used for Lifespan which includes the 
diffusion gradient vectors from the link in the sidebar on this page: 
http://protocols.humanconnectome.org/CCF/

Download The CCF Protocol Template - Human Connectome 
Project
protocols.humanconnectome.org
Downloadable scanner protocol for HCP-style image acquisition for CCF studies

Best,

Jenn

Jennifer Elam, Ph.D.
Scientific Outreach, Human Connectome Project
Washington University School of Medicine
Department of Neuroscience, Box 8108
660 South Euclid Avenue
St. Louis, MO 63110
314-362-9387
e...@wustl.edu
www.humanconnectome.org



From: hcp-users-boun...@humanconnectome.org 
 on behalf of Akram, Harith 

Sent: Thursday, August 9, 2018 9:59:43 AM
To: hcp-users@humanconnectome.org
Subject: [HCP-Users] HCP lifespan diffusion gradient table

Hi,

We are trying to set up the HCP imaging protocol (used for the lifespan study) 
on our Prisma (+32ch coil).

Is it possible to get the gradient table please?

\\USER\HCP-Lifespan\Lifespan\Session 1\DWI_dir98_AP
TA:5:35 PAT:Off Voxel size:1.5x1.5x1.5 mm Rel. SNR:1.00 :epse

\\USER\HCP-Lifespan\Lifespan\Session 1\DWI_dir98_PA
TA:5:35 PAT:Off Voxel size:1.5x1.5x1.5 mm Rel. SNR:1.00 :epse

\\USER\HCP-Lifespan\Lifespan\Session 1\DWI_dir99_AP
TA:5:38 PAT:Off Voxel size:1.5x1.5x1.5 mm Rel. SNR:1.00 :epse

\\USER\HCP-Lifespan\Lifespan\Session 1\DWI_dir99_PA
TA:5:38 PAT:Off Voxel size:1.5x1.5x1.5 mm Rel. SNR:1.00 :epse

Harith

___
HCP-Users mailing list
HCP-Users@humanconnectome.org
http://lists.humanconnectome.org/mailman/listinfo/hcp-users

___
HCP-Users mailing list
HCP-Users@humanconnectome.org
http://lists.humanconnectome.org/mailman/listinfo/hcp-users

Re: [HCP-Users] Download HCP1200 PTN data

[Elam, Jennifer]

Hi Taicheng,

The only way to download the PTN dataset is through ConnectomeDB using Aspera, 
which was not affected by the AWS bucket migration. It sounds like the problem 
is your firewall. Our Aspera server is configured to use TCP and UDP Port 33001 
for  file transfers instead of a common port such as Port 22. If you are  
downloading from behind an institutional firewall, please speak with  your IT 
people to ensure that Port 33001 is open to both TCP and UDP  traffic. You also 
might try downloading the dataset from home where you are likely to not have a 
firewall that presents a problem.


Best,

Jenn

Jennifer Elam, Ph.D.
Scientific Outreach, Human Connectome Project
Washington University School of Medicine
Department of Neuroscience, Box 8108
660 South Euclid Avenue
St. Louis, MO 63110
314-362-9387
e...@wustl.edu
www.humanconnectome.org



From: hcp-users-boun...@humanconnectome.org 
 on behalf of taicheng_huang 

Sent: Thursday, August 9, 2018 5:11:40 AM
To: hcp-users
Subject: [HCP-Users] Download HCP1200 PTN data

Dear HCP experts,

I would like to download HCP1200 PTN data from ConnectomeDB 
(https://db.humanconnectome.org/data/projects/HCP_1200). After configuring IBM 
Aspera Connect, I couldn't establish HTTP connection. The error message 
reported 'Error establishing HTTP conn... (check HTTP port and firewall)'. I 
also tried to use HTTP proxy for downloading this data but we still couldn't 
establish HTTP connection. I'm wondering if it's due to the recent data 
migration activity. Or could I download the HCP1200 PTN in other ways?
Thanks in advance!

Best,
Taicheng

___
HCP-Users mailing list
HCP-Users@humanconnectome.org
http://lists.humanconnectome.org/mailman/listinfo/hcp-users

___
HCP-Users mailing list
HCP-Users@humanconnectome.org
http://lists.humanconnectome.org/mailman/listinfo/hcp-users

[HCP-Users] HCP lifespan diffusion gradient table

[Akram, Harith]

Hi,

We are trying to set up the HCP imaging protocol (used for the lifespan study) 
on our Prisma (+32ch coil).

Is it possible to get the gradient table please?

\\USER\HCP-Lifespan\Lifespan\Session 
1\DWI_dir98_AP
TA:5:35 PAT:Off Voxel size:1.5x1.5x1.5 mm Rel. SNR:1.00 :epse

\\USER\HCP-Lifespan\Lifespan\Session 
1\DWI_dir98_PA
TA:5:35 PAT:Off Voxel size:1.5x1.5x1.5 mm Rel. SNR:1.00 :epse

\\USER\HCP-Lifespan\Lifespan\Session 
1\DWI_dir99_AP
TA:5:38 PAT:Off Voxel size:1.5x1.5x1.5 mm Rel. SNR:1.00 :epse

\\USER\HCP-Lifespan\Lifespan\Session 
1\DWI_dir99_PA
TA:5:38 PAT:Off Voxel size:1.5x1.5x1.5 mm Rel. SNR:1.00 :epse

Harith

___
HCP-Users mailing list
HCP-Users@humanconnectome.org
http://lists.humanconnectome.org/mailman/listinfo/hcp-users

[HCP-Users] Download HCP1200 PTN data

[taicheng_huang]

Dear HCP experts,

I would like to download HCP1200 PTN data from ConnectomeDB 
(https://db.humanconnectome.org/data/projects/HCP_1200). After configuring IBM 
Aspera Connect, I couldn't establish HTTP connection. The error message 
reported 'Error establishing HTTP conn... (check HTTP port and firewall)'. I 
also tried to use HTTP proxy for downloading this data but we still couldn't 
establish HTTP connection. I'm wondering if it's due to the recent data 
migration activity. Or could I download the HCP1200 PTN in other ways? 
Thanks in advance!

Best, 
Taicheng
___
HCP-Users mailing list
HCP-Users@humanconnectome.org
http://lists.humanconnectome.org/mailman/listinfo/hcp-users

Re: [HCP-Users] Finding a significant difference only in one hemisphere

[Glasser, Matthew]

Sure the inflated surface of the midthickness surface average of the entire 
sample would be fine.

Matt.

From: Darko Komnenić mailto:komnen...@gmail.com>>
Date: Thursday, August 9, 2018 at 6:12 AM
To: "Winkler, Anderson (NIH/NIMH) [E]" 
mailto:anderson.wink...@nih.gov>>
Cc: Matt Glasser mailto:glass...@wustl.edu>>, "Harms, 
Michael" mailto:mha...@wustl.edu>>, 
"hcp-users@humanconnectome.org" 
mailto:hcp-users@humanconnectome.org>>
Subject: Re: [HCP-Users] Finding a significant difference only in one hemisphere

Thank you all for the very useful feedback!
A quick question before I address the other points:
I used the average surface off all controls in one image, and average surface 
of all patients in the other. If I understood Anderson's comment correctly, 
it'd be better to use exactly the same inflated surface for both? What should 
be the underlay then? The inflated surface of the entire sample (patients + 
controls), even though the myelin maps are only obtained on parts of that total 
sample?
Best,
Darko

On Thu, Aug 9, 2018 at 12:14 AM, Winkler, Anderson (NIH/NIMH) [E] 
mailto:anderson.wink...@nih.gov>> wrote:
Hi Darko, hi all,

Are these average images using the exact same color scale? It’d help if the 
surface geometries were the same (e.g., the same inflated for all). But 
assuming that the color scale is the same, and trying to see over the geometry 
differences, looks like the averages between the two groups aren’t that 
massively different, or at least not that asymmetric, correct?

Then taking this together with the Cohen’s d maps, for me this raises the 
possibility that there could be a large outlier in one hemisphere – perhaps a 
subject who moved a bit too much laterally between collection of T1 and T2? Or 
maybe a subject whose right pial reconstruction wasn’t correct and led to a 
much thinner cortex, thus with the myelin profile taken too far away from the 
others (i.e., the mid-thickness too off)?

I guess you might want to investigate on a per-subject and per vertex basis... 
For example, take some vertices where the difference between L and R is large 
and check how the values vary between subjects, see if there are too 
influential observations, then open the images for this/these subjects and see 
if something went wrong along the pipeline… You can also open the residuals and 
see if there are large outliers with a lot of spatial spread.

These are just suggestions… hope they could help anyway…

All the best,

Anderson


From: "Glasser, Matthew" mailto:glass...@wustl.edu>>
Date: Wednesday, August 8, 2018 at 17:05
To: "Harms, Michael" mailto:mha...@wustl.edu>>, Darko 
Komnenić mailto:komnen...@gmail.com>>

Cc: "Winkler, Anderson (NIH/NIMH) [E]" 
mailto:anderson.wink...@nih.gov>>, 
"hcp-users@humanconnectome.org" 
mailto:hcp-users@humanconnectome.org>>
Subject: Re: [HCP-Users] Finding a significant difference only in one hemisphere

Also please post the T1w and T2w image protocols and make sure to use the 
second scaling option for the percentiles (which takes percentiles of the 
absolute value so that positive and negative colors mean the same thing—this 
probably should be the default.

Matt.

From: "Harms, Michael" mailto:mha...@wustl.edu>>
Date: Wednesday, August 8, 2018 at 2:13 PM
To: Darko Komnenić mailto:komnen...@gmail.com>>, Matt 
Glasser mailto:glass...@wustl.edu>>
Cc: "Winkler, Anderson (NIH/NIMH) [E]" 
mailto:anderson.wink...@nih.gov>>, 
"hcp-users@humanconnectome.org" 
mailto:hcp-users@humanconnectome.org>>
Subject: Re: [HCP-Users] Finding a significant difference only in one hemisphere


Hi,
Viewing on the inflated surface would probably be more helpful.

Also, how many subjects per group?

Cheers,
-MH

--
Michael Harms, Ph.D.
---
Associate Professor of Psychiatry
Washington University School of Medicine
Department of Psychiatry, Box 8134
660 South Euclid 
Ave.  
  Tel: 314-747-6173
St. Louis, MO  63110  Email: 
mha...@wustl.edu
From: Darko Komnenić mailto:komnen...@gmail.com>>
Date: Wednesday, August 8, 2018 at 2:08 PM
To: "Glasser, Matthew" mailto:glass...@wustl.edu>>
Cc: "Harms, Michael" mailto:mha...@wustl.edu>>, "Winkler, 
Anderson (NIH/NIMH) [E]" 
mailto:anderson.wink...@nih.gov>>, 
"hcp-users@humanconnectome.org" 
mailto:hcp-users@humanconnectome.org>>
Subject: Re: [HCP-Users] Finding a significant difference only in one hemisphere

Thanks a lot for the clarifications everyone!
The first two images attached are average myelin maps for controls and 
patients, respectively.
The third image is what I hope to be the effect size map. I added the -saveglm 
flag to palm, and then used command -cifti-create-dense-from-template to merge 

Re: [HCP-Users] Finding a significant difference only in one hemisphere

[Darko Komnenić]

Thank you all for the very useful feedback!
A quick question before I address the other points:
I used the average surface off all controls in one image, and average
surface of all patients in the other. If I understood Anderson's comment
correctly, it'd be better to use exactly the same inflated surface for
both? What should be the underlay then? The inflated surface of the entire
sample (patients + controls), even though the myelin maps are only obtained
on parts of that total sample?
Best,
Darko

On Thu, Aug 9, 2018 at 12:14 AM, Winkler, Anderson (NIH/NIMH) [E] <
anderson.wink...@nih.gov> wrote:

> Hi Darko, hi all,
>
>
>
> Are these average images using the exact same color scale? It’d help if
> the surface geometries were the same (e.g., the same inflated for all). But
> assuming that the color scale is the same, and trying to see over the
> geometry differences, looks like the averages between the two groups aren’t
> that massively different, or at least not that asymmetric, correct?
>
>
>
> Then taking this together with the Cohen’s d maps, for me this raises the
> possibility that there could be a large outlier in one hemisphere – perhaps
> a subject who moved a bit too much laterally between collection of T1 and
> T2? Or maybe a subject whose right pial reconstruction wasn’t correct and
> led to a much thinner cortex, thus with the myelin profile taken too far
> away from the others (i.e., the mid-thickness too off)?
>
>
>
> I guess you might want to investigate on a per-subject and per vertex
> basis... For example, take some vertices where the difference between L and
> R is large and check how the values vary between subjects, see if there are
> too influential observations, then open the images for this/these subjects
> and see if something went wrong along the pipeline… You can also open the
> residuals and see if there are large outliers with a lot of spatial spread.
>
>
>
> These are just suggestions… hope they could help anyway…
>
>
>
> All the best,
>
>
>
> Anderson
>
>
>
>
>
> *From: *"Glasser, Matthew" 
> *Date: *Wednesday, August 8, 2018 at 17:05
> *To: *"Harms, Michael" , Darko Komnenić <
> komnen...@gmail.com>
>
> *Cc: *"Winkler, Anderson (NIH/NIMH) [E]" , "
> hcp-users@humanconnectome.org" 
> *Subject: *Re: [HCP-Users] Finding a significant difference only in one
> hemisphere
>
>
>
> Also please post the T1w and T2w image protocols and make sure to use the
> second scaling option for the percentiles (which takes percentiles of the
> absolute value so that positive and negative colors mean the same
> thing—this probably should be the default.
>
>
>
> Matt.
>
>
>
> *From: *"Harms, Michael" 
> *Date: *Wednesday, August 8, 2018 at 2:13 PM
> *To: *Darko Komnenić , Matt Glasser <
> glass...@wustl.edu>
> *Cc: *"Winkler, Anderson (NIH/NIMH) [E]" , "
> hcp-users@humanconnectome.org" 
> *Subject: *Re: [HCP-Users] Finding a significant difference only in one
> hemisphere
>
>
>
>
>
> Hi,
>
> Viewing on the inflated surface would probably be more helpful.
>
>
>
> Also, how many subjects per group?
>
>
>
> Cheers,
>
> -MH
>
>
>
> --
>
> Michael Harms, Ph.D.
>
> ---
>
> Associate Professor of Psychiatry
>
> Washington University School of Medicine
>
> Department of Psychiatry, Box 8134
>
> 660 South Euclid Ave
> .
> Tel: 314-747-6173
>
> St. Louis, MO  63110  Email: mha...@wustl.edu
>
> *From: *Darko Komnenić 
> *Date: *Wednesday, August 8, 2018 at 2:08 PM
> *To: *"Glasser, Matthew" 
> *Cc: *"Harms, Michael" , "Winkler, Anderson (NIH/NIMH)
> [E]" , "hcp-users@humanconnectome.org" <
> hcp-users@humanconnectome.org>
> *Subject: *Re: [HCP-Users] Finding a significant difference only in one
> hemisphere
>
>
>
> Thanks a lot for the clarifications everyone!
>
> The first two images attached are average myelin maps for controls and
> patients, respectively.
>
> The third image is what I hope to be the effect size map. I added the
> -saveglm flag to palm, and then used command -cifti-create-dense-from-template
> to merge the dpv_cohen files for left and right hemisphere. Let me know if
> this does not work.
>
> Thanks in advance for any comments!
>
> Best,
>
> Darko
>
>
>
>
>
>
>
> On Wed, Aug 8, 2018 at 2:22 AM, Glasser, Matthew 
> wrote:
>
> What is an effect size map?  An effect size map is in the case of a t-test
> of difference between two groups is simply the difference in the means.  A
> standardized effect size maps is a Cohen’s d, which is helpful if you want
> to compare effect sizes of different measures that are not on the same
> scale.  Looking at both the difference between means and the means
> themselves could be helpful in tracking down artifacts.  Looking at maps of
> statistical significance is not helpful, despite what you see typically
> done in the neuroimaging literature.
>
>
>
> Matt.
>
>
>
> *From: * on behalf of "Harms,
> Michael" 
> *Date: