Ben, Just a quick follow-up. If you want to access the files in the S3 bucket, it might be worth looking in to installing S3cmd (http://s3tools.org/s3cmd). It is a command line tool for accessing files in S3 buckets, and I've successfully used it to access some files in the HCP OpenAccess bucket. Being a command line tool, it should be easy to use in a script.
Tim On Wed, Jun 24, 2015, at 15:02, Timothy B. Brown wrote: > Hi Ben, > > You should be able to access the files you are trying to access by > prepending the following to a URI for the files: > > https://db.humanconnectome.org/data/archive/projects/HCP_500/subjects/100307/experiments/100307_CREST/resources/100307_CREST/files > > from the end of that prefix, you can attach the path to your desired > file in the standard packages and Connectome-in-a-Box directory > structure. > > So to get the 100307_3T_tfMRI_EMOTION_LR.nii.gz file you would use: > > https://db.humanconnectome.org/data/archive/projects/HCP_500/subjects/100307/experiments/100307_CREST/resources/100307_CREST/files/unprocessed/3T/tfMRI_EMOTION_LR/100307_3T_tfMRI_EMOTION_LR.nii.gz > > (Yep, that's pretty long. But the expectation is that people are > mostly going to use this in scripts, so the length shouldn't be a > big issue.) > > Notice that from the directory name unprocessed on to the end of the > URI, the directory structure matches what is available in a subject's > directory in S3, in Connectome-in-a-Box distributions, and in unzipped > package files for the subject that are downloaded from the > db.humanconnectome.org web site. > > Since this data can only be accessed by authorized users of > ConnectomeDB who have accepted the data use terms, you will need to > supply credentials one way or another to successfully access a file > using a URI like the above. > > For example, if you are using CURL you could use the -u command line > option and supply a ConnectomeDB username and password in the form -u > username:password. > > If you are accessing a lot of file using this mechanism (say in a > script), in order to avoid proliferating a large number of sessions, > you should get a session id using commands something like: > > user="this_is_my_username" password="this_is_my_password" > jsession=`curl -u ${user}:${password} https://db.humanconnectome.org` > > Then you can using the -b command line option with CURL > > curl -b "JSESSIONID=$jsession" -O *<your-url-here>* > > As you would expect, you are not limited to using CURL for this. Any > mechanism that allows you to retrieve contents via a URI/URL should > work. So, for example, you could enter the above URL into the address > bar of a browser. Then you would be prompted to enter credentials > before you get access to the file. (Of course, this does you little > good as a scripting solution.) > > I am not familiar with Amazon's boto package, but it seems from a > follow up message that you sent, you may have already tracked that > problem to a boto bug. > > Hope that is helpful, > > Tim > > On Fri, Jun 12, 2015, at 19:30, Ben Cipollini wrote: >> Hi, >> >> I'm trying to access HCP files via scripting. I know the directory >> structure through the appendix III definitions. However, I am not >> sure how to access the resources through XNAT nor Amazon S3. >> >> >> In XNAT, I can't figure out what URL to construct to access any file. >> For example, what would be the URL to access the file 100307/tfMRI_E- >> MOTION_LR/100307/tfMRI_EMOTION_LR/100307_3T_tfMRI_EMOTION_LR.nii.gz ? >> >> >> Alternately, I'm trying to access the Amazon S3 files through >> Amazon's boto package. I can open a connection with my S3 >> credentials, and I see a "hcp-openaccess" bucket. When I try to list >> the bucket contents (for file_key in bucket.list(): print file_key), >> I get access denied. >> >> Has anybody tried accessing the S3 data in this way that can share an >> example script? Or any suggestions for other ways to script the >> access? >> >> >> Thanks, Ben >> _______________________________________________ >> HCP-Users">HCP-Users mailing list >> HCP-Users">HCP-Users@humanconnectome">HCP-Users">HCP- [email protected] >> http://lists.humanconnectome.org/mailman/listinfo/hcp-users > -- > Timothy B. Brown Business & Technology Application Analyst III > Pipeline Developer (Human Connectome Project) tbbrown(at)wustl.edu > ________________________________________ > The material in this message is private and may contain Protected > Healthcare Information (PHI). If you are not the intended recipient, > be advised that any unauthorized use, disclosure, copying or the > taking of any action in reliance on the contents of this information > is strictly prohibited. If you have received this email in error, > please immediately notify the sender via telephone or return mail. -- Timothy B. Brown Business & Technology Application Analyst III Pipeline Developer (Human Connectome Project) tbbrown(at)wustl.edu ________________________________________ The material in this message is private and may contain Protected Healthcare Information (PHI). If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ HCP-Users mailing list [email protected] http://lists.humanconnectome.org/mailman/listinfo/hcp-users
