Great suggestion Thomas. Aspera is probably faster in general, but the ability to pull specific files from S3 can greatly reduce the total volume.
I suspect a number of folks would appreciate seeing your code. Is it on github or somewhere similar that you could provide a link to? -D From: <hcp-users-boun...@humanconnectome.org<mailto:hcp-users-boun...@humanconnectome.org>> on behalf of "Arnold, Thomas" <arn...@psy.fsu.edu<mailto:arn...@psy.fsu.edu>> Date: Thursday, August 18, 2016 at 5:12 PM To: "hcp-users@humanconnectome.org<mailto:hcp-users@humanconnectome.org>" <hcp-users@humanconnectome.org<mailto:hcp-users@humanconnectome.org>> Subject: Re: [HCP-Users] use of Aspera command-line client for unattended downloads/updates? (Mark Bergman) Mark, I have attempted to do automated downloads with Aspera's command-line client, but have found using AWS S3 to be much easier. This method also allows you to select specific files for download, for instance if you only want to nifti images and not all intermediate steps included in the subjects folder. setup details: https://wiki.humanconnectome.org/display/PublicData/How+To+Connect+to+Connectome+Data+via+AWS If you just want it all: /pathto/Amazon/AWSCLI/aws s3 sync s3://hcp-openaccess/HCP_900 /pathto/folderwith/subjectIDs Whole subjects: /cygdrive/c/Program\ Files/Amazon/AWSCLI/aws s3 sync s3://hcp-openaccess/HCP_900/100307 100307 Specific subjects files: /pathto/Amazon/AWSCLI/aws s3 cp s3://hcp-openaccess/HCP_900/100307/MNINonLinear/Results/rfMRI_REST1_LR/rfMRI_REST1_LR_hp2000_clean.nii.gz rfMRI_REST1_LR_hp2000_clean.nii.gz replacing 100307 with the subjects you need to download. <https://wiki.humanconnectome.org/display/PublicData/How+To+Connect+to+Connectome+Data+via+AWS> Please let me know if this helps. I've written some code that allows users to iteratively download, process, and delete through the S900 subjects with minimal storage and memory requirements, perhaps the download portion could be useful for you. Thomas Campbell Arnold _____________________________________________________________________________________________________________________________________________ Message: 8 Date: Thu, 18 Aug 2016 12:35:51 -0400 From: Mark Bergman <mark.berg...@uphs.upenn.edu<mailto:mark.berg...@uphs.upenn.edu>> Subject: [HCP-Users] use of Aspera command-line client for unattended downloads/updates? To: hcp-users@humanconnectome.org<mailto:hcp-users@humanconnectome.org> Message-ID: <20160818123551.002ea...@uphs.upenn.edu<mailto:20160818123551.002ea...@uphs.upenn.edu>> Content-Type: text/plain; charset=US-ASCII Has anyone used the Aspera command-line client (for Linux) to script unattended downloads of HCP data and periodic updates? We've got the 2014 500-case release installed and would like to update that to the Dec 2015 900 case release (and supplement that with any additional data). I'm interested in doing the equivalent of 'rsync' via the Aspera client, and would appreciate any suggestions. Thanks, Mark -- Mark Bergman voice: 215-746-4061 mark.berg...@uphs.upenn.edu<mailto:mark.berg...@uphs.upenn.edu> fax: 215-614-0266 http://www.cbica.upenn.edu/ IT Technical Director, Center for Biomedical Image Computing and Analytics Department of Radiology University of Pennsylvania PGP Key: http://www.cbica.upenn.edu/sbia/bergman ------------------------------ _______________________________________________ HCP-Users mailing list HCP-Users@humanconnectome.org<mailto:HCP-Users@humanconnectome.org> http://lists.humanconnectome.org/mailman/listinfo/hcp-users _______________________________________________ HCP-Users mailing list HCP-Users@humanconnectome.org<mailto:HCP-Users@humanconnectome.org> http://lists.humanconnectome.org/mailman/listinfo/hcp-users ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ HCP-Users mailing list HCP-Users@humanconnectome.org http://lists.humanconnectome.org/mailman/listinfo/hcp-users
