Re: [ccp4bb] Open Access Repositories for Big Data?

2019-01-18 Thread John R Helliwell 
Dear Aaron,
I think that the Zenodo limit is as their email to you states, per dataset 
cited from an article ie equals one doi. I recall that at International Data 
Week in Denver 2016 I mentioned in open discussion at the session on data 
repositories the zenodo limit per dataset of 5 Gbytes and that 50 Gbytes would 
be much better and eg thereby allow for a time resolved diffraction sequence of 
datasets. One week later it had increased from 5 to 50 Gbytes per data set. 
Greetings,
John 
Emeritus Professor John R Helliwell DSc



> On 18 Jan 2019, at 16:41, Aaron Finke  wrote:
> 
> This is what Zenodo emailed me: "By default, we provide a one-time quota 
> increase up to 100GB for a dataset that will be cited from a peer-reviewed 
> article. Zenodo is a free-to-use service, an in order to keep it this way, we 
> have to restrict the incoming data volume rate as very large datasets 
> contribute significantly to the overall data volume in Zenodo. Unfortunately, 
> at this point, we also cannot receive payment for quota increases, though we 
> do hope that this will be possible in the future, at which point we will 
> announce this possibility.”
> 
> I suppose I could split the dataset over multiple DOIs but it feels like 
> “cheating the system” a bit. The CXIDB sounds promising, though!
> 
> Thanks,
> Aaron
> 
> --
> Aaron Finke
> Staff Scientist, MacCHESS
> Cornell University
> e-mail: af...@cornell.edu
> 
>> On Jan 18, 2019, at 11:16 AM, Herbert J. Bernstein  wrote:
>> 
>> The zenodo policies seem to the most workable as a start.  I would suggest 
>> contacting them for the cases that go over 50GB, but at worst splitting into 
>> 50GB chunks.  -- Herbert
>> 
>>> On Fri, Jan 18, 2019 at 10:49 AM Andreas Förster 
>>>  wrote:
>>> Hi Aaron,
>>> 
>>> can you slice your data and then link to the bits?
>>> 
>>> We're currently trying to find out what "unlimited Google Drive storage" 
>>> means by uploading pi in chunks of 70 GB or so.
>>> 
>>> All best.
>>> 
>>> 
>>> Andreas
>>> 
>>> 
>>> 
 On Fri, Jan 18, 2019 at 4:31 PM Aaron Finke  wrote:
 Dear CCP4ites,
 
 Is anyone aware of online repositories that will store huge sets of raw 
 data (>100 GB)? I’m aware of Zenodo and SBGrid, but Zenodo’s limit is 
 typically 50 GB and their absolute limit is 100 GB. SBGrid has yet to 
 respond to my emails.
 
 I could host them myself, but the involuntary dry heaving response I got 
 when I brought up the idea to our IT department implied they were less 
 enthused with the idea than I was. So a cloud service would be far more 
 preferable as a long term solution.
 
 Thanks,
 Aaron
 
 --
 Aaron Finke
 Staff Scientist, MacCHESS
 Cornell University
 e-mail: af...@cornell.edu
 
 
 To unsubscribe from the CCP4BB list, click the following link:
 https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1
 
>>> 
>>> 
>>> -- 
>>> Andreas Förster, Ph.D.
>>> Application Scientist Crystallography, Scientific Sales
>>> Phone: +41 56 500 21 00 | Direct: +41 56 500 21 76 | Email: 
>>> andreas.foers...@dectris.com
>>> DECTRIS Ltd. | Taefernweg 1 | 5405 Baden-Daettwil | Switzerland | 
>>> www.dectris.com
>>> 
>>> 
>>> 
>>>  
>>> 
>>> 
>>> Confidentiality Note: This message is intended only for the use of the 
>>> named recipient(s)
>>> and may contain confidential and/or privileged information. If you are not 
>>> the intended
>>> recipient, please contact the sender and delete the message. Any 
>>> unauthorized use of
>>> the information contained in this message is prohibited.
>>> 
>>> 
>>> 
>>> To unsubscribe from the CCP4BB list, click the following link:
>>> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1
>>> 
>> 
>> To unsubscribe from the CCP4BB list, click the following link:
>> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1
>> 
> 
> 
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1

Re: [ccp4bb] Open Access Repositories for Big Data?

2019-01-18 Thread Aaron Finke 
This is what Zenodo emailed me: "By default, we provide a one-time quota 
increase up to 100GB for a dataset that will be cited from a peer-reviewed 
article. Zenodo is a free-to-use service, an in order to keep it this way, we 
have to restrict the incoming data volume rate as very large datasets 
contribute significantly to the overall data volume in Zenodo. Unfortunately, 
at this point, we also cannot receive payment for quota increases, though we do 
hope that this will be possible in the future, at which point we will announce 
this possibility.”

I suppose I could split the dataset over multiple DOIs but it feels like 
“cheating the system” a bit. The CXIDB sounds promising, though!

Thanks,
Aaron

--
Aaron Finke
Staff Scientist, MacCHESS
Cornell University
e-mail: af...@cornell.edu

On Jan 18, 2019, at 11:16 AM, Herbert J. Bernstein 
mailto:yaya...@gmail.com>> wrote:

The zenodo policies seem to the most workable as a start.  I would suggest 
contacting them for the cases that go over 50GB, but at worst splitting into 
50GB chunks.  -- Herbert

On Fri, Jan 18, 2019 at 10:49 AM Andreas Förster 
mailto:andreas.foers...@dectris.com>> wrote:
Hi Aaron,

can you slice your data and then link to the bits?

We're currently trying to find out what "unlimited Google Drive storage" means 
by uploading pi in chunks of 70 GB or so.

All best.


Andreas



On Fri, Jan 18, 2019 at 4:31 PM Aaron Finke 
mailto:af...@cornell.edu>> wrote:
Dear CCP4ites,

Is anyone aware of online repositories that will store huge sets of raw data 
(>100 GB)? I’m aware of Zenodo and SBGrid, but Zenodo’s limit is typically 50 
GB and their absolute limit is 100 GB. SBGrid has yet to respond to my emails.

I could host them myself, but the involuntary dry heaving response I got when I 
brought up the idea to our IT department implied they were less enthused with 
the idea than I was. So a cloud service would be far more preferable as a long 
term solution.

Thanks,
Aaron

--
Aaron Finke
Staff Scientist, MacCHESS
Cornell University
e-mail: af...@cornell.edu




To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1


--

Andreas Förster, Ph.D.
Application Scientist Crystallography, Scientific Sales
Phone: +41 56 500 21 00 | Direct: +41 56 500 21 76 | Email: 
andreas.foers...@dectris.com
DECTRIS Ltd. | Taefernweg 1 | 5405 Baden-Daettwil | Switzerland | 
www.dectris.com


[https://www.dectris.com/files/content/images/signatur/logo_signatur.png]

[LinkedIn]
[facebook][https://www.dectris.com/files/content/images/signatur/twitter_20px.png]

Confidentiality Note: This message is intended only for the use of the named 
recipient(s)
and may contain confidential and/or privileged information. If you are not the 
intended
recipient, please contact the sender and delete the message. Any unauthorized 
use of
the information contained in this message is prohibited.





To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1




To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1

Re: [ccp4bb] Open Access Repositories for Big Data?

2019-01-18 Thread Herbert J. Bernstein 
The zenodo policies seem to the most workable as a start.  I would suggest
contacting them for the cases that go over 50GB, but at worst splitting
into 50GB chunks.  -- Herbert

On Fri, Jan 18, 2019 at 10:49 AM Andreas Förster <
andreas.foers...@dectris.com> wrote:

> Hi Aaron,
>
> can you slice your data and then link to the bits?
>
> We're currently trying to find out what "unlimited Google Drive storage"
> means by uploading pi in chunks of 70 GB or so.
>
> All best.
>
>
> Andreas
>
>
>
> On Fri, Jan 18, 2019 at 4:31 PM Aaron Finke  wrote:
>
>> Dear CCP4ites,
>>
>> Is anyone aware of online repositories that will store huge sets of raw
>> data (>100 GB)? I’m aware of Zenodo and SBGrid, but Zenodo’s limit is
>> typically 50 GB and their absolute limit is 100 GB. SBGrid has yet to
>> respond to my emails.
>>
>> I could host them myself, but the involuntary dry heaving response I got
>> when I brought up the idea to our IT department implied they were less
>> enthused with the idea than I was. So a cloud service would be far more
>> preferable as a long term solution.
>>
>> Thanks,
>> Aaron
>>
>> --
>> Aaron Finke
>> Staff Scientist, MacCHESS
>> Cornell University
>> e-mail: af...@cornell.edu
>>
>>
>> --
>>
>> To unsubscribe from the CCP4BB list, click the following link:
>> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1
>>
>
>
> --
> 
> Andreas Förster, Ph.D.
> Application Scientist Crystallography, Scientific Sales
> Phone: +41 56 500 21 00 | Direct: +41 56 500 21 76 | Email:
> andreas.foers...@dectris.com
> DECTRIS Ltd. | Taefernweg 1 | 5405 Baden-Daettwil | Switzerland |
> www.dectris.com
>
>
>
> 
> 
> 
> 
> [image: LinkedIn]
> 
> 
> 
> 
> 
> 
> [image:
> facebook] 
>  
> 
>
> *Confidentiality Note: This message is intended only for the use of the
> named recipient(s)*
> *and may contain confidential and/or privileged information. If you are
> not the intended*
> *recipient, please contact the sender and delete the message. Any
> unauthorized use of*
> *the information contained in this message is prohibited.*
>
>
>
> --
>
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1
>



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1

Re: [ccp4bb] Open Access Repositories for Big Data?

2019-01-18 Thread Thomas White 
Hi,

> Is anyone aware of online repositories that will store huge sets of
> raw data (>100 GB)? I’m aware of Zenodo and SBGrid, but Zenodo’s
> limit is typically 50 GB and their absolute limit is 100 GB. SBGrid
> has yet to respond to my emails.

The Coherent X-ray Imaging Data Bank may be appropriate:
http://cxidb.org

The definition of "imaging" is taken quite widely, e.g. the data
there includes many serial crystallography diffraction data sets.  A
lot of them are MUCH bigger than 100 GB.

Tom

-- 
Thomas White  
4E1F C14D 0E0A A014 FE5D 3FC6 C628 75D1 D4CA 4C30
Direct telephone: +49 (0)40 8998-5786



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1

Re: [ccp4bb] Open Access Repositories for Big Data?

2019-01-18 Thread Andreas Förster 
Hi Aaron,

can you slice your data and then link to the bits?

We're currently trying to find out what "unlimited Google Drive storage"
means by uploading pi in chunks of 70 GB or so.

All best.


Andreas



On Fri, Jan 18, 2019 at 4:31 PM Aaron Finke  wrote:

> Dear CCP4ites,
>
> Is anyone aware of online repositories that will store huge sets of raw
> data (>100 GB)? I’m aware of Zenodo and SBGrid, but Zenodo’s limit is
> typically 50 GB and their absolute limit is 100 GB. SBGrid has yet to
> respond to my emails.
>
> I could host them myself, but the involuntary dry heaving response I got
> when I brought up the idea to our IT department implied they were less
> enthused with the idea than I was. So a cloud service would be far more
> preferable as a long term solution.
>
> Thanks,
> Aaron
>
> --
> Aaron Finke
> Staff Scientist, MacCHESS
> Cornell University
> e-mail: af...@cornell.edu
>
>
> --
>
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1
>


-- 

Andreas Förster, Ph.D.
Application Scientist Crystallography, Scientific Sales
Phone: +41 56 500 21 00 | Direct: +41 56 500 21 76 | Email:
andreas.foers...@dectris.com
DECTRIS Ltd. | Taefernweg 1 | 5405 Baden-Daettwil | Switzerland |
www.dectris.com







[image: LinkedIn]






[image:
facebook] 
 


*Confidentiality Note: This message is intended only for the use of the
named recipient(s)*
*and may contain confidential and/or privileged information. If you are not
the intended*
*recipient, please contact the sender and delete the message. Any
unauthorized use of*
*the information contained in this message is prohibited.*



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1

Re: [ccp4bb] Open Access Repositories for Big Data?

2019-01-18 Thread graeme.win...@diamond.ac.uk 
Hi Aaron

I would guess most places would start to want $$ for storing multiples of 100 GB

Google, Amazon, Microsoft all offer this kind of thing. Getting the data in and 
out can be slow, and I would expect as the data size tends towards big and the 
time tends to a long time it would be comparable to doing it yourself in terms 
of cost

Unless you can tag yourself onto a CERN Tier1 
http://wlcg-public.web.cern.ch/tier-centres

Cheers Graeme

On 18 Jan 2019, at 15:31, Aaron Finke 
mailto:af...@cornell.edu>> wrote:

Dear CCP4ites,

Is anyone aware of online repositories that will store huge sets of raw data 
(>100 GB)? I’m aware of Zenodo and SBGrid, but Zenodo’s limit is typically 50 
GB and their absolute limit is 100 GB. SBGrid has yet to respond to my emails.

I could host them myself, but the involuntary dry heaving response I got when I 
brought up the idea to our IT department implied they were less enthused with 
the idea than I was. So a cloud service would be far more preferable as a long 
term solution.

Thanks,
Aaron

--
Aaron Finke
Staff Scientist, MacCHESS
Cornell University
e-mail: af...@cornell.edu




To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1


-- 
This e-mail and any attachments may contain confidential, copyright and or 
privileged material, and are for the use of the intended addressee only. If you 
are not the intended addressee or an authorised recipient of the addressee 
please notify us of receipt by returning the e-mail and do not use, copy, 
retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not 
necessarily of Diamond Light Source Ltd. 
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments 
are free from viruses and we cannot accept liability for any damage which you 
may sustain as a result of software viruses which may be transmitted in or with 
the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and 
Wales with its registered office at Diamond House, Harwell Science and 
Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom




To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1

[ccp4bb] Open Access Repositories for Big Data?

2019-01-18 Thread Aaron Finke 
Dear CCP4ites,

Is anyone aware of online repositories that will store huge sets of raw data 
(>100 GB)? I’m aware of Zenodo and SBGrid, but Zenodo’s limit is typically 50 
GB and their absolute limit is 100 GB. SBGrid has yet to respond to my emails.

I could host them myself, but the involuntary dry heaving response I got when I 
brought up the idea to our IT department implied they were less enthused with 
the idea than I was. So a cloud service would be far more preferable as a long 
term solution.

Thanks,
Aaron

--
Aaron Finke
Staff Scientist, MacCHESS
Cornell University
e-mail: af...@cornell.edu




To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1