Excellent point. In our case, we want S3 as a data mirror. I started to get into a discourse of backup issues, some of which should be solved with tape (and for which we maintain tape backups), and realized that it is far more complicated.
For digital archival files that would never be altered (although they could be superceded by new versions), I increasingly see having multiple live RAID arrays as much saner than tape, and possible to rig up sanely so that it isn't dependent on humans to remember to maintain the tapeloader and get the tapes to the offsite storage location. In an ideal world, we would probably network with two or more other archives, each located in a different part of the continent (or globe) and each would nightly (or weekly) update copies of each other's files. S3 is one way to get that first non-local copy, safe from local disasters) with relatively minimal fuss, with the added advantage that it comes with a good SLA and is part of a very large, well-tended datastore. On Nov 13, 2007 3:59 PM, Morgan, Matt <matt.morgan at metmuseum.org> wrote: > As far as I can tell, S3 is powerful online storage, not literal online > backup. You could use it as a data mirror, but it's not like saving the > contents of the server to multiple tape sets created over the course of > time > and storing them in multiple secure places. > > With S3 (and other online storage), any automated backup process could > irreversibly overwrite the "backup" copy with corrupted/truncated/deleted > versions of the original; you wouldn't have last week's tape set (or last > night's differential) to fall back on. That is, there's no redundancy, so > you can reach unrecoverable situations too easily. > > There are online services that provide the equivalent to full tape backup > with offsite storage, like > > http://www.usdatatrust.com/service/how_it_works.asp > > (the "historical archive" part is the key difference between them and S3). > These online backup services are not at all cheap (to back up just ten of > Ari's AVIs would cost $1000+/year!), but are probably not more expensive > than doing the same thing yourself, in many cases (you wouldn't have to > buy > or store tapes ...). > > --Matt > > On 11/6/07 8:23 AM, "Ari Davidow" <aridavidow at gmail.com> wrote: > > > There seems to be a burgeoning market in servers for consumer > purposes--ways > > to share video clips, photos, whatever. I think that places like > > rememble.com or mediamax seem to represent that paradigm. We have > looked, > > and continue to look, at setting up an alliance with other institutions > in > > geographically different areas from us for mutual file sharing. The only > > viable commercial service for needs like ours (hundreds of gigabytes or > even > > terabytes of files) seems to be Amazon's S3, which isn't cheap, but does > do > > a lot to ensure the integrity and security of data. Unfortunately, S3 > has a > > maximum file size of 5GB. With an hour of AVI clocking in around 13GB, > > that's a problem. > > > > As big a problem of backing up this data in a reasonably secure fashion > is, > > MOVING it is also difficult. It takes half an hour to move that 13GB > file > > around on our network. On our small business pipe to the internet? > > > > So, hoping to hear that others are exploring this area, or that they > have > > other solutions, > > ari > > > > On 11/5/07, Richard Urban <rjurban at uiuc.edu> wrote: > >> > >> Hi Ari, > >> > >> Sorry you won't make it to Chicago. I've been wondering about this > >> myself after a recent round of discussion on the RCAAM list - they > >> were debating the need for hard copy of collections records. Many of > >> the pro-paper set pointed to the easy loss of computer data during > >> disasters. I didn't ask, but I wondered where eveyone's back ups > >> were. > >> I think one of the things that came out of Katrina was that off-site > >> backups weren't enough and people needed to think regionally or > >> nationally. > >> > >> I do wonder about how these companies may be using the data stored on > >> their servers. I think there would be a number of concerns about > >> storing collections records and donor information with a third party, > >> even if they had clear data usage policies. > >> > >> The digital preservation project I worked on still relied on FedEx to > >> ship hard-drives back and forth as the fastest way to move large > >> files - even though we are at a major research university with many > >> fat tubes to the Interweb. > >> > >> Richard > >> rjurban at uiuc.edu > >> > >> > >> > >> On Nov 5, 2007, at 2:21 PM, Ari Davidow wrote: > >> > >>> So, we have all of these digital assets. They need to be backed up > >>> and they > >>> are huge. It looks as though online backup, using Amazon's S3 > >>> service, or a > >>> company such as MediaMax, is the way to go. It looks like S3 is really > >>> designed for companies that are doing more with their data than > >>> just parking > >>> it. That doesn't bother me, but it also looks like S3 is limited to > >>> files > >>> <5GB. We have AVI files that are closer to 15GB. MediaMax may have > >>> similar > >>> limits. > >>> > >>> In a worst case, I'd simply make plans to back up the working MPEGs > >>> (2-3GB > >>> each), and do my best to ensure that nothing bad happens to the tape > >>> originals of this video. But it occurs to me that people on this > >>> list may > >>> have more extensive experience. > >>> > >>> Who is using online storage? What works for you? What provider > >>> helps you > >>> sleep soundly (less unsoundly?) at night? What does it cost? Does > >>> anyone > >>> handle 15GB files? (We won't even talk about how long it would take to > >>> upload each one of those....) > >>> > >>> Thanks, > >>> (Think of this as my way of provoking conversation in a year when I > >>> can't > >>> get to MCN ;-).) > >>> ari > >>> _______________________________________________ > >>> You are currently subscribed to mcn-l, the listserv of the Museum > >>> Computer Network (http://www.mcn.edu) > >>> > >>> To post to this list, send messages to: mcn-l at mcn.edu > >>> > >>> To unsubscribe or change mcn-l delivery options visit: > >>> http://toronto.mediatrope.com/mailman/listinfo/mcn-l > >> > >> _______________________________________________ > >> You are currently subscribed to mcn-l, the listserv of the Museum > Computer > >> Network (http://www.mcn.edu) > >> > >> To post to this list, send messages to: mcn-l at mcn.edu > >> > >> To unsubscribe or change mcn-l delivery options visit: > >> http://toronto.mediatrope.com/mailman/listinfo/mcn-l > >> > > _______________________________________________ > > You are currently subscribed to mcn-l, the listserv of the Museum > Computer > > Network (http://www.mcn.edu) > > > > To post to this list, send messages to: mcn-l at mcn.edu > > > > To unsubscribe or change mcn-l delivery options visit: > > http://toronto.mediatrope.com/mailman/listinfo/mcn-l > > _______________________________________________ > You are currently subscribed to mcn-l, the listserv of the Museum Computer > Network (http://www.mcn.edu) > > To post to this list, send messages to: mcn-l at mcn.edu > > To unsubscribe or change mcn-l delivery options visit: > http://toronto.mediatrope.com/mailman/listinfo/mcn-l >
