[MCN-L] Offsite Digital Image Archive Options
I am writing to ask if any of you -- like us here at the Folger Shakespeare Library -- are at small institutions and without major digital asset management or IT infrastructures but nevertheless are engaged in active digital imaging? If so, are you willing to talk (offlist or on) about your backup / archiving schemes? We are working on establishing scalable systems architecture and backup strategies for digital images of collection materials, and would love to compare notes with others who have some version of a 2-3 tier backup strategy. Our current situation is this: For each digital image of collection material, our Photography and Digital Imaging lab produces a minimum of two images: a ca. 100-120 mb unretouched master, and a ca. 80-100 mb cropped color-corrected derivative. We are looking for a solution that will permit us to archive the masters offsite. We're currently running tape backups and taking them to a staff member's house. However, tapes sitting on the bookshelf in a Folger staff member's house isn't good enough anymore. What we'll need is at least 1.5 - 2 TB of space. This can be a dark archive because we won't need frequent access, though infrequent access would be necessary. I looked at the MCN-L archives and found one thread from November, in which some spoke of Amazon S3. Any thoughts on this, or a different service that's cost-effective? With thanks in advance, Mary Bloodworth Head of Information Services Folger Shakespeare Library
[MCN-L] Offsite Digital Image Archive Options
A thought: ...When deciding to go with an online solution, did you factor in bandwidth costs? It looks like you may be moving a fair amount of data around. May or may not be a problem depending on volume and the terms of your internet connection package. Should be checked though. I'd probably be looking at something simple and cheap involving SATA drives in removable draws or maybe several NAS appliances. I'd have one big working directory tree (500GB+?) and have something like Microsoft's free Robocopy tool scan it daily and move everything that hadn't been touched for 3 months (or whatever) to a second onsite archive drive. Maybe periodically mirror that to third, which you swap weekly with a fourth in a bank size II safety deposit box ($70 a year?). Just thinking out loud. I'd be happy to talk about this further off list if that's useful. An inexpensive, simple, archive solution would be good for a lot of small orgs with little IT. David === David Marsh Chief Technician System Administrator H.R. MacMillan Space Centre 1100 Chestnut Street, Vancouver, BC V6J 3J9 E sysadmin at hrmacmillanspacecentre.com T (604) 738 7827 ext. 229 C (604) 813 9667 F (604) 736 5665 === For your next special event or meeting, consider the unique atmosphere of the H.R. MacMillan Space Centre. For more information go to www.spacecentre.ca or call (604) 738-7827 (ext 233) P Please consider the environment before printing e-mails -Original Message- From: mcn-l-bounces at mcn.edu [mailto:mcn-l-boun...@mcn.edu] On Behalf Of Mary Bloodworth Sent: Thursday, September 11, 2008 12:42 PM To: mcn-l at mcn.edu Subject: [MCN-L] Offsite Digital Image Archive Options I am writing to ask if any of you -- like us here at the Folger Shakespeare Library -- are at small institutions and without major digital asset management or IT infrastructures but nevertheless are engaged in active digital imaging? If so, are you willing to talk (offlist or on) about your backup / archiving schemes? We are working on establishing scalable systems architecture and backup strategies for digital images of collection materials, and would love to compare notes with others who have some version of a 2-3 tier backup strategy. Our current situation is this: For each digital image of collection material, our Photography and Digital Imaging lab produces a minimum of two images: a ca. 100-120 mb unretouched master, and a ca. 80-100 mb cropped color-corrected derivative. We are looking for a solution that will permit us to archive the masters offsite. We're currently running tape backups and taking them to a staff member's house. However, tapes sitting on the bookshelf in a Folger staff member's house isn't good enough anymore. What we'll need is at least 1.5 - 2 TB of space. This can be a dark archive because we won't need frequent access, though infrequent access would be necessary. I looked at the MCN-L archives and found one thread from November, in which some spoke of Amazon S3. Any thoughts on this, or a different service that's cost-effective? With thanks in advance, Mary Bloodworth Head of Information Services Folger Shakespeare Library ___ You are currently subscribed to mcn-l, the listserv of the Museum Computer Network (http://www.mcn.edu) To post to this list, send messages to: mcn-l at mcn.edu To unsubscribe or change mcn-l delivery options visit: http://toronto.mediatrope.com/mailman/listinfo/mcn-l
[MCN-L] Offsite Digital Image Archive Options
We're probably the folks who brought up S3 back when. We have a RAID server in-house that will ultimately hold about 6GB (which is enough for current projects once we get them all in manageable, preservable forms) and are backing that up to S3. My experience with removable media and/or optical media is sufficiently negative that our policy is first back up to RAID inhouse, then S3, then LOCKSS with other media as convenient (in our case, probably DVD and minimal tape). We are also using this with Subversion for the accompanying documentation (transcripts, interview logs, metadata descriptions), etc. I'll be presenting on the subject at the upcoming conference in Washington, DC--do come by! We are currently working on a minimalist Fedora install that is going to be based on Amazon's services so that we never have to purchase (or maintain) yet another server for this purpose. The bandwidth/usage charges are not s cheap, but the result, at least in this phase, is a server that is easily accessible, easily secured, easily re-imaged separate from the server content (that archive on S3 and other online services) and for which we don't have to write useful specs at a time when we're not sure what the real requirements will be once things (if things) stabilize to where a commodity physical solution (and its upkeep) is more economical. I am hoping to avoid ever again to be putting in a hard disk that has been sitting around for six months (or six months) hoping to find a way to get it unstuck and working, or hoping that the last copy of the three we had in separate places on separate optical media will work. But we're at least a year off having enough of this implemented in enough places that I'll be able to sleep easily on this account. I may change my tune by then ;-). ari On Thu, Sep 11, 2008 at 3:42 PM, Mary Bloodworth MBLOODWORTH at folger.eduwrote: I am writing to ask if any of you -- like us here at the Folger Shakespeare Library -- are at small institutions and without major digital asset management or IT infrastructures but nevertheless are engaged in active digital imaging? If so, are you willing to talk (offlist or on) about your backup / archiving schemes? We are working on establishing scalable systems architecture and backup strategies for digital images of collection materials, and would love to compare notes with others who have some version of a 2-3 tier backup strategy. Our current situation is this: For each digital image of collection material, our Photography and Digital Imaging lab produces a minimum of two images: a ca. 100-120 mb unretouched master, and a ca. 80-100 mb cropped color-corrected derivative. We are looking for a solution that will permit us to archive the masters offsite. We're currently running tape backups and taking them to a staff member's house. However, tapes sitting on the bookshelf in a Folger staff member's house isn't good enough anymore. What we'll need is at least 1.5 - 2 TB of space. This can be a dark archive because we won't need frequent access, though infrequent access would be necessary. I looked at the MCN-L archives and found one thread from November, in which some spoke of Amazon S3. Any thoughts on this, or a different service that's cost-effective? With thanks in advance, Mary Bloodworth Head of Information Services Folger Shakespeare Library ___ You are currently subscribed to mcn-l, the listserv of the Museum Computer Network (http://www.mcn.edu) To post to this list, send messages to: mcn-l at mcn.edu To unsubscribe or change mcn-l delivery options visit: http://toronto.mediatrope.com/mailman/listinfo/mcn-l
[MCN-L] Offsite Digital Image Archive Options
We have three different methods and currently around 300Gb worth of assets: 1) our RAID server helps in case of hard drive failure (so far, so good!). Through a weird quirk of fate, we now have two different RAID arrays, for a combined total of about 1.25 Tb. 2) We just started backing up to a 1 Tb disk. Our IT consultant comes in weekly and swaps it out, putting it into a fireproof safe which is moved offsite to a data storage facility. We have three disks total. 3) I also backup the assets to San Diego Supercomputing Center, with whom we have an arrangement. I manually perform those backups, but because our infrastructure is sloow, it takes me approximately 3 weeks to send all 300G to SDSC. Fortunately, I rarely need to do this, since I normally only send the new or modified assets. But I just completely changed our directory structure, so I'm doing a fresh upload. Only 24 more hours to go! (the moral being, if backing up through offsite uploads, make sure your assets and file structures are consistent and in the format you want it, and you have a nice, fat, internet connection) SDSC does, I believe, offer storage for humanities institutions. There is usually a cost associated with this, though if the project is of particular interest to them, they will sometimes award free allocations: http://www.sdsc.edu/resources/Resources.html Perian Sully Collection Information and New Media Coordinator Judah L. Magnes Museum -Original Message- From: mcn-l-bounces at mcn.edu [mailto:mcn-l-boun...@mcn.edu] On Behalf Of Mary Bloodworth Sent: Thursday, September 11, 2008 12:42 PM To: mcn-l at mcn.edu Subject: [MCN-L] Offsite Digital Image Archive Options I am writing to ask if any of you -- like us here at the Folger Shakespeare Library -- are at small institutions and without major digital asset management or IT infrastructures but nevertheless are engaged in active digital imaging? If so, are you willing to talk (offlist or on) about your backup / archiving schemes? We are working on establishing scalable systems architecture and backup strategies for digital images of collection materials, and would love to compare notes with others who have some version of a 2-3 tier backup strategy. Our current situation is this: For each digital image of collection material, our Photography and Digital Imaging lab produces a minimum of two images: a ca. 100-120 mb unretouched master, and a ca. 80-100 mb cropped color-corrected derivative. We are looking for a solution that will permit us to archive the masters offsite. We're currently running tape backups and taking them to a staff member's house. However, tapes sitting on the bookshelf in a Folger staff member's house isn't good enough anymore. What we'll need is at least 1.5 - 2 TB of space. This can be a dark archive because we won't need frequent access, though infrequent access would be necessary. I looked at the MCN-L archives and found one thread from November, in which some spoke of Amazon S3. Any thoughts on this, or a different service that's cost-effective? With thanks in advance, Mary Bloodworth Head of Information Services Folger Shakespeare Library ___ You are currently subscribed to mcn-l, the listserv of the Museum Computer Network (http://www.mcn.edu) To post to this list, send messages to: mcn-l at mcn.edu To unsubscribe or change mcn-l delivery options visit: http://toronto.mediatrope.com/mailman/listinfo/mcn-l
[MCN-L] Offsite Digital Image Archive Options
Mary: We have greatly simplified our backup/archiving strategy by switching from storing our master images from the TIFF format to JPEG2000. The latter are much smalled and are thus easier to archive to optical media. Once a digitization project is complete we delete the TIFF images. We now are able to store copies of them in more places than before because they are so much smaller. We can keep lots of copies without stressing the the storage devices and tape backups. I was very nervous when I deleted my fist batch of TIFFs, but I've gotten over it. I hope I don't live to regret this irreversible action. Peter Mary Bloodworth wrote: I am writing to ask if any of you -- like us here at the Folger Shakespeare Library -- are at small institutions and without major digital asset management or IT infrastructures but nevertheless are engaged in active digital imaging? If so, are you willing to talk (offlist or on) about your backup / archiving schemes? We are working on establishing scalable systems architecture and backup strategies for digital images of collection materials, and would love to compare notes with others who have some version of a 2-3 tier backup strategy. Our current situation is this: For each digital image of collection material, our Photography and Digital Imaging lab produces a minimum of two images: a ca. 100-120 mb unretouched master, and a ca. 80-100 mb cropped color-corrected derivative. We are looking for a solution that will permit us to archive the masters offsite. We're currently running tape backups and taking them to a staff member's house. However, tapes sitting on the bookshelf in a Folger staff member's house isn't good enough anymore. What we'll need is at least 1.5 - 2 TB of space. This can be a dark archive because we won't need frequent access, though infrequent access would be necessary. I looked at the MCN-L archives and found one thread from November, in which some spoke of Amazon S3. Any thoughts on this, or a different service that's cost-effective? With thanks in advance, Mary Bloodworth Head of Information Services Folger Shakespeare Library ___ You are currently subscribed to mcn-l, the listserv of the Museum Computer Network (http://www.mcn.edu) To post to this list, send messages to: mcn-l at mcn.edu To unsubscribe or change mcn-l delivery options visit: http://toronto.mediatrope.com/mailman/listinfo/mcn-l -- */Peter MacDonald/* Library Information Systems Specialist Hamilton College Library pmacdona at hamilton.edu 315 859-4493
