[MCN-L] Advice About Cloud Storage
We are planning to use a cloud service to store large numbers of hi-res digital images for an online education platform. If any of you already use cloud services I'd be very grateful for recommendations or advice about what to look out for. Thank you, as always. snip, snip, snip? Background: Along with running My Tours where we store images and audio clips on S3 I also work for Realestate.co.nz where we store ~1.4TB of images and serve around 80GB of images a day to consumers. This is currently stored on block storage at a local 'cloudy' provider. We are currently in the process of selecting a vendor for cloud file storage and at the moment Amazon s3 with a Fastly CDN is our preferred option. On 19/04/2013, at 12:00 AM, mcn-l-request at mcn.edu wrote: We use Rackspace. They have a new part to their cloud offer, which I've not yet tried but which sounds helpful if you have large storage needs because it lets you buy what you need without scaling up the server as a whole. Cloud Blocks, I think it's called. Ah yes, here it is: http://www.rackspace.com/knowledge_center/article/cloud-block-storage-overview You have 2 basic types of cloud storage, object storage and block storage. Block storage is similar to your normal hard disks you would find in your computer or server. They 'look' just like a regular disk and perform the same. They tend to be faster than object storage and fit into traditional apps just like a normal disk would. Object storage is aimed at storing a large number of objects (images, audio, backups, and other assets). It is not a traditional file system in that you don't just copy files to a folder but use an API to put, requests and delete files. Both types of storage can scale but object based storage (Cloud Files from Rackspace and s3 from Amazon) is the right approach to storing large numbers of hi-res images. Another type of storage that may be also be of interest is Amazon's Glacier storage. For long term archival the price comes down to 1c/GB a month and is a great replacement for tape backups and long term storage that doesn't need to be accessed a lot. Amazon also hooks into a lot of other services such as the Amazon storage gateway - http://aws.amazon.com/storagegateway/). Worth looking at depending on your needs. And also don't forget a CDN - This helps deliver the assets out to the users as quickly as possible (s3 can be a little slow if you need fast access to the assets). Again Amazon makes it easy with CloudFront although in NZ we're looking at Fastly which has a local point of presence and has a really nice interface. We're using S3 for our collections images with a security policy to reserve the full-size versions. It wasn't a very researched decision except that we were already using AWS and like it. I'd be curious to hear from anyone who did more research and went another direction...? Nate I came out with Amazon as the top contender. Rackspace Cloud Files is a close second but I'm not sure they do signed requests for access to certain files. We use Azure for one project with a few thousand images. We took a close look at the charges for additions to the store as well as traffic charges - hosting the data is just one part of the cost. It worked for us, and has been reliable, but you need to do your sums. Azure - Good point. We didn't look at Azure purely because most of our services are in a Linux environment. If you are Windows then this might be a good option. (Blob Storage - http://www.windowsazure.com/en-us/home/features/data-management/) Feel free to ping me off list if you want to go into anything further - Happy to chat - glen at mytoursapp.com.
[MCN-L] Advice About Cloud Storage
Great summary of the difference between block object storage, thanks Glen! Nate On Thu, Apr 18, 2013 at 4:24 PM, My Tours glen at mytoursapp.com wrote: We are planning to use a cloud service to store large numbers of hi-res digital images for an online education platform. If any of you already use cloud services I'd be very grateful for recommendations or advice about what to look out for. Thank you, as always. snip, snip, snip? Background: Along with running My Tours where we store images and audio clips on S3 I also work for Realestate.co.nz where we store ~1.4TB of images and serve around 80GB of images a day to consumers. This is currently stored on block storage at a local 'cloudy' provider. We are currently in the process of selecting a vendor for cloud file storage and at the moment Amazon s3 with a Fastly CDN is our preferred option. On 19/04/2013, at 12:00 AM, mcn-l-request at mcn.edu wrote: We use Rackspace. They have a new part to their cloud offer, which I've not yet tried but which sounds helpful if you have large storage needs because it lets you buy what you need without scaling up the server as a whole. Cloud Blocks, I think it's called. Ah yes, here it is: http://www.rackspace.com/knowledge_center/article/cloud-block-storage-overview You have 2 basic types of cloud storage, object storage and block storage. Block storage is similar to your normal hard disks you would find in your computer or server. They 'look' just like a regular disk and perform the same. They tend to be faster than object storage and fit into traditional apps just like a normal disk would. Object storage is aimed at storing a large number of objects (images, audio, backups, and other assets). It is not a traditional file system in that you don't just copy files to a folder but use an API to put, requests and delete files. Both types of storage can scale but object based storage (Cloud Files from Rackspace and s3 from Amazon) is the right approach to storing large numbers of hi-res images. Another type of storage that may be also be of interest is Amazon's Glacier storage. For long term archival the price comes down to 1c/GB a month and is a great replacement for tape backups and long term storage that doesn't need to be accessed a lot. Amazon also hooks into a lot of other services such as the Amazon storage gateway - http://aws.amazon.com/storagegateway/). Worth looking at depending on your needs. And also don't forget a CDN - This helps deliver the assets out to the users as quickly as possible (s3 can be a little slow if you need fast access to the assets). Again Amazon makes it easy with CloudFront although in NZ we're looking at Fastly which has a local point of presence and has a really nice interface. We're using S3 for our collections images with a security policy to reserve the full-size versions. It wasn't a very researched decision except that we were already using AWS and like it. I'd be curious to hear from anyone who did more research and went another direction...? Nate I came out with Amazon as the top contender. Rackspace Cloud Files is a close second but I'm not sure they do signed requests for access to certain files. We use Azure for one project with a few thousand images. We took a close look at the charges for additions to the store as well as traffic charges - hosting the data is just one part of the cost. It worked for us, and has been reliable, but you need to do your sums. Azure - Good point. We didn't look at Azure purely because most of our services are in a Linux environment. If you are Windows then this might be a good option. (Blob Storage - http://www.windowsazure.com/en-us/home/features/data-management/) Feel free to ping me off list if you want to go into anything further - Happy to chat - glen at mytoursapp.com. ___ You are currently subscribed to mcn-l, the listserv of the Museum Computer Network (http://www.mcn.edu) To post to this list, send messages to: mcn-l at mcn.edu To unsubscribe or change mcn-l delivery options visit: http://mcn.edu/mailman/listinfo/mcn-l The MCN-L archives can be found at: http://toronto.mediatrope.com/pipermail/mcn-l/ -- Nate Solas Sr. New Media Developer and Head Technologist Walker Art Center 1750 Hennepin Ave MInneapolis, MN 55407 http://www.walkerart.org/
[MCN-L] Advice About Cloud Storage
Absolutely, thank you Glen. I'm wondering at the moment about high-res images intended for delivery through an IIPImage server. If we store large volumes of these we'll definitely want plenty of space at low cost, but they need to be in, effectively, local storage, so that the server can access them as part of the file system and then stream them out as tiles to users in real time. I guess for this scenario block storage is actually what we need? All the best, Jeremy Ottevanger Technical Web Manager Imperial War Museum Lambeth Road London SE1 6HZ -Original Message- From: mcn-l-bounces at mcn.edu [mailto:mcn-l-boun...@mcn.edu] On Behalf Of Nate Solas Sent: 19 April 2013 15:47 To: Museum Computer Network Listserv Subject: Re: [MCN-L] Advice About Cloud Storage Great summary of the difference between block object storage, thanks Glen! Nate On Thu, Apr 18, 2013 at 4:24 PM, My Tours glen at mytoursapp.com wrote: We are planning to use a cloud service to store large numbers of hi-res digital images for an online education platform. If any of you already use cloud services I'd be very grateful for recommendations or advice about what to look out for. Thank you, as always. snip, snip, snip... Background: Along with running My Tours where we store images and audio clips on S3 I also work for Realestate.co.nz where we store ~1.4TB of images and serve around 80GB of images a day to consumers. This is currently stored on block storage at a local 'cloudy' provider. We are currently in the process of selecting a vendor for cloud file storage and at the moment Amazon s3 with a Fastly CDN is our preferred option. On 19/04/2013, at 12:00 AM, mcn-l-request at mcn.edu wrote: We use Rackspace. They have a new part to their cloud offer, which I've not yet tried but which sounds helpful if you have large storage needs because it lets you buy what you need without scaling up the server as a whole. Cloud Blocks, I think it's called. Ah yes, here it is: http://www.rackspace.com/knowledge_center/article/cloud-block-storage- overview You have 2 basic types of cloud storage, object storage and block storage. Block storage is similar to your normal hard disks you would find in your computer or server. They 'look' just like a regular disk and perform the same. They tend to be faster than object storage and fit into traditional apps just like a normal disk would. Object storage is aimed at storing a large number of objects (images, audio, backups, and other assets). It is not a traditional file system in that you don't just copy files to a folder but use an API to put, requests and delete files. Both types of storage can scale but object based storage (Cloud Files from Rackspace and s3 from Amazon) is the right approach to storing large numbers of hi-res images. Another type of storage that may be also be of interest is Amazon's Glacier storage. For long term archival the price comes down to 1c/GB a month and is a great replacement for tape backups and long term storage that doesn't need to be accessed a lot. Amazon also hooks into a lot of other services such as the Amazon storage gateway - http://aws.amazon.com/storagegateway/). Worth looking at depending on your needs. And also don't forget a CDN - This helps deliver the assets out to the users as quickly as possible (s3 can be a little slow if you need fast access to the assets). Again Amazon makes it easy with CloudFront although in NZ we're looking at Fastly which has a local point of presence and has a really nice interface. - This email message has been delivered safely and archived online by Mimecast. For more information please visit http://www.mimecast.com -
[MCN-L] Advice About Cloud Storage
Sorry for the multiple emails, but I need to clarify something: IIP wasn't stripping color profiles, we'd erroneously not applied them when we originally generated the PTIFFS. So our choice to correct the zoomable images was heavy either way: re-generate the PTIFFS with the ICC profile, or cut tiles with a script. I'm planning a blog post about the process once we're done. Because this is such an easy mistake to make and not something most web teams are familiar with checking, I suspect there are a great number of institutions out there with (sometimes majorly) incorrect colors in their online artwork... (As we were for YEARS. Yikes!) Nate On Fri, Apr 19, 2013 at 11:27 AM, Nate Solas nate.solas at walkerart.orgwrote: Jumping back as it's currently 4:00 am in NZ and I'm sure Glen is sleeping but I want to keep this conversation going... We have actually just moved away from IIPImage, for some of the reasons you hint at. We ended up pre-processing tiles and putting them out on S3 and serving to the end user via Cloudfront CDN. This offered us a few advantages: custom image compression, color profiles applied correctly (I think IIP was losing them!), and actually leveraging a CDN for parallel downloads. It's (a lot) more work up front to cut the tiles rather than doing it dynamically, but it's automated and working well. IMO the only way to run IIPImage in the cloud is to use EC2 as an IIP server (we actually had this set up for a while) so you don't get hit with all the bandwidth of pulling the ptiff each time. It does work, but for us the dealbreakers were the lack of control over the final jpegs and the inability to spread the requests across multiple CDN domains. (although that's solvable). I don't think you could possibly have the full ptiffs on the cloud and serve the tiles locally, the overhead and bandwidth costs would kill you. Nate On Fri, Apr 19, 2013 at 10:14 AM, Jeremy Ottevanger JOttevanger at iwm.org.uk wrote: Absolutely, thank you Glen. I'm wondering at the moment about high-res images intended for delivery through an IIPImage server. If we store large volumes of these we'll definitely want plenty of space at low cost, but they need to be in, effectively, local storage, so that the server can access them as part of the file system and then stream them out as tiles to users in real time. I guess for this scenario block storage is actually what we need? All the best, Jeremy Ottevanger Technical Web Manager Imperial War Museum Lambeth Road London SE1 6HZ -Original Message- From: mcn-l-bounces at mcn.edu [mailto:mcn-l-bounces at mcn.edu] On Behalf Of Nate Solas Sent: 19 April 2013 15:47 To: Museum Computer Network Listserv Subject: Re: [MCN-L] Advice About Cloud Storage Great summary of the difference between block object storage, thanks Glen! Nate On Thu, Apr 18, 2013 at 4:24 PM, My Tours glen at mytoursapp.com wrote: We are planning to use a cloud service to store large numbers of hi-res digital images for an online education platform. If any of you already use cloud services I'd be very grateful for recommendations or advice about what to look out for. Thank you, as always. snip, snip, snip... Background: Along with running My Tours where we store images and audio clips on S3 I also work for Realestate.co.nz where we store ~1.4TB of images and serve around 80GB of images a day to consumers. This is currently stored on block storage at a local 'cloudy' provider. We are currently in the process of selecting a vendor for cloud file storage and at the moment Amazon s3 with a Fastly CDN is our preferred option. On 19/04/2013, at 12:00 AM, mcn-l-request at mcn.edu wrote: We use Rackspace. They have a new part to their cloud offer, which I've not yet tried but which sounds helpful if you have large storage needs because it lets you buy what you need without scaling up the server as a whole. Cloud Blocks, I think it's called. Ah yes, here it is: http://www.rackspace.com/knowledge_center/article/cloud-block-storage- overview You have 2 basic types of cloud storage, object storage and block storage. Block storage is similar to your normal hard disks you would find in your computer or server. They 'look' just like a regular disk and perform the same. They tend to be faster than object storage and fit into traditional apps just like a normal disk would. Object storage is aimed at storing a large number of objects (images, audio, backups, and other assets). It is not a traditional file system in that you don't just copy files to a folder but use an API to put, requests and delete files. Both types of storage can scale but object based storage (Cloud Files from Rackspace and s3 from Amazon) is the right approach to storing large numbers of hi-res images. Another type of storage that may be also be of interest is Amazon's Glacier
[MCN-L] Advice About Cloud Storage
We use Rackspace. They have a new part to their cloud offer, which I've not yet tried but which sounds helpful if you have large storage needs because it lets you buy what you need without scaling up the server as a whole. Cloud Blocks, I think it's called. Ah yes, here it is: http://www.rackspace.com/knowledge_center/article/cloud-block-storage-overview All the best, Jeremy Jeremy Ottevanger Technical Web Manager Imperial War Museum Lambeth Road London SE1 6HZ -Original Message- From: mcn-l-bounces at mcn.edu [mailto:mcn-l-boun...@mcn.edu] On Behalf Of Robin White Owen Sent: 17 April 2013 16:23 To: mcn-l at mcn.edu Subject: [MCN-L] Advice About Cloud Storage Hello List, We are planning to use a cloud service to store large numbers of hi-res digital images for an online education platform. If any of you already use cloud services I'd be very grateful for recommendations or advice about what to look out for. Thank you, as always. Robin Robin White Owen M: 917/407-7641 T: 646/472-5145 robin at mediacombo.net www.mediacombo.net http://mediacombo.net/blog twitter.com/rocombo - This email message has been delivered safely and archived online by Mimecast. For more information please visit http://www.mimecast.com -
[MCN-L] Advice About Cloud Storage
We're using S3 for our collections images with a security policy to reserve the full-size versions. It wasn't a very researched decision except that we were already using AWS and like it. I'd be curious to hear from anyone who did more research and went another direction...? Nate On Wed, Apr 17, 2013 at 10:23 AM, Robin White Owen robin at mediacombo.netwrote: Hello List, We are planning to use a cloud service to store large numbers of hi-res digital images for an online education platform. If any of you already use cloud services I'd be very grateful for recommendations or advice about what to look out for. Thank you, as always. Robin Robin White Owen M: 917/407-7641 T: 646/472-5145 robin at mediacombo.net www.mediacombo.net http://mediacombo.net/blog twitter.com/rocombo ___ You are currently subscribed to mcn-l, the listserv of the Museum Computer Network (http://www.mcn.edu) To post to this list, send messages to: mcn-l at mcn.edu To unsubscribe or change mcn-l delivery options visit: http://mcn.edu/mailman/listinfo/mcn-l The MCN-L archives can be found at: http://toronto.mediatrope.com/pipermail/mcn-l/
[MCN-L] Advice About Cloud Storage
We use Azure for one project with a few thousand images. We took a close look at the charges for additions to the store as well as traffic charges - hosting the data is just one part of the cost. It worked for us, and has been reliable, but you need to do your sums. Michael = Michael Stocking Managing Director Armadillo Systems 106 Cleveland Street London W1T 6NX +44 (0)20 7388 8757 michael at armadillosystems.com www.armadillosystems.com www.inquireresearch.co.uk www.ebooktreasures.org www.turningthepages.com http://digitalcultureonline.blogspot.com/ On 17 Apr 2013, at 16:27, Jeremy Ottevanger JOttevanger at iwm.org.uk wrote: We use Rackspace. They have a new part to their cloud offer, which I've not yet tried but which sounds helpful if you have large storage needs because it lets you buy what you need without scaling up the server as a whole. Cloud Blocks, I think it's called. Ah yes, here it is: http://www.rackspace.com/knowledge_center/article/cloud-block-storage-overview All the best, Jeremy Jeremy Ottevanger Technical Web Manager Imperial War Museum Lambeth Road London SE1 6HZ -Original Message- From: mcn-l-bounces at mcn.edu [mailto:mcn-l-bounces at mcn.edu] On Behalf Of Robin White Owen Sent: 17 April 2013 16:23 To: mcn-l at mcn.edu Subject: [MCN-L] Advice About Cloud Storage Hello List, We are planning to use a cloud service to store large numbers of hi-res digital images for an online education platform. If any of you already use cloud services I'd be very grateful for recommendations or advice about what to look out for. Thank you, as always. Robin Robin White Owen M: 917/407-7641 T: 646/472-5145 robin at mediacombo.net www.mediacombo.net http://mediacombo.net/blog twitter.com/rocombo - This email message has been delivered safely and archived online by Mimecast. For more information please visit http://www.mimecast.com - ___ You are currently subscribed to mcn-l, the listserv of the Museum Computer Network (http://www.mcn.edu) To post to this list, send messages to: mcn-l at mcn.edu To unsubscribe or change mcn-l delivery options visit: http://mcn.edu/mailman/listinfo/mcn-l The MCN-L archives can be found at: http://toronto.mediatrope.com/pipermail/mcn-l/
