Hi, I would personally recommend against AWS S3/EBS for anything of this scale as the I/O is pretty pathetic unless you invest in their very high end instances. We've set up a 4TB 'SAN' using glusterfs on AWS EC2 using 1TB EBS volumes and separate instances for each - the performance has been so poor that we have had to redesign our workflow to get copies of data onto EBS attached to each mapserver instance - for scaling that sucks and even then the I/O performance of EBS is not that great on the normal instances.
I'm not a hardware guy but I think the purpose of a dedicated SAN box is to provide high bandwidth access to large amounts storage so that the data can effectively be distributed to/from multiple machines over a network - ideal for scaling mapserver onto multiple servers but rendering from the same data. I read an article about a year ago from a company that provides petabyte storage for online storage, it details how they built their storage devices - they say $7867 for 67 terabytes http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/ Seems pretty geeky, but perhaps you are the hardware type or know someone who is :) On 2011-01-28, at 3:41 AM, [email protected] wrote: > Hello guys, we're a startup and new to mapserver. We're expecting large > amounts of data to come by (at least on our scale) around 40-60TB of raster > images for mapserver to render. My question is for the infrastructure, what > is the best way to store this (cost-efficiently)? > > - Do we just get a dedicated server with a lot of HDDs? I'm looking at a 48TB > setup in RAID 1+0 so i get 24TB right what happens now if we need more? Also, > how can we scale from the mapserver side? Is access to different storage > servers possible? > - I've considered SANs but then it's not practical right because only one > machine will access the storage? > - What about Amazon's S3? or EBS? Anything we can use on that? > > I wish to get awesome advice on this storage issue, basically what the > considered best practice is for the mapserver people :P Thanks > _______________________________________________ > mapserver-users mailing list > [email protected] > http://lists.osgeo.org/mailman/listinfo/mapserver-users __________________________________________ Paul Spencer Chief Technology Officer DM Solutions Group Inc http://research.dmsolutions.ca/ _______________________________________________ mapserver-users mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/mapserver-users
