If mapserver is to slow, almost everything else is to slow... I also
noticed your crosspost on the mapproxy list. mapproxy will even cache to
more then the 60TB youve estimated, because it will cache every wms
request instead of square, stitched tiles.
If I where you and caching is an issue, I would invest in a (small) farm
of fast cgi mapserver servers with for instance a squid proxy in front.
One single machine will always be to slow. You say nothing about the
hardware. I for instance use i7 950 servers with 12Gb ram and 3Tb
diskspace for €89 a month.
- Search for the best hardware at acceptable price
- Consider setting up less servers with more diskspace per server and
concider using cache if you find out that the cost of individual high
end large disks (with in total enough space to hold 120Tb or so) are
lower then setting up 4 individual machines with less diskspace barely
fitting your initial dataset, consider that too.
- Upscale when needed. You say you have 4.7 Tb of imagery, how big is
the target audience? Will they be viewing the entire image set up to the
highest detail or will there be a limited "Area of interest"? I would
say that it is pointless to create a infrastructure capable of holding
the highest level of detail given the amount of disk space when only a
hundred users will be active..
- Find an investor
In my opinion, you are on top of imagery that is of invaluable worth to
your audience. Why would they want to cut on the infrastructure costs?
Sounds a bit like; we want a money-transport truck, but we would not
want to invest in armoring it and giving it an engine to outrun any bandits.
I think someone needs to do some good presales work here and set up an
excellent business case.
Good luck!
Kind regards,
Milo van der Linden
karsten vennemann wrote:
Hi All,
I am seeking some advice/ alternative ideas about the following
project I am working on...
I have been tasked with researching the best and fastest options
serving huge raster datasets on a web map using OpenLayers o the fly
(using all Open Source software). We want to serve the US NAIP Aerials
in 1m resolution (which are a total of about 4.7 TB of MrSid/Jp2 data)
on a interactive web map as an optional map background. The are using
MapServer to serve our other (vector) data such as roads, rivers etc
as WMS to overlay onto this. Of course there are many ways to go
about this but one of the things we determined early on is that
MapServer is too slow to serve compressed imagery such as the native
MrSid Jp2 imagery on the fly for our needs. Thus, one option would be
to spare MapServer from having to decompress the images. We can then
also avoid having to convert them to tiff and adding overviews (using
gdaladdo for example). This would also "blow up" the total data volume
to something about 60 TB ...
Thus, we are in the process of researching options on how to serve the
compressed data as fast as possible "on the fly" and without the need
for caching them on disk (that means no TileCache nor GeoWebCache
should be used because that also would involve having to set up huge
storage spaces ...
One option I came about was using IIpimage server and this would then
involve converting the MrSid all to Jp2 format. One advantage is that
OpenLayers 2.9 already has natively the Zoomify layer support so that
we can easily add the images coming out of IIPImage Server Zoomify +
JPEG2000 server http://help.oldmapsonline.org/jpeg2000/
I also found that another option is the Djatoka Jpeg 2000 Image Server
http://sourceforge.net/apps/mediawiki/djatoka/index.php?title=Main_Page
and the J2K Tiler Renderer:
http://dltj.org/article/introducing-j2ktilerenderer/.
None of the above seem to enable output as WMS (correct me if I'm
wrong). One draw back is that all of those above are using the Kakadu
library which is great but not free for commercial use.
I also wanted to research how the use of this new proxy server
http://mapproxy.org/ could improve our speed in combination with e.g.
IIP Image server...
Anybody has experiences with any of the above or comments ?
Any input what you think would be the fastest option to serve
the compressed US NAIP onto a web map on the fly (without caching
tiles on disk) ?
Cheers
Karsten
------------------------------------------------------------------------
_______________________________________________
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss
_______________________________________________
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss
