Re: [OSGeo-Discuss] fastest option of serving huge imagery on web map on the fly
On 22.05.2010, at 21:26, Jason Birch wrote: I've often wondered if power-of-two was the best approach from a perception viewpoint. It definitely makes the most sense from a code perspective. Anyone know of any research on this? No research, but we invested some time while developing MapProxy and looked for alternatives. We came up with power-of-1.41421, the square root of two. Together with some other tricks (like bicubic resampling, etc) you can get pretty good results for vector data. The benefit of sqrt(2) is that every second level exactly matches the resolution of the traditional power-of-two pyramid, which is important if you use clients like OpenLayers. See http://osm.omniscale.net/proxy/service? wich uses the same data as http://osm.omniscale.de/ Regards, Oliver -- Oliver Tonnhofer o...@omniscale.de Omniscale - Dominik Helle, Oliver Tonnhofer GbR Nadorster Str. 60, 26123 Oldenburg Tel: +49(0)441/9392774-2 (Fax: 9) ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] fastest option of serving huge imagery on web map on the fly
I've often wondered if power-of-two was the best approach from a perception viewpoint. It definitely makes the most sense from a code perspective. Anyone know of any research on this? On 2010-05-22, Christopher Schmidt crschm...@crschmidt.net wrote: On Fri, May 21, 2010 at 06:24:26PM -0300, Fabio Renzo Panettieri wrote: On Fri, 2010-05-21 at 13:17 -0700, Karsten-3-2 wrote: Yes. What I want to do is simply to find out the fastest options to render on the fly from raw data imagery (no tiles whatsoever stored on disk in addition to the raw data ). I will check out what SpatialCache is... From raw aerial imagery: 1. Store everything as uncompressed tiffs. 2. Make images as large as possible. (This probably requires BigTIFF support.) 3. Use overviews -- usually one for every power-of-two level from the base image up to the point where you have 256 x 256 overviews (gdaladdo) 4. If you have too many images to make one large image practical, create one reduced size image that you use at lower zoom levels. All of this is based around serving with MapServer. I have no experience using other imagery servers to solve this problem. Regards, -- Christopher Schmidt Web Developer ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] fastest option of serving huge imagery on web map on the fly
On 20.05.2010, at 22:10, miblon wrote: If mapserver is to slow, almost everything else is to slow... I also noticed your crosspost on the mapproxy list. mapproxy will even cache to more then the 60TB youve estimated, because it will cache every wms request instead of square, stitched tiles. I don't understand that. MapProxy does cache square tiles and if 60TB are a valid estimate for TileCache and GeoWebCache, than this should also apply to MapProxy. But, you don't want to cache everything in advance, that would be a waste of resources. Caching the lower resolutions, the common parts where more users will access the same images would take of the load of the WMS server. Karsten mentioned OpenLayers, so i guess tiled services like TMS are an option. MapProxy, TileCache and GeoWebCache should all be able to handle that without caching in advance. MapProxy comes with full HTTP cache control, you can limit the resolution till images should be cached (other requests will be passed to the WMS) and if some clients require full WMS you can use MapProxy's WMS and benefit from the cached tiles. All points that are quite useful in this scenario. Regards, Oliver -- Oliver Tonnhofer o...@omniscale.de Omniscale - Dominik Helle, Oliver Tonnhofer GbR Nadorster Str. 60, 26123 Oldenburg Tel: +49(0)441/9392774-2 (Fax: 9) ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] fastest option of serving huge imagery on web map on the fly
In my opinion (but of course Karsten needs to answer that himself) he needs wms as I think he mentioned before. I may be wrong assuming that mapproxy caches every unique wms request as a unique image. I totally agree that he shouldn't preseed everything in advance, especially when budgets are to low to consider serious diskspace. Oliver Tonnhofer wrote: On 20.05.2010, at 22:10, miblon wrote: If mapserver is to slow, almost everything else is to slow... I also noticed your crosspost on the mapproxy list. mapproxy will even cache to more then the 60TB youve estimated, because it will cache every wms request instead of square, stitched tiles. I don't understand that. MapProxy does cache square tiles and if 60TB are a valid estimate for TileCache and GeoWebCache, than this should also apply to MapProxy. But, you don't want to cache everything in advance, that would be a waste of resources. Caching the lower resolutions, the common parts where more users will access the same images would take of the load of the WMS server. Karsten mentioned OpenLayers, so i guess tiled services like TMS are an option. MapProxy, TileCache and GeoWebCache should all be able to handle that without caching in advance. MapProxy comes with full HTTP cache control, you can limit the resolution till images should be cached (other requests will be passed to the WMS) and if some clients require full WMS you can use MapProxy's WMS and benefit from the cached tiles. All points that are quite useful in this scenario. Regards, Oliver ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] fastest option of serving huge imagery on web map on the fly
On 21.05.2010, at 09:57, miblon wrote: In my opinion (but of course Karsten needs to answer that himself) he needs wms as I think he mentioned before. Even then MapProxy should be faster. The overhead of MrSid/JP2 is larger compared to the simple access to cached tiles. I may be wrong assuming that mapproxy caches every unique wms request as a unique image. Then you could just use Squid :) No, MapProxy does cache single square tiles and then merges and transforms the tiles back to full WMS responses. Regards, Oliver -- Oliver Tonnhofer o...@omniscale.de Omniscale - Dominik Helle, Oliver Tonnhofer GbR Nadorster Str. 60, 26123 Oldenburg Tel: +49(0)441/9392774-2 (Fax: 9) ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
[OSGeo-Discuss] fastest option of serving huge imagery on web map on the fly
Hi Oliver I don't understand what you mean with without caching tiles on disk. You don't want to pre-generate the tiles? If that's the case, I you can probably try with SpatialCache. http://code.google.com/p/spatialcache/ ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
[OSGeo-Discuss] fastest option of serving huge imagery on web map on the fly
Hi All, I am seeking some advice/ alternative ideas about the following project I am working on... I have been tasked with researching the best and fastest options serving huge raster datasets on a web map using OpenLayers o the fly (using all Open Source software). We want to serve the US NAIP Aerials in 1m resolution (which are a total of about 4.7 TB of MrSid/Jp2 data) on a interactive web map as an optional map background. The are using MapServer to serve our other (vector) data such as roads, rivers etc as WMS to overlay onto this. Of course there are many ways to go about this but one of the things we determined early on is that MapServer is too slow to serve compressed imagery such as the native MrSid Jp2 imagery on the fly for our needs. Thus, one option would be to spare MapServer from having to decompress the images. We can then also avoid having to convert them to tiff and adding overviews (using gdaladdo for example). This would also blow up the total data volume to something about 60 TB ... Thus, we are in the process of researching options on how to serve the compressed data as fast as possible on the fly and without the need for caching them on disk (that means no TileCache nor GeoWebCache should be used because that also would involve having to set up huge storage spaces ... One option I came about was using IIpimage server and this would then involve converting the MrSid all to Jp2 format. One advantage is that OpenLayers 2.9 already has natively the Zoomify layer support so that we can easily add the images coming out of IIPImage Server Zoomify + JPEG2000 server http://help.oldmapsonline.org/jpeg2000/ I also found that another option is the Djatoka Jpeg 2000 Image Server http://sourceforge.net/apps/mediawiki/djatoka/index.php?title=Main_Page and the J2K Tiler Renderer: http://dltj.org/article/introducing-j2ktilerenderer/. None of the above seem to enable output as WMS (correct me if I'm wrong). One draw back is that all of those above are using the Kakadu library which is great but not free for commercial use. I also wanted to research how the use of this new proxy server http://mapproxy.org/ could improve our speed in combination with e.g. IIP Image server... Anybody has experiences with any of the above or comments ? Any input what you think would be the fastest option to serve the compressed US NAIP onto a web map on the fly (without caching tiles on disk) ? Cheers Karsten ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] fastest option of serving huge imagery on web map on the fly
If mapserver is to slow, almost everything else is to slow... I also noticed your crosspost on the mapproxy list. mapproxy will even cache to more then the 60TB youve estimated, because it will cache every wms request instead of square, stitched tiles. If I where you and caching is an issue, I would invest in a (small) farm of fast cgi mapserver servers with for instance a squid proxy in front. One single machine will always be to slow. You say nothing about the hardware. I for instance use i7 950 servers with 12Gb ram and 3Tb diskspace for €89 a month. - Search for the best hardware at acceptable price - Consider setting up less servers with more diskspace per server and concider using cache if you find out that the cost of individual high end large disks (with in total enough space to hold 120Tb or so) are lower then setting up 4 individual machines with less diskspace barely fitting your initial dataset, consider that too. - Upscale when needed. You say you have 4.7 Tb of imagery, how big is the target audience? Will they be viewing the entire image set up to the highest detail or will there be a limited Area of interest? I would say that it is pointless to create a infrastructure capable of holding the highest level of detail given the amount of disk space when only a hundred users will be active.. - Find an investor In my opinion, you are on top of imagery that is of invaluable worth to your audience. Why would they want to cut on the infrastructure costs? Sounds a bit like; we want a money-transport truck, but we would not want to invest in armoring it and giving it an engine to outrun any bandits. I think someone needs to do some good presales work here and set up an excellent business case. Good luck! Kind regards, Milo van der Linden karsten vennemann wrote: Hi All, I am seeking some advice/ alternative ideas about the following project I am working on... I have been tasked with researching the best and fastest options serving huge raster datasets on a web map using OpenLayers o the fly (using all Open Source software). We want to serve the US NAIP Aerials in 1m resolution (which are a total of about 4.7 TB of MrSid/Jp2 data) on a interactive web map as an optional map background. The are using MapServer to serve our other (vector) data such as roads, rivers etc as WMS to overlay onto this. Of course there are many ways to go about this but one of the things we determined early on is that MapServer is too slow to serve compressed imagery such as the native MrSid Jp2 imagery on the fly for our needs. Thus, one option would be to spare MapServer from having to decompress the images. We can then also avoid having to convert them to tiff and adding overviews (using gdaladdo for example). This would also blow up the total data volume to something about 60 TB ... Thus, we are in the process of researching options on how to serve the compressed data as fast as possible on the fly and without the need for caching them on disk (that means no TileCache nor GeoWebCache should be used because that also would involve having to set up huge storage spaces ... One option I came about was using IIpimage server and this would then involve converting the MrSid all to Jp2 format. One advantage is that OpenLayers 2.9 already has natively the Zoomify layer support so that we can easily add the images coming out of IIPImage Server Zoomify + JPEG2000 server http://help.oldmapsonline.org/jpeg2000/ I also found that another option is the Djatoka Jpeg 2000 Image Server http://sourceforge.net/apps/mediawiki/djatoka/index.php?title=Main_Page and the J2K Tiler Renderer: http://dltj.org/article/introducing-j2ktilerenderer/. None of the above seem to enable output as WMS (correct me if I'm wrong). One draw back is that all of those above are using the Kakadu library which is great but not free for commercial use. I also wanted to research how the use of this new proxy server http://mapproxy.org/ could improve our speed in combination with e.g. IIP Image server... Anybody has experiences with any of the above or comments ? Any input what you think would be the fastest option to serve the compressed US NAIP onto a web map on the fly (without caching tiles on disk) ? Cheers Karsten ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
