Ed, Yep. My layers in the map file are set up (using MIN/MAX SCALE) so no more than 4 to 6 image files are opened for any given map request (usually only one or two files are opened), regardless of the scale. Performance is about 2 to 4 seconds per request (for a 1k x 1k image).
Brent ----- Original Message ----- From: "Ed McNierney" <[EMAIL PROTECTED]> To: <[email protected]> Sent: Friday, October 14, 2005 1:27 PM Subject: Re: [UMN_MAPSERVER-USERS] Problem trying to serve large .tif dataset Brent - It's OK to break the rules - just as long as you know you're doing it <g>. Frank made another point which is also important, but less-often discussed. At a certain point pyramids/overviews begin to break down because a large image requested at a small scale ends up reading a large number of very small files. If you're opening more than a few files to serve an image request, things will slow down, and if you start reading from hundreds of files you're in trouble. The time to render the image will be dominated by the raw file-to-file seek time of your disk subsystem. Since finding a new file and opening it is one of the slowest things you can do in an application server, your performance gets very bad very quickly. At a certain point you need to take your tiled overview files and merge them, creating a smaller number of larger files. Very broadly speaking, it's probably a good idea to strive for image files whose extents are in the range of several times the size of the output images generated. If you're making 1,000x1,000 output images, and your files are 4,000x4,000 pixels, then (a) most image requests will be served by only opening one file, (b) a reasonable chunk of any file opened is used, and (c) you'll never need to open more than 4 files to render an image. If your files are shrunken overviews that are 100 pixels each, then your 1,000x1,000 image may require up to 121 files being opened, and will never require fewer than 100! - Ed Ed McNierney President and Chief Mapmaker TopoZone.com / Maps a la carte, Inc. 73 Princeton Street, Suite 305 North Chelmsford, MA 01863 [EMAIL PROTECTED] (978) 251-4242 -----Original Message----- From: UMN MapServer Users List [mailto:[EMAIL PROTECTED] On Behalf Of Brent Fraser Sent: Friday, October 14, 2005 10:50 AM To: [email protected] Subject: Re: [UMN_MAPSERVER-USERS] Problem trying to serve large .tif dataset Glen, As Frank has pointed out in his reply (and Ed has said for several years), the answer is to pyramid your data. This means constructing layers at different resolutions, then setting MINSCALE MAXSCALE for each of the layers in the map file. The goal is to improve the performance of Mapserver by limiting the number of files it has to open to render a requested image. For example, I've set up a site to render the NASA's Geocover global Landsat mosaic. To handle the range of requests from full resolution (14 meters per pixel) to full world-wide coverage (20km per pixel), I set up six levels (and I may add one more) of the pyramid (view in a fixed-pitched font such as Courier): Largest Number of Tile Lvl: Map Scale: Resolution: Tile Size: Pixels: Tiles Size Total Size: Format: ---- ---------- ----------- --------------------------- ---- ---- ---------- ----- ----------- ------ 6 1: 50k 14 meters 1:250k-ish (1/16 of 5x6 deg) 10k x 10k 14064 26mb = 338 gb ECW 5 250k 50 meters 1:250k 1 deg H x 2 deg W 2k x 2k 14921 3mb = 38.4 gb ECW 4 1m 200 meters ~1:1m 5 deg H x 6 deg W 3k x 2k 36x60 = 2160 20mb = 26.1 gb TIFF 3 5m 30 sec ( 1km) 15 deg H x 30 deg W 2k x 3k 12x12 = 144 20mb = 2.4 gb TIFF 2 20m 2 min ( 4km) 45 deg H x 90 deg W 1k x 2k 4x 4 = 16 10mb = 0.167 gb TIFF 1 100m 10 min (20km) 180 deg H x360 deg W 1k x 2k 1 = 1 7mb = 0.007 gb TIFF Disclaimer: The above is just an example of pyramiding a large raster dataset (tuning by adding levels or adjusting scale will likely be required). And I've broken Ed's Rule #3 ("For best performance, don't compress your data") since levels 5 and 6 are ECW, and Rule #4 ("Don't re-project your data on-the-fly") since levels 4,5,6 are in UTM. Brent Fraser GeoAnalytic Inc. Calgary, Alberta > I am having difficulties with mapserver 4.6 generating output with a > large .tif dataset. I'm attempting to setup a large dataset to be > served up via WMS. > > I have a .tif dataset of approximately 2000 .tif files totalling > around 160k pixels by 375k pixels. Each tif image is 5000 x 5000 pixels. > > Has anyone had any success dealing with datasets of this size? If so, > how have you implemented them? > > Any help/advise would be greatly appreciated. > > Thank you, > Glen Thompson
