On Wednesday 05 August 2015 09:43:14 Rahkonen Jukka wrote: > Hi, > > I started to experiment with WCS 2.0.1 with Mapserver 7.1-dev and I managed > to get WCS to work rather easily. However, I can only get data with rather > small GetCoverage requests when using image/tiff format. > > My environment: > MapServer 7.1-dev which I installed from the 32-bit Windows binaries from > gisinternals > http://download.gisinternals.com/sdk/downloads/release-1600-gdal-mapserver. > zip. For installation I used existing MS4W installation by making a new > cgi-bin directory where I dropped the new binaries (dll files and > mapserv.exe). My computer is 64-bit Windows 7 with 8 GB of memory. I am > stuck with 32-bit Mapserver because using MS4W on the bottom is the only > way to install that I can handle. > > Output is successful when the resulting image is 8000x8000 pixels (192 MB) > but if I try to get an image with 10000x10000 pixels (300 MB) I am > receiving an error: > > <ows:ExceptionText>msWCSWriteFile20(): General error message. msSaveImage() > failed msSaveImageGDAL(): General error message. Failed to create output > GTiff file. TIFFAppendToStrip:Write error at scanline 6500 > I have no problem at all with output size of 12000x12000 pixels if I use > image/png or image/jpeg. > > If I play with subset ranges I can see that the GeoTIFF write error happens > at different scanlines. It makes me think that it could be some memory > problem. But how could I give more resources for GDAL so it could make its > job? I have not yet thought how big images I will need from WCS but lets > say the aim is at 50000x50000 pixels which would make 7.5 gigabyte GeoTIFF > as an uncompressed, 8-bit, 3-band image.
Jukka, My understanding on what is involved is the following. MapServer will first allocate a memory buffer for the whole resulting image, do the rendering on it, create a GDAL MEM dataset with the content of the mapserver image, and then CreateCopy() it to the final in-memory image file and then stream it out through HTTP. So in the case of the 10K x 10K image, you'll have a 300 MB buffer for the mapserver image + 300 MB for the GDAL MEM dataset + 300 MB for the resulting TIFF stored in a /vsimem/ file. Due to the way the TIFF writer works, by appending data and growing the /vsimem/ file progressively by successive realloc(), I suspect that heap memory fragmentation occurs and that on the available 2 GB of the 32 bit process, much more than the theoretically 3x300 MB needed is consumed, reaching the 2 GB limit. I don't see any easy workaround, but a few possible improvements: - (GDAL) In the case of uncompressed TIFF where the final file size is known the GDAL GTiff writer could perhaps be improved to actually pre-allocate the whole buffer instead of growing it progressively, so as to avoid heap fragmentation. - (MapServer) Another/complementary option would be in the case of GTiff, and other formats that supports the GDALCreate() API, to avoid the use of the MEM dataset. - (MapServer) Another/complementary option would be to offer an option to have a on-disk temporary file to save some RAM. For non-virtual enabled GDAL formats, such as netCDF, this is what is used. - (MapServer) Another/complementary option would be to use in MapServer the new capability in GDAL 2.0 of the GTiff driver to generate directly streamable file in the case of a uncompressed file. But even in the best case (just the mapserver image buffer), the maximum file you could produce would be 2 GB with a 32bit MapServer build. In theory, I believe that WCS GetCoverage could generite arbitrary file size with modest RAM requirements but that would likely require major architectural changes in MapServer. Even > > -Jukka Rahkonen- > _______________________________________________ > mapserver-users mailing list > [email protected] > http://lists.osgeo.org/mailman/listinfo/mapserver-users -- Spatialys - Geospatial professional services http://www.spatialys.com _______________________________________________ mapserver-users mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/mapserver-users
