Hi, Thank you very much about the information. How much about the troubles remain if 64-bit system is used, and would there be any difference between Windows and Linux then? 2 GB outputs are not unrealistically small for a web service but WCS coverages can be very large and as far as I know WCS has no standard way for paging. I don't even feel very relaxed with paging by making subsequent GetCoverages by sliding the subset range without checking carefully how the pixels at the seams behave.
-Jukka Rahkonen- Even Rouault wrote: On Wednesday 05 August 2015 09:43:14 Rahkonen Jukka wrote: > Hi, > > I started to experiment with WCS 2.0.1 with Mapserver 7.1-dev and I > managed to get WCS to work rather easily. However, I can only get data > with rather small GetCoverage requests when using image/tiff format. > > My environment: > MapServer 7.1-dev which I installed from the 32-bit Windows binaries > from gisinternals > http://download.gisinternals.com/sdk/downloads/release-1600-gdal-mapserver. > zip. For installation I used existing MS4W installation by making a > new cgi-bin directory where I dropped the new binaries (dll files and > mapserv.exe). My computer is 64-bit Windows 7 with 8 GB of memory. I > am stuck with 32-bit Mapserver because using MS4W on the bottom is the > only way to install that I can handle. > > Output is successful when the resulting image is 8000x8000 pixels (192 > MB) but if I try to get an image with 10000x10000 pixels (300 MB) I am > receiving an error: > > <ows:ExceptionText>msWCSWriteFile20(): General error message. > msSaveImage() failed msSaveImageGDAL(): General error message. Failed > to create output GTiff file. TIFFAppendToStrip:Write error at scanline > 6500 I have no problem at all with output size of 12000x12000 pixels > if I use image/png or image/jpeg. > > If I play with subset ranges I can see that the GeoTIFF write error > happens at different scanlines. It makes me think that it could be > some memory problem. But how could I give more resources for GDAL so > it could make its job? I have not yet thought how big images I will > need from WCS but lets say the aim is at 50000x50000 pixels which > would make 7.5 gigabyte GeoTIFF as an uncompressed, 8-bit, 3-band image. Jukka, My understanding on what is involved is the following. MapServer will first allocate a memory buffer for the whole resulting image, do the rendering on it, create a GDAL MEM dataset with the content of the mapserver image, and then CreateCopy() it to the final in-memory image file and then stream it out through HTTP. So in the case of the 10K x 10K image, you'll have a 300 MB buffer for the mapserver image + 300 MB for the GDAL MEM dataset + 300 MB for the resulting TIFF stored in a /vsimem/ file. Due to the way the TIFF writer works, by appending data and growing the /vsimem/ file progressively by successive realloc(), I suspect that heap memory fragmentation occurs and that on the available 2 GB of the 32 bit process, much more than the theoretically 3x300 MB needed is consumed, reaching the 2 GB limit. I don't see any easy workaround, but a few possible improvements: - (GDAL) In the case of uncompressed TIFF where the final file size is known the GDAL GTiff writer could perhaps be improved to actually pre-allocate the whole buffer instead of growing it progressively, so as to avoid heap fragmentation. - (MapServer) Another/complementary option would be in the case of GTiff, and other formats that supports the GDALCreate() API, to avoid the use of the MEM dataset. - (MapServer) Another/complementary option would be to offer an option to have a on-disk temporary file to save some RAM. For non-virtual enabled GDAL formats, such as netCDF, this is what is used. - (MapServer) Another/complementary option would be to use in MapServer the new capability in GDAL 2.0 of the GTiff driver to generate directly streamable file in the case of a uncompressed file. But even in the best case (just the mapserver image buffer), the maximum file you could produce would be 2 GB with a 32bit MapServer build. In theory, I believe that WCS GetCoverage could generite arbitrary file size with modest RAM requirements but that would likely require major architectural changes in MapServer. Even > > -Jukka Rahkonen- > _______________________________________________ > mapserver-users mailing list > [email protected] > http://lists.osgeo.org/mailman/listinfo/mapserver-users -- Spatialys - Geospatial professional services http://www.spatialys.com _______________________________________________ mapserver-users mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/mapserver-users
