On Wednesday 05 August 2015 12:16:49 Rahkonen Jukka wrote: > Hi, > > Thank you very much about the information. How much about the troubles > remain if 64-bit system is used,
With a 64 bit system, you would still need to be able to allocate 3 times the size of the output file, either from RAM only or RAM + swap space. > and would there be any difference between > Windows and Linux then? On 64 bit probably little. On 32 bit, the Linux memory allocator could potentially be better at limiting fragmentation, but I'm not sure of that. > 2 GB outputs are not unrealistically small for a > web service but WCS coverages can be very large and as far as I know WCS > has no standard way for paging. I don't even feel very relaxed with paging > by making subsequent GetCoverages by sliding the subset range without > checking carefully how the pixels at the seams behave. There might indeed be border effects if resampling is involved, although ideally MapServer should query a bit more than the requested area (I think it does) > > -Jukka Rahkonen- > > > Even Rouault wrote: > > On Wednesday 05 August 2015 09:43:14 Rahkonen Jukka wrote: > > Hi, > > > > I started to experiment with WCS 2.0.1 with Mapserver 7.1-dev and I > > managed to get WCS to work rather easily. However, I can only get data > > with rather small GetCoverage requests when using image/tiff format. > > > > My environment: > > MapServer 7.1-dev which I installed from the 32-bit Windows binaries > > from gisinternals > > http://download.gisinternals.com/sdk/downloads/release-1600-gdal-mapserver > > . > > zip. For installation I used existing MS4W installation by making a > > new cgi-bin directory where I dropped the new binaries (dll files and > > mapserv.exe). My computer is 64-bit Windows 7 with 8 GB of memory. I > > am stuck with 32-bit Mapserver because using MS4W on the bottom is the > > only way to install that I can handle. > > > > Output is successful when the resulting image is 8000x8000 pixels (192 > > MB) but if I try to get an image with 10000x10000 pixels (300 MB) I am > > receiving an error: > > > > <ows:ExceptionText>msWCSWriteFile20(): General error message. > > msSaveImage() failed msSaveImageGDAL(): General error message. Failed > > to create output GTiff file. TIFFAppendToStrip:Write error at scanline > > 6500 I have no problem at all with output size of 12000x12000 pixels > > if I use image/png or image/jpeg. > > > > If I play with subset ranges I can see that the GeoTIFF write error > > happens at different scanlines. It makes me think that it could be > > some memory problem. But how could I give more resources for GDAL so > > it could make its job? I have not yet thought how big images I will > > need from WCS but lets say the aim is at 50000x50000 pixels which > > would make 7.5 gigabyte GeoTIFF as an uncompressed, 8-bit, 3-band image. > > Jukka, > > My understanding on what is involved is the following. MapServer will first > allocate a memory buffer for the whole resulting image, do the rendering on > it, create a GDAL MEM dataset with the content of the mapserver image, and > then CreateCopy() it to the final in-memory image file and then stream it > out through HTTP. So in the case of the 10K x 10K image, you'll have a 300 > MB buffer for the mapserver image + 300 MB for the GDAL MEM dataset + 300 > MB for the resulting TIFF stored in a /vsimem/ file. Due to the way the > TIFF writer works, by appending data and growing the /vsimem/ file > progressively by successive realloc(), I suspect that heap memory > fragmentation occurs and that on the available 2 GB of the 32 bit process, > much more than the theoretically 3x300 MB needed is consumed, reaching the > 2 GB limit. I don't see any easy workaround, but a few possible > improvements: > - (GDAL) In the case of uncompressed TIFF where the final file size is known > the GDAL GTiff writer could perhaps be improved to actually pre-allocate > the whole buffer instead of growing it progressively, so as to avoid heap > fragmentation. - (MapServer) Another/complementary option would be in the > case of GTiff, and other formats that supports the GDALCreate() API, to > avoid the use of the MEM dataset. - (MapServer) Another/complementary > option would be to offer an option to have a on-disk temporary file to save > some RAM. For non-virtual enabled GDAL formats, such as netCDF, this is > what is used. - (MapServer) Another/complementary option would be to use in > MapServer the new capability in GDAL 2.0 of the GTiff driver to generate > directly streamable file in the case of a uncompressed file. > > But even in the best case (just the mapserver image buffer), the maximum > file you could produce would be 2 GB with a 32bit MapServer build. In > theory, I believe that WCS GetCoverage could generite arbitrary file size > with modest RAM requirements but that would likely require major > architectural changes in MapServer. > > Even > > > -Jukka Rahkonen- > > _______________________________________________ > > mapserver-users mailing list > > [email protected] > > http://lists.osgeo.org/mailman/listinfo/mapserver-users > > -- > Spatialys - Geospatial professional services http://www.spatialys.com -- Spatialys - Geospatial professional services http://www.spatialys.com _______________________________________________ mapserver-users mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/mapserver-users
