Even, that was really helpful. Finally, I solved it without any need to modify GDAL.
I do the mmap() myself in my application. Then I create VSILFILE out of the mmapped block by VSIFileFromMemBuffer(). I can then GDALOpen() the /vsimem/ file and fork freely. The GTIFF_USE_MMAP is a cherry on the cake. I still have to evaluate performance impact, but it seems promising so far. Thanks for the advice. Jiri 2017-12-15 11:29 GMT+01:00 Even Rouault <[email protected]>: > On jeudi 14 décembre 2017 23:31:48 CET Jiri Drbalek wrote: > > > Dear Even, > > > > > > Thank you for your helpful answer. > > > > > > I forgot to mention that when I was testing the fork() situation with > > > libtiff alone, it was working fine when the tiff file was memory > > > mapped. Unfortunately, for some reason, GDAL doesn't support memory > > > mapping of compressed tiffs. Libtiff can read them, at least those > > > with deflate compression I've tested. What is the reason for that > > > restriction? > > > > mmap is platform specific, so there was a need for a more general > mechanism. And at the time the GeoTIFF driver was created, 32 bit processes > were still common but large files already existed, so even on Linux, mmap > wasn't really always usable. Another potential issue is that the OS might > not behave appropriately if you mmap() a file larger than the available RAM > and read it entirely. At least that was my experience with some older > kernels where the OS wouldn't unload cached pages aggressively enough, > making it irresponsive due to heavy cache swapping. > > > > Another option at the /vsi level I looked yesterday was the use of pread() > that takes both a file offset and size to read, and makes it possible to > use the same underlying file descriptor from multiple threads/processes. > But the caveat is that this translates directly as a system call, bypassing > file stream buffering. > > > > > Should I try to enable mmaping of compressed tiffs? > > > > That could be a solution worth investigating. Perhaps restricted to 64bit > posix platforms, and with a configuration option, such as the existing > GTIFF_USE_MMAP that you'll see if you look at frmts/gtiff/tifvsi.cpp. The > name is a bit misleading since that's currently only available for a > pseudo-mmap emulation for /vsimem/ files, that was added per > https://trac.osgeo.org/gdal/changeset/39555. As you may wonder why this > was done, the aim was to be able to test the code paths in libtiff that are > mmap() specific, when GDAL is tortured by oss-fuzz. > > > > Even > > > > -- > > Spatialys - Geospatial professional services > > http://www.spatialys.com >
_______________________________________________ gdal-dev mailing list [email protected] https://lists.osgeo.org/mailman/listinfo/gdal-dev
