On Donnerstag, 16. Dezember 2021 16:35:40 CET Kemp Watson wrote: > Milian, are these images for geomapping or digital pathology? I am guessing > so since you mentioned mipmap subIFDs.
It's for quantitative imaging, so digital pathology is probably the right category for us. > If so, there is a good 20-25 years of experience in how to efficiently > write and read these large files, and the solution has inevitably come down > to tiling the data, and absolutely not mmapping. I agree that tiling is ideal. I don't see why tiling would mean "no mmapping" - one could mmap the tiles, no? Furthermore, it also has nothing to do with mipmapping. Furthermore, to reiterate this once more - we are interacting with a thirdparty library which requires contiguous buffers. There is (sadly!) no easy way to change that to deal with discontiguous, potentially compressed, tiled image data. > In fact, when you say the size of the image leaves you no alternative, I > think perhaps you really should really think that the size of the images > means mmapping should be out of the question. I don't understand why that is, can you please elaborate? See above: Without mmapping, I would have to copy the full image into a contiguous heap memory buffer, which is stricly worse than using mmap where the kernel can evict the data as needed, instead of having to write the data to swap. > Are you generating the TIFFs yourself, or are they being generated as some > kind of well-known TIFF derivative? Both. For this question here it's about those parts where we are actually writing the data ourselves. > If so, there may well be SDKs or > well-known approaches - also, if you are not generating the data, can you > really ensure that the files meet all your criteria for mmapping? We can, and are, checking the criteria for mmapping. If they are all met, we can continue directly. Otherwise our software will create a mmappable buffer on disk in a cache diretly that can be mmapped directly. > If not, > you will need to read/write in memory anyway... similarly, if your other > software stacks are not all on the same machine, or if you are using GPUs > with limited memory. None of this applies to us. Thanks -- Milian Wolff | [email protected] | Senior Software Engineer KDAB (Deutschland) GmbH, a KDAB Group company Tel: +49-30-521325470 KDAB - The Qt, C++ and OpenGL Experts
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ Tiff mailing list [email protected] https://lists.osgeo.org/mailman/listinfo/tiff
