Re: [gdal-dev] Question on TIFF compression alternatives
Le vendredi 14 octobre 2016 13:01:50, Neumann, Andreas a écrit : > Hi Even, > > Thank you for your detailed reply. > > I should have added that my concern is mainly about random access to > parts of the images - for QGIS server and QGIS Desktop. > > I have internal pyramids. I am not so sure if it is tiled or not. Internal pyramids created by GDAL are tiled with tiles of 128x128 pixel (can be changed with the GDAL_TIFF_OVR_BLOCKSIZE configuration option: http://gdal.org/frmt_gtiff.html#overviews . But the default is generally fine) > Does > gdalinfo reveal whether the TIFF was created with "TILED=YES"? If you see something "Band 1 Block=256x256" in the gdalinfo output, the full resolution image is tiled. If the block width (the first figure after Block=) is the image width, then it is not. You definitely want to use a tiled image for random access. > > So from your analysis it looks like lzma is the slowest, but overall, > the differences aren't that huge. I don't know how random access to > parts of the image changes the game a bit, compared to your analysis > where you examine the decompression of the whole image - right? I used whole image decompression otherwise the timings would have been hard to measure accurately. You can expect the same ratios of performance to apply to partial image decompression. My timing were done on "hot" runs. Depending on use cases, I/O might be the overwhelming time. > > Thanks, > > Andreas > > On 2016-10-14 12:09, Even Rouault wrote: > > Le vendredi 14 octobre 2016 09:45:38, Neumann, Andreas a écrit : > >> Hi, > >> > >> I have a question on TIFF compression alternatives. > >> > >> My data of concern is 8bit gray value. Lots of white pixel (>95%), the > >> rest is black pixels (content) and some gray pixels because of > >> antialiasing. > >> > >> I would like to compress to save some storage, but I need a good > >> performance. What is the recommended compression that is fast on > >> decompression? > >> > >> I tried DEFLATE, but it is rather slow to decompress. Would LZW or > >> PACKBITS be preferred in this situation? How about LZMA? I would like to > >> avoid non-compressed data, because I use SSD storage for performance > >> reasons. > > > > My findings on a 37614 x 18816 8 bit image (a scan of a text page) that > > should be similar to what you describe. > > > > Decompression time is the result of the second run of "time gdalinfo - > > checksum" (so you can consider that I/O is cached by the OS and you > > measure essentially CPU time. storage is spinning disk) > > > > No tiling: > > > > SizeCompressionDecompression time > > 707895750out_nocompression.tif9.3 s > > > > 11239972out_deflate.tif10.8 s > > 17749306out_lzw.tif10.5 s > > 12337710out_lzma.tif12.6 s > > 36201320out_packbits.tif9.9 s > > > > Tiling: > > > > SizeCompressionDecompression time > > 8813833out_deflate_tiled.tif11.1 s > > 14614121out_lzw_tiled.tif11.0 s > > 8479238out_lzma_tiled.tif13.0 s > > 37018470out_packbits_tiled.tif10.2 s > > > > Note: in the tiling case, the gdalinfo -checksum benchmark might not be > > ideal as it proceeds line by line (and not block by block), thus there's > > a lot of copying between cached blocks and output buffer that is done. > > > > So I've done time gdal_translate XXX.tif /vsimem/out.tif -q : > > > > deflate tiled:2.5 s > > lzw tiled:2.3 s > > packbits tiled: 1.5 s > > lzma tiled:4.5 s > > > > Times for non tiled files are essentially the same. > > > > Note: if you use tiling and you have whole tiles that are at white value, > > if you use GDAL trunk, set the SPARSE_OK=YES creation option and set the > > nodata value to 255, then those tiles will be entirely omitted from the > > compressed file. This might save a few extra bytes, and also some > > decompression time. > > > > gdal_translate out_nocompression.tif out_packbits_tiled.tif -co tiled=yes > > -co compress=packbits > > gdal_translate out_nocompression.tif out_packbits_tiled_sparse.tif -co > > tiled=yes -a_nodata 255 -co compress=packbits -co sparse_ok=yes > > > > 37018470out_packbits_tiled.tif > > 30409074out_packbits_tiled_sparse.tif > > > > time gdal_translate out_packbits_tiled_sparse.tif /vsimem/out.tif -q: 1.3 > > s > > > > Even -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Question on TIFF compression alternatives
Hi Even, Thank you for your detailed reply. I should have added that my concern is mainly about random access to parts of the images - for QGIS server and QGIS Desktop. I have internal pyramids. I am not so sure if it is tiled or not. Does gdalinfo reveal whether the TIFF was created with "TILED=YES"? So from your analysis it looks like lzma is the slowest, but overall, the differences aren't that huge. I don't know how random access to parts of the image changes the game a bit, compared to your analysis where you examine the decompression of the whole image - right? Thanks, Andreas On 2016-10-14 12:09, Even Rouault wrote: > Le vendredi 14 octobre 2016 09:45:38, Neumann, Andreas a écrit : > >> Hi, >> >> I have a question on TIFF compression alternatives. >> >> My data of concern is 8bit gray value. Lots of white pixel (>95%), the >> rest is black pixels (content) and some gray pixels because of >> antialiasing. >> >> I would like to compress to save some storage, but I need a good >> performance. What is the recommended compression that is fast on >> decompression? >> >> I tried DEFLATE, but it is rather slow to decompress. Would LZW or >> PACKBITS be preferred in this situation? How about LZMA? I would like to >> avoid non-compressed data, because I use SSD storage for performance >> reasons. > > My findings on a 37614 x 18816 8 bit image (a scan of a text page) that > should > be similar to what you describe. > > Decompression time is the result of the second run of "time gdalinfo - > checksum" (so you can consider that I/O is cached by the OS and you measure > essentially CPU time. storage is spinning disk) > > No tiling: > > SizeCompressionDecompression time > 707895750out_nocompression.tif9.3 s > > 11239972out_deflate.tif10.8 s > 17749306out_lzw.tif10.5 s > 12337710out_lzma.tif12.6 s > 36201320out_packbits.tif9.9 s > > Tiling: > > SizeCompressionDecompression time > 8813833out_deflate_tiled.tif11.1 s > 14614121out_lzw_tiled.tif11.0 s > 8479238out_lzma_tiled.tif13.0 s > 37018470out_packbits_tiled.tif10.2 s > > Note: in the tiling case, the gdalinfo -checksum benchmark might not be ideal > as it proceeds line by line (and not block by block), thus there's a lot of > copying between cached blocks and output buffer that is done. > > So I've done time gdal_translate XXX.tif /vsimem/out.tif -q : > > deflate tiled:2.5 s > lzw tiled:2.3 s > packbits tiled: 1.5 s > lzma tiled:4.5 s > > Times for non tiled files are essentially the same. > > Note: if you use tiling and you have whole tiles that are at white value, if > you use GDAL trunk, set the SPARSE_OK=YES creation option and set the nodata > value to 255, then those tiles will be entirely omitted from the compressed > file. This might save a few extra bytes, and also some decompression time. > > gdal_translate out_nocompression.tif out_packbits_tiled.tif -co tiled=yes > -co > compress=packbits > gdal_translate out_nocompression.tif out_packbits_tiled_sparse.tif -co > tiled=yes -a_nodata 255 -co compress=packbits -co sparse_ok=yes > > 37018470out_packbits_tiled.tif > 30409074out_packbits_tiled_sparse.tif > > time gdal_translate out_packbits_tiled_sparse.tif /vsimem/out.tif -q: 1.3 s > > Even ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Question on TIFF compression alternatives
Le vendredi 14 octobre 2016 09:45:38, Neumann, Andreas a écrit : > Hi, > > I have a question on TIFF compression alternatives. > > My data of concern is 8bit gray value. Lots of white pixel (>95%), the > rest is black pixels (content) and some gray pixels because of > antialiasing. > > I would like to compress to save some storage, but I need a good > performance. What is the recommended compression that is fast on > decompression? > > I tried DEFLATE, but it is rather slow to decompress. Would LZW or > PACKBITS be preferred in this situation? How about LZMA? I would like to > avoid non-compressed data, because I use SSD storage for performance > reasons. My findings on a 37614 x 18816 8 bit image (a scan of a text page) that should be similar to what you describe. Decompression time is the result of the second run of "time gdalinfo - checksum" (so you can consider that I/O is cached by the OS and you measure essentially CPU time. storage is spinning disk) No tiling: SizeCompression Decompression time 707895750 out_nocompression.tif 9.3 s 11239972out_deflate.tif 10.8 s 17749306out_lzw.tif 10.5 s 12337710out_lzma.tif12.6 s 36201320out_packbits.tif9.9 s Tiling: SizeCompression Decompression time 8813833out_deflate_tiled.tif 11.1 s 14614121out_lzw_tiled.tif 11.0 s 8479238out_lzma_tiled.tif 13.0 s 37018470out_packbits_tiled.tif 10.2 s Note: in the tiling case, the gdalinfo -checksum benchmark might not be ideal as it proceeds line by line (and not block by block), thus there's a lot of copying between cached blocks and output buffer that is done. So I've done time gdal_translate XXX.tif /vsimem/out.tif -q : deflate tiled: 2.5 s lzw tiled: 2.3 s packbits tiled: 1.5 s lzma tiled: 4.5 s Times for non tiled files are essentially the same. Note: if you use tiling and you have whole tiles that are at white value, if you use GDAL trunk, set the SPARSE_OK=YES creation option and set the nodata value to 255, then those tiles will be entirely omitted from the compressed file. This might save a few extra bytes, and also some decompression time. gdal_translate out_nocompression.tif out_packbits_tiled.tif -co tiled=yes -co compress=packbits gdal_translate out_nocompression.tif out_packbits_tiled_sparse.tif -co tiled=yes -a_nodata 255 -co compress=packbits -co sparse_ok=yes 37018470out_packbits_tiled.tif 30409074out_packbits_tiled_sparse.tif time gdal_translate out_packbits_tiled_sparse.tif /vsimem/out.tif -q: 1.3 s Even -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] Question on TIFF compression alternatives
Hi, I have a question on TIFF compression alternatives. My data of concern is 8bit gray value. Lots of white pixel (>95%), the rest is black pixels (content) and some gray pixels because of antialiasing. I would like to compress to save some storage, but I need a good performance. What is the recommended compression that is fast on decompression? I tried DEFLATE, but it is rather slow to decompress. Would LZW or PACKBITS be preferred in this situation? How about LZMA? I would like to avoid non-compressed data, because I use SSD storage for performance reasons. Thanks for you reply, Andreas ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
