On Fri, Nov 08, 2013 at 03:46:12PM -0500, Jack Howarth wrote: > On Fri, Nov 08, 2013 at 03:31:05PM -0500, Jack Howarth wrote: > > On Fri, Nov 08, 2013 at 08:24:51PM +0100, Even Rouault wrote: > > > Le vendredi 08 novembre 2013 18:50:45, Jack Howarth a écrit : > > > > On Fri, Nov 08, 2013 at 09:12:05AM -0800, Kurt Schwehr wrote: > > > > > Jack, > > > > > > > > > > Is there an explanation someplace with more detail as to what is wrong > > > > > those png's, why older versions of libpng are okay with it and why > > > > > libpng has changed its behavior? > > > > > > > > > > Thanks, > > > > > -kurt > > > > > > > > Kurt, > > > > The following link... > > > > > > > > http://comments.gmane.org/gmane.comp.graphics.png.devel/6017 > > > > > > > > appears to discuss the issue we are seeing with the test.png file... > > > > > > > > > Libpng-1.6.0 through 1.6.2 use the CMF bytes at the beginning of an > > > > > zlib > > > > > datastream to set the window size for decoding. Libpng-1.5.x and > > > > > earlier ignored the CMF bytes and always used a 32kbyte window. > > > > > Libpng-1.6.x has uncovered the fact that there are hundreds of files > > > > > in > > > > > the wild (found in linux distributions such as gentoo) with a > > > > > too-small > > > > > CMF, which results in a "Too far back" error when decoding these > > > > > PNGs. > > > > > I have not yet been able to discover what application is producing > > > > > these. > > > > > > > > If you read completely through the gmane.comp.graphics.png.devel/6017 > > > > thread, you should be at least convinced that such corruption is a known > > > > issue. > > > > > > This is strange. I can't reproduce any issue on Linux with libpng 1.6.2 > > > or > > > libpng 1.6.3 with gdalicon.png or autotest/gdrivers/data/test.png that is > > > used > > > by png_1 test. If it matters, the zlib is still my system zlib 1.2.3. > > > > Another thread discussing the same issue is at... > > > > http://lilypond.1069038.n5.nabble.com/fixing-PNG-images-td146225.html > > > > as well as here... > > > > http://www.opendevs.org/qkup/libpng16-will-not-read-some-broken-png-images.pdf > > > > I can see the problem with the pngfix from libpng 1.6.6 as follows... > > > > # pngfix data/test.png > > IDAT TFB default 10 15 8575 160400 data/test.png > > > > whereas when I repair the file with optipng, I see... > > > > % optipng -nb -nc -np -fix test.png > > OptiPNG 0.6.3: Advanced PNG optimizer. > > Copyright (C) 2001-2009 Cosmin Truta. > > > > ** Processing: test.png > > 400x400 pixels, 8 bits/pixel, 16 colors (16 transparent) in palette > > Input IDAT size = 8575 bytes > > Input file size = 8732 bytes > > > > Trying: > > zc = 9 zm = 8 zs = 0 f = 0 IDAT size = 7349 > > > > Selecting parameters: > > zc = 9 zm = 8 zs = 0 f = 0 IDAT size = 7349 > > > > Output IDAT size = 7349 bytes (1226 bytes decrease) > > Output file size = 7494 bytes (1238 bytes = 14.18% decrease) > > > > # pngfix data/test.png > > IDAT OK maximum 15 15 7349 160400 data/test.png > > > > which is explained in the usage output from pngfix with no arguments as... > > > > The summary lines describe issues encountered with the zlib compressed > > stream of a chunk. They have the following format, which is SUBJECT TO > > CHANGE in the future: > > > > chunk reason comp-level p1 p2 p3 p4 file > > > > p1 through p4 vary according to the 'reason'. There are always 8 space > > separated fields. Reasons specific formats are: > > > > chunk ERR status code read-errno write-errno message file > > chunk SKP comp-level file-bits zlib-rc compressed message file > > chunk ??? comp-level file-bits ok-bits compressed uncompress file > > > > The various fields are > > > > $1 chunk: The chunk type of a chunk in the file or 'HEAD' if a problem > > is reported by libpng at the start of the IDAT stream. > > $2 reason: One of: > > CHK: A zlib header checksum was detected and fixed. > > TFB: The zlib too far back error was detected and fixed. > > OK : No errors were detected in the zlib stream and optimization > > was not requested, or was not possible. > > OPT: The zlib stream window bits value could be improved (and > > was). > > SKP: The chunk was skipped because of a zlib issue (zlib-rc) with > > explanation 'message' > > ERR: The read of the file was aborted. The parameters explain > > why. > > $3 status: For 'ERR' the accumulate status code from 'EXIT CODES' above. > > This is printed as a 2 digit hexadecimal value > > comp-level: The recorded compression level (FLEVEL) of a zlib stream > > expressed as a string {supfast,stdfast,default,maximum} > > $4 code: The file exit code; where stop was called, as a fairly terse > > string {warning,libpng,zlib,invalid,read,write,unexpected}. > > file-bits: The zlib window bits recorded in the file. > > $5 read-errno: A system errno value from a read translated by strerror(3). > > zlib-rc: A zlib return code as a string (see zlib.h). > > ok-bits: The smallest zlib window bits value that works. > > $6 write-errno:A system errno value from a write translated by strerror(3). > > compressed: The count of compressed bytes in the zlib stream, when the > > reason is 'SKP'; this is a count of the bytes read from the > > stream when the fatal error was encountered. > > $7 message: An error message (spaces replaced by _, as in all > > parameters), > > uncompress: The count of bytes from uncompressing the zlib stream; this > > may not be the same as the number of bytes in the image. > > $8 file: The name of the file (this may contain spaces). > > > > Note that the current test.png that you are shipping is reported as TFB > > for "The zlib too far back error was detected and fixed" by pngfix whereas > > the copy repaired by optipng reports OK for "No errors were detected in the > > zlib > > stream and optimization was not requested, or was not possible.". So far > > I haven't been able to puzzle out how to get pngfix to repair the file but > > I notice most are using optipng for that. > > Jack > > > > One last note. I reinstalled the older libpng16-1.6.3-1 packaging on fink and > it reports the same exact error for the bundled test.png file as the 1.6.6 > release does... > > % /sw/bin/pngfix test.png > IDAT TFB default 10 15 8575 160400 test.png.sav > > So you should be able to find this problem with libpng 1.6.3 or later. Note > that the detection of this issue was added during the libpng 1.6.3 development > cycle so you need to be sure use its pngfix utility and not that from 1.6.2. > > ---------------------------------------------------------------------------- > Changes since the last public release (1.6.2): > ... > > Added png-fix-itxt and png-fix-too-far-back to the built programs and > removed warnings from the source code and timepng that are revealed as > a result. > Detect wrong libpng versions linked to png-fix-too-far-back, which currently > only works with libpng versions that can be made to reliably fail when > the deflate data contains an out-of-window reference. This means only > 1.6 and later. > ... > Attempt to detect configuration issues with png-fix-too-far-back, which > requires both the correct libpng and the correct zlib to function > correctly. > --------------------------------------------------------------------------- > > Note on darwin we typically use the system libz which appears to be > derivation of 1.2.5.
I can reproduce this output from pngfix on a Fedora 15 box as follows... % tar -zxvf libpng-1.6.6.tar.gz % cd libpng-1.6.6 % ./configure % make % cp ~/autotest/gdrivers/data/test.png . % ./pngfix test.png IDAT TFB default 10 15 8575 160400 test.png > > > > > > > > > > $ ldd apps/gdalinfo | grep png > > > libpng16.so.16 => /home/even/install-libpng-1.6.2/lib/libpng16.so.16 > > > (0x00007fda95e39000) > > > > > > $ gdalinfo data/gdalicon.png -checksum > > > Driver: PNG/Portable Network Graphics > > > Files: data/gdalicon.png > > > Size is 32, 32 > > > Coordinate System is `' > > > Image Structure Metadata: > > > INTERLEAVE=PIXEL > > > Corner Coordinates: > > > Upper Left ( 0.0, 0.0) > > > Lower Left ( 0.0, 32.0) > > > Upper Right ( 32.0, 0.0) > > > Lower Right ( 32.0, 32.0) > > > Center ( 16.0, 16.0) > > > Band 1 Block=32x1 Type=Byte, ColorInterp=Red > > > Checksum=7617 > > > Mask Flags: PER_DATASET ALPHA > > > Band 2 Block=32x1 Type=Byte, ColorInterp=Green > > > Checksum=7808 > > > Mask Flags: PER_DATASET ALPHA > > > Band 3 Block=32x1 Type=Byte, ColorInterp=Blue > > > Checksum=7377 > > > Mask Flags: PER_DATASET ALPHA > > > Band 4 Block=32x1 Type=Byte, ColorInterp=Alpha > > > Checksum=7926 > > > > > > $ gdalinfo ../autotest/gdrivers/data/test.png -checksum > > > Driver: PNG/Portable Network Graphics > > > Files: ../autotest/gdrivers/data/test.png > > > ../autotest/gdrivers/data/test.wld > > > Size is 400, 400 > > > Coordinate System is `' > > > GeoTransform = > > > 700000.3050000001, 0.38, 0.01 > > > 4287500.695, -0.01, -0.38 > > > Corner Coordinates: > > > Upper Left ( 700000.305, 4287500.695) > > > Lower Left ( 700004.305, 4287348.695) > > > Upper Right ( 700152.305, 4287496.695) > > > Lower Right ( 700156.305, 4287344.695) > > > Center ( 700078.305, 4287422.695) > > > Band 1 Block=400x1 Type=Byte, ColorInterp=Palette > > > Checksum=57921 > > > NoData Value=0 > > > Color Table (RGB with 16 entries) > > > 0: 255,255,255,0 > > > 1: 255,255,208,255 > > > 2: 255,255,204,255 > > > 3: 153,204,255,255 > > > 4: 0,153,255,255 > > > 5: 102,102,102,255 > > > 6: 153,153,153,255 > > > 7: 150,150,150,255 > > > 8: 0,0,0,255 > > > 9: 100,160,210,255 > > > 10: 255,255,255,255 > > > 11: 226,226,226,255 > > > 12: 51,51,51,255 > > > 13: 0,0,0,255 > > > 14: 0,0,0,255 > > > 15: 0,0,0,255 > > > > > > > > > > > > > Jack > > > > > > > > > On Nov 8, 2013, at 8:02 AM, Jack Howarth <[email protected]> > > > > > wrote: > > > > > > On Fri, Nov 08, 2013 at 10:31:08AM -0500, Jack Howarth wrote: > > > > > >> On Thu, Nov 07, 2013 at 04:34:07PM -0500, Jack Howarth wrote: > > > > > >>> The failures on x86_64-apple-darwin13 for the png.py test suite > > > > > >>> from > > > > > >>> auto test for gdal 0.10.1 are the same failures described here... > > > > > >>> > > > > > >>> https://github.com/licq-im/licq/pull/32 > > > > > >>> <https://github.com/licq-im/licq/pull/32> > > > > > >>> > > > > > >>> The failures are completely suppressed by regenerating > > > > > >>> autotest/gdrivers/data/test.png with > > > > > >>> "optipng -nb -nc -np test.png"... > > > > > >> > > > > > >> Just to clarify, I used optipng 0.6.3 for this. The newer optipng > > > > > >> 0.7.4 release seems more problematic as none of the obvious > > > > > >> combination of options can fix the test.png file without changing > > > > > >> the > > > > > >> checksums and causing additional failures. So you will want to use > > > > > >> the optipng 0.6.x release series to fix the bad png files in gdal. > > > > > > > > > > > > Also, my tests here using optipng 0.7.4 to detect damaged png files > > > > > > indicate that only data/gdalicon.png in gdal-0.10.1 needs to be > > > > > > fixed. > > > > > > > > > > > > optipng -nb -nc -np -fix data/gdalicon.png > > > > > > ** Processing: data/gdalicon.png > > > > > > Warning: > > > > > > 32x32 pixels, 4x8 bits/pixel, RGB+alpha > > > > > > Recoverable errors found in input. Fixing... > > > > > > Input IDAT size = 1964 bytes > > > > > > Input file size = 2021 bytes > > > > > > > > > > > > Trying: > > > > > > zc = 9 zm = 8 zs = 0 f = 0 IDAT size = 240 > > > > > > > > > > > > Selecting parameters: > > > > > > zc = 9 zm = 8 zs = 0 f = 0 IDAT size = 240 > > > > > > > > > > > > Output IDAT size = 240 bytes (1724 bytes decrease) > > > > > > Output file size = 313 bytes (1708 bytes = 84.51% decrease) > > > > > > > > > > > > ** Status report > > > > > > 1 file(s) have been processed. > > > > > > 1 error(s) have been encountered. > > > > > > 1 erroneous file(s) have been fixed. > > > > > > > > > > > >>> % ./png.py > > > > > >>> > > > > > >>> TEST: png_1 ... success > > > > > >>> TEST: png_2 ... success > > > > > >>> TEST: png_3 ... success > > > > > >>> TEST: png_4 ... success > > > > > >>> TEST: png_5 ... success > > > > > >>> TEST: png_6 ... success > > > > > >>> TEST: png_7 ... success > > > > > >>> TEST: png_8 ... success > > > > > >>> TEST: png_9 ... success > > > > > >>> TEST: png_10 ... success > > > > > >>> TEST: png_11 ... success > > > > > >>> > > > > > >>> Test Script: png > > > > > >>> Succeeded: 11 > > > > > >>> Failed: 0 (0 blew exceptions) > > > > > >>> Skipped: 0 > > > > > >>> Expected fail:0 > > > > > >>> Duration: 0.08s > > > > > >>> > > > > > >>> _______________________________________________ > > > > > >>> gdal-dev mailing list > > > > > >>> [email protected] > > > > > >>> http://lists.osgeo.org/mailman/listinfo/gdal-dev > > > > > >> > > > > > >> _______________________________________________ > > > > > >> gdal-dev mailing list > > > > > >> [email protected] > > > > > >> http://lists.osgeo.org/mailman/listinfo/gdal-dev > > > > > > > > > > > > _______________________________________________ > > > > > > gdal-dev mailing list > > > > > > [email protected] > > > > > > http://lists.osgeo.org/mailman/listinfo/gdal-dev > > > > > > > > _______________________________________________ > > > > gdal-dev mailing list > > > > [email protected] > > > > http://lists.osgeo.org/mailman/listinfo/gdal-dev > > > > > > -- > > > Geospatial professional services > > > http://even.rouault.free.fr/services.html > > _______________________________________________ > > gdal-dev mailing list > > [email protected] > > http://lists.osgeo.org/mailman/listinfo/gdal-dev > _______________________________________________ > gdal-dev mailing list > [email protected] > http://lists.osgeo.org/mailman/listinfo/gdal-dev _______________________________________________ gdal-dev mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/gdal-dev
