Nick Knize created SIS-634:
------------------------------
Summary: GeoTIFF Deflate decompression fails on JDK 21.0.11+ / 23+
(zlib 1.3.1 strict header check)
Key: SIS-634
URL: https://issues.apache.org/jira/browse/SIS-634
Project: Spatial Information Systems
Issue Type: Bug
Components: Coverage, Storage
Affects Versions: 1.6, 2.0
Environment: h2. Affected JDK Versions
- JDK 23+ (ships with zlib 1.3.1)
- JDK 21.0.11+ (backported via
[JDK-8324632|https://bugs.openjdk.org/browse/JDK-8324632])
- JDK 17.0.x recent patches (same backport)
- Not affected: Older JDK patch levels with zlib < 1.3.1, which were lenient
about mismatched headers
Reporter: Nick Knize
Assignee: Martin Desruisseaux
h2. Problem
{{org.apache.sis.storage.geotiff.inflater.ZIP}} creates {{new Inflater()}}
(equivalent to {{new Inflater(false)}}), which expects zlib-wrapped Deflate
data (RFC 1950: 2-byte CMF+FLG header + Adler-32 checksum).
However, the TIFF specification for Deflate compression (compression tag 8 /
"Adobe Deflate") stores raw Deflate data (RFC 1951) without a zlib wrapper.
This causes {{java.util.zip.DataFormatException: incorrect header check}}
when reading tile data from any Deflate-compressed GeoTIFF produced by standard
tools (GDAL, libtiff, etc.) on JDKs with zlib >= 1.3.1.
h2. Root Cause
Reader ({{inflater/ZIP.java}} line 54):
{code:java}
inflater = new Inflater(); // nowrap=false — expects zlib header
{code}
Should be:
{code:java}
inflater = new Inflater(true); // nowrap=true — raw Deflate per TIFF spec
{code}
Writer ({{writer/ZIP.java}} line 48):
{code:java}
deflater = new Deflater(level); // nowrap=false — writes zlib header
{code}
Should be:
{code:java}
deflater = new Deflater(level, true); // nowrap=true — raw Deflate per TIFF
spec
{code}
The reader and writer are both wrong in a compatible way — SIS-written TIFFs
can be SIS-read because both use zlib-wrapped format. But SIS cannot read
Deflate-compressed TIFFs written by any other implementation (GDAL, libtiff,
ImageMagick, NASA, USGS, etc.) that correctly follows the TIFF spec and writes
raw Deflate.
h2. TIFF Specification Reference
TIFF Technical Note #2 (Adobe Deflate, compression tag 8):
{quote}
The Deflate compressed data is stored in a TIFF strip or tile as specified in
RFC 1951 (raw Deflate without zlib wrapper).
{quote}
The {{java.util.zip.Inflater(boolean nowrap)}} Javadoc confirms:
{{nowrap=true}} means "support the decompression of data that is not wrapped in
ZLIB format."
h2. Reproduction
{code:java}
// Simulate what SIS inflater/ZIP.java does internally
Inflater inflater = new Inflater(false); // SIS's current code
// Minimal raw-Deflate block (BFINAL=1, BTYPE=01 fixed Huffman)
byte[] rawDeflate = new byte[] { 0x63, 0x00, 0x00, 0x00, 0x01, 0x00, 0x01 };
inflater.setInput(rawDeflate);
byte[] out = new byte[16];
inflater.inflate(out); // throws DataFormatException: incorrect header check
{code}
This passes on JDK 21.0.4 (old zlib) but throws on JDK 21.0.11+ and JDK 23+
(zlib 1.3.1).
h2. Stack Trace (from real NASA COG)
{code}
java.io.IOException: java.util.zip.DataFormatException: incorrect header check
at org.apache.sis.storage.geotiff.inflater.ZIP.read(ZIP.java:98)
at
org.apache.sis.storage.geotiff.reader.CompressedSubset.readSlice(CompressedSubset.java:232)
at
org.apache.sis.storage.geotiff.reader.DataSubset.readSlice(DataSubset.java:340)
at
org.apache.sis.storage.geotiff.reader.DataSubset$Tile.compute(DataSubset.java:463)
...
Caused by: java.util.zip.DataFormatException: incorrect header check
{code}
h2. Impact
Every Deflate-compressed GeoTIFF produced by standard tools fails to read on
modern JDKs. This includes:
- NASA Earth science imagery (OMI, MODIS, Landsat COGs)
- USGS National Map data
- Any GDAL-produced COG with {{COMPRESS=DEFLATE}} (the most common
compression for COGs)
- Copernicus / Sentinel imagery
Only SIS-written Deflate TIFFs or TIFFs using other compression (LZW,
uncompressed, JPEG) are unaffected.
h2. Suggested Fix
Two one-line changes:
{{inflater/ZIP.java:54}}: {{new Inflater()}} → {{new Inflater(true)}}
{{writer/ZIP.java:48}}: {{new Deflater(level)}} → {{new Deflater(level,
true)}}
Note on backward compatibility: The writer fix means SIS-written Deflate
TIFFs will change from zlib-wrapped to raw Deflate (matching the spec).
Existing SIS-written files using the incorrect zlib-wrapped format would fail
to read with the fixed reader. Options:
- Fix only the reader for immediate ecosystem compatibility
- Make the reader try {{Inflater(true)}} first, fall back to
{{Inflater(false)}} for backward
compatibility with SIS-written files
- Fix both and document the breaking change
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
