Copilot commented on code in PR #3850:
URL: https://github.com/apache/avro/pull/3850#discussion_r3567333194
##########
lang/py/avro/codecs.py:
##########
@@ -123,7 +173,18 @@ def decompress(readers_decoder: avro.io.BinaryDecoder) ->
avro.io.BinaryDecoder:
data = readers_decoder.read_bytes()
# -15 is the log of the window size; negative indicates
# "raw" (no zlib headers) decompression. See zlib.h.
- uncompressed = zlib.decompress(data, -15)
+ limit = _max_decompress_length()
+ decompressor = zlib.decompressobj(-15)
+ # Request at most limit + 1 bytes so that an over-large block is
detected
+ # without allocating the whole (potentially huge) output. Accumulate
into
+ # a bytearray so the flush() output is appended in place rather than
+ # creating an extra full-size copy of the already-decompressed data.
+ uncompressed = bytearray(decompressor.decompress(data, limit + 1))
+ if len(uncompressed) > limit:
+ _raise_decompression_too_large(limit)
+ uncompressed += decompressor.flush()
+ if len(uncompressed) > limit:
+ _raise_decompression_too_large(limit)
Review Comment:
DeflateCodec now uses zlib.decompressobj with a size cap, but it never
verifies that the end-of-stream marker was reached. For truncated/incomplete
deflate input, `decompressobj.flush()` can return partial output without
raising, which would silently accept corrupt blocks. After flushing, check
`decompressor.eof` and raise InvalidAvroBinaryEncoding when false to preserve
the previous behavior of rejecting truncated data.
##########
lang/py/avro/codecs.py:
##########
@@ -158,7 +242,15 @@ def decompress(readers_decoder: avro.io.BinaryDecoder) ->
avro.io.BinaryDecoder:
# Compressed data includes a 4-byte CRC32 checksum
length = readers_decoder.read_long()
data = readers_decoder.read(length - 4)
+ limit = _max_decompress_length()
Review Comment:
SnappyCodec.decompress reads `length - 4` bytes for the compressed payload.
If the declared block length is < 4 (or negative), this currently falls through
to BinaryDecoder.read's generic "expected positive integer" error, which is
less actionable and doesn’t clearly identify the codec/block format issue.
Validate `length >= 4` and raise InvalidAvroBinaryEncoding with a
codec-specific message before attempting the read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
