Copilot commented on code in PR #3850:
URL: https://github.com/apache/avro/pull/3850#discussion_r3564410918
##########
lang/py/avro/codecs.py:
##########
@@ -123,7 +173,16 @@ def decompress(readers_decoder: avro.io.BinaryDecoder) ->
avro.io.BinaryDecoder:
data = readers_decoder.read_bytes()
# -15 is the log of the window size; negative indicates
# "raw" (no zlib headers) decompression. See zlib.h.
- uncompressed = zlib.decompress(data, -15)
+ limit = _max_decompress_length()
+ decompressor = zlib.decompressobj(-15)
+ # Request at most limit + 1 bytes so that an over-large block is
detected
+ # without allocating the whole (potentially huge) output.
+ uncompressed = decompressor.decompress(data, limit + 1)
+ if len(uncompressed) > limit:
+ _raise_decompression_too_large(limit)
+ uncompressed += decompressor.flush()
+ if len(uncompressed) > limit:
+ _raise_decompression_too_large(limit)
Review Comment:
`uncompressed += decompressor.flush()` concatenates immutable `bytes`,
creating an extra full-size copy of the already-decompressed data. For large
blocks (up to the 200 MiB default limit) this adds significant peak memory and
CPU overhead; using a `bytearray` avoids the copy while preserving the size
checks.
##########
lang/py/avro/codecs.py:
##########
@@ -184,6 +256,8 @@ def decompress(readers_decoder: avro.io.BinaryDecoder) ->
avro.io.BinaryDecoder:
if not chunk:
break
uncompressed.extend(chunk)
+ if len(uncompressed) > limit:
+ _raise_decompression_too_large(limit)
Review Comment:
The limit check happens after `uncompressed.extend(chunk)`, so the bytearray
can grow past `limit` (by up to one chunk) before raising. Checking
`len(uncompressed) + len(chunk)` first avoids allocating beyond the configured
cap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
