Copilot commented on code in PR #3850:
URL: https://github.com/apache/avro/pull/3850#discussion_r3568128140
##########
lang/py/avro/codecs.py:
##########
@@ -123,7 +180,31 @@ def decompress(readers_decoder: avro.io.BinaryDecoder) ->
avro.io.BinaryDecoder:
data = readers_decoder.read_bytes()
# -15 is the log of the window size; negative indicates
# "raw" (no zlib headers) decompression. See zlib.h.
- uncompressed = zlib.decompress(data, -15)
+ limit = _max_decompress_length()
+ decompressor = zlib.decompressobj(-15)
+ # Decompress in bounded steps: request at most (limit + 1 - produced)
+ # bytes each call so the accumulated output can never exceed the limit
by
+ # more than one byte before being rejected. decompress()'s max_length
+ # leaves unconsumed input in unconsumed_tail, and flush() would
otherwise
+ # emit the remainder unbounded, so drain the tail in a loop and bound
the
+ # final flush too.
+ uncompressed = bytearray()
+ pending = data
+ while True:
+ want = limit + 1 - len(uncompressed)
+ uncompressed += decompressor.decompress(pending, want)
+ if len(uncompressed) > limit:
+ _raise_decompression_too_large(limit)
+ pending = decompressor.unconsumed_tail
+ if not pending:
+ break
+ uncompressed += decompressor.flush(limit + 1 - len(uncompressed))
Review Comment:
When AVRO_MAX_DECOMPRESS_LENGTH is set to a very large value,
`_max_decompress_length()` clamps it to `sys.maxsize`, but the deflate path
still computes `limit + 1` for the `max_length` argument. If `limit ==
sys.maxsize`, `limit + 1` overflows `Py_ssize_t` and `zlib` can raise
`OverflowError`, defeating the intent of the clamping. Compute a capped `cap`
(no `+ 1` when `limit == sys.maxsize`) and ensure `want` stays >= 1 for
progress.
##########
lang/py/avro/codecs.py:
##########
@@ -139,7 +220,30 @@ def compress(data: bytes) -> Tuple[bytes, int]:
def decompress(readers_decoder: avro.io.BinaryDecoder) ->
avro.io.BinaryDecoder:
length = readers_decoder.read_long()
data = readers_decoder.read(length)
- uncompressed = bz2.decompress(data)
+ limit = _max_decompress_length()
+ uncompressed = bytearray()
+ decompressor = bz2.BZ2Decompressor()
+ input_data = data
+ while True:
+ # Request enough to detect exceeding the limit without
allocating
+ # the full (potentially huge) output.
+ want = limit + 1 - len(uncompressed)
+ if want <= 0:
+ want = 1
+ uncompressed += decompressor.decompress(input_data, want)
Review Comment:
Same `limit + 1` overflow risk as the deflate path: if the env override is
clamped to `sys.maxsize`, `want = limit + 1 - len(uncompressed)` can become
`sys.maxsize + 1` and `bz2.BZ2Decompressor.decompress(..., max_length=want)`
may raise `OverflowError`. Use a `cap` that avoids `+ 1` at `sys.maxsize`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
