iemejia commented on code in PR #3850:
URL: https://github.com/apache/avro/pull/3850#discussion_r3568148828
##########
lang/py/avro/codecs.py:
##########
@@ -123,7 +180,31 @@ def decompress(readers_decoder: avro.io.BinaryDecoder) ->
avro.io.BinaryDecoder:
data = readers_decoder.read_bytes()
# -15 is the log of the window size; negative indicates
# "raw" (no zlib headers) decompression. See zlib.h.
- uncompressed = zlib.decompress(data, -15)
+ limit = _max_decompress_length()
+ decompressor = zlib.decompressobj(-15)
+ # Decompress in bounded steps: request at most (limit + 1 - produced)
+ # bytes each call so the accumulated output can never exceed the limit
by
+ # more than one byte before being rejected. decompress()'s max_length
+ # leaves unconsumed input in unconsumed_tail, and flush() would
otherwise
+ # emit the remainder unbounded, so drain the tail in a loop and bound
the
+ # final flush too.
+ uncompressed = bytearray()
+ pending = data
+ while True:
+ want = limit + 1 - len(uncompressed)
+ uncompressed += decompressor.decompress(pending, want)
+ if len(uncompressed) > limit:
+ _raise_decompression_too_large(limit)
+ pending = decompressor.unconsumed_tail
+ if not pending:
+ break
+ uncompressed += decompressor.flush(limit + 1 - len(uncompressed))
Review Comment:
Fixed — added _decompress_read_ceiling(limit) which returns limit + 1
normally but sys.maxsize when limit == sys.maxsize, so the deflate max_length
never overflows Py_ssize_t.
##########
lang/py/avro/codecs.py:
##########
@@ -139,7 +220,30 @@ def compress(data: bytes) -> Tuple[bytes, int]:
def decompress(readers_decoder: avro.io.BinaryDecoder) ->
avro.io.BinaryDecoder:
length = readers_decoder.read_long()
data = readers_decoder.read(length)
- uncompressed = bz2.decompress(data)
+ limit = _max_decompress_length()
+ uncompressed = bytearray()
+ decompressor = bz2.BZ2Decompressor()
+ input_data = data
+ while True:
+ # Request enough to detect exceeding the limit without
allocating
+ # the full (potentially huge) output.
+ want = limit + 1 - len(uncompressed)
+ if want <= 0:
+ want = 1
+ uncompressed += decompressor.decompress(input_data, want)
Review Comment:
Fixed — the bz2 path uses the same _decompress_read_ceiling() so `want`
can't become sys.maxsize + 1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
