[
https://issues.apache.org/jira/browse/AVRO-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18102693#comment-18102693
]
ASF subversion and git services commented on AVRO-4247:
-------------------------------------------------------
Commit 478ed0b433d9bd1db3a2781f8b6b351b9b39d735 in avro's branch
refs/heads/branch-1.12 from Ismaël Mejía
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=478ed0b433 ]
AVRO-4290: [python] Enforce a maximum decompressed block size (#3850)
* AVRO-4290: [python] Enforce a maximum decompressed block size
When reading a data file, each block is decompressed according to the
file's codec. A block with a very high compression ratio (or a malformed
block) could expand to far more memory than its compressed size. Enforce a
configurable maximum decompressed size across the deflate, bzip2, snappy and
zstandard codecs, mirroring the Java SDK's decompression limit (AVRO-4247).
The limit defaults to 200 MiB and can be overridden with the
AVRO_MAX_DECOMPRESS_LENGTH environment variable; exceeding it raises
AvroDecompressionSizeException.
Assisted-by: GitHub Copilot:claude-opus-4.8
* AVRO-4290: [python] Address review: avoid deflate copy; zstd
check-before-extend
- DeflateCodec.decompress accumulates into a bytearray so the flush() output is
appended in place instead of creating an extra full-size copy of the already
decompressed data.
- ZstandardCodec.decompress checks len(uncompressed) + len(chunk) before
extending, so the buffer never grows past the configured limit.
Assisted-by: GitHub Copilot:claude-opus-4.8
* AVRO-4290: [python] Verify bzip2 stream EOF and drain fully
BZip2Codec.decompress now loops the BZ2Decompressor: it drains all buffered
output (bounded by the limit), verifies the stream reached EOF (rejecting a
truncated/corrupt block with InvalidAvroBinaryEncoding), and handles
concatenated bzip2 streams as bz2.decompress does. Add a truncated-block test.
Assisted-by: GitHub Copilot:claude-opus-4.8
* AVRO-4290: [python] Reject truncated deflate and short snappy blocks
Address review feedback:
- DeflateCodec: after flushing the decompressor, verify decompressor.eof so a
truncated/incomplete raw-deflate block is rejected with
InvalidAvroBinaryEncoding (zlib.decompress used to raise for this; the
decompressobj-based size cap otherwise silently accepted partial output).
- SnappyCodec: validate the block length is >= 4 before reading length - 4
bytes, raising a codec-specific InvalidAvroBinaryEncoding instead of falling
through to the generic decoder error.
Assisted-by: GitHub Copilot:claude-opus-4.8
* AVRO-4290: [python] Add a truncated-deflate-block rejection test
Cover the new DeflateCodec behavior (raising InvalidAvroBinaryEncoding when the
end-of-stream marker isn't reached) with a test that truncates a valid deflate
block and asserts decompress() rejects it, mirroring the bzip2 truncation test.
Assisted-by: GitHub Copilot:claude-opus-4.8
* AVRO-4290: [python] Bound the deflate flush() output too
The previous fix bounded decompress(data, limit+1) but then called
decompressor.flush() with no limit, so the residual output (from the
unconsumed input left by max_length) could still expand unbounded. Drain the
unconsumed_tail in a loop with a per-call max_length and pass a bounded length
to flush(), so the accumulated output can never exceed the limit by more than
one byte before being rejected.
Assisted-by: GitHub Copilot:claude-opus-4.8
* AVRO-4290: [python] Clamp AVRO_MAX_DECOMPRESS_LENGTH to sys.maxsize
_max_decompress_length() returned the raw parsed int. An absurdly large override
is passed as max_length to zlib/bz2 decompress(), which raise OverflowError when
it exceeds Py_ssize_t. Clamp to sys.maxsize so an oversized override is honored
as "effectively unbounded" instead of producing a confusing OverflowError.
Assisted-by: GitHub Copilot:claude-opus-4.8
* AVRO-4290: [python] Avoid limit+1 Py_ssize_t overflow at sys.maxsize
After clamping AVRO_MAX_DECOMPRESS_LENGTH to sys.maxsize, the deflate/bz2 paths
still computed `limit + 1` for the decompress() max_length, which overflows
Py_ssize_t (and raises OverflowError) when limit == sys.maxsize. Add
_decompress_read_ceiling(), which returns limit + 1 normally but sys.maxsize
when the limit is already sys.maxsize (no realistic block can exceed it), and
use it for the bounded read/flush in both paths.
Assisted-by: GitHub Copilot:claude-opus-4.8
> [java] Avro compression codecs should verify decompression size
> ---------------------------------------------------------------
>
> Key: AVRO-4247
> URL: https://issues.apache.org/jira/browse/AVRO-4247
> Project: Apache Avro
> Issue Type: Sub-task
> Components: java
> Affects Versions: 1.12.1
> Reporter: Ismaël Mejía
> Assignee: Steve Loughran
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.13.0, 1.12.2
>
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> {color:#1f2328}All Avro Java compression codecs (Deflate, Zstandard, XZ,
> BZip2, Snappy) decompress data without any size limit, allowing an attacker
> to craft a small Avro file (~50KB) that expands to an extremely large size
> (~50MB+), causing {color}{{OutOfMemoryError}}{color:#1f2328} and crashing the
> JVM.{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
