[
https://issues.apache.org/jira/browse/AVRO-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía updated AVRO-4300:
-------------------------------
Description:
When GenericDatumReader decodes an array it reads the block item count from the
stream and pre-allocates the backing store (newArray -> new Object[count])
before decoding any element. Several independent gaps let a tiny payload drive
an unbounded allocation or an unbounded skip loop and exhaust the heap:
1. Zero-byte elements (classic and fast reader).
Elements whose schema encodes to zero bytes (null, a zero-length fixed, or a
record with only zero-byte fields) consume no input, so
ensureAvailableCollectionBytes (AVRO-4241) skips the check for them
(minBytesPerElement == 0). The collection-length cap is Integer.MAX_VALUE-8, a
JVM array-size ceiling rather than a memory budget. So an array such as
{"type":"array","items":"null"} declaring a block count of 200,000,000 is a ~6
byte payload that allocates a 200M-slot array (~1.6 GB).
2. The fast reader never had the AVRO-4241 available-bytes guard at all.
AVRO-4241 only modified the classic GenericDatumReader (and BinaryDecoder /
Decoder / ValidatingDecoder); it never touched FastReaderBuilder, which is the
default decode path (avro.io.fastread defaults to true). As a result, on the
default reader even a non-zero-byte array such as array<long> or array<int>
with a huge block count and no data pre-allocated new GenericData.Array<>((int)
count) and exhausted the heap.
3. The skip path was unbounded.
BinaryDecoder.skipArray()/skipMap() returned doSkipItems() without applying the
collection cap, and GenericDatumReader.skip() looped over that count, so
skipping a huge block of zero-byte elements (e.g. a writer array<null> field
absent from the reader schema, skipped during projection) could loop
unboundedly even though skipping allocates nothing.
Fix (applied identically on the classic and fast reader paths):
- Add SystemLimitException.checkMaxCollectionAllocation, a heap-aware
cumulative cap for zero-byte elements (default maxMemory()/4/8 elements,
overridable via the org.apache.avro.limits.collectionItems.maxAllocation system
property, mirroring the existing decompression limit).
- Expose GenericDatumReader.ensureAvailableCollectionBytes and apply it,
together with the zero-byte cap, before allocating each array block in
FastReaderBuilder, so the fast and classic readers enforce identical guards.
- Bound the skip path: BinaryDecoder.skipArray()/skipMap() now apply the
structural collection cap (covering the resolving-decoder projection skip on
both reader types), and GenericDatumReader.skip() additionally bounds the
cumulative count using the heap-aware cap for zero-byte elements and the
structural cap otherwise.
- Maps were already safe against pre-allocation on both paths because each
entry carries a string key of at least one byte.
Verification: a full matrix of array<null|long|int> and map<null|long> at count
200,000,000 under -Xmx256m is rejected on both the fast and classic readers
(SystemLimitException for zero-byte elements, EOFException otherwise) instead
of OutOfMemoryError; the skip path is bounded on both readers including under
schema resolution; legitimate collections within the limit still decode. This
is part of AVRO-4292.
was:
When GenericDatumReader decodes an array it reads the block item count from the
stream and pre-allocates the backing store (newArray -> new Object[count])
before decoding any element. Two independent gaps let a tiny payload drive an
unbounded allocation and exhaust the heap:
1. Zero-byte elements (classic and fast reader).
Elements whose schema encodes to zero bytes (null, a zero-length fixed, or a
record with only zero-byte fields) consume no input, so
ensureAvailableCollectionBytes (AVRO-4241) skips the check for them
(minBytesPerElement == 0). The collection-length cap is Integer.MAX_VALUE-8, a
JVM array-size ceiling rather than a memory budget. So an array such as
{"type":"array","items":"null"} declaring a block count of 200,000,000 is a ~6
byte payload that allocates a 200M-slot array (~1.6 GB).
2. The fast reader never had the AVRO-4241 available-bytes guard at all.
AVRO-4241 only modified the classic GenericDatumReader (and BinaryDecoder /
Decoder / ValidatingDecoder); it never touched FastReaderBuilder, which is the
default decode path (avro.io.fastread defaults to true). As a result, on the
default reader even a non-zero-byte array such as array<long> or array<int>
with a huge block count and no data pre-allocated new GenericData.Array<>((int)
count) and exhausted the heap. Only the classic reader was protected.
Fix (applied identically on both the classic and fast reader paths):
- Add SystemLimitException.checkMaxCollectionAllocation, a heap-aware
cumulative cap for zero-byte elements (default maxMemory()/4/8 elements,
overridable via the org.apache.avro.limits.collectionItems.maxAllocation system
property, mirroring the existing decompression limit).
- Expose GenericDatumReader.ensureAvailableCollectionBytes and apply it,
together with the zero-byte cap, before allocating each array block in
FastReaderBuilder, so the fast and classic readers enforce identical guards.
- Maps were already safe on both paths because each entry carries a string key
of at least one byte (ensureAvailableMapBytes on the classic path; key reads
consume bytes on the fast path).
Verification: a full matrix of array<null|long|int> and map<null|long> at count
200,000,000 under -Xmx256m is rejected on both the fast and classic readers
(SystemLimitException for zero-byte elements, EOFException otherwise) instead
of OutOfMemoryError; legitimate collections within the limit still decode. Unit
tests cover all ten collection/reader combinations plus cumulative multi-block
and positive-decode cases. This completes AVRO-4277 by covering the Java SDK's
residual gaps.
Summary: [java] Bound array/map allocation and skipping when decoding
on both the classic and fast readers (zero-byte elements + missing fast-reader
available-bytes check) (was: [java] Bound array allocation when decoding on
both the classic and fast readers (zero-byte elements + missing fast-reader
available-bytes check))
> [java] Bound array/map allocation and skipping when decoding on both the
> classic and fast readers (zero-byte elements + missing fast-reader
> available-bytes check)
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: AVRO-4300
> URL: https://issues.apache.org/jira/browse/AVRO-4300
> Project: Apache Avro
> Issue Type: Sub-task
> Components: java
> Affects Versions: 1.11.5, 1.12.1
> Reporter: Ismaël Mejía
> Assignee: Ismaël Mejía
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.2
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When GenericDatumReader decodes an array it reads the block item count from
> the stream and pre-allocates the backing store (newArray -> new
> Object[count]) before decoding any element. Several independent gaps let a
> tiny payload drive an unbounded allocation or an unbounded skip loop and
> exhaust the heap:
> 1. Zero-byte elements (classic and fast reader).
> Elements whose schema encodes to zero bytes (null, a zero-length fixed, or a
> record with only zero-byte fields) consume no input, so
> ensureAvailableCollectionBytes (AVRO-4241) skips the check for them
> (minBytesPerElement == 0). The collection-length cap is Integer.MAX_VALUE-8,
> a JVM array-size ceiling rather than a memory budget. So an array such as
> {"type":"array","items":"null"} declaring a block count of 200,000,000 is a
> ~6 byte payload that allocates a 200M-slot array (~1.6 GB).
> 2. The fast reader never had the AVRO-4241 available-bytes guard at all.
> AVRO-4241 only modified the classic GenericDatumReader (and BinaryDecoder /
> Decoder / ValidatingDecoder); it never touched FastReaderBuilder, which is
> the default decode path (avro.io.fastread defaults to true). As a result, on
> the default reader even a non-zero-byte array such as array<long> or
> array<int> with a huge block count and no data pre-allocated new
> GenericData.Array<>((int) count) and exhausted the heap.
> 3. The skip path was unbounded.
> BinaryDecoder.skipArray()/skipMap() returned doSkipItems() without applying
> the collection cap, and GenericDatumReader.skip() looped over that count, so
> skipping a huge block of zero-byte elements (e.g. a writer array<null> field
> absent from the reader schema, skipped during projection) could loop
> unboundedly even though skipping allocates nothing.
> Fix (applied identically on the classic and fast reader paths):
> - Add SystemLimitException.checkMaxCollectionAllocation, a heap-aware
> cumulative cap for zero-byte elements (default maxMemory()/4/8 elements,
> overridable via the org.apache.avro.limits.collectionItems.maxAllocation
> system property, mirroring the existing decompression limit).
> - Expose GenericDatumReader.ensureAvailableCollectionBytes and apply it,
> together with the zero-byte cap, before allocating each array block in
> FastReaderBuilder, so the fast and classic readers enforce identical guards.
> - Bound the skip path: BinaryDecoder.skipArray()/skipMap() now apply the
> structural collection cap (covering the resolving-decoder projection skip on
> both reader types), and GenericDatumReader.skip() additionally bounds the
> cumulative count using the heap-aware cap for zero-byte elements and the
> structural cap otherwise.
> - Maps were already safe against pre-allocation on both paths because each
> entry carries a string key of at least one byte.
> Verification: a full matrix of array<null|long|int> and map<null|long> at
> count 200,000,000 under -Xmx256m is rejected on both the fast and classic
> readers (SystemLimitException for zero-byte elements, EOFException otherwise)
> instead of OutOfMemoryError; the skip path is bounded on both readers
> including under schema resolution; legitimate collections within the limit
> still decode. This is part of AVRO-4292.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
