Ismaël Mejía created AVRO-4283:
----------------------------------
Summary: Enforce a maximum decompressed block size across language
SDKs
Key: AVRO-4283
URL: https://issues.apache.org/jira/browse/AVRO-4283
Project: Apache Avro
Issue Type: Improvement
Components: csharp, javascript, perl, c, php, ruby, c++, python
Affects Versions: 1.13.0
Reporter: Ismaël Mejía
When reading an Avro container (data) file, each block is decompressed
according to the file's codec (deflate, bzip2, snappy, xz, zstandard). Several
SDKs read the compressed block into memory and decompress into an output buffer
that grows (or is sized from an attacker-declared length) without any upper
bound. A block with a very high compression ratio can therefore expand to an
amount of memory far larger than the input, leading to excessive allocation on
malformed or hostile input.
The Java SDK addressed this in AVRO-4247 by enforcing a maximum decompressed
size in every codec, defaulting to 200MB and configurable via
{{org.apache.avro.limits.decompress.maxLength}}. This umbrella tracks bringing
the C, C++, C#, JavaScript, Perl, PHP, Python, and Ruby SDKs in line with that
behavior so all SDKs bound decompression output consistently.
h3. Proposed approach
* Enforce a configurable maximum decompressed size while
inflating/decompressing each data-file block, across all codecs.
* Reject a block whose decompressed output would exceed the limit with a clear
error instead of continuing to allocate.
* Default the limit to a runtime-safe value; allow it to be tuned per SDK
(mirroring the Java property).
* Add regression tests: a block that decompresses to more than the limit must
fail with a bounded, well-defined error rather than exhausting memory.
An audit of the current decompression paths (for reference in the subtasks):
* C: {{lang/c/src/codec.c}} (deflate doubles the output buffer; snappy
allocates the attacker-declared uncompressed length; lzma doubles), driven from
{{lang/c/src/datafile.c}}.
* C++: {{lang/c++/impl/DataFile.cc}} (deflate/snappy) and
{{lang/c++/impl/ZstdDecompressWrapper.cc}} (zstd trusts the frame content
size). The existing {{BoundedInputStream}} bounds only the compressed input.
* C#: {{lang/csharp/src/apache/main/File/DataFileReader.cs}} and the codec
classes (deflate/snappy/zstandard/bzip2/xz), all via {{Stream.CopyTo}} into an
unbounded {{MemoryStream}}.
* JavaScript: {{lang/js/lib/files.js}} (deflate via {{zlib.inflateRaw}} with no
{{maxOutputLength}}).
* Perl: {{lang/perl/lib/Avro/DataFileReader.pm}} (a {{block_max_size}}
attribute exists but is never enforced).
* PHP: {{lang/php/lib/DataFile/AvroDataIOReader.php}}
(deflate/zstandard/snappy/bzip2).
* Python: {{lang/py/avro/codecs.py}} (deflate/bzip2/snappy/zstandard).
* Ruby: {{lang/ruby/lib/avro/data_file.rb}} (deflate/snappy/zstandard).
Subtasks: one per SDK (below).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
