Ismaël Mejía created AVRO-4303:
----------------------------------
Summary: Bound bytes/string allocation from length prefix on
non-seekable streams
Key: AVRO-4303
URL: https://issues.apache.org/jira/browse/AVRO-4303
Project: Apache Avro
Issue Type: Improvement
Components: csharp, javascript, perl, c, php, ruby, c++, java, python
Reporter: Ismaël Mejía
Assignee: Ismaël Mejía
h2. Summary
The "available bytes" guard added under AVRO-4292 rejects a declared {{bytes}}/
{{string}} length that exceeds the data actually remaining -- but *only when the
decoder can report the number of bytes remaining* (memory-backed or seekable
sources). On a *non-seekable stream* (socket, pipe, a decompression stream,
etc.) the check is a no-op, so a huge declared length still drives a single
large up-front allocation ({{new byte[len]}}) before any payload is read. A tiny
truncated input can therefore force a multi-hundred-MB (up to the max-length
limit) allocation.
h2. Details
For example, in the Java SDK {{BinaryDecoder.remainingBytes()}} returns {{-1}}
for any source other than {{ByteArrayInputStream}}/{{ByteBufferInputStream}}, so
{{ensureAvailableBytes()}} does nothing for the common streaming case; the C#
{{BinaryDecoder.RemainingBytes()}} likewise returns {{-1}} when
{{!stream.CanSeek}}. The C, C++ and Python stream decoders have the same
blind spot.
h2. Proposed approach
* When the remaining byte count is *unknown* (non-seekable stream), read the
{{bytes}}/{{string}} value into a *growing buffer in bounded chunks* rather
than allocating the full attacker-declared length up front. A truncated or
hostile stream then fails after a bounded allocation (at EOF) instead of
attempting one huge allocation. This mirrors the collection preallocation
clamp already applied for arrays/maps.
* Keep the existing single-allocation fast path when the remaining byte count
*is* known (the available-bytes check already bounds it there).
* Apply consistently across the SDKs that allocate a buffer sized from a length
prefix.
h2. Relationship
* AVRO-4292 -- collection/allocation limits ("available bytes"). This issue is
the residual protection for the streaming case that the available-bytes check
cannot cover, and re-touches the same decoder read paths, so it is best landed
after the current AVRO-4292 SDK PRs merge.
Recommend an umbrella with one subtask per affected SDK.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
