[
https://issues.apache.org/jira/browse/AVRO-4303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía reassigned AVRO-4303:
----------------------------------
> Bound bytes/string allocation from length prefix on non-seekable streams
> ------------------------------------------------------------------------
>
> Key: AVRO-4303
> URL: https://issues.apache.org/jira/browse/AVRO-4303
> Project: Apache Avro
> Issue Type: Improvement
> Components: csharp, javascript, perl, c, php, ruby, c++, java, python
> Reporter: Ismaël Mejía
> Assignee: Ismaël Mejía
> Priority: Major
>
> h2. Summary
> The "available bytes" guard added under AVRO-4292 rejects a declared
> {{bytes}}/
> {{string}} length that exceeds the data actually remaining -- but *only when
> the
> decoder can report the number of bytes remaining* (memory-backed or seekable
> sources). On a *non-seekable stream* (socket, pipe, a decompression stream,
> etc.) the check is a no-op, so a huge declared length still drives a single
> large up-front allocation ({{new byte[len]}}) before any payload is read. A
> tiny
> truncated input can therefore force a multi-hundred-MB (up to the max-length
> limit) allocation.
> h2. Details
> For example, in the Java SDK {{BinaryDecoder.remainingBytes()}} returns {{-1}}
> for any source other than {{ByteArrayInputStream}}/{{ByteBufferInputStream}},
> so
> {{ensureAvailableBytes()}} does nothing for the common streaming case; the C#
> {{BinaryDecoder.RemainingBytes()}} likewise returns {{-1}} when
> {{!stream.CanSeek}}. The C, C++ and Python stream decoders have the same
> blind spot.
> h2. Proposed approach
> * When the remaining byte count is *unknown* (non-seekable stream), read the
> {{bytes}}/{{string}} value into a *growing buffer in bounded chunks* rather
> than allocating the full attacker-declared length up front. A truncated or
> hostile stream then fails after a bounded allocation (at EOF) instead of
> attempting one huge allocation. This mirrors the collection preallocation
> clamp already applied for arrays/maps.
> * Keep the existing single-allocation fast path when the remaining byte count
> *is* known (the available-bytes check already bounds it there).
> * Apply consistently across the SDKs that allocate a buffer sized from a
> length
> prefix.
> h2. Relationship
> * AVRO-4292 -- collection/allocation limits ("available bytes"). This issue is
> the residual protection for the streaming case that the available-bytes
> check
> cannot cover, and re-touches the same decoder read paths, so it is best
> landed
> after the current AVRO-4292 SDK PRs merge.
> Recommend an umbrella with one subtask per affected SDK.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
