iemejia opened a new pull request, #3931:
URL: https://github.com/apache/avro/pull/3931
## What is the purpose of the change
Python SDK implementation of AVRO-4303 (parent). The available-bytes guard
added
under AVRO-4296 rejects a declared bytes/string length that exceeds the data
remaining only when the reader can report the number of bytes remaining (a
seekable source). On a non-seekable stream (socket, pipe, decompression
stream)
the check is skipped, and a single `reader.read(n)` for a huge declared `n`
allocates `n` bytes up front before any payload is validated, so a tiny
truncated or hostile input can force a large allocation.
When the remaining byte count is unknown, this reads the value into a buffer
that grows in bounded chunks rather than issuing a single `reader.read(n)`. A
truncated or hostile stream then fails with a bounded
`InvalidAvroBinaryEncoding` after a bounded allocation. The existing
single-read
fast path is kept when the remaining byte count is known.
## Verifying this change
This change added tests and can be verified as follows:
- Added `test_bounded_stream_read.py`: a near-2GB declared bytes/string
length on
a truncated non-seekable stream fails with a bounded
`InvalidAvroBinaryEncoding` (not a huge allocation), and a legitimately
large
value on a non-seekable stream still round-trips.
- Existing decoder tests (`test_io.py`) continue to pass; `ruff check`,
`ruff format` and `mypy --strict` pass.
## Documentation
- Does this pull request introduce a new feature? (no — DoS hardening)
- If yes, how is the feature documented? (docstrings on the new bounded-read
helper in `avro/io.py`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
