Ismaël Mejía created AVRO-4325:
----------------------------------
Summary: [Trevni] Validate column-file header counts and lengths
before allocating in the Trevni readers
Key: AVRO-4325
URL: https://issues.apache.org/jira/browse/AVRO-4325
Project: Apache Avro
Issue Type: Improvement
Components: trevni
Reporter: Ismaël Mejía
The Trevni columnar reader sizes several allocations directly from values read
from the file header/metadata without validating them against the input
actually available. For a malformed, corrupted, or truncated Trevni file these
counts/lengths may greatly exceed the bytes actually present, leading to
oversized eager allocations or an unexpected {{NegativeArraySizeException}}:
* {{ColumnFileReader.readHeader}} allocates {{new
ColumnDescriptor[columnCount]}} (and a {{HashMap}} sized to {{columnCount}})
from the 32-bit column count before any column metadata is read.
* {{ColumnDescriptor.ensureBlocksRead}} allocates {{new
BlockDescriptor[blockCount]}} from the block count.
* {{InputBuffer.readBytes}}/{{readString}} allocate {{new byte[readInt()]}}
from a length prefix.
* {{ColumnValues.startBlock}} computes {{new byte[compressedSize +
checksumSize]}}; the addition can overflow to a negative size
({{NegativeArraySizeException}}).
Unlike the Java SDK, Trevni has no shared limit/validation layer. We should:
* validate these counts/lengths against the remaining input (and a sane
maximum) before allocating,
* adopt a consistent bound across the Trevni readers ({{ColumnFileReader}},
{{ColumnDescriptor}}, {{InputBuffer}}, {{ColumnValues}}), similar in spirit to
the Java SDK's {{SystemLimits}}, and
* guard the {{compressedSize + checksum}} addition against overflow,
so that reading a malformed file fails fast with a clear exception. Valid files
continue to read normally.
Relevant code:
*
{{lang/java/trevni/core/src/main/java/org/apache/trevni/ColumnFileReader.java}}
— {{readHeader()}}.
*
{{lang/java/trevni/core/src/main/java/org/apache/trevni/ColumnDescriptor.java}}
— {{ensureBlocksRead()}}.
* {{lang/java/trevni/core/src/main/java/org/apache/trevni/InputBuffer.java}} —
{{readBytes()}}/{{readString()}}.
* {{lang/java/trevni/core/src/main/java/org/apache/trevni/ColumnValues.java}} —
{{startBlock()}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
