fallintoplace opened a new issue, #950: URL: https://github.com/apache/arrow-go/issues/950
### Describe the enhancement requested Parquet page headers provide signed 32-bit compressed and uncompressed sizes. The page reader rejects negative values but otherwise accepts positive values up to `math.MaxInt32`. Several paths then resize buffers from those declarations before proving that the body fits inside the remaining column chunk or that the uncompressed size is reasonable. Streaming support reduces the allocation risk for some data-page encodings, but dictionary pages and non-streaming paths still allocate from the declared size. Reader-side limits would provide a predictable resource boundary for corrupt or unexpectedly large files. ### Proposed direction Add reader properties for limits such as: - Maximum compressed page size. - Maximum uncompressed page size. - Optionally, maximum decompression ratio. Before allocating or decoding: - Check the compressed size against the remaining column-chunk bytes. - Validate compressed and uncompressed sizes against configured limits. - Use checked arithmetic for level and value-region sizes. - Bound streaming readers to the validated page body. - Preserve an explicit override for workloads with unusually large valid pages. The defaults need maintainer agreement because existing users may read pages larger than common recommendations. ### Regression coverage - `math.MaxInt32` compressed and uncompressed declarations. - A compressed page extending beyond its column chunk. - Small compressed input declaring a very large output. - Dictionary pages and data page versions 1 and 2. - Allocator accounting showing rejection before a large allocation. Observed on current `main` at `750586710988bc18f7af64fd29af8e0acdb252ac`. ### Component(s) Parquet -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
