github-actions[bot] commented on code in PR #70314:
URL: https://github.com/apache/doris/pull/70314#discussion_r4489054204
##########
be/src/format_v2/parquet/parquet_crypto.cpp:
##########
@@ -167,6 +167,19 @@ Status deserialize_crypto_metadata(std::span<const
uint8_t> bytes, uint32_t* con
return deserialize_bounded(bytes, consumed, metadata);
}
+Status deserialize_crypto_metadata(std::span<const uint8_t> bytes, uint32_t*
consumed,
+ tparquet::ColumnIndex* metadata) {
+ return deserialize_bounded(bytes, consumed, metadata);
Review Comment:
[P2] Use index-specific limits for ColumnIndex histograms
Routing `ColumnIndex` through `deserialize_bounded` inherits the footer's
65,536-element container cap and largest-footer-object charge per entry.
[Parquet
histograms](https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift)
contain `pages * (max_definition_level + 1)` bins, and [Parquet Java
1.15.2](https://github.com/apache/parquet-java/blob/apache-parquet-1.15.2/parquet-column/src/main/java/org/apache/parquet/internal/column/columnindex/ColumnIndexBuilder.java#L584-L587)
emits them. For example, `optional group s { optional int32 v; }` with 22,000
one-row pages has 66,000 definition bins, despite fitting both the encrypted
page-ordinal limit and the 64 MiB module budget. A usable top-level range
predicate with this leaf projected makes `load_native_page_indexes` parse that
index and propagate `INVALID_METADATA` (possibly hitting the synthetic charge
even earlier), so a valid file previously read sequentially now fails. Use
index-aware decoded-allocat
ion limits, or authenticate and discard unsupported histogram metadata, and
cover this small-wire nested case in a regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
