github-actions[bot] commented on code in PR #70248:
URL: https://github.com/apache/doris/pull/70248#discussion_r4478540464
##########
be/src/format_v2/parquet/reader/native/page_reader.cpp:
##########
@@ -317,9 +388,15 @@ Status PageReader<IN_COLLECTION,
OFFSET_INDEX>::get_page_data(Slice& slice) {
if (UNLIKELY(_io_ctx && _io_ctx->should_stop)) {
return Status::EndOfFile("stop");
}
- slice.size = _cur_page_header.compressed_page_size;
+ slice.size = _serialized_page_size;
RETURN_IF_ERROR(_reader->read_bytes(slice, _offset, _io_ctx));
_offset += slice.size;
+ if (_page_read_ctx.encryption != nullptr) {
+ RETURN_IF_ERROR(_page_read_ctx.encryption->decrypt_page(
+ std::span(reinterpret_cast<const uint8_t*>(slice.data),
slice.size),
+ _crypto_dictionary, _crypto_page, &_decrypted_page));
Review Comment:
[P2] Include decrypted pages in scratch reclamation
For a supported encrypted column chunk containing a 32 MiB page followed by
many small pages, `decrypt_module()` only resizes `_decrypted_page`, so its
capacity stays at least 32 MiB for the rest of the row group.
`ColumnChunkReader`'s retained/active scratch accounting and
`release_decoder_scratch()` never include this buffer, so the existing 4 MiB
reclamation policy cannot release it. Wide projections and concurrent readers
multiply this extra residency and can exhaust query memory after the large page
is no longer active. Integrate this storage with the existing scratch policy,
releasing oversized inactive capacity at a safe page boundary while preserving
live payload and V2 level views.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
