github-actions[bot] commented on code in PR #66200:
URL: https://github.com/apache/doris/pull/66200#discussion_r3670390861
##########
be/src/format_v2/parquet/reader/native/column_reader.cpp:
##########
@@ -766,6 +787,9 @@ void ScalarColumnReader<IN_COLLECTION,
OFFSET_INDEX>::release_batch_scratch(
// Persistent decoders also own batch-sized value/slice buffers, not
only the reader.
_chunk_reader->release_decoder_scratch(max_retained_bytes);
}
+ if (_serde != nullptr) {
+ _serde->release_parquet_raw_predicate_scratch(max_retained_bytes);
Review Comment:
[P2] Apply the scratch limit to the aggregate, not every buffer
`retained_batch_scratch_bytes()` now sums decoder, SerDe, and predicate
buffers to decide that a leaf exceeds 4 MiB, but this call gives the SerDe the
entire 4 MiB allowance after the decoder has already received the same
allowance; the following vectors do the same. A converted 65,535-value BSS FLBA
DECIMAL256 batch retains 2,097,120 bytes in the decoder, while the SerDe's
padded predicate POD array grows to 4,194,240 retained bytes—64 bytes below the
per-buffer `>` threshold. Those two buffers alone retain about 6 MiB, yet every
release-tested capacity remains below 4 MiB, so after the idle hysteresis every
release is a no-op. Allocate a remaining aggregate budget (or evict inactive
components until the total is under the limit) and test several individually
sub-limit buffers whose sum exceeds the limit.
##########
be/src/format_v2/parquet/reader/native/column_chunk_reader.cpp:
##########
@@ -1725,22 +1883,104 @@ Status ColumnChunkReader<IN_COLLECTION,
OFFSET_INDEX>::filter_fixed_width_values
selection.total_values);
}
+ VExprSPtrs raw_conjuncts;
+ VExprSPtrs null_map_conjuncts;
+ for (const auto& conjunct : conjuncts) {
+ if (conjunct->can_execute_on_null_map(_field_schema->data_type,
column_id)) {
+ null_map_conjuncts.push_back(conjunct);
+ } else {
+ raw_conjuncts.push_back(conjunct);
+ }
+ }
physical_matches->clear();
+ physical_conversion_nulls->clear();
if (selection.selected_values == 0) {
RETURN_IF_ERROR(_page_decoder->skip_values(selection.total_values));
+ *execution_kind = DirectPredicateExecutionKind::DEFINITION_LEVEL;
+ } else if (raw_conjuncts.empty()) {
+ if (serde.supports_parquet_raw_predicate(decode_context)) {
Review Comment:
[P1] Use the current page encoding before choosing the conversion branch
`decode_context.encoding` is only updated in `materialize_values()`, so a
decoder-direct first read still carries its default `PLAIN` value even when
`_current_encoding` is dictionary encoding. For a predicate-only `IS NULL` on a
dictionary-encoded INT32/INT64/FLOAT/DOUBLE (including DECIMAL32/64 admitted by
`has_identity_value`), this check therefore says raw conversion is supported
and `SelectedDecodeSource` reaches the dictionary decoder's default fixed-value
path, which returns `NotSupported` instead of filtering a valid page. Translate
`_current_encoding` into the context before these capability checks (or branch
on the decoder's actual dictionary state), and add a dictionary fixed-width
NULL-predicate test; for decimals, preserve strict/permissive conversion
validation rather than treating metadata alone as payload validation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
