github-actions[bot] commented on code in PR #66149:
URL: https://github.com/apache/doris/pull/66149#discussion_r3670595415
##########
be/src/exprs/hybrid_set.h:
##########
@@ -207,6 +207,27 @@ class HybridSetBase : public FilterBase {
}
}
+ virtual void find_batch_raw_fixed_negative(const uint8_t* values, size_t
rows,
+ size_t value_width, uint8_t*
matches) const {
+ for (size_t row = 0; row < rows; ++row) {
+ matches[row] &= find(values + row * value_width) ? 0 : 1;
+ }
+ }
+
+ virtual void find_batch_raw_binary(const StringRef* values, size_t rows,
+ uint8_t* matches) const {
+ for (size_t row = 0; row < rows; ++row) {
+ matches[row] &= find(values[row].data, values[row].size) ? 1 : 0;
Review Comment:
[P2] Avoid allocating a string for every raw IN probe
This new batch loop reaches `StringSet::find(data, size)` for
CHAR/VARCHAR/STRING runtime-filter IN predicates, and that overload constructs
an owning `std::string` before each hash lookup. Values beyond the small-string
buffer therefore allocate and copy once per selected row, even though this
decoder-direct path is intended to avoid intermediate string materialization.
Use a heterogeneous `StringRef`/byte-span lookup (or another reusable
non-owning probe) and add long-string multi-page coverage that verifies the raw
path does not allocate per value.
##########
be/src/format_v2/parquet/reader/native/column_chunk_reader.cpp:
##########
@@ -1725,22 +1890,104 @@ Status ColumnChunkReader<IN_COLLECTION,
OFFSET_INDEX>::filter_fixed_width_values
selection.total_values);
}
+ VExprSPtrs raw_conjuncts;
+ VExprSPtrs null_map_conjuncts;
+ for (const auto& conjunct : conjuncts) {
+ if (conjunct->can_execute_on_null_map(_field_schema->data_type,
column_id)) {
+ null_map_conjuncts.push_back(conjunct);
+ } else {
+ raw_conjuncts.push_back(conjunct);
+ }
+ }
physical_matches->clear();
+ physical_conversion_nulls->clear();
if (selection.selected_values == 0) {
RETURN_IF_ERROR(_page_decoder->skip_values(selection.total_values));
+ *execution_kind = DirectPredicateExecutionKind::DEFINITION_LEVEL;
+ } else if (raw_conjuncts.empty()) {
+ if (serde.supports_parquet_raw_predicate(page_decode_context)) {
+ // Definition levels alone cannot distinguish a present payload
that permissive SerDe
+ // conversion turns into NULL (or a strict conversion error).
Decode only the selected
+ // payloads through the same conversion kernel before evaluating
IS NULL/IS NOT NULL.
+ SelectedDecodeSource selected_source(*_page_decoder, selection);
+ FixedWidthPredicateConsumer consumer(raw_conjuncts,
_field_schema->data_type, column_id,
+ physical_matches,
projected_column,
+ physical_conversion_nulls);
+ RETURN_IF_ERROR(serde.read_parquet_raw_predicate(selected_source,
page_decode_context,
+
selection.selected_values,
+
enable_strict_mode, consumer));
+ // Conversion is needed only to refine the logical null map; no
value predicate ran.
+ *execution_kind = DirectPredicateExecutionKind::DEFINITION_LEVEL;
+ } else {
+
RETURN_IF_ERROR(_page_decoder->skip_values(selection.total_values));
+ physical_matches->resize_fill(selection.selected_values, 1);
+ *execution_kind = DirectPredicateExecutionKind::DEFINITION_LEVEL;
+ }
} else {
- FixedWidthPredicateConsumer consumer(conjuncts,
_field_schema->data_type, column_id,
- physical_matches,
projected_column);
- RETURN_IF_ERROR(_page_decoder->decode_selected_fixed_values(selection,
consumer));
+ const bool all_raw_binary = std::ranges::all_of(raw_conjuncts,
[&](const auto& conjunct) {
+ return
conjunct->can_execute_on_raw_binary_values(_field_schema->data_type, column_id);
+ });
+ const bool all_raw_fixed = std::ranges::all_of(raw_conjuncts,
[&](const auto& conjunct) {
+ return
conjunct->can_execute_on_raw_fixed_values(_field_schema->data_type, column_id);
+ });
+ if (all_raw_binary &&
+ supports_raw_binary_filter_encoding(_current_encoding,
_metadata.type)) {
+ BinaryPredicateConsumer consumer(raw_conjuncts,
_field_schema->data_type, column_id,
+ physical_matches,
projected_column,
+ &_binary_predicate_refs,
&_binary_projected_refs);
+ if (_metadata.type == tparquet::Type::BYTE_ARRAY) {
+
RETURN_IF_ERROR(_page_decoder->decode_selected_binary_values(selection,
consumer));
Review Comment:
[P2] Retire completed binary output before scratch aging
This new raw-binary call leaves completed decoder output at its last logical
size. For example, DELTA_BYTE_ARRAY retains its selected payload/ref buffers
after the consumer returns; converted SerDe buffers have the same lifecycle. A
later all-NULL fragment takes `skip_values(0)` and clears neither. If the
completed output exceeded 4 MiB, `active_batch_scratch_bytes()` remains over
budget and every amortized check resets the idle counter, so even three idle
checks cannot release the capacity. Clear disposable output sizes after
consumption (while retaining capacity), or report only memory still borrowed by
the current operation, and cover an oversized value batch followed by enough
value-less fragments to cross the idle cadence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
