SteNicholas commented on code in PR #504: URL: https://github.com/apache/paimon-cpp/pull/504#discussion_r4492392696
########## include/paimon/realtime/index/realtime_index_reader.h: ########## @@ -0,0 +1,58 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +#pragma once + +#include <memory> +#include <optional> +#include <vector> + +#include "paimon/predicate/function_visitor.h" +#include "paimon/utils/roaring_bitmap64.h" +#include "paimon/visibility.h" + +namespace paimon { + +/// Visits predicates on one indexed field, returning realtime offsets. +/// nullopt means no pruning; an empty bitmap means no candidates. False negatives are not allowed. +/// Unsupported predicates default to no pruning. Plugins override only the supported visits. +class PAIMON_EXPORT RealtimeIndexReader : public FunctionVisitor<std::optional<RoaringBitmap64>> { + public: + ~RealtimeIndexReader() override = default; + + Result<std::optional<RoaringBitmap64>> VisitIsNotNull() override; + Result<std::optional<RoaringBitmap64>> VisitIsNull() override; + Result<std::optional<RoaringBitmap64>> VisitEqual(const Literal& literal) override; + Result<std::optional<RoaringBitmap64>> VisitNotEqual(const Literal& literal) override; + Result<std::optional<RoaringBitmap64>> VisitLessThan(const Literal& literal) override; + Result<std::optional<RoaringBitmap64>> VisitLessOrEqual(const Literal& literal) override; + Result<std::optional<RoaringBitmap64>> VisitGreaterThan(const Literal& literal) override; + Result<std::optional<RoaringBitmap64>> VisitGreaterOrEqual(const Literal& literal) override; + Result<std::optional<RoaringBitmap64>> VisitIn(const std::vector<Literal>& literals) override; + Result<std::optional<RoaringBitmap64>> VisitNotIn( + const std::vector<Literal>& literals) override; + Result<std::optional<RoaringBitmap64>> VisitStartsWith(const Literal& literal) override; + Result<std::optional<RoaringBitmap64>> VisitEndsWith(const Literal& literal) override; + Result<std::optional<RoaringBitmap64>> VisitContains(const Literal& literal) override; + Result<std::optional<RoaringBitmap64>> VisitLike(const Literal& literal) override; + Result<std::optional<RoaringBitmap64>> VisitFullTextMatch( + const std::shared_ptr<FullTextMatch>& full_text_match) override; Review Comment: **Remove the obsolete full-text visitor override.** In the current merged tree, `FunctionVisitor` no longer declares `VisitFullTextMatch` or forward-declares `FullTextMatch`. Consequently, including this header alone fails with `use of undeclared identifier 'FullTextMatch'`; including `paimon/predicate/full_text_match.h` first still fails with `only virtual member functions can be marked 'override'`. Both failures reproduce with `c++ -std=c++17 -fsyntax-only -x c++ -I include -I src -I /opt/homebrew/include -` and the corresponding include directives on stdin. The [clang CI job](https://github.com/apache/paimon-cpp/actions/runs/42206084052/job/127853560006) reports the undeclared-type error as well. This blocks the ordinary core build regardless of whether full-text support is enabled, since `realtime_index_reader.cpp` is compiled unconditionally and this header is exported through `paimon/api.h`. Please remove the stale declaration and its implementation to match the current scalar visitor interface. ########## src/paimon/common/reader/predicate_batch_reader.cpp: ########## @@ -123,19 +126,43 @@ Status PredicateBatchReader::BindPredicateToArray(const arrow::Array& array) { return Status::OK(); } -Result<RoaringBitmap32> PredicateBatchReader::Filter(const std::shared_ptr<arrow::Array>& array) { +Result<RoaringBitmap32> PredicateBatchReader::Filter(const std::shared_ptr<arrow::Array>& array, + const RoaringBitmap32& selection) { if (!predicate_filter_) { PAIMON_RETURN_NOT_OK(BindPredicateToArray(*array)); } - PAIMON_ASSIGN_OR_RAISE(std::vector<char> result, - predicate_filter_->Test(*array, arrow_pool_.get())); - assert(result.size() == static_cast<size_t>(array->length())); + // At 25% selectivity or above, evaluate the batch once to avoid repeatedly initializing + // predicate lookup state for fragmented selections. Only selected rows may enter the result. + if (selection.Cardinality() >= array->length() * kFullBatchSelectivityThreshold) { Review Comment: **Handle fragmented selections below the 25% cutoff as well.** This cutoff fixes the dense case from the earlier review, but selecting every eighth row of an 8,192-row batch still takes the slice loop and calls `PredicateFilter::Test()` 1,024 times. Each call rebuilds the IN value-set array and initializes Arrow's lookup state, so ordinary file-index/DV reads can still regress severely compared with the previous single batch evaluation. An independent C++ reproduction with local Arrow 19.0.1, 1,024 IN values, and that 12.5% selection took about **0.032 ms per batch for whole-batch evaluation versus 11.2 ms for single-row slices**, with identical selected results. Even the new test's stride-64 case took about 0.032 ms versus 1.36 ms. These are kernel-level measurements, not Paimon Arrow 17 benchmarks, and reuse the value-set array rather than rebuilding it as this path does. Please gather fragmented candidates for one predicate call, reuse prepared lookup state, or account for the number of runs and initialization cost when choosing the path. The new correctness test alone cannot detect this regression. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
