SteNicholas commented on code in PR #504:
URL: https://github.com/apache/paimon-cpp/pull/504#discussion_r4492392696


##########
include/paimon/realtime/index/realtime_index_reader.h:
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#pragma once
+
+#include <memory>
+#include <optional>
+#include <vector>
+
+#include "paimon/predicate/function_visitor.h"
+#include "paimon/utils/roaring_bitmap64.h"
+#include "paimon/visibility.h"
+
+namespace paimon {
+
+/// Visits predicates on one indexed field, returning realtime offsets.
+/// nullopt means no pruning; an empty bitmap means no candidates. False 
negatives are not allowed.
+/// Unsupported predicates default to no pruning. Plugins override only the 
supported visits.
+class PAIMON_EXPORT RealtimeIndexReader : public 
FunctionVisitor<std::optional<RoaringBitmap64>> {
+ public:
+    ~RealtimeIndexReader() override = default;
+
+    Result<std::optional<RoaringBitmap64>> VisitIsNotNull() override;
+    Result<std::optional<RoaringBitmap64>> VisitIsNull() override;
+    Result<std::optional<RoaringBitmap64>> VisitEqual(const Literal& literal) 
override;
+    Result<std::optional<RoaringBitmap64>> VisitNotEqual(const Literal& 
literal) override;
+    Result<std::optional<RoaringBitmap64>> VisitLessThan(const Literal& 
literal) override;
+    Result<std::optional<RoaringBitmap64>> VisitLessOrEqual(const Literal& 
literal) override;
+    Result<std::optional<RoaringBitmap64>> VisitGreaterThan(const Literal& 
literal) override;
+    Result<std::optional<RoaringBitmap64>> VisitGreaterOrEqual(const Literal& 
literal) override;
+    Result<std::optional<RoaringBitmap64>> VisitIn(const std::vector<Literal>& 
literals) override;
+    Result<std::optional<RoaringBitmap64>> VisitNotIn(
+        const std::vector<Literal>& literals) override;
+    Result<std::optional<RoaringBitmap64>> VisitStartsWith(const Literal& 
literal) override;
+    Result<std::optional<RoaringBitmap64>> VisitEndsWith(const Literal& 
literal) override;
+    Result<std::optional<RoaringBitmap64>> VisitContains(const Literal& 
literal) override;
+    Result<std::optional<RoaringBitmap64>> VisitLike(const Literal& literal) 
override;
+    Result<std::optional<RoaringBitmap64>> VisitFullTextMatch(
+        const std::shared_ptr<FullTextMatch>& full_text_match) override;

Review Comment:
   **Remove the obsolete full-text visitor override.** In the current merged 
tree, `FunctionVisitor` no longer declares `VisitFullTextMatch` or 
forward-declares `FullTextMatch`. Consequently, including this header alone 
fails with `use of undeclared identifier 'FullTextMatch'`; including 
`paimon/predicate/full_text_match.h` first still fails with `only virtual 
member functions can be marked 'override'`. Both failures reproduce with `c++ 
-std=c++17 -fsyntax-only -x c++ -I include -I src -I /opt/homebrew/include -` 
and the corresponding include directives on stdin. The [clang CI 
job](https://github.com/apache/paimon-cpp/actions/runs/42206084052/job/127853560006)
 reports the undeclared-type error as well.
   
   This blocks the ordinary core build regardless of whether full-text support 
is enabled, since `realtime_index_reader.cpp` is compiled unconditionally and 
this header is exported through `paimon/api.h`. Please remove the stale 
declaration and its implementation to match the current scalar visitor 
interface.



##########
src/paimon/common/reader/predicate_batch_reader.cpp:
##########
@@ -123,19 +126,43 @@ Status PredicateBatchReader::BindPredicateToArray(const 
arrow::Array& array) {
     return Status::OK();
 }
 
-Result<RoaringBitmap32> PredicateBatchReader::Filter(const 
std::shared_ptr<arrow::Array>& array) {
+Result<RoaringBitmap32> PredicateBatchReader::Filter(const 
std::shared_ptr<arrow::Array>& array,
+                                                     const RoaringBitmap32& 
selection) {
     if (!predicate_filter_) {
         PAIMON_RETURN_NOT_OK(BindPredicateToArray(*array));
     }
-    PAIMON_ASSIGN_OR_RAISE(std::vector<char> result,
-                           predicate_filter_->Test(*array, arrow_pool_.get()));
-    assert(result.size() == static_cast<size_t>(array->length()));
+    // At 25% selectivity or above, evaluate the batch once to avoid 
repeatedly initializing
+    // predicate lookup state for fragmented selections. Only selected rows 
may enter the result.
+    if (selection.Cardinality() >= array->length() * 
kFullBatchSelectivityThreshold) {

Review Comment:
   **Handle fragmented selections below the 25% cutoff as well.** This cutoff 
fixes the dense case from the earlier review, but selecting every eighth row of 
an 8,192-row batch still takes the slice loop and calls 
`PredicateFilter::Test()` 1,024 times. Each call rebuilds the IN value-set 
array and initializes Arrow's lookup state, so ordinary file-index/DV reads can 
still regress severely compared with the previous single batch evaluation.
   
   An independent C++ reproduction with local Arrow 19.0.1, 1,024 IN values, 
and that 12.5% selection took about **0.032 ms per batch for whole-batch 
evaluation versus 11.2 ms for single-row slices**, with identical selected 
results. Even the new test's stride-64 case took about 0.032 ms versus 1.36 ms. 
These are kernel-level measurements, not Paimon Arrow 17 benchmarks, and reuse 
the value-set array rather than rebuilding it as this path does.
   
   Please gather fragmented candidates for one predicate call, reuse prepared 
lookup state, or account for the number of runs and initialization cost when 
choosing the path. The new correctness test alone cannot detect this regression.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to