SteNicholas commented on code in PR #504:
URL: https://github.com/apache/paimon-cpp/pull/504#discussion_r4457197784


##########
src/paimon/common/reader/predicate_batch_reader.cpp:
##########
@@ -123,19 +126,29 @@ Status PredicateBatchReader::BindPredicateToArray(const 
arrow::Array& array) {
     return Status::OK();
 }
 
-Result<RoaringBitmap32> PredicateBatchReader::Filter(const 
std::shared_ptr<arrow::Array>& array) {
+Result<RoaringBitmap32> PredicateBatchReader::Filter(const 
std::shared_ptr<arrow::Array>& array,
+                                                     const RoaringBitmap32& 
selection) {
     if (!predicate_filter_) {
         PAIMON_RETURN_NOT_OK(BindPredicateToArray(*array));
     }
-    PAIMON_ASSIGN_OR_RAISE(std::vector<char> result,
-                           predicate_filter_->Test(*array, arrow_pool_.get()));
-    assert(result.size() == static_cast<size_t>(array->length()));
+    PAIMON_ASSIGN_OR_RAISE(arrow::ArrayVector slices,
+                           ReaderUtils::GenerateFilteredArrayVector(array, 
selection));
     RoaringBitmap32 is_valid;
-    for (int32_t i = 0; i < static_cast<int32_t>(result.size()); i++) {
-        if (result[i]) {
-            is_valid.Add(i);
+    // Slices share the source buffers; selection maps their results to 
original batch rows.
+    auto iter = selection.Begin();
+    for (const std::shared_ptr<arrow::Array>& slice : slices) {
+        PAIMON_ASSIGN_OR_RAISE(std::vector<char> result,
+                               predicate_filter_->Test(*slice, 
arrow_pool_.get()));

Review Comment:
   **Preserve batched predicate evaluation for fragmented selections**: An 
alternating selection such as `{0, 2, 4, ...}` produces one slice per selected 
row, so this loop calls `PredicateFilter::Test()` thousands of times for a 
single input batch. `MultiLiteralsLeafFunction::Test()` rebuilds the IN value 
set and invokes Arrow `IsIn` for each call, repeatedly initializing the lookup 
state. The work therefore grows with the number of selected runs multiplied by 
the IN-list size. Since this is the common reader, existing file-index and 
deletion-vector reads are affected too.
   
   An isolated C++ experiment with the locally available Arrow 19.0.1, 8,192 
input rows, an alternating selection, and 1,024 IN values took about 0.035 ms 
for one whole-batch call versus 40 ms for the single-row calls, with identical 
selected results. This is a kernel-level reproduction, not a benchmark of 
Paimon's Arrow 17 build; it also reused the value-set array, whereas this path 
rebuilds it.
   
   Could we gather fragmented candidates once, evaluate the predicate in a 
batch, and map the results back, or reuse the initialized predicate lookup 
state across slices? Please add performance coverage for a sparse selection 
combined with a large IN list.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to