SteNicholas commented on code in PR #504:
URL: https://github.com/apache/paimon-cpp/pull/504#discussion_r4457197784


##########
src/paimon/common/reader/predicate_batch_reader.cpp:
##########
@@ -123,19 +126,29 @@ Status PredicateBatchReader::BindPredicateToArray(const 
arrow::Array& array) {
     return Status::OK();
 }
 
-Result<RoaringBitmap32> PredicateBatchReader::Filter(const 
std::shared_ptr<arrow::Array>& array) {
+Result<RoaringBitmap32> PredicateBatchReader::Filter(const 
std::shared_ptr<arrow::Array>& array,
+                                                     const RoaringBitmap32& 
selection) {
     if (!predicate_filter_) {
         PAIMON_RETURN_NOT_OK(BindPredicateToArray(*array));
     }
-    PAIMON_ASSIGN_OR_RAISE(std::vector<char> result,
-                           predicate_filter_->Test(*array, arrow_pool_.get()));
-    assert(result.size() == static_cast<size_t>(array->length()));
+    PAIMON_ASSIGN_OR_RAISE(arrow::ArrayVector slices,
+                           ReaderUtils::GenerateFilteredArrayVector(array, 
selection));
     RoaringBitmap32 is_valid;
-    for (int32_t i = 0; i < static_cast<int32_t>(result.size()); i++) {
-        if (result[i]) {
-            is_valid.Add(i);
+    // Slices share the source buffers; selection maps their results to 
original batch rows.
+    auto iter = selection.Begin();
+    for (const std::shared_ptr<arrow::Array>& slice : slices) {
+        PAIMON_ASSIGN_OR_RAISE(std::vector<char> result,
+                               predicate_filter_->Test(*slice, 
arrow_pool_.get()));

Review Comment:
   [P2] Preserve batched predicate evaluation for fragmented selections
   
   An alternating selection such as `{0, 2, 4, ...}` produces one slice per 
selected row, so this loop calls `PredicateFilter::Test()` thousands of times 
for a single input batch. `MultiLiteralsLeafFunction::Test()` rebuilds the IN 
value set and invokes Arrow `IsIn` for each call, repeatedly initializing the 
lookup state. The work therefore grows with the number of selected runs 
multiplied by the IN-list size. Since this is the common reader, existing 
file-index and deletion-vector reads are affected too.
   
   An isolated C++ experiment with the locally available Arrow 19.0.1, 8,192 
input rows, an alternating selection, and 1,024 IN values took about 0.035 ms 
for one whole-batch call versus 40 ms for the single-row calls, with identical 
selected results. This is a kernel-level reproduction, not a benchmark of 
Paimon's Arrow 17 build; it also reused the value-set array, whereas this path 
rebuilds it.
   
   Could we gather fragmented candidates once, evaluate the predicate in a 
batch, and map the results back, or reuse the initialized predicate lookup 
state across slices? Please add performance coverage for a sparse selection 
combined with a large IN list.



##########
docs/source/user_guide/realtime.rst:
##########
@@ -398,6 +403,34 @@ context's reclaimed progress for relevant active stores. 
It does not restore
 an old in-memory state and is not unrestricted historical time travel. For a
 disk-only historical query, do not attach the real-time context.
 
+In-memory Predicate Indexes
+--------------------------
+
+The default Arrow store supports optional predicate indexes on key or value 
fields
+in append and append deduplicate modes. PK modes are unsupported. For example:
+
+.. code-block:: text
+
+   realtime.enabled = true
+   realtime.index.<index_name>.columns = id
+   realtime.store.stats-mode = full
+
+Replace ``<index_name>`` with the registered index plugin name. Use a
+comma-separated list such as ``id,payload`` to index multiple fields. Supported
+field types and predicates depend on the plugin; indexing ``_REALTIME_OFFSET``
+is unsupported.

Review Comment:
   [P3] Document the built-in hash plugin and its supported operations
   
   This asks users to substitute a registered plugin name, but the guide never 
identifies the built-in `hash` plugin introduced by this PR or describes its 
capabilities. Please provide a usable example such as 
`realtime.index.hash.columns=id,payload`, list equality/IN support and the 
accepted field types, and explicitly mention the current DECIMAL, TIMESTAMP, 
and nested-type restrictions.
   
   It would also help to state that the hash index returns candidates rather 
than exact matches: postings are keyed only by `Literal::HashCode()`, so hash 
collisions can retain nonmatching rows. Residual filtering is needed for exact 
results even when batch statistics are disabled. The explanation below 
currently motivates filtering mainly in terms of mixed batches and statistics.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to