SteNicholas commented on code in PR #504:
URL: https://github.com/apache/paimon-cpp/pull/504#discussion_r4457197784
##########
src/paimon/common/reader/predicate_batch_reader.cpp:
##########
@@ -123,19 +126,29 @@ Status PredicateBatchReader::BindPredicateToArray(const
arrow::Array& array) {
return Status::OK();
}
-Result<RoaringBitmap32> PredicateBatchReader::Filter(const
std::shared_ptr<arrow::Array>& array) {
+Result<RoaringBitmap32> PredicateBatchReader::Filter(const
std::shared_ptr<arrow::Array>& array,
+ const RoaringBitmap32&
selection) {
if (!predicate_filter_) {
PAIMON_RETURN_NOT_OK(BindPredicateToArray(*array));
}
- PAIMON_ASSIGN_OR_RAISE(std::vector<char> result,
- predicate_filter_->Test(*array, arrow_pool_.get()));
- assert(result.size() == static_cast<size_t>(array->length()));
+ PAIMON_ASSIGN_OR_RAISE(arrow::ArrayVector slices,
+ ReaderUtils::GenerateFilteredArrayVector(array,
selection));
RoaringBitmap32 is_valid;
- for (int32_t i = 0; i < static_cast<int32_t>(result.size()); i++) {
- if (result[i]) {
- is_valid.Add(i);
+ // Slices share the source buffers; selection maps their results to
original batch rows.
+ auto iter = selection.Begin();
+ for (const std::shared_ptr<arrow::Array>& slice : slices) {
+ PAIMON_ASSIGN_OR_RAISE(std::vector<char> result,
+ predicate_filter_->Test(*slice,
arrow_pool_.get()));
Review Comment:
[P2] Preserve batched predicate evaluation for fragmented selections
An alternating selection such as `{0, 2, 4, ...}` produces one slice per
selected row, so this loop calls `PredicateFilter::Test()` thousands of times
for a single input batch. `MultiLiteralsLeafFunction::Test()` rebuilds the IN
value set and invokes Arrow `IsIn` for each call, repeatedly initializing the
lookup state. The work therefore grows with the number of selected runs
multiplied by the IN-list size. Since this is the common reader, existing
file-index and deletion-vector reads are affected too.
An isolated C++ experiment with the locally available Arrow 19.0.1, 8,192
input rows, an alternating selection, and 1,024 IN values took about 0.035 ms
for one whole-batch call versus 40 ms for the single-row calls, with identical
selected results. This is a kernel-level reproduction, not a benchmark of
Paimon's Arrow 17 build; it also reused the value-set array, whereas this path
rebuilds it.
Could we gather fragmented candidates once, evaluate the predicate in a
batch, and map the results back, or reuse the initialized predicate lookup
state across slices? Please add performance coverage for a sparse selection
combined with a large IN list.
##########
docs/source/user_guide/realtime.rst:
##########
@@ -398,6 +403,34 @@ context's reclaimed progress for relevant active stores.
It does not restore
an old in-memory state and is not unrestricted historical time travel. For a
disk-only historical query, do not attach the real-time context.
+In-memory Predicate Indexes
+--------------------------
+
+The default Arrow store supports optional predicate indexes on key or value
fields
+in append and append deduplicate modes. PK modes are unsupported. For example:
+
+.. code-block:: text
+
+ realtime.enabled = true
+ realtime.index.<index_name>.columns = id
+ realtime.store.stats-mode = full
+
+Replace ``<index_name>`` with the registered index plugin name. Use a
+comma-separated list such as ``id,payload`` to index multiple fields. Supported
+field types and predicates depend on the plugin; indexing ``_REALTIME_OFFSET``
+is unsupported.
Review Comment:
[P3] Document the built-in hash plugin and its supported operations
This asks users to substitute a registered plugin name, but the guide never
identifies the built-in `hash` plugin introduced by this PR or describes its
capabilities. Please provide a usable example such as
`realtime.index.hash.columns=id,payload`, list equality/IN support and the
accepted field types, and explicitly mention the current DECIMAL, TIMESTAMP,
and nested-type restrictions.
It would also help to state that the hash index returns candidates rather
than exact matches: postings are keyed only by `Literal::HashCode()`, so hash
collisions can retain nonmatching rows. Residual filtering is needed for exact
results even when batch statistics are disabled. The explanation below
currently motivates filtering mainly in terms of mixed batches and statistics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]