SteNicholas commented on code in PR #504:
URL: https://github.com/apache/paimon-cpp/pull/504#discussion_r4457197784
##########
src/paimon/common/reader/predicate_batch_reader.cpp:
##########
@@ -123,19 +126,29 @@ Status PredicateBatchReader::BindPredicateToArray(const
arrow::Array& array) {
return Status::OK();
}
-Result<RoaringBitmap32> PredicateBatchReader::Filter(const
std::shared_ptr<arrow::Array>& array) {
+Result<RoaringBitmap32> PredicateBatchReader::Filter(const
std::shared_ptr<arrow::Array>& array,
+ const RoaringBitmap32&
selection) {
if (!predicate_filter_) {
PAIMON_RETURN_NOT_OK(BindPredicateToArray(*array));
}
- PAIMON_ASSIGN_OR_RAISE(std::vector<char> result,
- predicate_filter_->Test(*array, arrow_pool_.get()));
- assert(result.size() == static_cast<size_t>(array->length()));
+ PAIMON_ASSIGN_OR_RAISE(arrow::ArrayVector slices,
+ ReaderUtils::GenerateFilteredArrayVector(array,
selection));
RoaringBitmap32 is_valid;
- for (int32_t i = 0; i < static_cast<int32_t>(result.size()); i++) {
- if (result[i]) {
- is_valid.Add(i);
+ // Slices share the source buffers; selection maps their results to
original batch rows.
+ auto iter = selection.Begin();
+ for (const std::shared_ptr<arrow::Array>& slice : slices) {
+ PAIMON_ASSIGN_OR_RAISE(std::vector<char> result,
+ predicate_filter_->Test(*slice,
arrow_pool_.get()));
Review Comment:
**Preserve batched predicate evaluation for fragmented selections**: An
alternating selection such as `{0, 2, 4, ...}` produces one slice per selected
row, so this loop calls `PredicateFilter::Test()` thousands of times for a
single input batch. `MultiLiteralsLeafFunction::Test()` rebuilds the IN value
set and invokes Arrow `IsIn` for each call, repeatedly initializing the lookup
state. The work therefore grows with the number of selected runs multiplied by
the IN-list size. Since this is the common reader, existing file-index and
deletion-vector reads are affected too.
An isolated C++ experiment with the locally available Arrow 19.0.1, 8,192
input rows, an alternating selection, and 1,024 IN values took about 0.035 ms
for one whole-batch call versus 40 ms for the single-row calls, with identical
selected results. This is a kernel-level reproduction, not a benchmark of
Paimon's Arrow 17 build; it also reused the value-set array, whereas this path
rebuilds it.
Could we gather fragmented candidates once, evaluate the predicate in a
batch, and map the results back, or reuse the initialized predicate lookup
state across slices? Please add performance coverage for a sparse selection
combined with a large IN list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]