Tarun-kishore opened a new pull request, #20042:
URL: https://github.com/apache/pinot/pull/20042
Part of #20041: this PR covers the inverted index. A separate PR covers the
H3, JSON_MATCH and null value vector filters.
### Problem
On a consuming segment, `numDocs` is read when the query is planned, but
`InvertedIndexFilterOperator` reads its bitmaps from the mutable inverted index
later, when the query runs. Those bitmaps can hold rows ingested in between:
- For `<>` and `NOT IN`, the flip over `[0, numDocs)` keeps the new rows
that hold an excluded value, and reports them as matches. In an AND with a
scan-based filter, they reach the result through
`RangelessBitmapDocIdIterator`, so a query with `env <> 'staging'` returns
staging rows.
- The count shortcuts and `getBitmaps()` mix the planning-time `numDocs`
with the live bitmaps, so `COUNT(*)` is off.
See #20041 for details.
### Change
`InvertedIndexFilterOperator` now limits every bitmap it reads from the
inverted index to `[0, numDocs)`, through a new private `getDocIds(int dictId)`
helper. This covers the matching rows, the flip for exclusive predicates, the
count shortcuts, and `getBitmaps()`.
- **Immutable segments:** no change in behavior or cost. A bitmap whose last
row is below `numDocs` is returned as is (one `isEmpty()` and one `last()`
call), which is always the case on an immutable segment.
- **Consuming segments:** the mutable inverted index already returns a copy
of its bitmap, so that copy is trimmed in place. Any other bitmap type is
copied before it is trimmed.
Alternatives considered:
- Bounding the iterators instead (`RangelessBitmapDocIdIterator`,
`applyAnd()`) would fix the returned rows, but not the flip, the counts, or the
bitmaps handed to AND/OR.
- Reading `numDocs` at execution time would not help either: the index can
still move on after it is read.
### Tests
- `InvertedIndexFilterOperatorConsumingSegmentTest` (new) plans each query
on a real consuming segment, ingests more rows, and then runs it. It uses a
single-value and a multi-value column, and covers:
- the rows returned for `<>`, `NOT IN`, `=` and `IN` ANDed with a scan
filter
- a group-by with `<>`
- the fast `COUNT(*)` path for `<>`, `NOT IN`, `NOT (=)`, and AND and OR
of inverted index predicates
- on the multi-value column, the count with 1, 2 and 3 dictionary ids,
which take different paths
- the fast `COUNT(*)` with null handling, with a null row among the
planned ones
- `InvertedIndexFilterOperatorTest.testDocsAtOrAboveNumDocsAreIgnored` (new)
uses a mocked reader whose bitmaps hold rows at and above `numDocs`. It covers
single- and multi-value columns, with inclusive and exclusive predicates.
Without the fix, all 19 cases of the new class fail, and so does the new
unit test. For example:
- `<>` returns the new staging rows, and the group-by returns a `staging`
group.
- `COUNT(*)` with `<>` returns 2 instead of 4.
- `COUNT(*)` with `IN` on the multi-value column returns 10 instead of 6.
With the fix, they pass, along with the full pinot-core unit test suite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
