pdeaudney opened a new pull request, #69048:
URL: https://github.com/apache/doris/pull/69048
### What problem does this PR solve?
Issue Number: draft for discussion
Related PR: draft for duscussion
Problem Summary: Queries over large shared datasets need a way to supply a
posting list of candidate `resource_id` values at query time. The list may
represent facets, categories, access rules, or another application-defined
candidate set.
This PR adds `bitmap_from_roaring_base64` for portable Roaring32 and
Go-compatible portable Roaring64 input. It also adds early bitmap membership
filtering in native scans and an opt-in compressed runtime filter for
subquery-produced bitmaps. The exact bitmap semi join remains the final
membership check.
The benchmark uses 20 million rows in the selected `org_id` partition, 10
million rows in another partition, and candidate lists of 250,000 to 10 million
IDs. In the Iceberg rerun, a 1-million-ID broad-text query averaged 11,225 ms
with the default bitmap join and 674 ms with the scan filter. A selective-text
query averaged 180 ms and 191 ms respectively. Iceberg still read the same
Parquet row groups and bytes, so the measured gain is from avoiding downstream
join work.
A Go client also generated portable Roaring64 outside Doris and sent it
through the MySQL query interface. Its 1-million-ID Base64 payload was 2.67 MB;
the broad-text query averaged 130 ms on native storage and 207 ms on Iceberg
from that local client. These are separate query shapes and runs, not a direct
comparison with the subquery runtime filter.
The runtime filter is opt-in through
`enable_bitmap_subquery_runtime_filter`. Enable it only after FE and BE nodes
are upgraded. Distributed transfer, memory use, and remote client latency still
need evaluation.
### Release note
Add `bitmap_from_roaring_base64` to decode portable Roaring bitmap input for
bitmap queries. Add an opt-in compressed bitmap runtime filter for positive
bitmap subqueries.
### Check List (For Author)
- Test:
- [x] Regression test: native and Iceberg bitmap suites, including
result checks against ordinary `IN` queries
- [x] Manual test: Go-produced portable Roaring64 at 250,000, 1 million,
5 million, and 10 million candidate IDs on native and Iceberg tables
- [x] Go test and vet for the benchmark client
- Behavior changed:
- [x] Yes. Adds a SQL function and bitmap scan paths; the subquery
runtime filter requires an explicit session setting.
- Does this need documentation?
- [x] Yes. The PR includes a Go sample README and benchmark results.
User-facing Doris documentation remains to be prepared.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
