pdeaudney commented on PR #69048: URL: https://github.com/apache/doris/pull/69048#issuecomment-6159859131
### Bitmap candidate-filter benchmarks Setup: one local macOS arm64 FE/BE in Release mode; 20 million rows for the selected org and 10 million for another org. Each query had one warm-up and five timed executions. Counts and SUM(resource_id) were verified. **Go-produced portable Roaring64:** The client generated the bitmap outside Doris and sent it through the MySQL query interface. Times are local client-to-result means. | Candidate IDs | Base64 payload | Native, broad text | Iceberg, broad text | | ---: | ---: | ---: | ---: | | 250k | 670 KB | 60 ms | 158 ms | | 1m | 2.67 MB | 130 ms | 207 ms | | 5m | 3.34 MB | 101 ms | 198 ms | | 10m | 3.35 MB | 101 ms | 208 ms | At 1m IDs, Go generation and encoding added about 9 ms, giving a 216 ms generation-plus-query mean on Iceberg. Candidate retrieval and remote network latency were not measured. **Subquery-produced bitmap on Iceberg:** With broad text, the opt-in scan runtime filter reduced work passed to the bitmap join. | Candidate IDs | Default join | Scan filter | | ---: | ---: | ---: | | 250k | 11,121 ms | 220 ms | | 1m | 11,225 ms | 674 ms | | 5m | 11,243 ms | 2,967 ms | | 10m | 11,277 ms | 5,855 ms | At 1m IDs with selective text, the scan filter averaged 191 ms versus 180 ms for the default join. Both plans still read all 20 million selected-org rows, all 18 Parquet row groups, and 44.82 MB. The measured gain was from avoiding downstream join work, not reducing Parquet I/O. The Go constant-bitmap query and subquery runtime-filter query use different plans, so their timings are not a controlled comparison of filter delivery. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
