pdeaudney commented on PR #69048: URL: https://github.com/apache/doris/pull/69048#issuecomment-6159911660
# Comparison against sub-query IN with local data The fixture had 20 million rows for `org_id = 101` and 10 million for another org. Candidate bitmaps contained 250,000 to 10 million `resource_id` values. Queries also applied `payload LIKE '%needle%'`. Each shape was warmed and timed five times; bitmap and `IN` queries returned the same final count and `SUM(resource_id)`. **Direct Base64 bitmap versus an ordinary `IN` subquery** (mean query latency, ms): | Candidate IDs | Native bitmap | Native `IN` | Iceberg bitmap | Iceberg `IN` | | ---: | ---: | ---: | ---: | ---: | | 250k | 45.4 | 25.9 | 119.1 | 106.7 | | 1m | 123.5 | 37.2 | 217.5 | 148.1 | | 5m | 106.2 | 92.1 | 196.1 | 204.8 | | 10m | 107.9 | 137.9 | 194.1 | 244.7 | At 10m IDs, the bitmap query was about 22% faster than `IN` on native storage and 21% faster on Iceberg. At 250k and 1m IDs, `IN` was faster. Bitmap construction happened before the timed bitmap query; the `IN` query read the allowlist table, so these are useful query-shape results rather than a controlled comparison of filter delivery. **Subquery-produced bitmap on native storage** (mean query latency, ms): | Candidate IDs | Default bitmap join | Compressed scan runtime filter | | ---: | ---: | ---: | | 250k | 97.4 | 54.4 | | 1m | 102.0 | 73.6 | | 5m | 122.0 | 81.4 | | 10m | 125.0 | 94.0 | At 250k IDs, the compressed filter's scan profile reported 19.75 million rows rejected by bitmap membership, leaving 250,000 rows for later expressions. These were warm, single-host macOS arm64 measurements. The scattered matching IDs did not reduce fact-column bytes read. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
