zhangyue19921010 opened a new issue, #11130:
URL: https://github.com/apache/paimon/issues/11130

   ### Search before asking
   
   - [x] I searched the issues and found no matching optimization.
   
   ### Motivation
   
   Bitmap global-index range predicates scan and decode every dictionary block 
in each selected index file, even when only a few keys can match. The writer 
already orders keys and records the first key of each dictionary block.
   
   On current master (`31f3db767`), a local Java reader benchmark with 50,000 
integer keys (-25,000 through 24,999) and four rows per key reads 450,188 bytes 
to return 40 rows for `BETWEEN -5 AND 4`. The dictionary blocks use the default 
16 KiB target size, without compression.
   
   ### Solution
   
   Use the existing logical-key binary search to restrict dictionary scans to 
the blocks covering the bounds. Keep the existing exact predicate checks within 
those blocks. This preserves the file format, NULL semantics and configured 
fallback budget.
   
   A prototype reduces that lookup to 16,991 bytes and 13 seeks, from 40 seeks. 
Warm median latency is 2.56 ms before and 0.64 ms after. The all-key control 
reads the same bytes and measures 61.6 ms before and 64.9 ms after; the per-JVM 
median ranges overlap (58.6–65.6 ms and 58.7–67.6 ms).
   
   ### Anything else?
   
   Measurements use OpenJDK 11 on an 8-vCPU Xeon Platinum 8260 VM, a local 
filesystem, a 512 MiB heap, three alternating base/head JVM runs, three warmups 
and seven measured queries per case. Returned row IDs are checked outside 
timing. This measures one index reader, not end-to-end query performance. LZ4 
and a 1,000-key control were also checked. Heap allocation was not measured; 
the change adds logarithmic bound comparisons and loads fewer dictionary blocks 
for selective ranges.
   
   AI-generated with OpenAI Codex.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to