chishui opened a new pull request, #16836: URL: https://github.com/apache/lucene/pull/16836
### Description The change targets reducing stored fields retrieval performance through optimizing LZ4 decompression performance. Instead of reading data bytes to bytes, it reads a batch data and store into a heap object then every read is from the heap object. ### Benchmark results ``` Real data (stock codec, the same index read by a main jar and a branch jar in alternating JVMs, one pinned core; checksums identical): ┌───────────────────────────────┬─────────┬───────┬───────────────────────┬────────────┐ │ Dataset │ Avg doc │ Ratio │ Random access │ Merge read │ ├───────────────────────────────┼─────────┼───────┼───────────────────────┼────────────┤ │ msmarco sparse vectors (JSON) │ 1.7 KB │ 1.6× │ 27.5 → 18.9 µs (−32%) │ +42% │ ├───────────────────────────────┼─────────┼───────┼───────────────────────┼────────────┤ │ geonames │ 306 B │ 4.0× │ 10.8 → 8.4 µs (−22%) │ +30% │ ├───────────────────────────────┼─────────┼───────┼───────────────────────┼────────────┤ │ http_logs │ 133 B │ 4.3× │ 10.4 → 8.2 µs (−22%) │ +28% │ ├───────────────────────────────┼─────────┼───────┼───────────────────────┼────────────┤ │ noaa │ 287 B │ 2.6× │ 14.5 → 10.1 µs (−30%) │ +42% │ ├───────────────────────────────┼─────────┼───────┼───────────────────────┼────────────┤ │ nyc_taxis │ 498 B │ 4.3× │ 9.3 → 6.9 µs (−26%) │ +36% │ ├───────────────────────────────┼─────────┼───────┼───────────────────────┼────────────┤ │ so (text + code) │ 690 B │ 1.6× │ 20.4 → 14.7 µs (−28%) │ +40% │ ├───────────────────────────────┼─────────┼───────┼───────────────────────┼────────────┤ │ pmc (full-text articles) │ 22.8 KB │ 1.8× │ 68.8 → 52.4 µs (−24%) │ +35% │ ├───────────────────────────────┼─────────┼───────┼───────────────────────┼────────────┤ │ dense vector, 768-d JSON │ 8.6 KB │ 1.4× │ 53.2 → 35.5 µs (−33%) │ +51% │ ├───────────────────────────────┼─────────┼───────┼───────────────────────┼────────────┤ │ dense vector, 768-d binary │ 3 KB │ 1.0× │ 3.70 → 3.62 µs (−2%) │ +1% │ ├───────────────────────────────┼─────────┼───────┼───────────────────────┼────────────┤ │ random bytes │ 1 KB │ 1.0× │ 2.76 → 2.78 µs (+1%) │ 0% │ └───────────────────────────────┴─────────┴───────┴───────────────────────┴────────────┘ JMH, main vs branch (random access = µs per 10-doc batch; merge = full sequential read): ┌────────────────────┬──────────────────────┬───────────────┐ │ docType │ randomAccess │ mergeInstance │ ├────────────────────┼──────────────────────┼───────────────┤ │ json_logs │ 110.6 → 78.4 (−29%) │ −28% │ ├────────────────────┼──────────────────────┼───────────────┤ │ text │ 174.9 → 112.3 (−36%) │ −32% │ ├────────────────────┼──────────────────────┼───────────────┤ │ sparse_vector_json │ 310.5 → 216.1 (−30%) │ −29% │ ├────────────────────┼──────────────────────┼───────────────┤ │ dense_vector_json │ 514.5 → 350.6 (−32%) │ −32% │ ├────────────────────┼──────────────────────┼───────────────┤ │ random │ 25.8 → 24.9 (−4%) │ −3% │ └────────────────────┴──────────────────────┴───────────────┘ ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
