rajat315315 commented on PR #16361: URL: https://github.com/apache/lucene/pull/16361#issuecomment-5013858015
I have run benchmarking on local machine to double check on real world queries. Below is the script used for benchmarking. [localrun_automata.py](https://github.com/user-attachments/files/30159698/localrun_automata.py) This benchmark evaluates the performance impact of replacing the linear scan in `CompiledAutomaton.addTail()` with a binary search to find the largest transition that is `< leadLabel`. ## Benchmark Configuration - **Dataset**: Wikipedia 100k lines corpus (114,745 documents) - **Iterations**: 3 JVM iterations per competitor - **Warmups / Repeat Count**: 5 query executions per JVM - **Baseline Branch**: `main` - **Feature Branch**: `automata_opt` (patched version) --- ## 1. Throughput Comparison (Queries Per Second - QPS) Higher QPS represents better performance: | Query Type / Task | Baseline QPS | Feature QPS (Optimized) | Throughput Change | p-value | | :-------------------------------| :------------:| :-----------------------:| :-----------------:| :-------:| | **Fuzzy1** (Edit distance = 1) | 60.60 | 82.06 | **+35.4%** | 0.223 | | **Wildcard** | 115.02 | 175.91 | **+52.9%** | 0.085 | | **Fuzzy2** (Edit distance = 2) | 13.28 | 21.29 | **+60.3%** | 0.059 | | **PKLookup** | 49.76 | 80.33 | **+61.4%** | 0.079 | --- ## 2. Latency Comparison (Percentiles in Milliseconds) Lower latency represents better performance: | Query Type / Task | Metric | Baseline (ms) | Feature (ms) | Latency Reduction | | :------------------| :-------------| :-------------:| :------------:| :-----------------:| | **Fuzzy1** | P50 (Median) | 43.18 | 16.39 | **-62.1%** | | | P90 | 49.75 | 16.81 | **-66.2%** | | | P99 | 49.75 | 16.81 | **-66.2%** | | | P100 (Max) | 188.35 | 102.37 | **-45.6%** | | **Wildcard** | P50 (Median) | 20.09 | 7.36 | **-63.4%** | | | P90 | 29.18 | 7.65 | **-73.8%** | | | P99 | 29.18 | 7.65 | **-73.8%** | | | P100 (Max) | 160.16 | 113.05 | **-29.4%** | | **Fuzzy2** | P50 (Median) | 112.14 | 62.56 | **-44.2%** | | | P90 | 259.04 | 111.48 | **-57.0%** | | | P99 | 259.04 | 111.48 | **-57.0%** | | | P100 (Max) | 377.63 | 179.04 | **-52.6%** | --- -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
