luizmlima opened a new pull request, #16416: URL: https://github.com/apache/lucene/pull/16416
### Description `JapaneseCompletionFilter.CompletionTokenGenerator` uses `outputs` as a FIFO queue. The current implementation uses an `ArrayList` with `remove(0)`, which shifts the remaining elements after each removal. This change replaces it with an `ArrayDeque` and uses `addLast()` and `removeFirst()`. There is no change to the public API or to the generated tokens, their order, offsets, or position increments. ### Benchmark I used JMH to compare the current `ArrayList` implementation with the proposed `ArrayDeque` implementation. The benchmark runs the full `JapaneseCompletionAnalyzer` pipeline. For each input, it creates a token stream and consumes all generated tokens. Both versions were tested with the same benchmark code, JDK, JVM options, hardware, and JMH configuration. #### Configuration - Mode: Average time - Unit: microseconds per operation - Warmup: 5 iterations of 1 second - Measurement: 10 iterations of 1 second - Forks: 3 - JVM: `-Xms1g -Xmx1g -XX:+AlwaysPreTouch` Lower values are better. #### Environment - JDK: OpenJDK Temurin 25.0.3 LTS - Runtime: `Temurin-25.0.3+9` - OS: Pop!_OS / Linux - Kernel: `7.0.11-76070011-generic` - Architecture: `x86_64` - CPU: Intel Core i5-10300H @ 2.50 GHz - 4 physical cores - 8 logical CPUs - 2 threads per core #### Inputs - `東京` - `ドラえもん` - `シンシンシン` - `シンシンシンシンシン` The first two are short, more typical inputs. The last two were included to generate larger output batches and make the queue behavior more visible. #### Results | Input | ArrayList | ArrayDeque | Difference | |---|---:|---:|---:| | `東京` | 1.309 ± 0.024 us/op | 1.322 ± 0.026 us/op | +0.98% | | `ドラえもん` | 2.557 ± 0.013 us/op | 2.600 ± 0.051 us/op | +1.70% | | `シンシンシン` | 4.809 ± 0.074 us/op | 4.656 ± 0.075 us/op | -3.19% | | `シンシンシンシンシン` | 8.288 ± 0.138 us/op | 7.743 ± 0.079 us/op | -6.57% | Negative values mean that `ArrayDeque` completed the operation in less time. For the shorter inputs, the error ranges overlap, so there was no meaningful performance difference. For the larger output batches, `ArrayDeque` performed better: - about 3.2% faster for `シンシンシン` - about 6.6% faster for `シンシンシンシンシン` The largest case went from `8.288 ± 0.138 us/op` with `ArrayList` to `7.743 ± 0.079 us/op` with `ArrayDeque`. This matches the way the collection is used. `ArrayList.remove(0)` shifts the remaining elements, while `ArrayDeque.removeFirst()` does not. The change does not improve typical short inputs in a measurable way, but it better matches the FIFO access pattern and avoids the increasing cost of repeated removals from the beginning of an `ArrayList`. ### Validation The following commands completed successfully: - `./gradlew tidy` - `./gradlew -p lucene/analysis/kuromoji test --tests "org.apache.lucene.analysis.ja.TestJapaneseCompletionFilter"` - `./gradlew -p lucene/analysis/kuromoji test --tests "org.apache.lucene.analysis.ja.TestJapaneseCompletionAnalyzer"` - `./gradlew -p lucene/analysis/kuromoji check` - `./gradlew check` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
