costin opened a new pull request, #16381: URL: https://github.com/apache/lucene/pull/16381
Pack short ASCII stop words (up to 8 chars) into `long` values and use `LongHashSet` for O(1) lookup, avoiding `char[]` pointer chasing and the per-codepoint `Character.toLowerCase()` hashing in `CharArrayMap`. When all stop words fit (ASCII, ≤8 chars — which covers all standard English stop word lists), the constructor packs each word into a `long` (one byte per char) and builds a `LongHashSet`. The `accept()` method packs the incoming token the same way and does a single `LongHashSet.contains()`. Non-packable sets (non-ASCII or words >8 chars) fall back to the existing `CharArraySet` path unchanged. ### Benchmark Raw lookup throughput — 2000 tokens per iteration, `english` = 50% stop words, `technical` = 20%: ``` Benchmark (textType) Mode Cnt Score Error Units StopFilterBenchmark.charArraySet english thrpt 5 15.328 ± 0.235 ops/µs StopFilterBenchmark.packedLongHashSet english thrpt 5 40.127 ± 0.892 ops/µs StopFilterBenchmark.charArraySet technical thrpt 5 12.864 ± 0.198 ops/µs StopFilterBenchmark.packedLongHashSet technical thrpt 5 35.741 ± 0.547 ops/µs ``` ~2.6x throughput on English stop words, ~2.8x on technical text. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
