kamthorn opened a new pull request, #16722:
URL: https://github.com/apache/lucene/pull/16722
### Description
Closes #16721
This PR introduces user/custom dictionary support to `ThaiTokenizer` and
`ThaiTokenizerFactory`, allowing users to supply domain-specific terms, proper
nouns, jargon, and loanwords (e.g. "พารากอน", "คลาวด์เนทีฟ", "คนขับรถ") that
take precedence over default `BreakIterator` segmentation boundaries.
#### Motivation:
`ThaiTokenizer` uses JRE's `BreakIterator` with a fixed built-in dictionary.
In domain-specific search use cases (e.g. e-commerce, technical documentation,
enterprise search), words not present in the default dictionary or multi-word
compound terms are often fragmented into sub-words or single characters:
- `"ไปพารากอนกัน"` was fragmented into `["ไป", "พา", "รา", "กอน", "กัน"]-
`"คนขับรถ"` was fragmented into `["คน", "ขับ", "รถ"]
Providing user dictionary support brings `ThaiTokenizer` in line with other
language tokenizers in Lucene (such as Kuromoji for Japanese and Nori for
Korean).
#### Changes:
1. **`ThaiTokenizer`**:
- Added constructors accepting `CharArraySet userDictionary`.
- In `incrementWord()`, when a user dictionary is provided, prefix
matches from `userDictionary` take precedence, cleanly spanning custom terms
and adjusting segmentation boundaries.
2. **`ThaiTokenizerFactory`**:
- Implemented `ResourceLoaderAware`.
- Added support for the `dictionary` parameter (e.g. `<tokenizer
class="solr.ThaiTokenizerFactory" dictionary="custom_words.txt"/>`).
3. **Tests**:
- Added `TestThaiTokenizer` testing default segmentation, user dictionary
term recognition, multi-term offset correctness, and empty dictionary handling.
- Updated `TestThaiTokenizerFactory` with `testCustomDictionary()` using
a test dictionary resource (`customThaiDictionary.txt`).
4. **Backwards Compatibility**:
- Retained all existing constructors and behaviors when no user
dictionary is supplied.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
