gritmind opened a new pull request #576: LUCENE-8631: Longest-Matching for User words in Nori Tokenizer URL: https://github.com/apache/lucene-solr/pull/576 The issue is described in [LUCENE-8631](https://issues.apache.org/jira/browse/LUCENE-8631). The existing nori tokenizer is able to do Longest-Matching for system words, but not for user words. The reason why it doesn’t work for user words is because their large costs severely mitigate bigram costs. Longest-Matching is a natural behavior in that system words, whose costs were generated from large Korean corpus, can be applied. Also, it is needed when to search exact words. So, we want it also apply to user words. Thankfully, Jim Ferenczi, who is a nori developer, gave to me some suggestions to fix it (ref. [LUCENE-8631](https://issues.apache.org/jira/browse/LUCENE-8631)). I followed the “Implement longest-match first in the dictionary” because it is simplest and efficient way. As Jim mentioned, it provides not only Longest-Matching, but also speed-up because it retains a single path for user words. The idea is, whenever to iterate the input stream by character step, the longest word is only stored and its subset words are ignored. There are two steps to implement this. First, before break in the for-loop, we call at once add function with posAheadMax info. Second, we remember global_posAheadMax during while-loop and use it to avoid storing subset words at different positions. For checking how robust is the implementation, we tested using about 100,000 Korean words, where subset words are included, and all test cases were passed. Yeongsu Kim [email protected] Wemakeprice Inc., South Korea https://www.wemakeprice.com
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
