gritmind opened a new pull request #576: LUCENE-8631: Longest-Matching for User 
words in Nori Tokenizer
URL: https://github.com/apache/lucene-solr/pull/576
 
 
   The issue is described in 
[LUCENE-8631](https://issues.apache.org/jira/browse/LUCENE-8631). 
   
   The existing nori tokenizer is able to do Longest-Matching for system words, 
but not for user words. The reason why it doesn’t work for user words is 
because their large costs severely mitigate bigram costs. Longest-Matching is a 
natural behavior in that system words, whose costs were generated from large 
Korean corpus, can be applied. Also, it is needed when to search exact words. 
So, we want it also apply to user words.
   
   Thankfully, Jim Ferenczi, who is a nori developer, gave to me some 
suggestions to fix it (ref. 
[LUCENE-8631](https://issues.apache.org/jira/browse/LUCENE-8631)). I followed 
the “Implement longest-match first in the dictionary” because it is simplest 
and efficient way. As Jim mentioned, it provides not only Longest-Matching, but 
also speed-up because it retains a single path for user words.
   
   The idea is, whenever to iterate the input stream by character step, the 
longest word is only stored and its subset words are ignored. There are two 
steps to implement this. First, before break in the for-loop, we call at once 
add function with posAheadMax info. Second, we remember global_posAheadMax 
during while-loop and use it to avoid  storing subset words at different 
positions. 
   
   For checking how robust is the implementation, we tested using about 100,000 
Korean words, where subset words are included, and all test cases were passed. 
   
   Yeongsu Kim
   [email protected]
   Wemakeprice Inc., South Korea
   https://www.wemakeprice.com

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[email protected]


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to