[
https://issues.apache.org/jira/browse/LUCENE-9064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16981773#comment-16981773
]
Robert Muir commented on LUCENE-9064:
-------------------------------------
That one is probably easiest because you wouldn't have to convert japanese
wikipedia to "line-docs" format. Although the "analyzer benchmark" in
luceneutil that produces this chart:
https://people.apache.org/~mikemccand/lucenebench/analyzers.html is probably
more robust:
https://github.com/mikemccand/luceneutil/blob/master/src/main/perf/TestAnalyzerPerf.java
The main thing is just to give it actual japanese/korean docs when testing, as
these analyzers try to optimize for the languages they target with the caches
in question.
> Can we remove the FST cache in Kuromoji and Nori analyzers?
> -----------------------------------------------------------
>
> Key: LUCENE-9064
> URL: https://issues.apache.org/jira/browse/LUCENE-9064
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Bruno Roustant
> Priority: Minor
>
> Is the ~30k han cache in kuromoji redundant after LUCENE-8920?
> [https://github.com/apache/lucene-solr/blob/813ca77250db29116812bc949e2a466a70f969a3/lucene/analysis/kuromoji/src/java/org/apache/lucene/analysis/ja/dict/TokenInfoFST.java#L35-L38])
> The entire linked file's purpose is all around this caching, so if its not
> needed anymore it would be a nice cleanup. But it was definitely needed for
> good performance before, so we shoudl be careful. Nori analyzer has the exact
> same thing (file has the same name) for ~10k hangul syllables.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
