[
https://issues.apache.org/jira/browse/LUCENE-6254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14325688#comment-14325688
]
Erlend Garåsen commented on LUCENE-6254:
----------------------------------------
This lemmatizer can do POS-tagging if it's enabled (and that the dictionary has
information about word-classes). Ambiguous forms can either be indexed or
reduced to one lemma. depending on how it is configured.
We have tested this lemmatizer by indexing 200,000 larger texts with a
dictionary containing 700,000 entries. It does not take any longer time
compared to one of the other available stemmers such as Hunspell.
I guess morphological analysis will be more time-consuming and require more
memory at index time?
> Dictionary-based lemmatizer
> ---------------------------
>
> Key: LUCENE-6254
> URL: https://issues.apache.org/jira/browse/LUCENE-6254
> Project: Lucene - Core
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Erlend Garåsen
> Labels: patch
> Fix For: 5.0
>
> Attachments: LUCENE-6254.patch
>
>
> The only way to achieve lemmatization today is to use the
> SynonymFilterFactory. The available stemmers are also inaccurate since they
> are only following simplistic rules.
> A dictionary-based lemmatizer will be more precise because it has the
> opportunity to know the part of speech. Thus it provides a more precise
> method to stem words compared to other dictionary-based stemmers such as
> Hunspell.
> This is my effort to develop such a lemmatizer for Apache Lucene. The
> documentation is temporarily placed here:
> http://folk.uio.no/erlendfg/solr/lemmatizer.html
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
