[
https://issues.apache.org/jira/browse/LUCENE-2899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Rowe updated LUCENE-2899:
-------------------------------
Attachment: LUCENE-2899.patch
Patch, lots of changes (see below), I think it's ready to go (precommit and all
Lucene/Solr tests pass). My plan is to wait a couple days for review, then
commit if there are no objections.
Changes since the last patch:
* Corrected some test model training data.
* Switched OpenNLPTokenizer to require both the sentence and tokenizer models -
I couldn't think of a reason to support specification of only one of them.
* OpenNLPTokenizer now extends SegmentingTokenizerBase, and uses an OpenNLP
sentence segmenter via shim class OpenNLPSentenceBreakIterator.
End-of-sentence tokens are marked by setting a bit on the FlagsAttribute. All
OpenNLP-based filters operate on sentences.
* OpenNLPLemmatizerFilter now supports dictionary-based and/or model-based
lemmatization; if both are configured, dictionary-based tokenization is
performed first, and then out-of-vocabulary tokens are lemmatized using the
model.
* Each analysis operation is now in its own component (previously OpenNLPFilter
did multiple things).
* Removed the end-of-sentence hack in OpenNLPPOSFilter (described in an earlier
comment on this issue) - periods are no longer appended to sentences prior to
pos tagging.
* Named entity recognition is now handled in an update request processor in the
analysis-extras Solr contrib (though the NER model loading machinery remains in
OpenNLPOpsFactory in the lucene/analysis/opennlp module).
* Added ref guide docs.
* Added CHANGES.txt entries.
> Add OpenNLP Analysis capabilities as a module
> ---------------------------------------------
>
> Key: LUCENE-2899
> URL: https://issues.apache.org/jira/browse/LUCENE-2899
> Project: Lucene - Core
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Grant Ingersoll
> Assignee: Grant Ingersoll
> Priority: Minor
> Fix For: 4.9, 6.0
>
> Attachments: LUCENE-2899-6.1.0.patch, LUCENE-2899-RJN.patch,
> LUCENE-2899.patch, LUCENE-2899.patch, LUCENE-2899.patch, LUCENE-2899.patch,
> LUCENE-2899.patch, LUCENE-2899.patch, OpenNLPFilter.java,
> OpenNLPTokenizer.java
>
>
> Now that OpenNLP is an ASF project and has a nice license, it would be nice
> to have a submodule (under analysis) that exposed capabilities for it. Drew
> Farris, Tom Morton and I have code that does:
> * Sentence Detection as a Tokenizer (could also be a TokenFilter, although it
> would have to change slightly to buffer tokens)
> * NamedEntity recognition as a TokenFilter
> We are also planning a Tokenizer/TokenFilter that can put parts of speech as
> either payloads (PartOfSpeechAttribute?) on a token or at the same position.
> I'd propose it go under:
> modules/analysis/opennlp
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
