[
https://issues.apache.org/jira/browse/SOLR-908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Rutherglen updated SOLR-908:
----------------------------------
Attachment: SOLR-908.patch
We ran an indexing performance test with CommonGramFilter and a
heavily modified/optimized filter state machine from the book
Building Search Applications that does not emit stop words. I
tried to change CommonGramFilter to not emit stop words, however
BufferedTokenStream seems to only move forwards.
CommonGramFilter was 10% faster with the StopWordsFilter at the
end of the chain compared with the book's version. The book has
a GNU license for it's code so I cannot post it here.
The attached patch just cleans up a few things. Perhaps more of
the test code can be shared?
To mark the issue as 1.4 you'd edit it, change affected version
to 1.3, fix version to 1.4.
> Port of Nutch CommonGrams filter to Solr
> -----------------------------------------
>
> Key: SOLR-908
> URL: https://issues.apache.org/jira/browse/SOLR-908
> Project: Solr
> Issue Type: Wish
> Components: Analysis
> Reporter: Tom Burton-West
> Priority: Minor
> Attachments: CommonGramsPort.zip, SOLR-908.patch, SOLR-908.patch,
> SOLR-908.patch, SOLR-908.patch, SOLR-908.patch
>
>
> Phrase queries containing common words are extremely slow. We are reluctant
> to just use stop words due to various problems with false hits and some
> things becoming impossible to search with stop words turned on. (For example
> "to be or not to be", "the who", "man in the moon" vs "man on the moon" etc.)
>
> Several postings regarding slow phrase queries have suggested using the
> approach used by Nutch. Perhaps someone with more Java/Solr experience might
> take this on.
> It should be possible to port the Nutch CommonGrams code to Solr and create
> a suitable Solr FilterFactory so that it could be used in Solr by listing it
> in the Solr schema.xml.
> "Construct n-grams for frequently occuring terms and phrases while indexing.
> Optimize phrase queries to use the n-grams. Single terms are still indexed
> too, with n-grams overlaid."
> http://lucene.apache.org/nutch/apidocs-0.8.x/org/apache/nutch/analysis/CommonGrams.html
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
