I can see an argument for NGramTokenFilter incrementing position for each ngram, because they really are an ordered scan across the text. Pure ngrams are a different text representation than words. That could be an option on the token filter.
LUCENE-1224 is mostly concerned with ngram searching of CJK. That is a useful thing -- we had it in Ultraseek. It might be less important now that we have better tokenizers for those languages. I'd probably address that with a dedicated bigram tokenizer for CJK. This bug is about EdgeNGramTokenFilter, which is a way of precomputing trailing (or leading) wildcards. Each shorter sequence is a partial synonym for the original token. If the text is "molecular biology", I'd like the query "mol bio" to match that as a phrase, with the edge-ngrams in sequential positions. The current behavior puts each edge-ngram in a new position, so "bio" could be 8 positions after "mol" (with mingram=1 and maxgram=1024). That won't match with a tight phrase slop, even though the original tokens are next to each other. Personally, I need this to make phrase autocomplete really fast. wunder On Mar 1, 2013, at 8:46 PM, Otis Gospodnetic wrote: > Wunder, you may be thinking of LUCENE-1224 from a few years ago? > > http://search-lucene.com/?q=ngram&fc_project=Lucene&fc_type=issue > > Otis > -- > Solr & ElasticSearch Support > http://sematext.com/ > > > > > > On Fri, Mar 1, 2013 at 1:41 PM, Walter Underwood <[email protected]> > wrote: > I'm fixing position increment in EdgeNgramTokenFilter to act like synonyms, > with each ngram at the same position as the source token. Currently, the > position is incremented for each output token, which breaks phrase searching > with edge ngrams. > > I could not find a current Jira issue for this. Is there one? > > We are still on 3.3, but I'll submit a patch for 4.x. > > Thanks to whoever converted EdgeNgramTokenFilter to use TokenStream. > > wunder > -- > Walter Underwood > [email protected] > > > > -- Walter Underwood [email protected]
