Hi, See org.apache.lucene.search.PhraseWildcardQuery in Lucene's sandbox module. It was recently added by my amazing colleague Bruno. At this time there is no query parser that uses it in Lucene unfortunately but you can rectify this for your own purposes. I hope this query "graduates" to Lucene core some day. It's placement in sandbox is why it can't be added to any of Lucene's query parsers like complex phrase.
~ David Smiley Apache Lucene/Solr Search Developer http://www.linkedin.com/in/davidwsmiley On Wed, Feb 12, 2020 at 11:07 AM <baris.ka...@oracle.com> wrote: > Hi,- > > Regarding this mechanisms below i mentioned, > > does this class offer any Shingling capability embedded to it? > > I could not find any api within this class ComplexPhraseQueryParser for > that purpose. > > > For instance does this class offer the most commonly used words api? > > i can then use one of those words as to use the second and third char > from it to search like > > term1 term2FirstCharTerm2SecondChar* (where i would look up > term2FirstChar in my dictionary hashmap for the most common word value > and bring its second char into the search query) > > > Having second char in the search query reduces search time by 20 times. > > > Otherwise, do i have to use the following at index time? i already have > TextField index with my custom analyzer. > > How should i embed the shingling filter into my current custom analyzer? > i dont want to disturb my current indexing. > > All i want to do is to find most common word in my data for each letter > in the alphabet. > > Should i do this at search time? That would be costly, right? > > > view-source: > http://www.philippeadjiman.com/blog/2009/11/02/writing-a-token-n-grams-analyzer-in-few-lines-of-code-using-lucene/ > > > <p><a href="http://lucene.apache.org/";><img class="alignleft size-full > wp-image-524" title="lucene_green_300" > src=" > http://www.philippeadjiman.com/blog/wp-content/uploads/2009/11/lucene_green_3001.gif"; > > alt="lucene_green_300" hspace="15" width="300" height="46" align="left" > /></a> If you need to parse the tokens n-grams of a string, you may use > the facilities offered by lucene analyzers.</p> > <p>What you simply have to do is to build you own analyzer using a > ShingleMatrixFilter with the parameters that suits you needs. For > instance, here the few lines of code to build a token bi-grams > analyzer:</p> > <pre lang="java">public class NGramAnalyzer extends Analyzer { > @Override > public TokenStream tokenStream(String fieldName, Reader reader) { > return new StopFilter(new LowerCaseFilter(new > ShingleMatrixFilter(new StandardTokenizer(reader),2,2,' ')), > StopAnalyzer.ENGLISH_STOP_WORDS); > } > }</pre> > <p>The parameters of the ShingleMatrixFilter simply states the minimum > and maximum shingle size. “Shingle” is just another name for > token N-Grams and is popular to be the basic units to help solving > problems in spell checking, near-duplicate detection and others.<br /> > Note also the use of a StandardTokenizer to deal with basic special > characters like hyphens or other “disturbers”. </p> > <p>To use the analyzer, you can for instance do:</p> > <pre lang="java"> > public static void main(String[] args) { > try { > String str = "An easy way to write an analyzer for tokens > bi-gram (or even tokens n-grams) with lucene"; > Analyzer analyzer = new NGramAnalyzer(); > > TokenStream stream = analyzer.tokenStream("content", new > StringReader(str)); > Token token = new Token(); > while ((token = stream.next(token)) != null){ > System.out.println(token.term()); > } > > } catch (IOException ie) { > System.out.println("IO Error " + ie.getMessage()); > } > } > </pre> > <p>The output will print:</p> > <pre lang="none"> > an easy > easy way > way to > to write > write an > an analyzer > analyzer for > for tokens > tokens bi > bi gram > gram or > or even > even tokens > tokens n > n grams > grams with > with lucene > </pre> > <p>Note that the text “bi-gram” was treated like two > different tokens, as a desired consequence of using a StandardTokenizer > in the ShingleMatrixFilter initialization.</p> > > > Best regards > > On 2/4/20 11:14 AM, baris.ka...@oracle.com wrote: > > > > Thanks but i thought this class would have a mechanism to fix this issue. > > Thanks > > > >> On Feb 4, 2020, at 4:14 AM, Mikhail Khludnev <m...@apache.org> wrote: > >> > >> It's slow per se, since it loads terms positions. Usual advices are > >> shingling or edge ngrams. Note, if this is not a text but a string or > enum, > >> it probably let to apply another tricks. Another idea is perhaps > >> IntervalQueries can be smarter and faster in certain cases, although > they > >> are backed on the same slow positions. > >> > >>> On Tue, Feb 4, 2020 at 7:25 AM <baris.ka...@oracle.com> wrote: > >>> > >>> How can this slowdown be resolved? > >>> is this another limitation of this class? > >>> Thanks > >>> > >>>>> On Feb 3, 2020, at 4:14 PM, baris.ka...@oracle.com wrote: > >>>> Please ignore the first comparison there. i was comparing there > {term1 > >>> with 2 chars} vs {term1 with >= 5 chars + term2 with 1 char} > >>>> > >>>> The slowdown is > >>>> > >>>> The query "term1 term2*" slows down 400 times (~1500 millisecs) > compared > >>> to "term1*" when term1 has >5 chars and term2 is still 1 char. > >>>> Best regards > >>>> > >>>> > >>>>> On 2/3/20 4:13 PM, baris.ka...@oracle.com wrote: > >>>>> Hi,- > >>>>> > >>>>> i hope everyone is doing great. > >>>>> > >>>>> I saw this issue with this class such that if you search for "term1*" > >>> it is good, (i.e., 4 millisecs when it has >= 5 chars and it is ~250 > >>> millisecs when it is 2 chars) > >>>>> but when you search for "term1 term2*" where when term2 is a single > >>> char, the performance degrades too much. > >>>>> The query "term1 term2*" slows down 50 times (~200 millisecs) > compared > >>> to "term1*" case when term 1 has >5 chars and term2 is still 1 char. > >>>>> The query "term1 term2*" slows down 400 times (~1500 millisecs) > >>> compared to "term1*" when term1 has >5 chars and term2 is still 1 char. > >>>>> Is there any suggestion to speed it up? > >>>>> > >>>>> Best regards > >>>>> > >>>>> > >>>>> > >>>>> --------------------------------------------------------------------- > >>>>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >>>>> For additional commands, e-mail: java-user-h...@lucene.apache.org > >>>>> > >>>> --------------------------------------------------------------------- > >>>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >>>> For additional commands, e-mail: java-user-h...@lucene.apache.org > >>>> > >>> > >>> --------------------------------------------------------------------- > >>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >>> For additional commands, e-mail: java-user-h...@lucene.apache.org > >>> > >>> > >> -- > >> Sincerely yours > >> Mikhail Khludnev > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > >
![http://www.philippeadjiman.com/blog/wp-content/uploads/2009/11/lucene_green_3001.gif"](http://www.philippeadjiman.com/blog/wp-content/uploads/2009/11/lucene_green_3001.gif"){kind=link}