Hi Steve, I'm sorry. That's also CustomAnalyzer. public class CustomAnalyzer extends Analyzer { > @Override > protected Analyzer.TokenStreamComponents createComponents(final String > fieldName, final Reader reader) { > final WhitespaceTokenizer src = new WhitespaceTokenizer(getVersion(), > reader); > TokenStream tok = new ShingleFilter(src, 2, 3); > tok = new ClassicFilter(tok); > tok = new LowerCaseFilter(tok); > // tok = new SynonymFilter(tok,SynonymDictionary. > getSynonymMap(),true); > return new Analyzer.TokenStreamComponents(src, tok); > } > } > > public class Test { > public static void main(String[] args) throws Exception { > CustomAnalyzer analyzer = new CustomAnalyzer(); > String queryStr = "cup board"; > TokenStream ts = new CustomAnalyzer().tokenStream("n", new > StringReader(queryStr)); > ts.reset(); > System.out.println("Tokens are :"); > while (ts.incrementToken()) { > System.out.print(ts.getAttribute(CharTermAttribute.class) + > ", "); > } > QueryParser parser = new QueryParser("n", analyzer); > Query query = null; > query = parser.parse(queryStr); > System.out.println("\nQuery is"); > System.out.print(query.toString()); > } > }
Output: > Tokens are : > cup, cup board, board > Query is n > n:cup n:board > On Mon, Jul 24, 2017 at 11:08 PM, Steve Rowe <sar...@gmail.com> wrote: > Hi hariram, > > There may be other problems, but at a minimum you have two different > analysis classes here. You’re printing the output stream from one > (CustomSynynymAnalyzer, the source of which is not shown in your email), > but constructing a query from a different one (CustomAnalyzer). > > -- > Steve > www.lucidworks.com > > > On Jul 24, 2017, at 10:53 AM, hariram ravichandran < > hariramravichan...@gmail.com> wrote: > > > > I'm using Lucene 4.10.4 and trying to construct (shingles) combinations > of > > tokens. > > > > > > Code: > > > > public class CustomAnalyzer extends Analyzer { > > @Override > > protected Analyzer.TokenStreamComponents createComponents(final String > > fieldName, final Reader reader) { > > final WhitespaceTokenizer src = new > > WhitespaceTokenizer(getVersion(), reader); > > TokenStream tok = new ShingleFilter(src, 2, 3); > > tok = new ClassicFilter(tok); > > tok = new LowerCaseFilter(tok); > > // tok = new > > SynonymFilter(tok,SynonymDictionary.getSynonymMap(),true); > > return new Analyzer.TokenStreamComponents(src, tok); > > } > > } > > > > public class Test { > > public static void main(String[] args) throws Exception { > > CustomSynonymAnalyzer analyzer = new CustomSynonymAnalyzer(); > > String queryStr = "cup board"; > > TokenStream ts = new CustomAnalyzer().tokenStream("n", new > > StringReader(queryStr)); > > ts.reset(); > > System.out.println("Tokens are :"); > > while (ts.incrementToken()) { > > System.out.print(ts.getAttribute(CharTermAttribute.class) + > ", > > "); > > } > > QueryParser parser = new QueryParser("n", analyzer); > > Query query = null; > > query = parser.parse(queryStr); > > System.out.println("\nQuery is"); > > System.out.print(query.toString()); > > } > > } > > > > > > > >> Output: > >> Tokens are : > >> cup, cup board, board > >> Query is n > >> n:cup n:board > >> > > > > Tokens are printed as expected. And expecting the resulting query to be > *n:cup > > n:board n:cup board*. But tokens formed by shingle filter are not > appended > > to the query. I get only *n:cup n:board.* Where is my mistake? > > > > Thanks. > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > >