[Lucene-dev] Allowing an Analyzer to choose a parsing strategy based on context

[Scott Ganyo]

I've made the following simple and backward-compatible changes to a couple
of classes in order to allow an Analyzer to choose a parsing strategy based
on Document and/or Field:

I changed DocumentWriter.java, line 123 from:
TokenStream stream = analyzer.tokenStream(reader);

To:
TokenStream stream = analyzer.tokenStream(doc, field, reader);


...and I changed Analyzer.java implementation to add tokenStream(Document,
Field, Reader) method:

abstract public class Analyzer {
  /** Creates a TokenStream which tokenizes all the text in the provided
    Reader.  Default implementation forwards to tokenStream(Reader) for 
    compatibility with older version.  Override to allow Analyzer to choose 
    strategy based on document and/or field. */
  public TokenStream tokenStream(Document doc, Field field, Reader reader)
  {
          // implemented for backward compatibility
          return tokenStream(reader);
  }
  
  /** Creates a TokenStream which tokenizes all the text in the provided
    Reader. Can be implemented as a noop if overriding tokenStream(Document,
    Field, Reader). */
  abstract public TokenStream tokenStream(Reader reader);

Any objections to this change?

Thanks,
Scott


_______________________________________________
Lucene-dev mailing list
[EMAIL PROTECTED]
http://lists.sourceforge.net/lists/listinfo/lucene-dev