Re: adding attributes to TokenStream

[Michael Sokolov]

On 12/31/2012 11:39 AM, Itai Peleg wrote:
Hi all,
Can someone please post a simple example showing how to add additional
attributes to token in a TokenStream (inside IncrementToken for example?).

I'm working on entity extraction and want to flag specific tokens an
entities, but I'm having problems.

Thanks in advance,
Itai

Here's a simple example of a filter that adds an atytribute saying 
whether a token is "the"

class YourTokenStream extends TokenFilter {
  private final YourAttribute att;
  private final CharTermAttribute term;
  private final TokenStream source;

  public YourTokenStream (TokenStream upstream) {
     att = addAttribute (YourAttribute.class);
     term = addAttribute (CharTermAttribute.class);
     source = upstream;
  }

  public boolean incrementToken () {
    if (source.incrementToken()) ?? {
      if ("the".equals (new String(term.buffer())) {
        att.setIsAnEnglishArticle(true);
        return true;
    }
    return false;
  }

}



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org