Thinking through this a little more, I don't see an alternative to the tokenizer clearing all attributes at the start of incrementToken().
Consider a DefaultPayloadTokenFilter that only sets a payload if one isn't already set - it's clear that this filter can't clear the payload attribute, so it must be cleared by the head of the chain - the tokenizer. Right? -Yonik http://www.lucidimagination.com --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
