CharTokenizer.incrementToken() clears *all* attributes in the entire tokenizer chain. StandardTokenizer.incrementToken() clears only the term attribute.
So... which is right? Seems like the tokenizer should be responsible? On a performance related note, CharTokenizer.clearAttribtes() could be more efficient - 2 new objects (the unmodifiable map and the iterator object) are created for every incrementToken. -Yonik http://www.lucidimagination.com --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
