Hi Rui, I think ClassicTokenizerImpl.jflex file is good start for understanding tokenizers.
http://svn.apache.org/repos/asf/lucene/dev/trunk/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/ClassicTokenizerImpl.jflex Please see other *.jflex files in source tree. But usually you can manipulate tokenizer behaviour with chatFilters without creating a new tokenizer. Can you eleborate more? On Wednesday, March 5, 2014 1:00 PM, epnRui <rui_banda...@hotmail.com> wrote: I have managed to understand how to properly implement and change the words on a CharFilter and a Filter, but I fail to understand how the Tokenizer works... I also fail to find any tutorials on the thing.. Could you provide some example implementation of incrementToken and how to manipulate the tokens? Is there any documentation on this? Thanks! -- View this message in context: http://lucene.472066.n3.nabble.com/Implementing-a-customised-tokenizer-tp4121355.html Sent from the Solr - User mailing list archive at Nabble.com.
