Hi guys,
If I am to tokenize an email address like "John Smith" <
<mailto:[EMAIL PROTECTED]> [EMAIL PROTECTED]> into
[ <mailto:[EMAIL PROTECTED]> [EMAIL PROTECTED]
[John]
[Smith]
[J.Smith]
[london.gb.world.net]
[gb.world.net]
[world.net]
[world]
[net]
Is it possible to have a different Position increment for each of these
tokens? If it is, could you please help me with the same sample, with
numbers against each token.
Also could you please point me to a skeleton code for a custom Tokenizer.
Many Thanks
Dino
