RE: Writing custom Tokenizer

[Siddhartha Singh Sandhu]

Hi Everyone,

I wanted to write a custom tokenizer and wanted a generic direction and
some guidance on how I should go about achieving this goal.

Your input will be much appreciated.

Regards,

Sid.