Plain easy to do customize with lambdas! E.g., an elegant way to create a 
tokenizer which behaves exactly as WhitespaceTokenizer and LowerCaseFilter is:

Tokenizer tok = 

Adjust with Lambdas and you can create any tokenizer based on any character 
check, so to check for whitespace or underscore:

Tokenizer tok = CharTokenizer.fromSeparatorCharPredicate(ch -> 
Character.isWhitespace || ch == '_');


Uwe Schindler
Achterdiek 19, D-28357 Bremen

> -----Original Message-----
> From: Armins Stepanjans []
> Sent: Monday, January 8, 2018 11:30 AM
> To:
> Subject: Looking For Tokenizer With Custom Delimeter
> Hi,
> I am looking for a tokenizer, where I could specify a delimiter by which
> the words are tokenized, for example if I choose the delimiters as ' ' and
> '_' the following string:
> "foo__bar doo"
> would be tokenized into:
> "foo", "", "bar", "doo"
> (The analyzer could further filter empty tokens, since having the empty
> string token is not critical).
> Is such functionality built into Lucene (I'm working with 7.1.0) and does
> this seem like the correct approach to the problem?
> Regards,
> Armīns

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to