It has been brought to my attention that ICUTokenizerFactory drops tokens like the ++ in "The C++ Programming Language." Is there any way to persuade it to preserve these types of tokens?
thanks, Demian
- Preserving punctuation tokens with ICUTokenizerFactory Demian Katz
