Can StandardTokenizerFactory works well for Chinese and English (Bilingual)?

[Zheng Lin Edwin Yeo]

Hi,

Would like to check, will StandardTokenizerFactory works well for indexing
both English and Chinese (Bilingual) documents, or do we need tokenizers
that are customised for chinese (Eg: HMMChineseTokenizerFactory)?


Regards,
Edwin