rmuir commented on PR #16722: URL: https://github.com/apache/lucene/pull/16722#issuecomment-5957468739
its not gonna make anything worse. for example, let's take the tokenizer change. for the tokenizer change, i made it try to "only break on sentences" because most CJK models are trained on whole sentences. So it was just a bias there. But for thai it doesn't make sense: the way the dictionary-based-break-iterator works, it doesn't make use of a whole sentence. So for thai this was just causing truncated analysis (like "words split in half"). Hence, if before you were getting words truncated due to that, its not going to make anything worse to no longer do it. if you want to take advantage of the improved analysis, you have to reindex in order to get the benefits. then you won't have such truncated tokens anymore. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
