Re: [Assp-test] soft hyphen fooling Bayesian analysis

2022-09-07 Thread K Post
Thanks again for the explanation. Looking forward to a future release when soft-hyphens (and additional control characters?) are essentially ignored. On Wed, Sep 7, 2022 at 9:14 AM Thomas Eckardt wrote: > If unicode normalization NFKC does'nt fulfill your requirement, you may > enable

Re: [Assp-test] soft hyphen fooling Bayesian analysis

2022-09-07 Thread Thomas Eckardt
If unicode normalization NFKC does'nt fulfill your requirement, you may enable 'DoTransliterate' - by accepting some performance penalties. The "Unicode Technical Standard #39" http://www.unicode.org/reports/tr39/ will give you some more information and