On Tue, 2009-09-15 at 16:48 +0900, Olaf Meeuwissen wrote: > For that matter, splitting on whitespace sucks when you use Japanese > (which normally doesn't use whitespace). Splitting should be done on > word boundaries. Hopefully 0.12 will address at least some of these > issues for non-POSIX locales.
I am not doubting the complexity. Perhaps it could be as simple as a configured parameter where one could add characters, beyond the ones defined by the locale, that are to be treated as whitespace/punctuation. If you are happy with the current result, just leave the list alone. ANSI C defines ispunct(), which is locale-sensitive. Perhaps that is of some use here. Is that also available in python? As to Japanese, I am guessing that word boundaries at sentence punctuation is ok. It is just spaces that are the problem. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Trac Users" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/trac-users?hl=en -~----------~----~----~----~------~----~------~--~---
