Michel Pelletier wrote:
>The splitter should really be a modular component. That's what
>vocabularies were origninally for, to store language specific artifacts
>like word lists and splitters. For example, stripping the "ing" suffix
>obviously only makes sense in English. so if you want to change this
>behavior, make your own vocabulary with its own custom splitter.
>This is because each language has very different splitting requirements,
>and even different meanings of the word "word". Imagine, for example,
>splitting Japanese or one of the Chinese languages (based textualy on
Just imagine German! There are composite words without spaces or other
non-aphanumeric characters between them.
>Identifying words in Kanji is a very hard problem. In romance langauge,
>it's easy, words are seperated by spaces, but in Kanji words are
>diferentiated by the context of the surrounding characters, there are no
>"spaces". Splitting Kanji text requres a pre-existing dictionary and some
>interesting heuristic matching algorithms. And that's only half of
>Japanese itself, really, since there are two other alphabets (hiragana and
>katagana) that *are* character-phonetic like romance langauges, and all
>three alphabets are commonly mixed together in the same sentence! Chinese
>language may also have these phonetic alphabets.
The same applies for German: You'd need a huge dictionary with word stems,
exceptions, and stop words.
Stems of many words change in different cases, too.
>In other words, it's not an easy problem! There is going to be an
>unimaginable culture clash when asian and other non-romance languages
>catch up to the volume of romance language content on the web.
Well, English or German in fact aren't romance languages, they're germanic
Zope-Dev maillist - [EMAIL PROTECTED]
** No cross posts or HTML encoding! **
(Related lists -