On Tue, Nov 15, 2005 at 06:50:33PM +0100, [EMAIL PROTECTED] wrote: > Do you know if search engines such as google has different code to > support different languages?
I don't know about Google specifically, but certainly some engines process text in different languages differently. Specifically, they use different stemming algorithms, term splitting algorithms (particularly for languages such as German which really need word-splitting, and languages which need multibyte characters). Also, accents in english tend to be best ignored (cafe has the same meaning with or without an accent), while in other languages they tend to be quite important. Automatic language guessing is not actually that tricky to implement - there tend to be characteristic words which occur frequently only in a certain language, or infrequently in a certain language. You can find these words by taking reasonably large sets of documents in each language you wish to be able to detect, and then analysing the frequencies of words in each language. It is usually then possible to extract a fairly small set of words and associated probabilities for each language, and compare these probabilities with the frequencies of words in a sample document, and get a reasonably reliable guess as to the language of the sample document. This technique can even be used with some success on reasonably sized paragraphs of text, so mixed language documents could be processed appropriately. -- Richard _______________________________________________ Dashboard-hackers mailing list [email protected] http://mail.gnome.org/mailman/listinfo/dashboard-hackers
