On Tue, 9 Aug 2005 16:11:39 +0430, mohsen ali momeni
<[EMAIL PROTECTED]> posted to gmane.comp.misc.persiancomputing:
 > How can I auto-detect language of a webpage without knowing it's
 > charset? (suppose language and charset is not defined in header)
 > Is there a simple (not time-consuming) method to detect a page charset?

Sorry for the late reply -- I just stumbled over your question via Google.
I'm hoping I might be able to help, a little bit.

There are tools which attempt to match both the language and the
character set of a piece of text at the same time, which indeed kind
of solves the "chicken and egg" problem you stumble into if you
attempt to tackle the problems separately. A popular free tool is
TextCat, which contains language (and incidentally charset)
identification for almost 50 languages, although not for Farsi.

... However, it would seem that the Arabic language detection which is
included in the distribution has in fact been trained with Farsi data
to some extent. Also it would seem to be getting the charset wrong (as
per the garbage in / garbage out principle).

I know neither Farsi nor Arabic so I would appreciate if somebody
could confirm this information. (I'm hoping this message will go to
the list too, but I'm not subscribed, so it might get rejected.)

In any event, the language models which are included in the TextCat
distribution are "proof of concept" ones, not really production
quality. Another tool which comes with better models is mguesser. And
in fact, the mguesser models can be used more or less directly by
TextCat (and vice versa).

If you have a sizable corpus of good-quality Persian / Farsi text
(sorry, I'm vague about the possible difference) to train the tool
with, it would be most helpful if you could make it (the raw data, or
your training results) available.

     Perl source + (small) models + on-line demo + good documentation

     C source + models; site is somewhat poor quality but the software is OK

The TextCat site also has a page with a survey of the competition
(although mguesser is mysteriously missing from the list).

Hope this helps,

/* era */

If this were a real .signature, it would suck less.  Well, maybe not.

PersianComputing mailing list

Reply via email to