On Aug 1, 2005, at 5:31 PM, Ken Krugler wrote:
Or you can derive the language from the host URL, if it includes a
country code.
That's not really sufficient... many Japanese sites also have pages
in English. Actually, that's true for most non-English sites from
what I've seen.
Yes - this is just a last-gasp fallback, in case you're forced to
guess. Statistically it will be better than always picking en :)
It's hard to detect all the various encodings... EUC-JP,
SHIFT-JIS, ISO-2022-KR/JP, BIG5, etc. and many servers do not
correctly identify the encodings.
See the latest release of ICU (3.4), which now supports charset detection.
Yes, I forgot about that... but even then I wonder how well it will
do. For largish blocks of text (1k or so) it's not bad... you can
use statistical modelling to give you accurate probabilities, but
for smallish blocks (e.g. query strings) you have a much harder time.
Yes - small chunks of untagged text are going to be a problem, no
matter what you do. But if you're referring to query strings from an
HTML page, the default is to use the encoding of the page (which from
Nutch defaults to UTF-8). And you can use the accept-charset form
attribute to explicitly specify UTF-8.
-- Ken
--
Ken Krugler
TransPac Software, Inc.
<http://www.transpac.com>
+1 530-470-9200
-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
Nutch-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
