Emmanuel Charpentier wrote:
On Mon, 08 Sep 2008 01:45:51 +0200, Peter Dalgaard wrote :
Emmanuel Charpentier wrote:
Dear list,
[ Snip ... ]
This looks reasonably sane, I think. The last loop could be d[] <-
lapply(d, conv1, from, to), but I think that is cosmetic. You can't
really do much better because there is no simple way of distinguishing
between the various 8-bit character sets.
Thank you Peter !
Could you point me to some not-so-simple (or even doubleplusunsimple)
ways ? I get the problem not so rarely, and I'd like to pull this chard
outta my poor tired foot one and for all... and I suppose that I am not
alone in this predicament.
In full generality it is impossible, but you might get something if you
make certain assumptions. If you can convert from UTF8, then it probably
is UTF8 (or ASCII but in either case, you're done). Otherwise it is a
single-byte 8-bit encoding if the language can be assumed to be
French. If it uses characters between 0x80 and 0x9f, then it is not
latin1 but rather cp437, 850, or 1252. The tricky bit is that although
the presence of say 0x82 suggests that it is not cp1252, but rather 437
or 850 (e aigu) , it just might be 1252 after all (single low quote).
Some sort of naive Bayes classifier might work.
You could presumably setup
some heuristics. like the fact that the occurrence of 0x82 or 0x8a
probably indicates cp437, but it gets tricky. (At least, in French, you
don't have the Danish/Norwegian peculiarity that upper/lowercase o-slash
were missing in cp437, and therefore often replaced yen and cent symbols
in matrix printer ROMs. We still get the occational parcel addressed to
"¥ster Farimagsgade".)
Peter, you're gravely underestimating the ingenuity of some Excel
l^Husers... (and your story is a possible candidate for a fortune()
entry...)
(If so, please respell "occasional". Ouch! And actually, the cent/yen
thing is also a difference between cp437 and cp850, so the story may be
a bit too colourful.)
--
O__ ---- Peter Dalgaard Øster Farimagsgade 5, Entr.B
c/ /'_ --- Dept. of Biostatistics PO Box 2099, 1014 Cph. K
(*) \(*) -- University of Copenhagen Denmark Ph: (+45) 35327918
~~~~~~~~~~ - ([EMAIL PROTECTED]) FAX: (+45) 35327907
______________________________________________
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
