Hello, Dan! Hello, Jarkko! Hello, Nick! I'm a bit confused with [EMAIL PROTECTED] Is that a better place for our conversation? Is it alive? Has any traffic?
First of all I'm terribly glad to have somebody with a .jp address to step into the discussion! :-))) 1) Yes, I have read lately a lot. And I have caught the ideas behind ISO 2022 and EUC. 2) Indeed I have sent plenty of mails to [EMAIL PROTECTED] about JIS 0201, JIS 0208, JIS0212, GB 1988, GB 2312 in Encode.pm My first reaction to seeing them in Encode.pm was the same as Dan's: "what are they doing there?" (using terminology from the http://www.debian.org.ru/doc/manuals/intro-i18n/ch-codes.html page) "they are Coded Character Sets, not Character Encoding Schemes!" "Wipe 'em out of Encode.pm!" 3) But I did not meet much approval for this. Here are some of NIS == Nick Ing-Simmons <[EMAIL PROTECTED]> 's comments: AT> GB 2312 ... AT> I would advise to exclude it from Encode.pm NIS> We have it due to fact we inherited stuff from Tcl/Tk - which needs NIS> it for fonts e.g.: NIS> NIS> nick@bactrian 1051$ xlsfonts | grep 2312 NIS> -cc-song-medium-r-normal-jiantizi-40-400-75-75-c-400-gb2312.1980-0 AT> GB 12345 (GB/T 12345-9) NIS> Used for fonts I assume. AT> CNS 11643-1992 NIS> I see a trend here - perhaps a separate bundle for font encodings? AT> GB 18030 AT> BIG5PLUS NIS> They are mostly inherited from Tcl AT> is GB 2312 valid as a parameter to Encode::encode? NIS> We have an gb2312.enc NIS> NIS> FWIW (and worth has to be -ve) I get a daily pile of SPAM with Subjects NIS> like: NIS> NIS> Subject: =?GB2312?B?0LvQu8Tjo6E=?= NIS> NIS> So something thinks it is an encoding. Since I dit not have an immediate 100% approval to wipe 'me out of Encode.pm I have suspected that I was wrong and continued my searches for new knowledge. And seemed 2 me today that I have understood why JIS 0201, JIS 0208, JIS 0212, GB 2312 are in Encode.pm 4) We have a terminology mess here. What is a "coded character set"? rfc1345 (http://www.ietf.org/rfc/rfc1345.txt): From RFC1345: The ISO definition of the term "coded character set" is as follows: "A set of unambiguous rules that establishes a character set and the one-to-one relationship between the characters of the set and their coded representation." and this definition may be subject to different interpretations. This memo does not put further restrictions on the term of "coded character set" than the following: "A coded character set is a set of rules that unambiguously and completely determines which sequence of characters, if any, is represented by each possible sequence of n-bit bytes for a certain value of n." This implies that e.g. a coded character set extended with one or more other coded character sets by means of the extension techniques of ISO 2022 constitutes a coded character set in its own right. In this memo the term "charset" is used to refer to the above interpretation of the ISO term "coded character set". on the other hand http://www.debian.org.ru/doc/manuals/intro-i18n/ch-codes.html uses Coded Character Set and Character Encoding Scheme separately (though I haven't found where the terms used in that page are defined) What meaning of a "coded character set" should we stick to? Let me use - "CCS" (acronym of Coded Character Set) when I mean abstract enumeration of characters in general (f.e. the JIS X 0208 94 x 94 table enumerates chars via the KUTEN (row-cell) code, then this 94 x 94 table specifies a CCS) - "CES" (acronum of Character Encoding Scheme) when I mean mapping from sequences of 8-bit (or 7-bit) bytes into character sequences and back This terminology comes from http://www.debian.org.ru/doc/manuals/intro-i18n/ch-codes.html On the other hand let me use "coded character set" in the meaning that the cited fragment of rfc1345 has attached to it. So "CES" === "coded character set" "CCS" ne "coded character set" "CES" ne "CCS" http://www.debian.org.ru/doc/manuals/intro-i18n/ch-codes.html says "ISO 8859 is both a series of CCS and a series of encodings" That is the salt. The CCS and CES at the same time. The question: are JIS X 0201, JIS X 0208, JIS X 0212 only "CCS"-es or "CCS"-es and "CES"-es at the same time? Dan's mail said they are only "CCS"-es. Encode.pm operates on "CES"-es. That was my original opinion too. But: http://www.iana.org/assignments/character-sets is a "Character Sets Registry" What meaning does this registry attach to "coded character set" term? I believe that this is "CES" after all. It contains things like Shift_JIS and no one will doubt that Shift_JIS is a "CES". But it (the IANA registry) also contains things like Name: JIS_C6220-1969-jp [RFC1345,KXS2] Source: ECMA registry Alias: JIS_C6220-1969 Alias: iso-ir-13 Alias: katakana Name: JIS_C6220-1969-ro [RFC1345,KXS2] Source: ECMA registry Alias: iso-ir-14 Alias: jp Alias: ISO646-JP Alias: csISO14JISC6220ro How do we understand this? Is JIS_C6220-1969-jp a "CES"? For the answer we should look at iso-ir-13. I have no access to the ISO documents, but RFC1345 seems to retell the ISO documents. RFC1345 explains to us what iso-ir-13 is. From RFC1345: It is the intention of this memo to document precisely the mapping between all characters and their corresponding coded representations in various coded character sets, and give names to these coded character sets, so they can be referenced unambiguously in Internet standards. ... The coded character sets covered include all parts of ISO 8859, ISO 6937-2 and all ISO 646 conforming coded character sets in the ISO character set registry managed by ECMA according to ISO 2375. Almost all graphic coded character sets in the ECMA registry (1) are covered. ... The East-Asian 16-bit character sets from the ECMA registry is also included in this memo. So rfc1345 actually specifies a number of "coded character sets". this is what it has for iso-ir-13: &charset JIS_C6220-1969-jp &rem source: ECMA registry &alias JIS_C6220-1969 &alias iso-ir-13 &alias katakana &alias x0201-7 &g0esc x2849 &g1esc x2949 &g2esc x2a49 &g3esc x2b49 &code 0 NU SH SX EX ET EQ AK BL BS HT LF VT FF CR SO SI DL D1 D2 D3 D4 NK SY EB CN EM SB EC FS GS RS US SP ._ <' >' ,_ .6 Wo a6 i6 u6 e6 o6 YA YU YO TU -6 A6 I6 U6 E6 O6 Ka Ki Ku Ke Ko Sa Si Su Se So Ta Ti Tu Te To Na Ni Nu Ne No Ha Hi Hu He Ho Ma Mi Mu Me Mo Ya Yu Yo Ra Ri Ru Re Ro Wa N6 "5 05 ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? DT It has taken me quite a bit of time to understand this, but I have done that :-) - &g0esc... line is not very interesting to us, it only lists the ESC sequences that are used in ISO 2022 to plug in JIS_C6220-1969-jp as G0/G1/G2/G3 - &code 0 is more interesting. This is how &code0 is defined: From RFC1345: "&code" has one parameter indicating the byte number allocated to the following character mnemonic. After the "&code" specification the characters are listed with their mnemonic in ascending order. A character mnemonic of "??" indicates that the position is unused. This means that RFC1345 defines "CES"-es too! Look, &code 0 mean that that bytes numbered starting with 0 are allocated to the following character mnemonics!! Relation between sequences of bytes and characters!! The mnemonics NU SH SX EX ET.. stand for C0 control chars, Ta Ti Tu Te To Na.. stand for Katakana characters. (RFC1345 explains the mnemonics in section 3, but it so boring to read :-) From RFC1345: "&code2" has 2 parameters specifying the row and column in certain 16-bit character sets. The value 32 must be added to obtain the first and second byte respectively. This is how RFC1345 deals with JIS X 0208: &charset JIS_C6226-1983 &rem source: ECMA registry &alias iso-ir-87 &bits 16 &g0esc x2442 &g1esc x242942 &g2esc x242a42 &g3esc x242b42 &alias x0208 &alias JIS_X0208-1983 &code2 1 1 SP ,_ ._ , . .6 : ; ! "5 05 '' '! ': '> '- _ *6 +6 *5 +5 +" so we see that SP (space) in JIS_C6226-1983 gets two byte code. We get it this way: take &code2's "1 1", conclude it is row 1, cell 1, add 0x20 to each of them, get 0x21 0x21 and take this to be the two-byte code for SP in JIS_C6226-1983 5) So, my conclusion: everything listed in http://www.iana.org/assignments/character-sets are "CES"-es. All coded character sets mentioned in RFC 1345 are CES-es too. This is what both RFC 1345 and IANA registry have: &charset JIS_X0201 &alias X0201 Name: JIS_X0201 [RFC1345,KXS2] Alias: X0201 This _must_ be a CES. What is it? If we look into RFC, we'll see a table with 256 mnemonics. First half is JIS X 0201 Roman, the second half is JIS X 0201 Katakana. The IANA registry usually just names the encoding. It does not explain what it is. But in some cases there's an explanation. JIS_X0201 is such a case: Name: JIS_X0201 [RFC1345,KXS2] Source: JIS X 0201-1976. One byte only, this is equivalent to JIS/Roman (similar to ASCII) plus eight-bit half-width Katakana Just what we could conclude from RFC1345. Hurray! :-))) Then the only thing we should do about this is to come up with a better name: JIS 0201 is bad. Would it better be JIS_X0201? X0201? 'JIS X 0201'? 'JIS-X-0201'? Same thing with JIS 0208, JIS 0212. They are registered both in RFC1345 and in the IANA registry. Why should we wipe them from Encode.pm if these are CES-es and they are doing something reasonable? 6) The situation is worse with GB 1988. Unlike JIS X 0201 it defines only one set of characters: ASCII: $->Yuan. But our 1988.enc seems to use the codes 0xA1-0xFE the same way as for JIS X 0201, for Katakana. Is this reasonable? Do Chinese use Katakana? (But this was in another mail of mine) 7) GB-2312_80 is just in the same situation: at the first glance it is only a CCS, but at the second one it is a CES too! It is registered both at RFC1345 and in IANA registry. The only thing we have to do with it is think of a better name. (See my other name to see why GB 2312 is ambigeous) 8) The situation with CNS 11643 GB 12345 is worse because they are registered neither in RFC1345 nor at the IANA registry. And they also look like CCS-es at the first glance. But them being in Encode.pm seems to indicate they are full weight CES-es on their own, just like all others mentioned so far. Uff.. It hasn't been a piece of cake to understand this, but once I have done this I'm more or less content :-)) The only things left to do are - fix the comments in Encode.pm: JIS 0201,JIS 0208,JIS 0212,GB 2312 are not good names - GB 1988 may be just broken (should there be katakana in GB 1988?) - put at least _some_ of the _this_ knowledge into Encode.pm's comments (this is what my patch is partially about) - submit a bugreport saying that aliasing mechanisms are broken in Encode.pm And have some sleep :-))) My best regards, Anton DK> Aanton, DK> You may not know this but JISX2\d\d does not determine encoding at at DK> all. They are tables that actual encodings base upon. In other words, DK> RAW JISX2\d\d are NEVER used in actual encodings. Here is how EUC-JP DK> uses ALL of them. DK> EUC-JP consists of.... DK> ASCII: 0x00-0x7f DK> JIS X 0201-1978: 0x8E + (X0201 Code beyond 0x80) DK> JIS X 0208-1990: (X0208 Code) + 0x8080 DK> JIS X 0212-1990: 0x8F . (X0212 Code) DK> In other words, virtually all multibyte encodings are not only DK> multibyte, but also multitabled. DK> Then how come there are Encode/jis02\d\d ? Good question. They are DK> used by Encode::Tcl to implement 7-bit jis. Encode/7bit-jis.enc looks DK> like this. DK> # Encoding file: 7bit-jis, escape-driven DK> E DK> name 7bit-jis DK> init {} DK> final {} DK> ascii \x1b(B DK> ascii \x1b(J DK> 7bit-kana \x1b(I DK> jis0208 \x1b$B DK> jis0208 \x1b$@ DK> jis0208 \x1b&@\x1b$B DK> jis0212 \x1b$(D DK> This is how ISO-2022 implements a given encoding. It switches the DK> "current" encoding by escape sequence. Since escape sequence is used, DK> "raw" encodings are directly applied, while EUC turns Most significant DK> bit (MSB) on. DK> As a transfer encoding, ISO-2022 is great because in theory it can DK> swallow any number of subcodings. However, it is pain to the neck to DK> use it as internal encoding because you can't tell in what subcoding we DK> are now in just by looking a given byte. In case of EUC you tell if it DK> is single byte or multibyte just by looking at MSB. And Unicode was DK> born out of even more ambition by making all character double-byte (but DK> this has failed (or compromised if you don't like the word) when DK> surrogate pair was introduced). DK> Dan the Man with Too Many Codings DK> On Wednesday, March 20, 2002, at 04:28 , Jarkko Hietaniemi wrote: >> More from Anton... >> >> ----- Forwarded message from Anton Tagunov <[EMAIL PROTECTED]> ----- >> >> Subject: Better names for JIS X 0201/0208/0212? (was: ISO-8859-1 vs >> ISO 8859-1 (typo + UTF8 case too :) >> From: Anton Tagunov <[EMAIL PROTECTED]> >> Date: Tue, 19 Mar 2002 20:32:26 +0300 >> Message-ID: <[EMAIL PROTECTED]> >> To: Nick Ing-Simmons <[EMAIL PROTECTED]> >> Cc: [EMAIL PROTECTED], [EMAIL PROTECTED] >> In-reply-To: <[EMAIL PROTECTED]> >> >> Hello, Nick! Hello, all! >> >> I certainly think that the names like 'JIS 0201' >> are embarrassing. >> >> Here's rfc1345 >> >> &charset JIS_X0201 >> &alias X0201 >> ...8-bit, JIS-Roman (0xA1-0x7E) + JIS-Katakana (0xA1-0xFE) >> >> &charset JIS_C6226-1983 >> &alias iso-ir-87 >> &bits 16 >> &alias x0208 >> &alias JIS_X0208-1983 >> >> &charset JIS_X0212-1990 >> &alias x0212 >> &alias iso-ir-159 >> &bits 16 >> >> here's IANA registry >> >> Name: JIS_X0201 [RFC1345,KXS2] >> MIBenum: 15 >> Source: JIS X 0201-1976. One byte only, this is equivalent to >> JIS/Roman (similar to ASCII) plus eight-bit half-width >> Katakana >> Alias: X0201 >> >> Name: JIS_C6226-1983 [RFC1345,KXS2] >> Alias: iso-ir-87 >> Alias: x0208 >> Alias: JIS_X0208-1983 >> >> Name: JIS_X0212-1990 [RFC1345,KXS2] >> MIBenum: 98 >> Alias: x0212 >> Alias: iso-ir-159 >> >> >> Are >> >> JIS_X0201 / X0201 / >> JIS_C6226-1983 / X0208 / >> JIS_X0212 / X0212 >> >> better candidates? >> >> - Anton >> >> P.S. I have also seen JIS_X0201 referred to as >> >> JIS X 0201-1976 >> JIS X 0201 Katakana/JIS X 0201 Roman >> JISX0201. >> >> P.P.S. (currently >> JIS 0201/JIS 0208/JIS 0212 do not seem to work for me: >> >> perl15173 -MEncode -MEncode::JP -we "print Encode::decode('JIS >> 0210','aaa')" >> >> gives me Unknown encoding 'JIS 210' at -e line 1, only >> Encode::encode('JIS0201','aaa') behaves okay.. >> >> and if they are not working, we're free to change the names for anything >> we like ;-) >> >> >> ----- End forwarded message ----- >> >> -- >> $jhi++; # http://www.iki.fi/jhi/ >> # There is this special biologist word we use for 'stable'. >> # It is 'dead'. -- Jack Cohen -- Best regards, Anton mailto:[EMAIL PROTECTED]
