On 2002.01.31, at 17:13, Nick Ing-Simmons wrote: >> Now the problem is escape-based codings such as ISO-2022. > > Can you explain the way those work? > I can imagine two ways for decode: > A - keep going with current sub-encoding till we get a fail, > then look at next few octets for an escape sequence. > B. - Scan ahead for next escape sequence (or end of available input) > then translate up to that.
To answer these questions, let's see what the existing utilities do. Here I will discuss NKF, jcode.pl and my humble Jcode. NKF (Network Kanji Filter) ftp://ftp.ie.u- ryukyu.ac.jp/pub/software/kono/ * 1st appeared in 1987. Still maintained. * Handles EUC-JP, JIS (ISO-2022-JP) and SHIFT_JIS. * No Unicode support to the date. This is understandable because other "Legacy" encodings needs no conversion table since they are all based upon JISX2xx. * Stream based. No buffer allocation and such (later changed when NKF.pm was added to the distribution. But even this case NKF.xs just does buffer handling and nkf(1) does no in-memory conversion). * Uses method B for ISO-2022 (or my ungetc() !). jcode.pl ftp://ftp.iij.ad.jp/pub/IIJ/dist/utashiro/perl/ * 1st appeared in 1992, BEFORE Perl5 * still maintained; still widely used for the same reason cgi-lib.pl is used instead of CGI.pm * Written 100% in perl * No Unicode support * Method C? Just like method B but it uses regex to grab between escape boundaries (see Jcode.pm) below Jcode.pm http://www.openlab.gr.jp/Jcode/ * 1st appeared in 1999 * Unicode support added (XS and NoXS both supported) * object model * internal routines based upon jcode.pl * and here is the piece of sub that does JIS -> EUC conversion sub jis_euc { my $thingy = shift; my $r_str = ref $thingy ? $thingy : \$thingy; $$r_str =~ s( ($RE{JIS_0212}|$RE{JIS_0208}|$RE{JIS_ASC}|$RE{JIS_KANA}) ([^¥e]*) ) { my ($esc, $str) = ($1, $2); if ($esc !~ /$RE{JIS_ASC}/o) { $str =~ tr/\x21-\x7e/\xa1-\xfe/; if ($esc =~ /$RE{JIS_KANA}/o) { $str =~ s/([\xa1-\xdf])/\x8e$1/og; } elsif ($esc =~ /$RE{JIS_0212}/o) { $str =~ s/([\xa1-\xfe][\xa1-\xfe])/\x8f$1/og; } } $str; }geox; $$r_str; } > You could bundle several encodings in one XS (the way Encode itself > bundles ASCII, ios-8859-* and koi8). I know. But it is bulky and another problem is that Tcl has a different notion of 'Escape' (like euc_jp_0212, which is not exactly an escape but an extension) which needs to be corrected for the practical use. > If any of the bundled encodings have similar sequences of code points > then we will get overall table size reductions too. > > In the limit one could have Encode::CJK, but perhaps > Encode::JP / Encode::CN / Encode::KR makes more sense ??? Right. From a user's point of view distinct package space for each (human) language is better. But again, this can be implemented like Encode::EUC (does all euc-based conversion) Encode::JP (Wrapper module that calls Encode::EUC and Encode::ISO2022) Encode::KR Encode::ZN and so forth. Actually even more table reduction can be done between SHIFT_JIS and EUC. They are all based upon JISX0208 (and 0201 and 0212) so simple calculation converts one another. > We could ship things pre-compiled (with origianal .ucm's gzipped, or > provide a way to extract a .ucm from the compiled form). > Also the compile process is all in perl and has not really been tunned. > It spends a lot of time trying to find common "strings" (which gets > tables > down in size so is a win.) Right. How we do that we still need more experiments but this is what should be done.... Dan
