Hi Charles,
The MS936 charset is long overdue for a update. See CR#6183404. The
mapping need
to be re-generated from MS's latest 936 table (not, MS936 should just
follow MS's mapping
table, not GB18030) As noted in MS936.map, the existing mapping table
uses 1894 entries
from GBK UDC block for EUDC mapping, as suggested by IBM engineer back
to 1999, which
was a reasonable approach back then.
I will try to generate a new MS936 for JDK8.
-Sherman
On 5/23/2012 1:03 AM, Charles Lee wrote:
Hi guys,
We have a simple test case:
for (String cname : new String[] { "GBK", "MS936", "GB18030" }) {
Charset charset = Charset.forName(cname);
System.out.println("charset: " + charset.name());
CharsetEncoder ce = charset.newEncoder();
char[] chars = new char[] { 0xE585, 0xE586, 0xE592 };
CharBuffer cb = CharBuffer.wrap(chars);
ByteBuffer bb = ce.encode(cb);
for (char c : chars) {
System.out.printf("\\u%04x", (int) c);
}
System.out.print(" -> ");
for (byte b : bb.array())
if (b != 0x0) {
System.out.printf("\\x%02x", (int) b & 0xFF);
}
System.out.println("");
}
The output is
charset: GBK
\ue585\ue586\ue592 -> \xa2\xa0\xa2\xab\xa3\x40
charset: x-mswin-936
\ue585\ue586\ue592 -> \xa2\xa0\xa2\xab\xa3\x40
charset: GB18030
\ue585\ue586\ue592 -> \xa2\xa0\xa3\x40\xa3\x4c
From the msdn[1], U+E000 -> U+F8FF is in the EUDC scope. So U+E586 is
in the EUDC scope. But the mapped code in MS936/GBK is 0xA2AB, it is
not in the EUDC scope.
With another simple test case, you can find there are more codes that
is not mapped right:
for (int i = 0xE000; i < 0xE000 + 1894; i++) {
String s = new String(new char[] { (char) i });
byte[] bs = s.getBytes("MS936");
int b0 = (int) bs[0] & 0xFF;
int b1 = (int) bs[1] & 0xFF;
if ((b0 >= 0xAA && b0 <= 0xAF) && (b1 >= 0xA1 && b1 <= 0xFE))
continue;
if ((b0 >= 0xF8 && b0 <= 0xFE) && (b1 >= 0xA1 && b1 <= 0xFE))
continue;
if ((b0 >= 0xA1 && b0 <= 0xA7) && (b1 >= 0x40 && b1 <= 0xA0))
continue;
System.out.printf("\\u%04X -> \\x%02X\\x%02X%n", i, b0, b1);
}
I have written a generator in C#[2] which outputs the mapping code in
GB2312[3] and GB18030[4] in scope U+E000 and U+F8FF to find that most
of code are the same. Hereby I suggest we may follow the code from
GB2312 and the changed map file in openjdk can be found [5][6].
Would anyone help to take a look on this issue?
[1]
http://msdn.microsoft.com/en-us/library/windows/desktop/dd317837%28v=vs.85%29.aspx
[2] http://cr.openjdk.java.net/~littlee/OJDK-63/webrev.00/Program.cs
[3] http://cr.openjdk.java.net/~littlee/OJDK-63/webrev.00/gb2312Map.txt
[4] http://cr.openjdk.java.net/~littlee/OJDK-63/webrev.00/gb18030Map.txt
[5] http://cr.openjdk.java.net/~littlee/OJDK-63/webrev.00/GBK.map.new
[6] http://cr.openjdk.java.net/~littlee/OJDK-63/webrev.00/MS936.map.new
P.S: Sorry for the late notice.
On 03/29/2011 03:00 PM, Charles Lee wrote:
On 03/28/2011 11:06 PM, Alan Bateman wrote:
Charles Lee wrote:
:
It looks similar. How can I find the patch quickly? I notice it
says "the list is attached to this CR". Is it CR-6183404? Since cr
has the pattern cr.openjdk.java.net/~username/id, how can I know
who is the committer to this CR?
cr.openjdk.java.net is the place where we push webrevs when a patch
is out for review. I don't think this one is one anyone's list for
jdk7 and the list attached to the bug is likely the list of
incorrect mappings. If this is fixed then I assume the fix will
update the mappings in jdk/make/tools/CharsetMapping/MS936.map.
-Alan
I have output more bytes[1] to see whether other bytes are encoded
correctly. But unfortunately it is not. It is kind of like, on
windows, using ms936, PUA of ms936 use the PUA of gb18030. In
wikipedia, it says gb18030 is compatible with gbk which ms936
implemented. Can we conclude that ms936 should follow the gb18030's
behavior?
[1] 0xE585, 0xE586, 0xE587, 0xE588, 0xE589, 0xE58a, 0xE58b, 0xE58c,
0xE58d, 0xE58e, 0xE58f, 0xE590, 0xE591, 0xE592, 0xE593, 0xE594,
0xE595, 0xE596, 0xE597, 0xE598, 0xE599, 0xE59a, 0xE59b, 0xE59c,
0xE59d, 0xE59e, 0xe79f.
Using MS936 charset, we expect:
\xa2\xa0\xa3\x40\xa3\x41\xa3\x42\xa3\x43\xa3\x44\xa3\x45\xa3\x46\xa3\x47\xa3\x48\xa3\x49\xa3\x4a\xa3\x4b\xa3\x4c\xa3\x4d\xa3\x4e\xa3\x4f\xa3\x50\xa3\x51\xa3\x52\xa3\x53\xa3\x54\xa3\x55\xa3\x56\xa3\x57\xa3\x58\xa6\xfe
but we got:
\xa2\xa0\xa2\xab\xa2\xac\xa2\xad\xa2\xae\xa2\xaf\xa2\xb0\xa2\xe3\xa2\xe4\xa2\xef\xa2\xf0\xa2\xfd\xa2\xfe\xa3\x40\xa3\x41\xa3\x42\xa3\x43\xa3\x44\xa3\x45\xa3\x46\xa3\x47\xa3\x48\xa3\x49\xa3\x4a\xa3\x4b\xa3\x4c\xa7\xa0
--
Yours Charles
