Hi,
[I combined the proposed charge for #7082884, in which no one appears to be
interested:-) into this one]
Unicode Standard added "Addition Constraints on conversion of ill-formed
UTF-8"
in version 5.1 [1] and updated in 6.0 again with further "clarification"
[2] regarding
how a "conformance" implementation should handle ill-formed UTF-8 byte
sequence. Basically it says
(1) the conversion process should not interpret any ill-formed code
unit sequence
(2) such process must not treat any adjacent well-formed code unit
sequences
as being part of those ill-formed code unit sequences
(3) and recommend a "best practice" of "maximal valid sub-part" for
replacement
The new UTF-8 charset implementation we put in JDK7 (and back-ported to
previous
release since then) follows the new constraints in most cases, except
(1) The decoder still accepts "historical" 3 bytes surrogates and 6
bytes surrogate
pair (the encoder never outputs such sequence). Unicode Standard
"tightened" its
UTF-8 definition in ver 3.2 [3], as
"Most notable among the corrigenda to the Standard is a further
tightening
of the definition of UTF-8, to eliminate irregular UTF-8 and to
bring the
Unicode specification of UTF-8 more completely into line with other
specifications of UTF-8."
So the 3-byte/6-byte surrogates are now defined as "ill-formed" code unit
sequence, instead of "irregular" [5] in ver 3.1
(2) While no longer accepting the "historical" 5-byte, 6-byte UTF-8 byte
sequence,
the decoder treats these 5/6-byte sequence as ONE malformed unit. As a
result
these bytes get replaced by one replacement character, when "replace for
malformed" is desirable (as in new String(bytes), for example).
According the
latest Unicode standard [2], however, because the leading byte of these
5/6-byte
sequence is no longer an illegal appearance of the UTF-8, these bytes
should be
treated as 5-6 individual ill-formed bytes.
(3)Corner case like ill-formed byte sequence ED 31 is not handled correctly/
consistently, as described in #7082884 [6]
The reason behind (1) and (2) is mostly the compatibility concern. As
suggested
in TR#26 [4] (in which it defines CESU-8, a separate UTF encoding scheme
that
uses 3-6-byte sequence for supplementary characters, instead of 4-byte
sequence
in UTF-8), there are apps/data over there that do use surrogates pair in
"UTF-8"
form. To change the UTF-8 charset to follow standard obviously will break
someone's code when they migrate/upgrade from JDK/JRE N to N+1, something
we try really hard to avoid.
That said, given almost decade has passed and we are now at Unicode 6, I
think
the possibility of breaking someone's code/date of upgrading UTF-8 to do the
"right thing" is small/minor. So I proposed here
(1) to upgrade the JDK8 UTF-8 implementation to strictly follow the
standard to
a) reject 3-byte surrogate/6-byte surrogate pair
b) treats 5/6-byte surrogate as individual ill-formed bytes.
c) fix the corner case bug #7082884
(2) to add CESU-8 charset into JDK/JRE's charset repository (for those still
prefer/work on 3-6 bytes surrogate, in "UTF-8" form)
Here is the webrev. The change will need to go through some "in-compatible
change" review process, but I think we can start the code review/discussion
here first.
http://cr.openjdk.java.net/~sherman/7096080/webrev/
<http://cr.openjdk.java.net/%7Esherman/7096080/webrev/>
-Sherman
[1] http://www.unicode.org/versions/Unicode5.1.0/#Notable_Changes
[2] http://www.unicode.org/versions/Unicode6.0.0/#Conformance_Changes
[3] http://www.unicode.org/reports/tr28/tr28-3.html
[4] http://unicode.org/reports/tr26/
[5] http://unicode.org/versions/corrigendum1.html
[6]
http://mail.openjdk.java.net/pipermail/core-libs-dev/2011-September/007722.html
