Hi
This is a long standing "regression" from 1.3.1 on how
OutputStreamWriter.flush()/flushBuffer()
handles escape or shift sequence in some of the charset/encoding, for
example the ISO-2022-JP.
ISO-2022-JP is encoding that starts with ASCII mode and then switches
between ASCII andJapanese
characters through an escape sequence. For example, the escape sequence ESC
$ B (0x1B, 0x24 0x42)
is used to indicate the following bytes are Japanese (switch from ASCII
mode to Japanese mode), and
the ESC ( B (0x1b 0x28 0x42) is used to switch back to ASCII.
In Java's sun.io.CharToByteConvert (old generation charset converter) and
the nio.io.charset.CharsetEncoder
usually switches back forth between ASCII and Japanese modes based on the
input character sequence
(for example, if you are in ASCII mode, and your next input character is a
Japanese, you add the
ESC $ B into the output first and then followed the converted input
character, or if you are in Japanese
mode and your next input is ASCII, you output ESC ( B first to switch the
mode and then the ASCII) and
switch back to ASCII mode (if the last mode is non-Japanese) if either the
encoding is ending or the
flush() method gets invoked.
In JDK1.3.1, OutputStreamWriter.flushBuffer() explicitly invokes
sun.io.c2b's flushAny() to switch
back to ASCII mode every time the flush() or flushBuffer() (from
PrintStream) gets invoked, as
showed at the end of this email. For example, as showed below, the code
uses OutputStreamWriter
to "write" a Japanese character \u6700 to the underlying stream with
iso-2022jp,
ByteArrayOutputStream bos = new ByteArrayOutputStream();
String str = "\u6700";
OutputStreamWriter osw = new OutputStreamWriter(bos, "iso-2022-jp");
osw.write(str, 0, str.length());
Since the iso-2022-jp starts with ASCII mode, we now have a Japanese, so we
need to
switch into Japanese mode first (the first 3 bytes) and then the encoded
Japanese
character (the following 2 bytes)
0x1b 0x24 0x42 0x3a 0x47
and then the code invokes
osw.flush();
since we are now in Japanese, the writer continues to write out
0x1b 0x28 0x 42
to switch back to ASCII mode. The total output is 8 bytes after write() and
flush().
However, when all encoidng/charset related codes were migrated from 1.3.1's
sun.io based to
1.4's java.nio.charset based implementation (1.4, 1.4.1 and 1.4.2, we
gradually migrated from
sun.io to java.nio.charset), the "c2b.flushAny()" invocation obviously was
dropped in
sun.nio.cs.StreamEncoder. It results in that the "switch back to ASCII
mode" sequence is no longer
output when OutputStreamWriter.flush() or PrintStream.write(String) is
invoked.
This does not trigger problem for most use scenario, if the "stream"
finally gets closed
(in which the StreamEncoder does invoke encoder's flush() to output the
escape sequence
to switch back to ASCII) or PrintStream.println(String) is used (in which
it outputs a \n character,
since this \n is in ASCII range, it "accidentally " switches the mode back
to ASCII).
But it obviously causes problem when you can't not close the
OutputStreamWriter after
you're done your iso2022-jp writing (for example, you need continue to use
the underlying
OutputStream for other writing, but not "this" osw), for 1.3.1, these apps
invoke osw.flush()
to force the output switch back to ASCII, this no longer works when we
switch to java.nio.charset
in jdk1.4.2. (we migrated iso-2022-jp to nio.charset in 1.4.2). This is
what happened in JavaMail,
as described in the bug report.
The solution is to re-store the "flush the encoder" mechanism in
StreamEncoder's flushBuffer().
I have been hesitated to make this change for a while, mostly because this
regressed behavior
has been their for 3 releases, and the change triggers yet another
"behavior change". But given
there is no obvious workaround and it only changes the behavior of the
charsets with this
shift in/out mechanism, mainly the iso-2022 family and those IBM
EBCDIC_DBCS charsets, I
decided to give it a try.
Here is the webreview
http://cr.openjdk.java.net/~sherman/6995537/webrev
Sherman
---------------------------------1.3.1
OutputStreamWriter-----------------------
/**
* Flush the output buffer to the underlying byte stream, without flushing
* the byte stream itself. This method is non-private only so that it may
* be invoked by PrintStream.
*/
void flushBuffer() throws IOException {
synchronized (lock) {
ensureOpen();
for (;;) {
try {
nextByte += ctb.flushAny(bb, nextByte, nBytes);
}
catch (ConversionBufferFullException x) {
nextByte = ctb.nextByteIndex();
}
if (nextByte == 0)
break;
if (nextByte > 0) {
out.write(bb, 0, nextByte);
nextByte = 0;
}
}
}
}
/**
* Flush the stream.
*
* @exception IOException If an I/O error occurs
*/
public void flush() throws IOException {
synchronized (lock) {
flushBuffer();
out.flush();
}
}
