Geir,
I've also tried the same changes suggested by Michael with our test
templates (Russian/ISO-8859-5, Japanese/Shift_JIS, Chinese/GBK,
Hebrew/ISO-8859-8 plus the same ones with UTF-8) and the modified ASCII
stream worked well. ParserTokenManager required the "UNICODE_INPUT=true"
option, as noted by Michael, otherwise it didn't accept any non-zero high
order bytes.
As Velocity lets InputStreamReader to perform the encoding, the characters
returned by it are always correct 16 bit Unicode (if the given encoding
corresponds to the one used when writing the templates). It should be safe
to use them as they are without modifications performed by the stream. The
ASCII stream without stripping is a suitable kind of raw stream for that.
I didn't manage to make the USER_CHAR_STREAM option to work as it didn't
produce a working parser (JavaCC 1.1), but a Parser.java class with missing
methods. Also, the reuse of a customized stream wasn't supported
automatically. If you have solved those problems, I think that this solution
is more reliable than the previous one. Actually, I haven't yet understood
why the original ASCII stream works at all with 16 bit encodings even though
the high-order bytes are lost?
-- Ilkka
----- Original Message -----
From: Geir Magnusson Jr. <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Sent: Thursday, May 24, 2001 12:41
Subject: Re: Velocity v1.1-rc1 released
> Ilkka Priha wrote:
> >
> > It's great that Michael has detected the problem and even solved it.
We've
> > been also testing some simple Chinese templates and didn't notice it.
> >
> > Just for motivation Geir, there's not any other even near as good tool
for
> > multilingual web application development as Velocity after the problems
are
> > solved. So it would be cool to have this one already in Velocity
1.1/Turbine
> > 2.1. I don't mean to put any pressure, reliability comes always first.
> >
> > But it would be cool :-)
> >
>
> Ok. I think I am pretty satisifed - Michael's change works fine - I
> have a 'custom' VelocityCharStream which is ASCII_CharStream with the
> changes.
>
> When things get back to normal (prollie friday) I will commit into the
> HEAD CVS, and we can all beat on it for a bit.
>
> If everyone is satisfied, we should discuss putting into 1.1 - if people
> aren't comfortable (and I am not yet fully, btw) then we just follow 1.1
> with a 1.2 soon after.
>
> geir
>
> --
> Geir Magnusson Jr. [EMAIL PROTECTED]
> System and Software Consulting
> Developing for the web? See http://jakarta.apache.org/velocity/
> "still climbing up to the shoulders..."
