I have managed to
knock up a self contained example which I have attached,
an example string is Descripci�n -
although you will need to
have XML::Simple installed.
The example takes an
input string and then prints it twice - one with concatenation another just
displaying the inputted string. The mangling occurs when you concatenate an XML
string with a CGI string.
I'm not sure why
this happens but here is a first attempt at a possible theory. All XML parsing
is done in UTF8, but perl has no idea of encodings for incomding CGI streams and
assumes them to be iso-88591 (latin1) - I read this somewhere don't know if
its correct. String operations upgrade none UTF8 strings to UTF8, so perl tries
to convert the CGI string from iso-88591 to UTF8 thus mangling it as its already
UTF8.
Ivan - thank your for your example example, I think it
shows the same issue as mine. I'm not sure how your fix would help with mine as
the concatonation happens at compiled template stage - and we would have to
change template toolkit to work with sprintf which I expect is not desirable. Is
there some way to tag an incoming value as UTF8 so that it doesn't get mangled
when it is upgraded during the concatenation?
Barry - I'm still trying to digest what you said, I'm
about to start reading through the unicode site you linked too. How does this
issue relate to the two example from Ivan and myself?
Thanks
Mark
testUTF8.pl
Description: Binary data
