Martijn van Oosterhout wrote:
On Sat, Mar 17, 2007 at 11:46:01AM -0400, Andrew Dunstan wrote:
How can we fix this? Frankly, the statement in the docs warning about
making sure that escaped sequences are valid in the server encoding is a
cop-out. We don't accept invalid data elsewhere, and this should be no
different IMNSHO. I don't see why this should be any different from,
say, date or numeric data. For years people have sneered at MySQL
because it accepted dates like Feb 31st, and rightly so. But this seems
to me to be like our own version of the same problem.
It seems to me that the easiest solution would be to forbid \x?? escape
sequences where it's greater than \x7F for UTF-8 server encodings.
Instead introduce a \u escape for specifying the unicode character
directly. Under the basic principle that any escape sequence still has
to represent a single character. The result can be multiple bytes, but
you don't have to check for consistancy anymore.
Have a nice day,
The escape processing is actually done in the lexer in the case of
literals. We have to allow for bytea literals there too, regardless of
encoding. The lexer naturally has no notion of the intended destination
of the literal, So we need to defer the validity check to the *in
functions for encoding-aware types. And it as Tom has noted, COPY does
its own escape processing but does it before the transcoding.
So ISTM that any solution other than something like I have proposed will
probably involve substantial surgery.
It does also seem from my test results that transcoding to MB charsets
(or at least to utf-8) is surprisingly expensive, and that this would be
a good place to look at optimisation possibilities. The validity tests
can also be somewhat expensive.
But correctness matters most, IMNSHO.
---------------------------(end of broadcast)---------------------------
TIP 5: don't forget to increase your free space map settings