GitHub user zellerh opened a pull request:
https://github.com/apache/incubator-trafodion/pull/986
[TRAFODION-2477] Invalid characters in translation are ignored
Right now we ignore such invalid characters and also may truncate
the string at the point of the invalid char. The expected behavior
would be an error.
The only type of invalid data I could create with regular SQL syntax
is an invalid UTF-16 surrogate pair. We have no checks that detect
those when we enter the data. Invalid UTF-8, on the other hand, is
rejected when we try to insert it in the database (at least in the
case I tried).
The fix adds a check to generate an error (file conversionLocale.cpp).
It also adds two CQDs to suppress the error (remaining code files)
and to replace the invalid character with a replacement character.
Right now we use "?", even for Unicode, which has a special replacement
character, see TRAFODION-2515.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zellerh/incubator-trafodion bug/cses_jan-17
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-trafodion/pull/986.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #986
----
commit 079b2107bc6c4475192d21deba9cac3f1f6687dd
Author: Hans Zeller <[email protected]>
Date: 2017-03-01T22:33:44Z
[TRAFODION-2477] Invalid characters in translation are ignored
Right now we ignore such invalid characters and also may truncate
the string at the point of the invalid char. The expected behavior
would be an error.
The only type of invalid data I could create with regular SQL syntax
is an invalid UTF-16 surrogate pair. We have no checks that detect
those when we enter the data. Invalid UTF-8, on the other hand, is
rejected when we try to insert it in the database (at least in the
case I tried).
The fix adds a check to generate an error (file conversionLocale.cpp).
It also adds two CQDs to suppress the error (remaining code files)
and to replace the invalid character with a replacement character.
Right now we use "?", even for Unicode, which has a special replacement
character, see TRAFODION-2515.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
