[
https://issues.apache.org/jira/browse/AVRO-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14166625#comment-14166625
]
Hatem Helal commented on AVRO-1593:
-----------------------------------
JsonGenerator::doEncodeString relies on calling "iscntrl" to determine whether
a given byte is a control character. In the windows-1252 code page the byte
"\x81" is a control character but not in the C locale which breaks the UTF-8
byte sequence.
A workaround is to temporarily set the LC_CTYPE locale to "C", do the json
encoding, and then reset the LC_CTYPE back to its previous value but modifying
the global locale settings isn't thread safe. A better approach would be to
use the newer "std::iscntrl" function which accepts a locale object. A patch
is available here:
http://git.io/XetN-w
> C++ json encoder assumes "C" locale and generates invalid UTF-8 sequence
> -------------------------------------------------------------------------
>
> Key: AVRO-1593
> URL: https://issues.apache.org/jira/browse/AVRO-1593
> Project: Avro
> Issue Type: Bug
> Components: c++
> Environment: windows-1252 encoding
> Reporter: Hatem Helal
> Priority: Critical
>
> encoding a multibyte UTF-8 code point such as:
> "\xEF\xBD\x81"
> Incorrectly becomes:
> "\xEF\xBD\U0081"
> When encoded in the service running in the windows-1252 locale. This isn¹t a
> valid UTF-8 sequence so we end up with Mojibake when reading back the JSON
> encoded string.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
