arib06 opened a new pull request, #3973: URL: https://github.com/apache/avro/pull/3973
## What is the purpose of the change `JsonGenerator::doEncodeString` (the C++ JSON value encoder) decodes each multi-byte UTF-8 sequence of a string value and re-escapes it, but it never checks for code points in the UTF-16 surrogate range (U+D800..U+DFFF). Surrogates are ill-formed in UTF-8 and have no JSON representation. The binary decoder reads string bytes without validating UTF-8, so a string value carrying WTF-8/CESU-8 surrogate bytes (for example `ED A0 80` for U+D800) is serialized as a lone `\ud800`, which is invalid JSON. The JSON decoder already rejects lone surrogates (#3841); this makes the encoder consistent by throwing on a decoded surrogate code point. Valid UTF-8 never contains surrogate encodings, so well-formed values are unaffected. ## Verifying this change This change added tests and can be verified as follows: - Added `testEncodeSurrogate` in `JsonTests.cc`, which parses a string holding raw surrogate bytes (U+D800 and U+DFFF) and asserts that serializing it throws instead of emitting a lone `\u` surrogate. Without the fix the encoder produces `"\ud800"` / `"\udfff"`; with it the full `JsonTests` suite passes. ## Documentation - Does this pull request introduce a new feature? no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
