Edit report at https://bugs.php.net/bug.php?id=62010&edit=1
ID: 62010 Comment by: masakielastic at gmail dot com Reported by: tklingenberg at lastflood dot net Summary: json_decode produces invalid byte-sequences Status: Open Type: Bug Package: JSON related Operating System: Windows PHP Version: 5.3.13 Block user comment: N Private report: N New Comment: Here is RFC 3629's description about UTF-8 definition. The definition of UTF-8 prohibits encoding character numbers between U+D800 and U+DFFF, which are reserved for use with the UTF-16 encoding form (as surrogate pairs) and do not directly represent characters. http://tools.ietf.org/html/rfc3629 The following patch solve the part of problem, The isolated low surrogate pairs(U+DC00 U+DFFF) are replaced with U+FFFD, The imrovement for high surrogate pairs (U+D800 - U+DBFF) is needed. https://gist.github.com/masakielastic/5985383 var_dump( "\xef\xbf\xbd" === json_decode('"\udc00"'), "\xef\xbf\xbd"."\xed\xa0\x80" === json_decode('"\ud800\ud800"'), "\xed\xa0\x80" === json_decode('"\ud800"') ); The consistency for the following options (under the discussion) is needed too. json_encode's option for replacing ill-formd byte sequences with substitute characters https://bugs.php.net/bug.php?id=65082 Previous Comments: ------------------------------------------------------------------------ [2013-01-11 09:44:55] votefordevnull at gmail dot com Successfully reproduced on Linux ------------------------------------------------------------------------ [2012-05-11 22:46:34] tklingenberg at lastflood dot net Looks like that #41067 https://bugs.php.net/bug.php?id=41067 was not fully fixed. ------------------------------------------------------------------------ [2012-05-11 22:12:42] tklingenberg at lastflood dot net Description: ------------ It's a typical case the JSON *and* UTF-16 specifications warn about: decoding of non-existing UTF-16 code-points: json_decode('"\ud834"') shoud give NULL because \ud834 is *invalid*. But instead it starts some party, get's boozed and offers this as UTF-8 byte-sequence: 1110 1101 1010 0000 1011 0100 1110 xxxx 10xx xxxx 10xx xxxx 1101 1000 0011 0100 D8 34 U+D834 is not a valid unicode character. Test script: --------------- if (NULL !== json_decode('"\ud834"')) { echo "json_decode is still broken."; } Expected result: ---------------- NULL because the json is invalid. Actual result: -------------- PHP tries to create UTF-8 out of it and fails by creating invalid UTF-8 unicode byte-sequences. ------------------------------------------------------------------------ -- Edit this bug report at https://bugs.php.net/bug.php?id=62010&edit=1
