[issue11489] json.dumps not parsable by json.loads (on Linux only)

Changes by Serhiy Storchaka : -- resolution: -> fixed stage: -> committed/rejected status: open -> closed ___ Python tracker ___ ___

[issue11489] json.dumps not parsable by json.loads (on Linux only)

Roundup Robot added the comment: New changeset 02d186e3af09 by Serhiy Storchaka in branch '2.7': Fixed JSON tests on wide build when ran from *.pyc files (issue #11489). http://hg.python.org/cpython/rev/02d186e3af09 -- ___ Python tracker

[issue11489] json.dumps not parsable by json.loads (on Linux only)

Arfrever Frehtes Taifersar Arahesis added the comment: test_json_surrogates.patch fixes these tests. -- ___ Python tracker ___ ___ Pyt

[issue11489] json.dumps not parsable by json.loads (on Linux only)

Serhiy Storchaka added the comment: Thank you Arfrever. Does this patch fix the test? -- Added file: http://bugs.python.org/file32922/test_json_surrogates.patch ___ Python tracker __

[issue11489] json.dumps not parsable by json.loads (on Linux only)

Arfrever Frehtes Taifersar Arahesis added the comment: ... when code is loaded from .pyc files (i.e. when `make test` runs tests the second time). -- ___ Python tracker ___

[issue11489] json.dumps not parsable by json.loads (on Linux only)

Arfrever Frehtes Taifersar Arahesis added the comment: New tests fail on 2.7 branch, at least with Python configured with --enable-unicode=ucs4 (which is default in Gentoo): == FAIL: test_surrogates (json.tests.test_scanstring.

[issue11489] json.dumps not parsable by json.loads (on Linux only)

Changes by Serhiy Storchaka : -- resolution: -> fixed stage: patch review -> committed/rejected status: open -> closed ___ Python tracker ___ ___

[issue11489] json.dumps not parsable by json.loads (on Linux only)

Roundup Robot added the comment: New changeset c85305a54e6d by Serhiy Storchaka in branch '2.7': Issue #11489: JSON decoder now accepts lone surrogates. http://hg.python.org/cpython/rev/c85305a54e6d New changeset 8abbdbe86c01 by Serhiy Storchaka in branch '3.3': Issue #11489: JSON decoder now ac

[issue11489] json.dumps not parsable by json.loads (on Linux only)

Serhiy Storchaka added the comment: I there are no objections I'll commit this patch soon. -- assignee: -> serhiy.storchaka Added file: http://bugs.python.org/file32718/json_decode_lone_surrogates_3-3.4.patch ___ Python tracker

[issue11489] json.dumps not parsable by json.loads (on Linux only)

Taras Prokopenko added the comment: You should use ensure_ascii=False option to json.dumps, ie import json unicode_bytes = '\xed\xa8\x80' unicode_string = unicode_bytes.decode("utf8") json_encoded = json.dumps(unicode_string, ensure_ascii=False) json.loads(json_encoded),unicode_string (u'\uda00

[issue11489] json.dumps not parsable by json.loads (on Linux only)

Changes by Raymond Hettinger : -- assignee: rhettinger -> ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http

[issue11489] json.dumps not parsable by json.loads (on Linux only)

Changes by Serhiy Storchaka : -- components: +Extension Modules stage: needs patch -> patch review ___ Python tracker ___ ___ Python-b

[issue11489] json.dumps not parsable by json.loads (on Linux only)

Changes by Serhiy Storchaka : Added file: http://bugs.python.org/file30235/json_decode_lone_surrogates_2-2.7.patch ___ Python tracker ___ ___

[issue11489] json.dumps not parsable by json.loads (on Linux only)

Serhiy Storchaka added the comment: Here are updated patches from issue17906. Updated tests, fixed a bug reported by Bob Ippolito in msg188857 and fixed inconsistency noted by Ezio Melotti on Rietveld (Python implementation now raises same exception as C implementation on illegal hexadecimal e

[issue11489] json.dumps not parsable by json.loads (on Linux only)

Serhiy Storchaka added the comment: I forgot about this issue and open a new issue17906. There is a patch for it. Simplejson has accepted it in https://github.com/simplejson/simplejson/issues/62. RFC 4627 does not make exceptions for the range 0xD800-0xDFFF (unescaped = %x20-21 / %x23-5B / %x

[issue11489] json.dumps not parsable by json.loads (on Linux only)

Serhiy Storchaka added the comment: About patch. I think "with" is unnecessary here. One-line self.assertRaises(UnicodeEncodeError, self.dumps, ch) looks better for me. -- stage: -> needs patch ___ Python tracker

[issue11489] json.dumps not parsable by json.loads (on Linux only)

Ezio Melotti added the comment: Attached failing test. -- keywords: +patch Added file: http://bugs.python.org/file27369/issue11489.diff ___ Python tracker ___ ___

[issue11489] json.dumps not parsable by json.loads (on Linux only)

Serhiy Storchaka added the comment: > It's Unicode that considers unpaired surrogates invalid, not UTF-8 by itself. It's UTF-8 too. See RFC 3629: The definition of UTF-8 prohibits encoding character numbers between U+D800 and U+DFFF, which are reserved for use with the UTF-16 encoding

[issue11489] json.dumps not parsable by json.loads (on Linux only)

Petri Lehtinen added the comment: Bear in mind that Douglas Crockford thinks a JSON document is valid even if it contains unpaired surrogates: http://tech.groups.yahoo.com/group/json/message/1603 http://tech.groups.yahoo.com/group/json/message/1583 It's Unicode that considers unpaired

[issue11489] json.dumps not parsable by json.loads (on Linux only)

Changes by Raymond Hettinger : -- assignee: -> rhettinger priority: high -> normal ___ Python tracker ___ ___ Python-bugs-list mailin

[issue11489] json.dumps not parsable by json.loads (on Linux only)

Changes by Ezio Melotti : -- nosy: +pitrou, tchrist versions: -Python 2.6 ___ Python tracker ___ ___ Python-bugs-list mailing list Un

[issue11489] json.dumps not parsable by json.loads (on Linux only)

Ezio Melotti added the comment: RFC 4627 doesn't say much about lone surrogates: A string is a sequence of zero or more Unicode characters [UNICODE]. [...] All Unicode characters may be placed within the quotation marks except for the characters that must be escaped: quotation ma

[issue11489] json.dumps not parsable by json.loads (on Linux only)

STINNER Victor added the comment: print(repr(json.loads(json.dumps({u"my_key": u'\uda00'}))['my_key'])): - displays u'\uda00' in Python 2.7, 3.2 and 3.3 - raises a ValueError('Invalid \u escape: ...') on loads() in Python 2.6 - raises a ValueError('Unpaired high surrogate: ...') on loads(

[issue11489] json.dumps not parsable by json.loads (on Linux only)

Changes by Ezio Melotti : -- nosy: +ezio.melotti ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.py

[issue11489] json.dumps not parsable by json.loads (on Linux only)

Changes by Ezio Melotti : Removed file: http://bugs.python.org/file21135/unnamed ___ Python tracker ___ ___ Python-bugs-list mailing list Unsu

[issue11489] json.dumps not parsable by json.loads (on Linux only)

Brian added the comment: On Mon, Mar 14, 2011 at 4:09 PM, Raymond Hettinger wrote: > > Raymond Hettinger added the comment: > > > We seem to be in the worst of both worlds right now > > as I've generated and stored a lot of json that can > > not be read back in > > This is unfortunate. The du

[issue11489] json.dumps not parsable by json.loads (on Linux only)

Raymond Hettinger added the comment: > We seem to be in the worst of both worlds right now > as I've generated and stored a lot of json that can > not be read back in This is unfortunate. The dumps() should have never worked in the first place. I don't think that loads() should be changed t

[issue11489] json.dumps not parsable by json.loads (on Linux only)

Brian Merrell added the comment: >I am not sure this should be fixed in 2.x. Lone surrogates seem to >round-trip >just fine in 2.x and there likely to be existing code that >relies on this. I generally agree but am then at a loss as to how to detect and deal with lone surrogates(eg "ignore",

[issue11489] json.dumps not parsable by json.loads (on Linux only)

Alexander Belopolsky added the comment: > It appears this is an invalid unicode character. > Shouldn't this be caught by decode("utf8") It should and it is in Python 3.x: >>> b'\xed\xa8\x80'.decode("utf8") Traceback (most recent call last): File "", line 1, in UnicodeDecodeError: 'utf8' cod

[issue11489] json.dumps not parsable by json.loads (on Linux only)

New submission from Brian Merrell : The following works on Win7x64 Python 2.6.5 and breaks on Ubuntu 10.04x64-2.6.5. This raises three issues: 1) Shouldn't anything generated by json.dumps be parsed by json.loads? 2) It appears this is an invalid unicode character. Shouldn't this be caught