[issue21213] Memory bomb by incorrect custom serializer to json.dumps
Changes by Ezio Melotti ezio.melo...@gmail.com: -- stage: - resolved ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21213 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21213] Memory bomb by incorrect custom serializer to json.dumps
Changes by Raymond Hettinger raymond.hettin...@gmail.com: -- resolution: - not a bug status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21213 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21213] Memory bomb by incorrect custom serializer to json.dumps
Changes by Raymond Hettinger raymond.hettin...@gmail.com: -- nosy: +bob.ippolito ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21213 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21213] Memory bomb by incorrect custom serializer to json.dumps
Bob Ippolito added the comment: I agree with ebfe. It's a case that only comes up if you're writing your own default handlers, and there's not a reasonable solution to avoid this issue. You would've gotten a RuntimeError: maximum recursion depth exceeded if it wasn't for the behavior of repr here. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21213 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21213] Memory bomb by incorrect custom serializer to json.dumps
Lukas Lueg added the comment:
The behavior is triggered in Modules/_json.c:encoder_listencode_obj(). It
actually has nothing to do with the TypeError itself, any object that produces
a new string representation of itself will do.
The function encoder_listencode_obj() calls the user-supplied function with the
instance to get a string, float, integer or whatever it knows to how convert to
json by itself. As the function keeps returning new instances of TypeError, the
recursion builds up. The MemoryError is ultimately triggered by the fact that
repr() keeps escaping all single quotes from the previous repr(), generating a
huge string. Also see repr(repr(repr(')))
Testing with 2gb of ram and no swap (disable to to prevent starvation instead
of immediate crash!), cpython dies within 34 recursion levels. The
obj-parameter for encoder_listencode_obj() looks like Foo(obj='Foo \'Foo
\\\'Foo \\\'Foo Foo \\\'object object at
0x77f52100\\\'\\\'\\\'\'').
My two cents: This is expected behavior. The json-module has no way to tell in
advance if the encoding-function never returns. The fact that repr() causes
this blowup here can't be fixed.
--
nosy: +ebfe
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21213
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21213] Memory bomb by incorrect custom serializer to json.dumps
saaj added the comment: Well, as far as I see the question here is whether it makes sense to allow the default function to return JSON-incompatible objects. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21213 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21213] Memory bomb by incorrect custom serializer to json.dumps
Lukas Lueg added the comment: It's perfectly fine for the function to return an object that can't be put directly into a json string. The function may not convert the object directly but in multiple steps; the encoder will call the function again with the new object until everything boils down to a str, an integer etc.. If one keeps returning objects that never converge to one of those basic types, the interpreter faces death by infinite recursion. The situation described here adds the oom condition caused by repr() blowing up. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21213 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21213] Memory bomb by incorrect custom serializer to json.dumps
saaj added the comment: I'll try to be more specific at my point. There're two cases: 1. Scalar: NoneType, int, bool, float, str. Ended immediately. 2. Non-scalar: list/tuple, dict. Recursively traversed, which may result in subsequent calls to the custom function. If the return value is restricted to given types (what the encoder is capable on its own), it is harder to shoot oneself in the foot. In other words what's the point of returning arbitrary Python object from the function? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21213 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21213] Memory bomb by incorrect custom serializer to json.dumps
Changes by Ezio Melotti ezio.melo...@gmail.com: -- nosy: +ezio.melotti, pitrou, rhettinger type: - behavior versions: +Python 3.4, Python 3.5 -Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21213 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21213] Memory bomb by incorrect custom serializer to json.dumps
New submission from saaj: I was chaning an implementation of the function that is passed to json.dumps to extend serializable types. By a mistake (**return** instead of **raise**) it turned into, which at its minum can be expressed as:: def d(obj): return TypeError(repr(obj)) json.dumps(1j, default = d) After a few moments by laptop froze, though after a minute I could open shell in separate session, and top command showed that python interpretter is consuming about 4GiB of memory and 50% of 4 logical cores. Worst about it it doesn't end with any exception, it just keeps running. Without ``repr`` it ends up with somewhat expected ``RuntimeError: maximum recursion depth exceeded while getting the str of an object``. The same behaviour is on python3, where it just consumes memory with less speed. OS: Linux Mint 15 Olivia Linux 3.8.0-31-generic #46-Ubuntu SMP Tue Sep 10 20:03:44 UTC 2013 x86_64 Packages are last available: python 2.7.4-0ubuntu1 python3 3.3.1-0ubuntu1 P.S. Sorry for confirming on console at python.org. -- components: Library (Lib) messages: 216071 nosy: saaj priority: normal severity: normal status: open title: Memory bomb by incorrect custom serializer to json.dumps versions: Python 2.7, Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21213 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
