[issue23215] MemoryError with custom error handlers and multibyte codecs
Roundup Robot added the comment: New changeset af8089217cc6 by Serhiy Storchaka in branch '2.7': Issue #23215: Multibyte codecs with custom error handlers that ignores errors https://hg.python.org/cpython/rev/af8089217cc6 New changeset 4dc8b7ed8973 by Serhiy Storchaka in branch '3.4': Issue #23215: Multibyte codecs with custom error handlers that ignores errors https://hg.python.org/cpython/rev/4dc8b7ed8973 New changeset 5620691ce26b by Serhiy Storchaka in branch 'default': Issue #23215: Multibyte codecs with custom error handlers that ignores errors https://hg.python.org/cpython/rev/5620691ce26b -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23215 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23215] MemoryError with custom error handlers and multibyte codecs
Changes by Serhiy Storchaka storch...@gmail.com: -- resolution: - fixed stage: patch review - resolved status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23215 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23215] MemoryError with custom error handlers and multibyte codecs
Serhiy Storchaka added the comment: Thank you for your patch Aleksi. It LGTM in general. Updated patch just moves the test to Lib/test/test_multibytecodec.py where it can reuse ALL_CJKENCODINGS and fixes few other minor bugs in multibyte codecs. -- Added file: http://bugs.python.org/file38150/python_codec_crash_fix_2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23215 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23215] MemoryError with custom error handlers and multibyte codecs
Changes by Serhiy Storchaka storch...@gmail.com: -- assignee: - serhiy.storchaka ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23215 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23215] MemoryError with custom error handlers and multibyte codecs
Roundup Robot added the comment: New changeset 3a9b1e5fe179 by R David Murray in branch '3.4': #23215: reflow paragraph. https://hg.python.org/cpython/rev/3a9b1e5fe179 New changeset 52a06812d5da by R David Murray in branch 'default': Merge: #23215: note that time.sleep affects the current thread only. https://hg.python.org/cpython/rev/52a06812d5da -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23215 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23215] MemoryError with custom error handlers and multibyte codecs
R. David Murray added the comment: Oops, typoed the issue number. That should have been 23251. -- nosy: +r.david.murray ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23215 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23215] MemoryError with custom error handlers and multibyte codecs
Changes by Aleksi Torhamo alexerion+pythonb...@gmail.com: -- keywords: +patch Added file: http://bugs.python.org/file37660/python_codec_crash_fix.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23215 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23215] MemoryError with custom error handlers and multibyte codecs
Changes by Serhiy Storchaka storch...@gmail.com: -- nosy: +haypo, lemburg, loewis, serhiy.storchaka stage: - patch review versions: -Python 3.2, Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23215 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23215] MemoryError with custom error handlers and multibyte codecs
New submission from Aleksi Torhamo: Using a multibyte codec and a custom error handler that ignores errors to encode a string that contains characters not representable in said encoding causes exponential growth of the output buffer, raising MemoryError. The problem is in multibytecodec_encerror() and REQUIRE_ENCODEBUFFER() in Modules/cjkcodecs/multibytecodec.c. multibytecodec_encerror() always uses REQUIRE_ENCODEBUFFER() to ensure there's enough space for the replacement string, and if more space is needed, REQUIRE_ENCODEBUFFER() calls expand_encodebuffer(), which in turn always grows the buffer by at least 50%. However, if size 1, REQUIRE_ENCODEBUFFER() doesn't check if more space is actually needed. (It's used with negative values in other places) I have no idea why the condition was originally size 1 instead of size 0, but changing it seems to fix this. The replacement string case is also the only use of the macro that may use 0 as the argument. In the patch, I've instead wrapped the REQUIRE_ENCODEBUFFER() (and memcpy) in a if(size 0), since that's what the corresponding part in multibytecodec_decerror() did in the past: https://hg.python.org/cpython/file/1c3f8d044589/Modules/cjkcodecs/multibytecodec.c#l438 Not sure which one makes more sense. As for the tests, I'm not sure if 1) all of the affected encodings should be tested or only one (or even all encodings, affected or not?) and 2) whether it should be a new test or if I should just add it to test_longstrings in Lib/test/test_codeccallbacks.py. (Structurally it's a perfect fit, but it really isn't a long string test as it can happen with 50 characters) At the moment, the patch is testing affected encodings in a separate test. Is the test philosophy as thorough as possible or as fast as possible? -- components: Interpreter Core files: python_codec_crasher.py messages: 233800 nosy: alexer priority: normal severity: normal status: open title: MemoryError with custom error handlers and multibyte codecs type: resource usage versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4, Python 3.5 Added file: http://bugs.python.org/file37659/python_codec_crasher.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23215 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
