[issue11763] assertEqual memory issues with large text inputs
Ezio Melotti added the comment: The second problem I mentioned in msg134530 got fixed in #18996, making the patches on this issue obsolete. Since the other problem is now tracked in #19217, I'm going to close this. -- stage: patch review - resolved status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11763 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11763] assertEqual memory issues with large text inputs
Ezio Melotti added the comment: Attached an updated patch that addresses Serhiy comments. The tests in the previous patch have been committed already, so this might need new tests. The first problem I mentioned in msg134530 has been reported in #19217. -- Added file: http://bugs.python.org/file32104/issue11763-3.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11763 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11763] assertEqual memory issues with large text inputs
Ezio Melotti added the comment: There are some tests in the issue11763-2.diff patch. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11763 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11763] assertEqual memory issues with large text inputs
Serhiy Storchaka added the comment: I left comments on Rietveld. Tests needed. -- nosy: +serhiy.storchaka versions: +Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11763 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11763] assertEqual memory issues with large text inputs
Sara Magliacane sara.magliac...@gmail.com added the comment: Here is a small patch addressing the second point of the last message, but the tests are still missing. -- nosy: +sara.magliacane Added file: http://bugs.python.org/file22462/issue11763_safe_repr.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11763 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11763] assertEqual memory issues with large text inputs
Michael Foord mich...@voidspace.org.uk added the comment: The basic idea of the patch is good, but instead of introducing _MAX_LENGTH, maxDiff should be reused. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11763 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11763] assertEqual memory issues with large text inputs
Michael Foord mich...@voidspace.org.uk added the comment: Sorry, ignore that. I see that the patch already passes maxDiff to truncate_str. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11763 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11763] assertEqual memory issues with large text inputs
Roundup Robot devnull@devnull added the comment: New changeset 8dbf661c0a63 by Ezio Melotti in branch '2.7': #11763: don't use difflib in TestCase.assertMultiLineEqual if the strings are too long. http://hg.python.org/cpython/rev/8dbf661c0a63 New changeset 04e64f77c6c7 by Ezio Melotti in branch '3.1': #11763: don't use difflib in TestCase.assertMultiLineEqual if the strings are too long. http://hg.python.org/cpython/rev/04e64f77c6c7 New changeset b316019638df by Ezio Melotti in branch '3.2': #11763: merge with 3.1. http://hg.python.org/cpython/rev/b316019638df New changeset df154c872b0c by Ezio Melotti in branch 'default': #11763: merge with 3.2. http://hg.python.org/cpython/rev/df154c872b0c -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11763 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11763] assertEqual memory issues with large text inputs
Ezio Melotti ezio.melo...@gmail.com added the comment: I committed a slightly modified version of the patch, so the issue should be fixed now. There are two related problems though: 1) difflib is used in other places, so those should probably be checked too; 2) _baseAssertEqual should check if the len of the msg (or the sum of the lengths of the two args) is greater than maxDiff and use safe_repr(arg, short=True) to avoid long diffs. -- resolution: - fixed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11763 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11763] assertEqual memory issues with large text inputs
New submission from Michael Foord mich...@voidspace.org.uk: s = x * (2**29) case.assertEqual(s + a, s + b) Traceback (most recent call last): File stdin, line 1, in module File /home/antoine/cpython/default/Lib/unittest/case.py, line 643, in assertEqual assertion_func(first, second, msg=msg) File /home/antoine/cpython/default/Lib/unittest/case.py, line 984, in assertMultiLineEqual secondlines = [second + '\n'] MemoryError assertEqual delegates to assertMultilineEqual for comparing text which uses difflib for comparisons. This has performance issues (as well as memory issues) for very large inputs, so should fallback to a simple comparison (or simpler diff generation technique) for very large inputs. -- assignee: michael.foord messages: 132965 nosy: ezio.melotti, michael.foord, pitrou priority: normal severity: normal status: open title: assertEqual memory issues with large text inputs versions: Python 2.7, Python 3.2, Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11763 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11763] assertEqual memory issues with large text inputs
Ezio Melotti ezio.melo...@gmail.com added the comment: Attached patch adds a _diffThreshold attribute of 2**16 and uses _baseAssertEqual whenever one of the two string is longer than 2**16 chars. -- keywords: +patch Added file: http://bugs.python.org/file21535/issue11763.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11763 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11763] assertEqual memory issues with large text inputs
Antoine Pitrou pit...@free.fr added the comment: Rather than hardwiring `self.addCleanup(lambda: setattr(self, '_diffThreshold', 2**16))`, you should retrieve the previous value. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11763 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11763] assertEqual memory issues with large text inputs
Changes by Ezio Melotti ezio.melo...@gmail.com: -- assignee: michael.foord - ezio.melotti components: +Library (Lib) nosy: +rhettinger stage: - patch review type: - behavior Added file: http://bugs.python.org/file21536/issue11763-2.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11763 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
