[issue33186] Memory corruption with urllib.parse
Ivan Zakharyaschev <i...@altlinux.org> added the comment: Actually, Alexey shared this problem with me orally first, and then asked to have a look at his report, and I felt that just describing the technical details about what is going on is not enough, and suggested to include a brief sentence which would state what kind of assumptions seemed to be broken to Alexey. And it was: > a list contains elements that have never been appended. And this statement actually appeared quite helpful for me: in a few hours after this statement was made, I got an idea that this statement might actually be not true in this case. And this quickly lead me to the demonstration what really happens here, and that it was correct Python behavior. Without this brief formal statement of the broken assumptions, I had simply had no ideas (because I supposed that something was going wrong according to Alexey's words, and didn't question whether it is really wrong or we might expect these results under some conditions as correct results). (Actually, I'm a fan of type-checking, and what is more, of languages with expressive types like Agda or Idris where the programmer is to make provable statements of the expected properties written down as types; sometimes, they can be derived automatically, but nevertheless my point is the attitude to programming, even without formal tools: think how the expected properties would be proved, how they would be decomposed into simpler statements during the proof. Well, this is offtopic here, so, Alexey, let's not continue this discussion here.) -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33186> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33186] Memory corruption with urllib.parse
Ivan Zakharyaschev <i...@altlinux.org> added the comment: > 1. The described result can be valid if there are "doubly quoted" strings in > the input list. To check this I've modified your testcase.py: diff --git a/testcase.py b/testcase.py index b597205..fefcef2 100755 --- a/testcase.py +++ b/testcase.py @@ -53,6 +53,11 @@ for entry in input_list: # Conveting conv_entry=urllib.parse.unquote(entry) + if conv_entry in bad_boys: + flag=True + print("ABOUT TO APPEND A BAD BOY") + print("Conversion:",entry,id(entry),end=" ") + # Printing (debug) if flag: print(">>",conv_entry,id(conv_entry)) @@ -61,7 +66,7 @@ for entry in input_list: output_list.append(conv_entry) # Printing (debug) - if entry in bad_boys: + if flag: print("Just appended:", output_list[-1], id(output_list[-1])) flag=False for x in output_list: Let's run it and look for my new message: $ ./testcase.py | fgrep -A1 -e ABOUT ABOUT TO APPEND A BAD BOY Conversion: eil%252fx2_firmware 139920242273280 >> eil%2fx2_firmware 139920244465360 -- ABOUT TO APPEND A BAD BOY Conversion: seil%252fb1_firmware 139920242325448 >> seil%2fb1_firmware 139920241899608 -- ABOUT TO APPEND A BAD BOY Conversion: seil%252fx1_firmware 139920242325520 >> seil%2fx1_firmware 139920241899752 -- ABOUT TO APPEND A BAD BOY Conversion: seil%252fx2_firmware 139920242325592 >> seil%2fx2_firmware 139920241899896 -- ABOUT TO APPEND A BAD BOY Conversion: seil%252fx86_firmware 139920242356704 >> seil%2fx86_firmware 139920241901408 -- ABOUT TO APPEND A BAD BOY Conversion: seil%252fturbo_firmware 139920242416280 >> seil%2fturbo_firmware 139920242993744 -- ABOUT TO APPEND A BAD BOY Conversion: seil%252fneu_2fe_plus_firmware 139920242065856 >> seil%2fneu_2fe_plus_firmware 139920243007536 -- ABOUT TO APPEND A BAD BOY Conversion: intelligent_platforms_proficy_hmi%252fscada_cimplicity 139920241710560 >> intelligent_platforms_proficy_hmi%2fscada_cimplicity 139920244342480 $ fgrep -ne 'seil%252fneu_2fe_plus_firmware' data.txt 15191:seil%252fneu_2fe_plus_firmware $ So, there is indeed a "doubly" quoted data, at 15191 in data.txt. This is not a bug, everything work correctly. This issue needs to be closed. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33186> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33186] Memory corruption with urllib.parse
Ivan Zakharyaschev <i...@altlinux.org> added the comment: > a list contains elements that have never been appended Why do we think that it's true? 1. The described result can be valid if there are "doubly quoted" strings in the input list. 2. Also, there could be a bug in unquote() which doesn't corrupt the list as a side effect, but simply returns a wrong result value, which is indeed appended to the output list at its turn. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33186> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33186] Memory corruption with urllib.parse
Ivan Zakharyaschev <i...@altlinux.org> added the comment: The bad results are also reproducible with: # dpkg -l python3.2 Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version ArchitectureDescription +++-=-===-===-=== ii python3.2 3.2.3-7 ia64Interactive high-level object-oriented language (version 3.2) -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33186> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33186] Memory corruption with urllib.parse
Ivan Zakharyaschev <i...@altlinux.org> added the comment: I've tested this on a e2k machine (e2k is a VLIW/EPIC architecture, similar to ia64), where python3 3.5 has been compiled with a completely different proprietary compiler, and got the same bad result. python3-3.5.1-alt7.3 (This has also been tested with the same results on an Ubuntu machine.) So, this probably means that this is not a matter of wrong compiler optimizations. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33186> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33186] Memory corruption with urllib.parse
Change by Ivan Zakharyaschev <i...@altlinux.org>: -- nosy: +imz ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33186> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33153] interpreter crash when multiplying large tuples
Ivan Zakharyaschev <i...@altlinux.org> added the comment: It was run in i586 chroot on x86_64. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33153> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33153] interpreter crash when multiplying large tuples
Ivan Zakharyaschev <i...@altlinux.org> added the comment: The traceback: [builder@localhost ~]$ python -c 'x = ("a", "b") * 2**20; x *= 2**20' Segmentation fault (core dumped) [builder@localhost ~]$ gdb python core.23284 GNU gdb (GDB) 7.9-alt4 (ALT) Copyright (C) 2015 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i586-alt-linux". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from python...Reading symbols from /usr/lib/debug/usr/bin/python2.7.debug...done. done. warning: core file may not match specified executable file. [New LWP 23284] Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: generic error Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: generic error warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: generic error Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: generic error warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. Core was generated by `python -c x = ("a", "b") * 2**20; x *= 2**20'. Program terminated with signal SIGSEGV, Segmentation fault. #0 tuplerepeat (a=0xf694301c, n=1048576) at Objects/tupleobject.c:503 503 *p = items[j]; (gdb) bt #0 tuplerepeat (a=0xf694301c, n=1048576) at Objects/tupleobject.c:503 #1 0xf7546d47 in sequence_repeat (n=0x80a83ac, seq=0xf694301c, repeatfunc=0xf75b24a0 ) at Objects/abstract.c:1210 #2 PyNumber_InPlaceMultiply (v=0xf694301c, w=0x80a83ac) at Objects/abstract.c:1374 #3 0xf7602ff8 in PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:1653 #4 0xf7608fee in PyEval_EvalCodeEx (co=0xf71a8ba8, globals=0xf7240714, locals=0xf7240714, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3589 #5 0xf760916e in PyEval_EvalCode (co=0xf71a8ba8, globals=0xf7240714, locals=0xf7240714) at Python/ceval.c:669 #6 0xf762c508 in run_mod (arena=0x8068320, flags=0xffa1a82c, locals=0xf7240714, globals=0xf7240714, filename=0xf765fec9 "", mod=) at Python/pythonrun.c:1376 #7 PyRun_StringFlags (str=0x804b160 "x = (\"a\", \"b\") * 2**20; x *= 2**20\n", start=257, globals=0xf7240714, locals=0xf7240714, flags=0xffa1a82c) at Python/pythonrun.c:1339 #8 0xf762e160 in PyRun_SimpleStringFlags (command=0x804b160 "x = (\"a\", \"b\") * 2**20; x *= 2**20\n", flags=0xffa1a82c) at Python/pythonrun.c:974 #9 0xf764498e in Py_Main (argc=, argv=) at Modules/main.c:589 #10 0x080484f7 in main (argc=3, argv=0xffa1a974) at Modules/python.c:20 (gdb) quit [builder@localhost ~]$ It was built like this -- http://git.altlinux.org/tasks/archive/done/_188/193020/build/100/i586/log : i586-alt-linux-gcc -pthread -c -fno-strict-aliasing -pipe -Wall -g -O3 -march=i586 -mtune=generic -DNDEBUG -pipe -Wall -g -O3 -march=i586 -mtune=generic -I. -IInclude -I./Include -fPIC -DPy_BUILD_CORE -o Objects/tupleobject.o Objects/tupleobject.c The same happens with gcc7-7.3.1-alt3 here -- http://git.altlinux.org/tasks/202598/build/100/i586/log . -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33153> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33153] interpreter crash when multiplying large tuples
New submission from Ivan Zakharyaschev <i...@altlinux.org>: The issue https://bugs.python.org/msg314475 has arisen for tuples (but not for lists, as in the example there) in 2.7.14 for me. How should we fix it in a better way? This bug is not reproducible in python 3.5.4. [builder@localhost ~]$ python Python 2.7.14 (default, Nov 7 2017, 17:07:17) [GCC 6.3.1 20170118 (ALT 6.3.1-alt2)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> x = [0] * 2**20 >>> x *= 2**20 Traceback (most recent call last): File "", line 1, in MemoryError >>> x = [0,0,0,0,0,0] * 2**20 >>> x *= 2**20 Traceback (most recent call last): File "", line 1, in MemoryError >>> x = ('a', 'b') >>> x = ('a', 'b') * 2**20 >>> x *= 2**20 Segmentation fault [builder@localhost ~]$ python --version Python 2.7.14 [builder@localhost ~]$ python Python 2.7.14 (default, Nov 7 2017, 17:07:17) [GCC 6.3.1 20170118 (ALT 6.3.1-alt2)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.maxsize 2147483647 >>> sys.maxint 2147483647 >>> [builder@localhost ~]$ python RPM/BUILD/Python-2.7.14/Lib/test/test_tuple.py test_addmul (__main__.TupleTest) ... ok test_bigrepeat (__main__.TupleTest) ... Segmentation fault [builder@localhost ~]$ -- components: Interpreter Core messages: 314508 nosy: imz priority: normal severity: normal status: open title: interpreter crash when multiplying large tuples type: crash versions: Python 2.7 ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33153> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1704621] interpreter crash when multiplying large lists
Ivan Zakharyaschev <i...@altlinux.org> added the comment: New issue filed: https://bugs.python.org/issue33153 -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue1704621> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33153] interpreter crash when multiplying large tuples
Ivan Zakharyaschev <i...@altlinux.org> added the comment: I meant the old issue https://bugs.python.org/issue1704621 . -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33153> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1704621] interpreter crash when multiplying large lists
Ivan Zakharyaschev <i...@altlinux.org> added the comment: Hi! This is broken for tuples (but not for lists, as in the example here) in 2.7.14 for me. Should we reopen this issue and fix it better? [builder@localhost ~]$ python Python 2.7.14 (default, Nov 7 2017, 17:07:17) [GCC 6.3.1 20170118 (ALT 6.3.1-alt2)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> x = [0] * 2**20 >>> x *= 2**20 Traceback (most recent call last): File "", line 1, in MemoryError >>> x = [0,0,0,0,0,0] * 2**20 >>> x *= 2**20 Traceback (most recent call last): File "", line 1, in MemoryError >>> x = ('a', 'b') >>> x = ('a', 'b') * 2**20 >>> x *= 2**20 Segmentation fault [builder@localhost ~]$ python --version Python 2.7.14 [builder@localhost ~]$ python Python 2.7.14 (default, Nov 7 2017, 17:07:17) [GCC 6.3.1 20170118 (ALT 6.3.1-alt2)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.maxsize 2147483647 >>> sys.maxint 2147483647 >>> [builder@localhost ~]$ python RPM/BUILD/Python-2.7.14/Lib/test/test_tuple.py test_addmul (__main__.TupleTest) ... ok test_bigrepeat (__main__.TupleTest) ... Segmentation fault [builder@localhost ~]$ -- nosy: +imz ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue1704621> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6083] Reference counting bug in PyArg_ParseTuple and PyArg_ParseTupleAndKeywords
Ivan Zakharyaschev <i...@altlinux.org> added the comment: And will the next call be effective (do anything), if we have already set the limit with the testing call? -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue6083> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6083] Reference counting bug in PyArg_ParseTuple and PyArg_ParseTupleAndKeywords
Ivan Zakharyaschev <i...@altlinux.org> added the comment: Thanks! I also thought about this simplest way. What about this: diff --git a/Python/Lib/test/test_resource.py b/Python/Lib/test/test_resource.py index de29d3b..bec4440 100644 --- a/Python/Lib/test/test_resource.py +++ b/Python/Lib/test/test_resource.py @@ -102,16 +102,21 @@ class ResourceTest(unittest.TestCase): # Issue 6083: Reference counting bug def test_setrusage_refcount(self): +howmany = 100 try: limits = resource.getrlimit(resource.RLIMIT_CPU) except AttributeError: self.skipTest('RLIMIT_CPU not available') +try: +resource.setrlimit(resource.RLIMIT_CPU, (howmany, howmany)) +except _: +self.skipTest('Setting RLIMIT_CPU not possible') class BadSequence: def __len__(self): return 2 def __getitem__(self, key): if key in (0, 1): -return len(tuple(range(100))) +return len(tuple(range(howmany))) raise IndexError resource.setrlimit(resource.RLIMIT_CPU, BadSequence()) What should I write instead of _? -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue6083> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6083] Reference counting bug in PyArg_ParseTuple and PyArg_ParseTupleAndKeywords
Ivan Zakharyaschev <i...@altlinux.org> added the comment: >>> import resource >>> resource.getrlimit(resource.RLIMIT_CPU) (7200, 7260) -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue6083> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6083] Reference counting bug in PyArg_ParseTuple and PyArg_ParseTupleAndKeywords
Ivan Zakharyaschev <i...@altlinux.org> added the comment: > New changeset a4c85f9b8f58 by Serhiy Storchaka in branch '2.7': Issue #6083: Fix multiple segmentation faults occured when PyArg_ParseTuple http://hg.python.org/cpython/rev/a4c85f9b8f58 This test has a problem: though it tests not the ability to set a CPU hard limit, it fails if the hard limit is limited. Perhaps, ignore any exception there? Could you please help me re-write it correctly, so that I can run it on gyle--ALT's builder host--successfully): # Issue 6083: Reference counting bug def test_setrusage_refcount(self): try: limits = resource.getrlimit(resource.RLIMIT_CPU) except AttributeError: self.skipTest('RLIMIT_CPU not available') class BadSequence: def __len__(self): return 2 def __getitem__(self, key): if key in (0, 1): return len(tuple(range(100))) raise IndexError resource.setrlimit(resource.RLIMIT_CPU, BadSequence()) The failure: [builder@team ~]$ python /usr/lib64/python2.7/test/test_resource.py test_args (__main__.ResourceTest) ... ok test_fsize_enforced (__main__.ResourceTest) ... ok test_fsize_ismax (__main__.ResourceTest) ... ok test_fsize_toobig (__main__.ResourceTest) ... ok test_getrusage (__main__.ResourceTest) ... ok test_setrusage_refcount (__main__.ResourceTest) ... ERROR == ERROR: test_setrusage_refcount (__main__.ResourceTest) -- Traceback (most recent call last): File "/usr/lib64/python2.7/test/test_resource.py", line 117, in test_setrusage_refcount resource.setrlimit(resource.RLIMIT_CPU, BadSequence()) ValueError: not allowed to raise maximum limit -- Ran 6 tests in 0.085s FAILED (errors=1) Traceback (most recent call last): File "/usr/lib64/python2.7/test/test_resource.py", line 123, in test_main() File "/usr/lib64/python2.7/test/test_resource.py", line 120, in test_main test_support.run_unittest(ResourceTest) File "/usr/lib64/python2.7/test/support/__init__.py", line 1577, in run_unittest _run_suite(suite) File "/usr/lib64/python2.7/test/support/__init__.py", line 1542, in _run_suite raise TestFailed(err) test.support.TestFailed: Traceback (most recent call last): File "/usr/lib64/python2.7/test/test_resource.py", line 117, in test_setrusage_refcount resource.setrlimit(resource.RLIMIT_CPU, BadSequence()) ValueError: not allowed to raise maximum limit [builder@team ~]$ -- nosy: +imz ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue6083> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26912] test/test_email/torture_test.py (and test_asian_codecs.py) has a broken import
New submission from Ivan Zakharyaschev: (An issue similar to http://bugs.python.org/issue26911 , but with a different source file from the tests) Lib/test/test_email/torture_test.py:15:from email.test.test_email import TestEmailBase should be: from test.test_email import TestEmailBase because the first variant has a non-existing path. (This makes the auto-requirements generator in ALT Sisyphus to generate a broken/unsatisfiable dependency on email.test.test_email when building the package from http://git.altlinux.org/people/imz/packages/python3.git .) Similarly, Lib/test/test_email/test_asian_codecs.py:8:from test.test_email.test_email import TestEmailBase should better not have an extra "test_email" component in the path. All other imports of test_email in Python 3.5.1 sources are of the form I'm suggesting: $ git --no-pager grep -Fn .test_email Lib/test/test_email/__init__.py:9:from test.test_email import __file__ as landmark Lib/test/test_email/__main__.py:1:from test.test_email import load_tests Lib/test/test_email/test__encoded_words.py:4:from test.test_email import TestEmailBase Lib/test/test_email/test__header_value_parser.py:6:from test.test_email import TestEmailBase, parameterize Lib/test/test_email/test_asian_codecs.py:8:from test.test_email.test_email import TestEmailBase Lib/test/test_email/test_contentmanager.py:2:from test.test_email import TestEmailBase, parameterize Lib/test/test_email/test_defect_handling.py:6:from test.test_email import TestEmailBase Lib/test/test_email/test_email.py:42:from test.test_email import openfile, TestEmailBase Lib/test/test_email/test_generator.py:8:from test.test_email import TestEmailBase, parameterize Lib/test/test_email/test_headerregistry.py:8:from test.test_email import TestEmailBase, parameterize Lib/test/test_email/test_inversion.py:11:from test.test_email import TestEmailBase, parameterize Lib/test/test_email/test_message.py:5:from test.test_email import TestEmailBase, parameterize Lib/test/test_email/test_parser.py:5:from test.test_email import TestEmailBase Lib/test/test_email/test_pickleable.py:9:from test.test_email import TestEmailBase, parameterize Lib/test/test_email/torture_test.py:15:from email.test.test_email import TestEmailBase Misc/HISTORY:1733:- Issue #8315: (partial fix) python -m unittest test.test_email now works. BTW, I'm not sure these two files are used anywhere, because grep gives no results for their names. -- components: Tests messages: 264667 nosy: imz priority: normal severity: normal status: open title: test/test_email/torture_test.py (and test_asian_codecs.py) has a broken import type: compile error versions: Python 3.5 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26912> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com