[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

miss-islington added the comment: New changeset c9f83c173b0cc62d6fcdc363e9ab05f6664ff8f3 by Miss Islington (bot) in branch '3.9': bpo-40958: Avoid 'possible loss of data' warning on Windows (GH-20970) https://github.com/python/cpython/commit/c9f83c173b0cc62d6fcdc363e9ab05f6664ff8f3

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

miss-islington added the comment: New changeset 861efc6e8fe7f030b1e193989b13287b31385939 by Lysandros Nikolaou in branch 'master': bpo-40958: Avoid 'possible loss of data' warning on Windows (GH-20970) https://github.com/python/cpython/commit/861efc6e8fe7f030b1e193989b13287b31385939 ---

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Change by miss-islington : -- pull_requests: +20176 pull_request: https://github.com/python/cpython/pull/21001 ___ Python tracker ___ __

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Change by neonene : -- nosy: +christian.heimes -neonene ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Change by Lysandros Nikolaou : -- pull_requests: +20148 pull_request: https://github.com/python/cpython/pull/20970 ___ Python tracker ___ __

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Change by Lysandros Nikolaou : -- pull_requests: +20146 pull_request: https://github.com/python/cpython/pull/20968 ___ Python tracker ___ __

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Lysandros Nikolaou added the comment: > WIll make a PR soon No need. Already got something ready. -- ___ Python tracker ___ ___ Py

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Pablo Galindo Salgado added the comment: Hu, that is because now lineno, col_offset, end_lineno, end_col_offset are Py_ssize_t so we probably need to adapt the other assignments. WIll make a PR soon -- ___ Python tracker

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

neonene added the comment: FYI, since PR 20875/20919, msvc(x64) has warned C4244 (conversion from 'Py_ssize_t' to 'int', possible loss of data). parse.c especially gets more than 700. -- nosy: +neonene -christian.heimes, miss-islington ___ Python t

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Pablo Galindo Salgado added the comment: New changeset 7795ae8f05a5b1134a576a372f64699176cac5fb by Miss Islington (bot) in branch '3.9': bpo-40958: Avoid buffer overflow in the parser when indexing the current line (GH-20875) (GH-20919) https://github.com/python/cpython/commit/7795ae8f05a5b1

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Change by Pablo Galindo Salgado : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___ ___

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Change by miss-islington : -- nosy: +miss-islington nosy_count: 3.0 -> 4.0 pull_requests: +20098 pull_request: https://github.com/python/cpython/pull/20919 ___ Python tracker _

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Pablo Galindo Salgado added the comment: New changeset 51c5896b6205911d29ac07f167ec7f3cf1cb600d by Pablo Galindo in branch 'master': bpo-40958: Avoid buffer overflow in the parser when indexing the current line (GH-20875) https://github.com/python/cpython/commit/51c5896b6205911d29ac07f167ec7

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Change by Pablo Galindo Salgado : -- pull_requests: +20063 pull_request: https://github.com/python/cpython/pull/20875 ___ Python tracker ___ ___

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Change by Pablo Galindo Salgado : -- keywords: +patch pull_requests: +20035 stage: -> patch review pull_request: https://github.com/python/cpython/pull/20842 ___ Python tracker __

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Pablo Galindo Salgado added the comment: > This line is wrong though, since PyUnicode_GET_LENGTH returns the length in > code points Whoops! Thanks for pointing that out. In any case, the patch still shows that the ASAN error goes away by limiting the access to the string so I suppose that

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Lysandros Nikolaou added the comment: > +Py_ssize_t linesize = PyUnicode_GET_LENGTH(line); This line is wrong though, since PyUnicode_GET_LENGTH returns the length in code points and PyUnicode_DecodeUTF8 expects the number of bytes. For non-ascii input this would push the caret further to

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Pablo Galindo Salgado added the comment: Ok, I was able to reproduce: ❯ gcc --version gcc (GCC) 10.1.0 Copyright (C) 2020 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Lysandros Nikolaou added the comment: > Can you run the test suite before/after of pyflakes to make sure we don't > introduce any regression to double check? If everything looks fine, open a PR > :) The exact same errors before and after! I'll wait on the PR though till someone has checked

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Pablo Galindo Salgado added the comment: Can you run the test suite before/after of pyflakes to make sure we don't introduce any regression to double check? If everything looks fine, open a PR :) -- ___ Python tracker

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Lysandros Nikolaou added the comment: I'm guessing that some parts of the conversion code were only there to circumvent issues in displaying the error messages that weren't pegen's fault. These were fixed by Guido in GH-20072, so I think we can delete some of them. For example, this patch wo

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Pablo Galindo Salgado added the comment: Well, I think the solution here is to clip the col_offset to the line length and maybe adding assert that the difference is not bigger than 1. -- ___ Python tracker

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Lysandros Nikolaou added the comment: > What happens with: >>> "Ṕýţĥòñ" + Exact same thing. The offset is 16 at the start and gets decreased to 15 in the line I linked to in my previous post. And then col_offset gets converted to col_number which is 9, which seems correct. Although it is co

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Pablo Galindo Salgado added the comment: > returns an object that is one character shorter than the col_offset and > that's how we get to the situation you mentioned. What happens with: >>> "Ṕýţĥòñ" + -- ___ Python tracker

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Lysandros Nikolaou added the comment: > Note that although we could just exit if the length of the line is smaller > than the column offset before calling > https://github.com/python/cpython/blob/master/Parser/pegen.c#L148 (I assume > that is the problem) is more important to understand how

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Pablo Galindo Salgado added the comment: > Is the only way to get gcc-10.1 to build from source? Because currently my > internet connection is too slow to get all of gcc downloaded and `apt install > gcc-10` installs 10.0.1, which I'm getting totally unrelated erros with. If your package man

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Lysandros Nikolaou added the comment: Is the only way to get gcc-10.1 to build from source? Because currently my internet connection is too slow to get all of gcc downloaded and `apt install gcc-10` installs 10.0.1, which I'm getting totally unrelated erros with. --

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Pablo Galindo Salgado added the comment: Oh, I was confused by this: > Running Pablo's examples in the REPL does not reproduce the errors either: I thought you meant that you could not reproduce the crash also with my patch. For the ASAN, you may need the newer gcc and other stuff. I was no

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Lysandros Nikolaou added the comment: > Lysandros, are you using a debug build? The assert() won't trigger if you > don't have run configure with --with-pydebug. BTW I'm not talking about the assert not triggering. I'm only saying that ASAN/UBSAN do not report an error when running `yield fr

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Lysandros Nikolaou added the comment: I now configured Python with ./configure --with-pydebug --with-address-sanitizer --with-undefined-behavior-sanitizer. -- ___ Python tracker

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Lysandros Nikolaou added the comment: Let me try that. I'm currently using gcc 9.3. -- ___ Python tracker ___ ___ Python-bugs-list

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Pablo Galindo Salgado added the comment: Lysandros, are you using a debug build? The assert() won't trigger if you don't have run configure with --with-pydebug. I would recommend to: make distclean && ./configure --with-pydebug && make -- ___ Pyth

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Christian Heimes added the comment: Did you clean and compile with asan/ubsan? Older compilers may not detect the issue. My system has GCC 10: libubsan-10.1.1-1.fc32.x86_64 gcc-10.1.1-1.fc32.x86_64 libasan-10.1.1-1.fc32.x86_64 -- ___ Python tracker

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Lysandros Nikolaou added the comment: Running Pablo's examples in the REPL does not reproduce the errors either: ➜ cpython git:(master) ✗ LSAN_OPTIONS="suppressions=asan-suppression.txt,print_suppressions=0" ./python Modules/posixmodule.c:14682:9: runtime error: left shif

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Lysandros Nikolaou added the comment: I cannot reproduce this on my Ubuntu 20.04: ➜ cpython git:(master) ✗ LSAN_OPTIONS="suppressions=asan-suppression.txt,print_suppressions=0" ./python -m test -v test_eof Modules/posixmodule.c:14682:9: runtime error: left shift of 34 by 26 places cannot be

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Christian Heimes added the comment: I'm using an ASAN UBSAN build of Python: $ make clean $ ./configure --with-address-sanitizer --with-undefined-behavior-sanitizer $ cat > asan-suppression.txt << EOF # Python/initconfig.c leak:_PyWideStringList_Copy EOF $ LSAN_OPTIONS="suppressions=asan-suppr

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Pablo Galindo Salgado added the comment: I theorize that this happens when the error points to the end of the line. I assume that the difference between col_offset and PyUnicode_GET_LENGTH(line) may be still bigger than 1 if the line contains some Unicode stuff like with: >> "Ṕýţĥòñ" + but I

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Pablo Galindo Salgado added the comment: Correcting my previous post (deleted): You can get an assert failure without ASAN with this patch: diff --git a/Parser/pegen.c b/Parser/pegen.c index e29910bf86..65fa44921f 100644 --- a/Parser/pegen.c +++ b/Parser/pegen.c @@ -145,6 +145,9 @@ byte_offse

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Pablo Galindo Salgado added the comment: You can get an assert failure without ASAN with this patch: diff --git a/Parser/pegen.c b/Parser/pegen.c index e29910bf86..65fa44921f 100644 --- a/Parser/pegen.c +++ b/Parser/pegen.c @@ -145,6 +145,9 @@ byte_offset_to_character_offset(PyObject *line, in

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Change by Pablo Galindo Salgado : -- Removed message: https://bugs.python.org/msg371364 ___ Python tracker ___ ___ Python-bugs-list

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Pablo Galindo Salgado added the comment: Seems that just this (at top level): >> yield from is able to produce a situation where the column offset is bigger than the line length: Col offset is 11 and line is 10 -- ___ Python tracker

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Lysandros Nikolaou added the comment: Agreed. For that I'd probably need the input Christian is getting the error with. I'm currently trying things out, but haven't gotten the error yet. -- ___ Python tracker _

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Pablo Galindo Salgado added the comment: Note that although we could just exit if the length of the line is smaller than the column offset before calling https://github.com/python/cpython/blob/master/Parser/pegen.c#L148 (I assume that is the problem) is more important to understand how are w

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Lysandros Nikolaou added the comment: > Lysandros, could you take a look? Yup, I'm on it. -- ___ Python tracker ___ ___ Python-bug

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Pablo Galindo Salgado added the comment: Christian, what test are you running to get the error? -- ___ Python tracker ___ ___ Pytho

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Pablo Galindo Salgado added the comment: Lysandros, could you take a look? -- ___ Python tracker ___ ___ Python-bugs-list mailing l

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

Change by Pablo Galindo Salgado : -- nosy: +lys.nikolaou ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http

[issue40958] ASAN/UBSAN: heap-buffer-overflow in pegen.c

New submission from Christian Heimes : ASAN/UBSAN has detected a heap-buffer-overflow in pegen.c ==1625693==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60626b71 at pc 0x0073574d bp 0x7fff297284f0 sp 0x7fff297284e0 READ of size 1 at 0x60626b71 thread T0 #0 0x73574