[Python-Dev] Re: The Python 2 death march
On Fri, Mar 27, 2020, at 11:49, Sumana Harihareswara wrote: > Benjamin: now that PyCon 2020 has been cancelled, are you considering > releasing 2.7.18 slightly earlier? The plan is to follow the dates in PEP 373. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/AZFKC7QND67AFYO5ATY5TKOMQV3GF37X/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: How to enable tracemalloc for the test suite?
[Skip Montanaro ]
> ...
> I thought setting PYTHONTRACEMALLOC should provoke some useful output,
> but I was confused into thinking I was (am?) still missed something
> because it continued to produce this message:
>
> Enable tracemalloc to get the memory block allocation traceback
Ah, I overlooked that.
> which suggests to me tracemalloc still isn't enabled. That's emitted
> from Modules/_tracemalloc.c and seems to be properly protected:
>
> if (!_Py_tracemalloc_config.tracing) {
> PUTS(fd, "Enable tracemalloc to get the memory block "
> "allocation traceback\n\n");
> return;
> }
>
> so I think there is still more to do.
Agreed - for whatever reason, tracemalloc is not enabled for you at
the time your run dies, despite your
> % PYTHONTRACEMALLOC=5 ./python ./Tools/scripts/run_tests.py -R
5:50:reflog.txt test_rattlesnake
command line. Have to leave that mystery for Victor!
So between the "obvious" possibilities:
- something is going nuts spraying 0 all over memory
- the pointer passed to free() wasn't actually obtained from a
malloc/realloc function
there's nothing in the output to favor one over the other. Except
that dereferencing addresses near the pointer didn't blow up (null
bytes were returned), so the pointer is at least in the process's
address space.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/74XUF5UNOID5YZ2EOUJD5YXOSNHIBEGT/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Set PyEval_SetTrace to method from class
Hello. I need to set a PyEval_SetTrace with a new class instance. All examples use simple function. PyEval_SetTrace(trace_trampoline, obj); Following my problem: I start three threads, but I need to check trace C just one thread. If I run a while on C trace on thread, it stops all threads. So, I've been thinking to run Trace with a new instance class C. Any suggestions? Att. Leandro Müller ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/CDH7BTAYPJ6QRTXI32QRJVEPKWXEPYF4/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: How to enable tracemalloc for the test suite?
Victor> I wrote the feature (both tracemalloc and query tracemalloc when a
Victor> buffer overflow is detected), so I should be able to help you ;-)
Yes, I thought you might. :-)
I've attached the output of a more complete run. The command is
% PYTHONTRACEMALLOC=5 ./python ./Tools/scripts/run_tests.py -R
5:50:reflog.txt test_rattlesnake
where test_rattlesnake.py has been cut down to a single unit test,
which presumably is the one which exercises the problematic code. I
don't get the error about writing off either end of the buffer unless
I set the second arg pretty high (it succeeds at 40, fails at 45).
I'll also quote one part of Tim's response:
Tim> To my eyes, you left out the most important part ;-) A traceback
Tim> showing who made the fatal free() call to begin with.
Which is correct as far as that goes, but I hadn't yet given up all
hope of figuring things out. I was more concerned with why I couldn't
(and still can't) get tracemalloc to sing and dance. :-)
I thought setting PYTHONTRACEMALLOC should provoke some useful output,
but I was confused into thinking I was (am?) still missed something
because it continued to produce this message:
Enable tracemalloc to get the memory block allocation traceback
which suggests to me tracemalloc still isn't enabled. That's emitted
from Modules/_tracemalloc.c and seems to be properly protected:
if (!_Py_tracemalloc_config.tracing) {
PUTS(fd, "Enable tracemalloc to get the memory block "
"allocation traceback\n\n");
return;
}
so I think there is still more to do. I was worried enough that I
might have misspelled the environment variable name that I ran again
after copying it from the documentation. No such Doh! moment for me,
though I suspect it might be coming. :-/
FWIW, the register branch of my CPython fork:
https://github.com/smontanaro/cpython/tree/register
is what I'm working with at the moment. Current master has been merged
to it. It should be exactly what is failing for me. (Note that I'm not
asking for help there, just pointing out for the curious where my
busted code is.)
Thanks for both of your responses.
Skip
typescript
Description: Binary data
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/BO5672ZXBFVFPEHWUBQMNAZDTEC6XT54/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: How to enable tracemalloc for the test suite?
[Skip Montanaro ] > I've got a memory issue in my modified Python interpreter I'm trying > to debug. Output at the end of the problematic unit test looks like this: To my eyes, you left out the most important part ;-) A traceback showing who made the fatal free() call to begin with. In debug mode, allocations are padded on both ends with various stuff: FORBIDDENBYTEs (0xfd), a count of the number of bytes originally requested, and a serial number incremented by 1 each time alloc/realloc is called. The memory you requested is entirely filled with CLEANBYTEs ()xcd). On deallocation, that stuff is checked (that's where your run is dying), and the memory is filled with DEADBYTEs (0xdd) (a weak but nevertheless often effective way to detect "double free"s). Now in your case, absolutely every byte shown is 0x00. There are no useful clues at all remaining. It's not just a short buffer overrun, _everything_ has been NULLed out, including before the requested memory. So something has gone nuts spraying 0 all over memory ... or the pointer passed to free() was nonsense to begin with. If it were a case of double free, we'd usually expect to see some DEADBYTEs in the debug output. But there weren't any. However, you did not get any output from tracemalloc, despite that it sounds like you set up the right magic to activate it. Which leans toward the "complete nonsense" hypothesis. If tracemalloc doesn't find the address in its dict of currently-allocated addresses, looks like it fails silently, simply not producing any output. Which is what you're experiencing. So where was free() called and what was passed to it? All the evidence is consistent with the idea that what was passed to free() was never obtained from a malloc or reallloc call to begin with. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/A32E5ZVJ4IZBNDHMLOXH2OOAB2CKAYJN/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: How to enable tracemalloc for the test suite?
Hi Skip, I wrote the feature (both tracemalloc and query tracemalloc when a buffer overflow is detected), so I should be able to help you ;-) Le dim. 5 avr. 2020 à 00:27, Skip Montanaro a écrit : > Looking at the tracemalloc module docs and trying various command line args > (-X tracemalloc=5) or environment variables (PYTHONTRACEMALLOC=5) Yes, that's the right way to enable it. The env var is inherited by subprocesses. > I'm unable to provoke any different output. Maybe your test runs Python with -I or -E which ignores the environment variable. Which command do do you run to run tests? It's unclear from your output which test and which process triggers the "Debug memory block at address" bug. Maybe you are out of lock and the buffer corruption is only detected (the memory is only deallocated) after tracemalloc was disabled by _PyTraceMalloc_Fini() call in Py_FinalizeEx(). You may try to hack Python by commenting this _PyTraceMalloc_Fini() call. FYI there is an unit test on the debug hooks on memory allocator to ensure that it detects buffer overflow: test_capi.test_buffer_overflow() checks for "Debug memory block at address (...)". Example: $ cat bug.py import _testcapi _testcapi.pymem_buffer_overflow() $ python3 bug.py Segmentation fault (core dumped) $ python3 -X tracemalloc=5 -X dev bug.py Debug memory block at address p=0x7f225fd1c100: API 'm' 16 bytes originally requested The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected. The 8 pad bytes at tail=0x7f225fd1c110 are not all FORBIDDENBYTE (0xfd): at tail+0: 0x78 *** OUCH at tail+1: 0xfd at tail+2: 0xfd at tail+3: 0xfd at tail+4: 0xfd at tail+5: 0xfd at tail+6: 0xfd at tail+7: 0xfd The block was made by call #56367 to debug malloc/realloc. Data at p: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd Memory block allocated at (most recent call first): File "bug.py", line 2 Fatal Python error: bad trailing pad byte Current thread 0x7f226cda56c0 (most recent call first): File "bug.py", line 2 in Aborted (core dumped) Tracemalloc adds "Memory block allocated at" traceback. By the way, I started to also suggest enabling tracemalloc when a ResourceWarning is logged, to see where the leaked resource was allocated. Victor ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/EQD3RJOKL5H2IF53V7PA44KCO324OOWR/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] How to enable tracemalloc for the test suite?
I've got a memory issue in my modified Python interpreter I'm trying to debug. Output at the end of the problematic unit test looks like this: ... == Tests result: FAILURE then SUCCESS == 1 test OK. 1 re-run test: test_rattlesnake Total duration: 2.9 sec Tests result: FAILURE then SUCCESS Debug memory block at address p=0x55a227969080: API '' 0 bytes originally requested The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd): at p-7: 0x00 *** OUCH at p-6: 0x00 *** OUCH at p-5: 0x00 *** OUCH at p-4: 0x00 *** OUCH at p-3: 0x00 *** OUCH at p-2: 0x00 *** OUCH at p-1: 0x00 *** OUCH Because memory is corrupted at the start, the count of bytes requested may be bogus, and checking the trailing pad bytes may segfault. The 8 pad bytes at tail=0x55a227969080 are not all FORBIDDENBYTE (0xfd): at tail+0: 0x00 *** OUCH at tail+1: 0x00 *** OUCH at tail+2: 0x00 *** OUCH at tail+3: 0x00 *** OUCH at tail+4: 0x00 *** OUCH at tail+5: 0x00 *** OUCH at tail+6: 0x00 *** OUCH at tail+7: 0x00 *** OUCH The block was made by call #0 to debug malloc/realloc. Enable tracemalloc to get the memory block allocation traceback Looking at the tracemalloc module docs and trying various command line args (-X tracemalloc=5) or environment variables (PYTHONTRACEMALLOC=5) I'm unable to provoke any different output. The module docs give a tantalizing suggestion: Example of output of the Python test suite: but I see no command line args related to running the test suite with tracemalloc enabled. Pointers appreciated. Skip ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4WNQ5EERCQST4AIL6HLV4GDGM5LAVV6Q/ Code of Conduct: http://python.org/psf/codeofconduct/
