[issue43145] Leak of locks from multiprocessing.Process
Boris Staletic added the comment:
The `multiprocessing.Process`, on Linux, ends up doing something like this:
pid = os.fork()
if pid == 0: os._exit()
Translated to C:
int main() {
Py_Initialize();
PyOS_BeforeFork();
pid_t pid = fork();
if (pid == 0) {
PyOS_AfterFork_Child(); // Reinitializes stuff.
_exit(0); // Child process exits without cleanup.
}
PyOS_AfterFork_Parent();
Py_Finalize();
}
The call to `_exit()` happens in Lib/multiprocessing/popen_fork.py#L73
My attempts at cleaning this up resulted in even more problems.
--
___
Python tracker
<https://bugs.python.org/issue43145>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43145] Leak of locks in a subprocess
Boris Staletic added the comment:
Slightly simpler C example:
#include
int main()
{
Py_Initialize();
PyObject* multiprocessing = PyImport_ImportModule("multiprocessing");
PyObject* Process = PyObject_GetAttrString(multiprocessing, "Process");
PyObject* p = PyObject_CallNoArgs(Process);
PyObject* start = PyObject_GetAttrString(p, "start");
PyObject* join = PyObject_GetAttrString(p, "join");
Py_DECREF(PyObject_CallNoArgs(start));
Py_DECREF(PyObject_CallNoArgs(join));
Py_DECREF(join);
Py_DECREF(start);
Py_DECREF(p);
Py_DECREF(Process);
Py_DECREF(multiprocessing);
Py_Finalize();
}
--
___
Python tracker
<https://bugs.python.org/issue43145>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43145] Leak of locks in a subprocess
New submission from Boris Staletic :
The following C code leaks 7 locks allocated with PyThread_allocate_lock:
#include
int main()
{
Py_Initialize();
PyObject* multiprocessing = PyImport_ImportModule("multiprocessing");
PyObject* Process = PyObject_GetAttrString(multiprocessing, "Process");
PyObject* args = PyTuple_New(0);
PyObject* kw = PyDict_New();
PyDict_SetItemString(kw, "target", Process);
PyObject* p = PyObject_Call(Process, args, kw);
PyObject* start = PyObject_GetAttrString(p, "start");
PyObject* join = PyObject_GetAttrString(p, "join");
PyObject_CallNoArgs(start);
PyObject_CallNoArgs(join);
Py_DECREF(join);
Py_DECREF(start);
Py_DECREF(p);
Py_DECREF(kw);
Py_DECREF(args);
Py_DECREF(Process);
Py_DECREF(multiprocessing);
Py_Finalize();
}
The following locks are leaked:
1.
https://github.com/python/cpython/blob/196d4deaf4810a0bba75ba537dd40f2d71a5a634/Python/pystate.c#L78
2.
https://github.com/python/cpython/blob/196d4deaf4810a0bba75ba537dd40f2d71a5a634/Python/pystate.c#L84
3.
https://github.com/python/cpython/blob/196d4deaf4810a0bba75ba537dd40f2d71a5a634/Python/pystate.c#L90
4. https://github.com/python/cpython/blob/master/Python/ceval.c#L810
5. https://github.com/python/cpython/blob/master/Python/import.c#L126
6. and 7.
https://github.com/python/cpython/blob/master/Modules/_threadmodule.c#L597
In the attachment is valgrind's output.
--
components: C API
files: log
messages: 386558
nosy: bstaletic
priority: normal
severity: normal
status: open
title: Leak of locks in a subprocess
type: resource usage
versions: Python 3.10, Python 3.9
Added file: https://bugs.python.org/file49793/log
___
Python tracker
<https://bugs.python.org/issue43145>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42961] Use-after-free (of a heap type) during finalization
Boris Staletic added the comment:
Oops... I uploaded (and pasted) the wrong file. The /correct/ example can be
found here:
https://github.com/pybind/pybind11/pull/2797/#pullrequestreview-570541151
However, I have just realized that the example doesn't really need the embedded
module. The following also shows the use-after-free:
#include
static void pybind11_object_dealloc(PyObject *self) {
auto type = Py_TYPE(self);
type->tp_free(self);
Py_DECREF(type);
}
static PyType_Slot base_slots[] = {{Py_tp_dealloc,
(void*)pybind11_object_dealloc}, {0, nullptr}};
static PyType_Spec base_spec{"B", sizeof(PyObject), 0, Py_TPFLAGS_BASETYPE |
Py_TPFLAGS_HEAPTYPE, base_slots};
int main() {
Py_InitializeEx(1);
auto base_type = PyType_FromSpec(_spec);
auto globals = PyDict_New();
PyDict_SetItemString(globals, "B", base_type);
auto derived_t = PyRun_String("def f():\n"
" class C:\n"
"class D(B):pass\n"
"b=D()\n"
"f()", Py_file_input, globals, nullptr);
Py_DECREF(globals);
Py_DECREF(derived_t);
Py_Finalize();
}
--
___
Python tracker
<https://bugs.python.org/issue42961>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42961] Use-after-free (of a heap type) during finalization
[issue42176] Valgrind reports "Conditional jump or move depends on uninitialised value(s)" in `PyUnicode_AsEncodedString` and `PyUnicode_Decode`
Boris Staletic added the comment: Thanks for looking into this. > Looks like a bug in valgrind. That actually explains why I wasn't able to reproduce this problem on my local machine. Ubuntu 20.04 comes with valgrind 3.15.0, while my local machine has 3.16.1. Upgrading valgrind on Ubuntu 20.04 does fix the issue. This is good enough for me and I guess this can be closed as "not a bug". -- ___ Python tracker <https://bugs.python.org/issue42176> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42176] Valgrind reports "Conditional jump or move depends on uninitialised value(s)" in `PyUnicode_AsEncodedString` and `PyUnicode_Decode`
Boris Staletic added the comment: I can also reproduce the same problem with the ubuntu packaged python3, which is 3.8.5 on Ubuntu 20.04. The only problem is that, with a stripped library, you don't get line numbers in valgrind's output. Steps to repro: 1. apt install valgrind gcc python3-config 2. Save the same attached file from the first comment as test.c. 3. gcc $(python3-config --includes) $(python3-config --ldflags) -lpython3.8 -o python-error 4. PYTHONMALLOC=malloc valgrind ./python-error Valgrind output: ==1200== Conditional jump or move depends on uninitialised value(s) ==1200==at 0x4A7B37B: PyUnicode_Decode (in /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0) ==1200==by 0x109264: main (in /python-error) ==1200== ==1200== Conditional jump or move depends on uninitialised value(s) ==1200==at 0x4A7AE57: PyUnicode_AsEncodedString (in /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0) ==1200==by 0x109280: main (in /python-error) I have not checked earlier versions of python. -- versions: +Python 3.8 ___ Python tracker <https://bugs.python.org/issue42176> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42176] Valgrind reports "Conditional jump or move depends on uninitialised value(s)" in `PyUnicode_AsEncodedString` and `PyUnicode_Decode`
New submission from Boris Staletic : When running valgrind on a C code that calls `PyUnicode_AsEncodedString` and `PyUnicode_Decode`, valgrind reports that there's a conditional jump based on uninitialized variable, if the encoding is "latin1". I am able to replicate the error 100% of the time, on Ubuntu 20.04, with python 3.9.0 installed with pyenv. I also have repro'd the error in my CI (link below). Steps to repro: 1. docker run -it ubuntu:20.04 /bin/bash 2. apt update 3. apt install valgrind gcc build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev python-openssl git 4. curl https://pyenv.run | bash 5. export PATH="/root/.pyenv/bin:$PATH" 6. eval "$(pyenv init -)" 7. PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.9.0 8. Take the attached C file. 9. gcc -ggdb3 -I/root/.pyenv/versions/3.9.0/include/python3.9 -L/root/.pyenv/versions/3.9.0/lib test2.c -lpython3.9 10. LD_LIBRARY_PATH=/root/.pyenv/versions/3.9.0/lib/ PYTHONMALLOC=malloc valgrind ./a.out Valgrind output: ==22783== Conditional jump or move depends on uninitialised value(s) ==22783==at 0x49ABE64: PyUnicode_Decode (unicodeobject.c:3443) ==22783==by 0x49ABE64: PyUnicode_Decode (unicodeobject.c:3398) ==22783==by 0x109251: main (test2.c:5) ==22783== ==22783== Conditional jump or move depends on uninitialised value(s) ==22783==at 0x499A294: PyUnicode_AsEncodedString (unicodeobject.c:3732) ==22783==by 0x499A294: PyUnicode_AsEncodedString (unicodeobject.c:3688) ==22783==by 0x10926D: main (test2.c:6) CI log: https://dev.azure.com/borisstaletic/3ce92110-caa5-4c49-b8c3-44a433da676b/_apis/build/builds/1338/logs/6 Repository for testing the bug: https://github.com/bstaletic/ycmd/tree/python-error -- components: Interpreter Core files: test.c messages: 379790 nosy: bstaletic priority: normal severity: normal status: open title: Valgrind reports "Conditional jump or move depends on uninitialised value(s)" in `PyUnicode_AsEncodedString` and `PyUnicode_Decode` type: compile error versions: Python 3.9 Added file: https://bugs.python.org/file49542/test.c ___ Python tracker <https://bugs.python.org/issue42176> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
