[issue26387] Race condition in sqlite module

[Filipp Andjelo]

New submission from Filipp Andjelo:

Race condition in sqlite close/dealloc crashes the application with double 
free(). The pointer is set to NULL outside of mutexed zone, so if close and 
dealloc follow each other very shortly application crashes. Please see the 
attached patch.

--
components: Library (Lib)
files: sqlite_connection.c.patch
keywords: patch
messages: 260472
nosy: scorp
priority: normal
severity: normal
status: open
title: Race condition in sqlite module
type: crash
versions: Python 3.2, Python 3.3, Python 3.4, Python 3.5, Python 3.6
Added file: http://bugs.python.org/file41958/sqlite_connection.c.patch

___
Python tracker 
<http://bugs.python.org/issue26387>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26387] Race condition in sqlite module

[Filipp Andjelo]

Filipp Andjelo added the comment:

Hm, this is strange, because close() can only be called from the same thread 
and dealloc() can be called only if reference count is 0, right? So why is 
there a mutex again each other anyway? And how can that be, that 
sqlite3_close() is called multiple times with the same pointer? We are getting 
a crash, which is only possible if dealloc() is called where close() is not yet 
done. Is this possible? Or do I understand something wrong? For such cases we 
also have to set database pointer to NULL in dealloc(), but it looks like the 
problem is somewhere different, the whole code looks strange for me.

--
Added file: http://bugs.python.org/file41970/sqlite_connection.c.patch

___
Python tracker 
<http://bugs.python.org/issue26387>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26387] Crash calling sqlite3_close with invalid pointer

[Filipp Andjelo]

Filipp Andjelo added the comment:

Ok, the race condition seems not to be the real problem, since the patches 
didn't help. Don't want to do any assumptions anymore, but it seems like the 
python objects memory is deallocated before the sqlite connection was closed. 
The thing is, that this problem appears only on very heavy load with 40 
parallel python processes and overall memory consumption of more than 240GiB. 
This high number is due to deactivated GC on our side, due to performance 
problems, we call it once the complete work is done. Unfortunately, I can't 
reproduce this error in "small", but here is the crash report (the crash comes 
from different parts of free(), sometimes with double free, sometimes invalid 
pointer or similar):

*** glibc detected *** 
/share/nds-buildtools/python/python-3.4.2-64_5/bin/python: free(): invalid next 
size (normal): 0x058b72b0 ***
=== Backtrace: =
/lib64/libc.so.6[0x36e6275f4e]
/lib64/libc.so.6[0x36e6278cf0]
/share/nds-buildtools/python/python-3.4.2-64_5/lib/python3.4/lib-dynload/_sqlite3.cpython-34m.so(+0x23918)[0x7f785100a918]
/share/nds-buildtools/python/python-3.4.2-64_5/lib/python3.4/lib-dynload/_sqlite3.cpython-34m.so(sqlite3_free+0x63)[0x7f785100b6b6]
/share/nds-buildtools/python/python-3.4.2-64_5/lib/python3.4/lib-dynload/_sqlite3.cpython-34m.so(+0x3193d)[0x7f785101893d]
/share/nds-buildtools/python/python-3.4.2-64_5/lib/python3.4/lib-dynload/_sqlite3.cpython-34m.so(+0x31a93)[0x7f7851018a93]
/share/nds-buildtools/python/python-3.4.2-64_5/lib/python3.4/lib-dynload/_sqlite3.cpython-34m.so(+0x31e6d)[0x7f7851018e6d]
/share/nds-buildtools/python/python-3.4.2-64_5/lib/python3.4/lib-dynload/_sqlite3.cpython-34m.so(+0x32735)[0x7f7851019735]
/share/nds-buildtools/python/python-3.4.2-64_5/lib/python3.4/lib-dynload/_sqlite3.cpython-34m.so(+0x3115d)[0x7f785101815d]
/share/nds-buildtools/python/python-3.4.2-64_5/lib/python3.4/lib-dynload/_sqlite3.cpython-34m.so(+0x3119e)[0x7f785101819e]
/share/nds-buildtools/python/python-3.4.2-64_5/lib/python3.4/lib-dynload/_sqlite3.cpython-34m.so(+0x33df2)[0x7f785101adf2]
/share/nds-buildtools/python/python-3.4.2-64_5/lib/python3.4/lib-dynload/_sqlite3.cpython-34m.so(+0x36441)[0x7f785101d441]
/share/nds-buildtools/python/python-3.4.2-64_5/lib/python3.4/lib-dynload/_sqlite3.cpython-34m.so(+0x40e21)[0x7f7851027e21]
/share/nds-buildtools/python/python-3.4.2-64_5/lib/python3.4/lib-dynload/_sqlite3.cpython-34m.so(+0xbbddf)[0x7f78510a2ddf]
/share/nds-buildtools/python/python-3.4.2-64_5/lib/python3.4/lib-dynload/_sqlite3.cpython-34m.so(+0xbbcff)[0x7f78510a2cff]
/share/nds-buildtools/python/python-3.4.2-64_5/lib/python3.4/lib-dynload/_sqlite3.cpython-34m.so(sqlite3_close+0x1d)[0x7f78510a2d23]
/share/nds-buildtools/python/python-3.4.2-64_5/lib/python3.4/lib-dynload/_sqlite3.cpython-34m.so(pysqlite_connection_close+0x79)[0x7f7851002af9]
/share/nds-buildtools/python/python-3.4.2-64_5/bin/python(PyEval_EvalFrameEx+0x64e3)[0x4c52e3]
/share/nds-buildtools/python/python-3.4.2-64_5/bin/python(PyEval_EvalFrameEx+0x6e38)[0x4c5c38]
/share/nds-buildtools/python/python-3.4.2-64_5/bin/python(PyEval_EvalFrameEx+0x6e38)[0x4c5c38]
/share/nds-buildtools/python/python-3.4.2-64_5/bin/python(PyEval_EvalFrameEx+0x6e38)[0x4c5c38]
/share/nds-buildtools/python/python-3.4.2-64_5/bin/python(PyEval_EvalFrameEx+0x6e38)[0x4c5c38]
/share/nds-buildtools/python/python-3.4.2-64_5/bin/python(PyEval_EvalFrameEx+0x6e38)[0x4c5c38]
/share/nds-buildtools/python/python-3.4.2-64_5/bin/python(PyEval_EvalCodeEx+0x8a7)[0x4c72a7]
/share/nds-buildtools/python/python-3.4.2-64_5/bin/python(PyEval_EvalCode+0x3b)[0x4c736b]
/share/nds-buildtools/python/python-3.4.2-64_5/bin/python[0x4e7c61]
/share/nds-buildtools/python/python-3.4.2-64_5/bin/python(PyRun_FileExFlags+0x115)[0x4e9a85]
/share/nds-buildtools/python/python-3.4.2-64_5/bin/python(PyRun_SimpleFileExFlags+0xfa)[0x4eb2ea]
/share/nds-buildtools/python/python-3.4.2-64_5/bin/python(Py_Main+0xa74)[0x41d8e4]
/share/nds-buildtools/python/python-3.4.2-64_5/bin/python(main+0x155)[0x41a465]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x36e621ed5d]
/share/nds-buildtools/python/python-3.4.2-64_5/bin/python[0x41a249]
=== Memory map: 
0040-005fe000 r-xp  00:1a 491451 
/share/nds-buildtools/python/python-3.4.2-64_5/bin/python3.4
007fd000-00857000 rw-p 001fd000 00:1a 491451 
/share/nds-buildtools/python/python-3.4.2-64_5/bin/python3.4
00857000-00872000 rw-p  00:00 0 
00c84000-0dd92000 rw-p  00:00 0  [heap]
36e5e0-36e5e2 r-xp  fd:00 2964   
/lib64/ld-2.12.so
36e601f000-36e602 r--p 0001f000 fd:00 2964   
/lib64/ld-2.12.so
36e602-36e6021000 rw-p 0002 fd:00 2964   
/lib64/ld-2.12.so
36e6021000-36e6022000 rw-p  00:00 0 
36e620-36e638a000 r-xp  fd:00 3884  

[issue26387] Crash calling sqlite3_close with invalid pointer

[Filipp Andjelo]

Filipp Andjelo added the comment:

Hi Serhiy,

unfortunately not, we have one heavy application which is crashing with the 
given error non deterministically after about 25 hours of running on 40 CPUs 
with 280GiB memory (!) in use. We already made all kind of test to get it 
reproducible "in a small", but it just doesn't crash. If I have more 
information, I'll post it here.

--

___
Python tracker 
<http://bugs.python.org/issue26387>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26387] Crash calling sqlite3_close with invalid pointer

[Filipp Andjelo]

Filipp Andjelo added the comment:

The issue26494 seems really suspicious, could be our case. And regarding my 
patch, nice to know, you are the same opinion. However, the code there is 
looking very strange. I really don't understand, why there are mutex guards 
anyway, since close() can only be called from one thread (there is a check) and 
dealloc() can only be called, when the object is not in use anymore.

--

___
Python tracker 
<http://bugs.python.org/issue26387>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26494] Double deallocation on iterator exhausting

[Filipp Andjelo]

Filipp Andjelo added the comment:

Hi Serhiy,

I tried the short example you gave, but it doesn't crash. I'm getting:

Exception ignored in: 
Traceback (most recent call last):
  File "./test.py", line 5, in __del__
next(it)
StopIteration
Exception ignored in: 
Traceback (most recent call last):
  File "./test.py", line 5, in __del__
next(it)
StopIteration
Traceback (most recent call last):
  File "./test.py", line 8, in 
next(it)
StopIteration

Am I missing something?

--
nosy: +scorp

___
Python tracker 
<http://bugs.python.org/issue26494>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com