Hi Richard
Thanks for taking care of the issue.
I was using the the ubuntu 16.04.3 with
/usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6,
it's shipped byubuntu.
I compiled the latest sqlite3 with the suggested params, I force the python
to load the debug sqlite3.so and my app is not crashing now,
Unfortunately, my tests are blocked, by a lock in sqlite3.c "#2
0x00007fde23735089 in pthreadMutexEnter (p=0x285d220) at sqlite3.c:24023"
from trace of my python app, an "INSERT"(thread 3) and an "SELECT" (thread
2) failed into a dead lock and never had a chance to break up.
::
(gdb) thread apply all py-bt
Thread 3 (Thread 0x7fde232a6700 (LWP 4135)):
Traceback (most recent call first):
Waiting for the GIL
<built-in method execute of sqlite3.Cursor object at remote
0x7fde236fa490>
<built-in method execute of sqlite3.Connection object at remote
0x7fde23c799d0>
File "/home/user/Documents/persist-queue/persistqueue/sqlbase.py", line
17, in _execute
tran.execute(stat, param)
File "/home/user/Documents/persist-queue/persistqueue/sqlqueue.py", line
39, in put
self._insert_into(obj, _time.time())
File "/home/user/Documents/persist-queue/tests/test_sqlqueue.py", line
99, in producer
m_queue.put('var%d' % i)
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 882, in _bootstrap
self._bootstrap_inner()
Thread 2 (Thread 0x7fde222a4700 (LWP 4134)):
Traceback (most recent call first):
<built-in method execute of sqlite3.Cursor object at remote
0x7fde236fa340>
<built-in method execute of sqlite3.Connection object at remote
0x7fde23c799d0>
File "/home/user/Documents/persist-queue/persistqueue/sqlbase.py", line
149, in _select
return self._getter.execute(self._sql_select, args).fetchone()
File "/home/user/Documents/persist-queue/persistqueue/sqlqueue.py", line
49, in _pop
row = self._select()
File "/home/user/Documents/persist-queue/persistqueue/sqlqueue.py", line
71, in get
pickled = self._pop()
File "/home/user/Documents/persist-queue/tests/test_sqlqueue.py", line
105, in consumer
x = m_queue.get(block=True)
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 882, in _bootstrap
---Type <return> to continue, or q <return> to quit---
self._bootstrap_inner()
Thread 1 (Thread 0x7fde276f4700 (LWP 4121)):
Traceback (most recent call first):
<built-in method acquire of _thread.lock object at remote 0x7fde23c3e0d0>
File "/usr/lib/python3.5/threading.py", line 1070, in
_wait_for_tstate_lock
elif lock.acquire(block, timeout):
File "/usr/lib/python3.5/threading.py", line 1054, in join
self._wait_for_tstate_lock()
File "/home/user/Documents/persist-queue/tests/test_sqlqueue.py", line
112, in test_multi_threaded_parallel
p.join()
File "/usr/lib/python3.5/unittest/case.py", line 600, in run
testMethod()
File "/usr/lib/python3.5/unittest/case.py", line 648, in __call__
return self.run(*args, **kwds)
File "/usr/lib/python3.5/unittest/suite.py", line 122, in run
test(result)
File "/usr/lib/python3.5/unittest/suite.py", line 84, in __call__
return self.run(*args, **kwds)
File "/usr/lib/python3.5/unittest/suite.py", line 122, in run
test(result)
File "/usr/lib/python3.5/unittest/suite.py", line 84, in __call__
return self.run(*args, **kwds)
File "/usr/lib/python3.5/unittest/suite.py", line 122, in run
test(result)
File "/usr/lib/python3.5/unittest/suite.py", line 84, in __call__
return self.run(*args, **kwds)
File
"/home/user/Documents/persist-queue/.tox/py35/lib/python3.5/site-packages/nose2/runner.py",
line 41, in <lambda>
executor = lambda suite, result: suite(result)
File
"/home/user/Documents/persist-queue/.tox/py35/lib/python3.5/site-packages/nose2/runner.py",
line 53, in run
executor(test, result)
File
"/home/user/Documents/persist-queue/.tox/py35/lib/python3.5/site-packages/nose2/main.py",
line 271, in runTests
self.result = runner.run(self.test)
File "/usr/lib/python3.5/unittest/main.py", line 94, in __init__
self.runTests()
File
"/home/user/Documents/persist-queue/.tox/py35/lib/python3.5/site-packages/nose2/main.py",
line 100, in __init__
---Type <return> to continue, or q <return> to quit---
super(PluggableTestProgram, self).__init__(**kw)
File
"/home/user/Documents/persist-queue/.tox/py35/lib/python3.5/site-packages/nose2/main.py",
line 306, in discover
return main(*args, **kwargs)
File "/home/user/Documents/persist-queue/.tox/py35/bin/nose2", line 11,
in <module>
sys.exit(discover())
====
Reproduce:
1. checkout my repo:
$ git clone https://github.com/peter-wangxu -b memory_shared_cache
2. run the tox
$ tox -e py35 -- --verbose tests.test_sqlqueue.SQLite3QueueInMemory
Above will reproduce the hung issue.
Thanks
Peter
On Tue, Oct 31, 2017 at 10:28 PM, Richard Hipp <[email protected]> wrote:
> On 10/31/17, Wang, Peter (Xu) <[email protected]> wrote:
> > Hi
> > I am trying to provide a asychnonous queue based on sqlite3 in Python
> > Current, I already passted the file based tests with my queue
> > When switching file to in-memory db, I keep meeting a "segfault" issue
> when
> > running the same tests test suite
> >
> > Can anyone help me out of this situation?
>
> Can you get us a reproducible test case?
>
> Can you tell us what version of SQLite you are using?
>
> Can you recompile the SQLite shared library using -DSQLITE_DEBUG and
> -DSQLITE_ENABLE_API_ARMOR and see how that changes the outcome?
>
> >
> >
> > I tested the in-memory with multi-thread (but inserts/deletes are in a
> lock)
> > Here is the trace of the core file:
> >
> > (gdb) bt
> > #0 sqlite3_value_type (pVal=0x0) at sqlite3.c:72512
> > #1 0x00007fda2a34fd86 in sqlite3_column_type (pStmt=0x7fda08004178,
> > i=<optimized out>) at sqlite3.c:73318
> > #2 0x00007fda2a607987 in _pysqlite_fetch_one_row
> > (self=self@entry=0x7fda2a204b20) at
> > /build/python2.7-1tJBSB/python2.7-2.7.12/Modules/_sqlite/cursor.c:352
> > #3 0x00007fda2a6089c1 in _pysqlite_query_execute (self=0x7fda2a204b20,
> > multiple=<optimized out>, args=<optimized out>)
> > at /build/python2.7-1tJBSB/python2.7-2.7.12/Modules/_
> sqlite/cursor.c:711
> > #4 0x00000000004b0cb3 in PyObject_Call () at ../Objects/abstract.c:2546
> > #5 0x00000000004ce5d0 in PyEval_CallObjectWithKeywords () at
> > ../Python/ceval.c:4219
> > #6 0x00007fda2a609106 in pysqlite_connection_execute (self=<optimized
> out>,
> > args=('SELECT _id, data FROM queue_default ORDER BY _id ASC LIMIT 1',
> ()))
> > at
> > /build/python2.7-1tJBSB/python2.7-2.7.12/Modules/_
> sqlite/connection.c:1262
> > #7 0x00000000004cb945 in call_function (oparg=<optimized out>,
> > pp_stack=0x7fda19ff9fc0) at ../Python/ceval.c:4350
> > #8 PyEval_EvalFrameEx () at ../Python/ceval.c:2987
> > #9 0x00000000004c2765 in PyEval_EvalCodeEx () at ../Python/ceval.c:3582
> > #10 0x00000000004ca8d1 in fast_function (nk=0, na=<optimized out>,
> > n=<optimized out>, pp_stack=0x7fda19ffa1d0, func=<function at remote
> > 0x7fda2a83c050>)
> > at ../Python/ceval.c:4445
> > #11 call_function (oparg=<optimized out>, pp_stack=0x7fda19ffa1d0) at
> > ../Python/ceval.c:4370
> > #12 PyEval_EvalFrameEx () at ../Python/ceval.c:2987
> > #13 0x00000000004c9d8f in fast_function (nk=<optimized out>,
> na=<optimized
> > out>, n=<optimized out>, pp_stack=0x7fda19ffa320,
> > func=<function at remote 0x7fda2a8416e0>) at ../Python/ceval.c:4435
> > #14 call_function (oparg=<optimized out>, pp_stack=0x7fda19ffa320) at
> > ../Python/ceval.c:4370
> > #15 PyEval_EvalFrameEx () at ../Python/ceval.c:2987
> > #16 0x00000000004c2765 in PyEval_EvalCodeEx () at ../Python/ceval.c:3582
> > #17 0x00000000004ca099 in fast_function (nk=1, na=<optimized out>,
> > n=<optimized out>, pp_stack=0x7fda19ffa530, func=<function at remote
> > 0x7fda2a841758>)
> > at ../Python/ceval.c:4445
> > #18 call_function (oparg=<optimized out>, pp_stack=0x7fda19ffa530) at
> > ../Python/ceval.c:4370
> > #19 PyEval_EvalFrameEx () at ../Python/ceval.c:2987
> > #20 0x00000000004c2765 in PyEval_EvalCodeEx () at ../Python/ceval.c:3582
> > #21 0x00000000004de8b8 in function_call.lto_priv () at
> > ../Objects/funcobject.c:523
> > #22 0x00000000004b0cb3 in PyObject_Call () at ../Objects/abstract.c:2546
> > #23 0x00000000004c6ad1 in ext_do_call (nk=<optimized out>, na=<optimized
> > out>, flags=<optimized out>, pp_stack=0x7fda19ffa7e8,
> > func=<function at remote 0x7fda29fd6c80>) at ../Python/ceval.c:4664
> > #24 PyEval_EvalFrameEx () at ../Python/ceval.c:3026
> > #25 0x00000000004c9d8f in fast_function (nk=<optimized out>,
> na=<optimized
> > out>, n=<optimized out>, pp_stack=0x7fda19ffa930,
> > func=<function at remote 0x7fda2da20758>) at ../Python/ceval.c:4435
> > #26 call_function (oparg=<optimized out>, pp_stack=0x7fda19ffa930) at
> > ../Python/ceval.c:4370
> > #27 PyEval_EvalFrameEx () at ../Python/ceval.c:2987
> > #28 0x00000000004c9d8f in fast_function (nk=<optimized out>,
> na=<optimized
> > out>, n=<optimized out>, pp_stack=0x7fda19ffaa80,
> > ---Type <return> to continue, or q <return> to quit---
> > func=<function at remote 0x7fda2da208c0>) at ../Python/ceval.c:4435
> > #29 call_function (oparg=<optimized out>, pp_stack=0x7fda19ffaa80) at
> > ../Python/ceval.c:4370
> > #30 PyEval_EvalFrameEx () at ../Python/ceval.c:2987
> > #31 0x00000000004c2765 in PyEval_EvalCodeEx () at ../Python/ceval.c:3582
> > #32 0x00000000004de6fe in function_call.lto_priv () at
> > ../Objects/funcobject.c:523
> > #33 0x00000000004b0cb3 in PyObject_Call () at ../Objects/abstract.c:2546
> > #34 0x00000000004f492e in instancemethod_call.lto_priv () at
> > ../Objects/classobject.c:2602
> > #35 0x00000000004b0cb3 in PyObject_Call () at ../Objects/abstract.c:2546
> > #36 0x00000000004ce5d0 in PyEval_CallObjectWithKeywords () at
> > ../Python/ceval.c:4219
> > #37 0x0000000000597e32 in t_bootstrap () at ../Modules/threadmodule.c:620
> > #38 0x00007fda2d77f6ba in start_thread (arg=0x7fda19ffb700) at
> > pthread_create.c:333
> > #39 0x00007fda2d4b53dd in clone () at
> > ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
> >
> > Here is the python trace:
> >
> > (gdb) py-bt
> > Traceback (most recent call first):
> > File "/home/user/Documents/persist-queue/persistqueue/sqlbase.py",
> line
> > 140, in _select
> > def _select(self, *args):
> > File "/home/user/Documents/persist-queue/persistqueue/sqlqueue.py",
> line
> > 49, in _pop
> > row = self._select()
> > File "/home/wangp11/Documents/persist-queue/persistqueue/sqlqueue.py",
> > line 67, in get
> > pickled = self._pop()
> > File "/home/user/Documents/persist-queue/tests/test_sqlqueue.py", line
> > 105, in consumer
> > x = m_queue.get(block=True)
> > File "/usr/lib/python2.7/threading.py", line 754, in run
> > self.__target(*self.__args, **self.__kwargs)
> > File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
> > self.run()
> > File "/usr/lib/python2.7/threading.py", line 774, in __bootstrap
> > self.__bootstrap_inner()
> > _______________________________________________
> > sqlite-users mailing list
> > [email protected]
> > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
> >
>
>
> --
> D. Richard Hipp
> [email protected]
>
_______________________________________________
sqlite-users mailing list
[email protected]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
