Re: [sqlite] Core dump with python multi thread programming using sqlite3 DB

2019-03-01 Thread Keith Medcalf 
On Thursday, 28 February, 2019 21:09, Rishi Mutnuru  
wrote:

>I am developing a Multi threaded solution using sqlite3 DB. I know
>there are some issues
>With SQlite3 and python multithreads.

I have never seen a single multithreading problem with Python and SQLite3, at 
least not for at least the last 5 years or more.  So unless you are using a 
REALLY OLD version of Python and SQLite3 you are unlikely to run into any 
problems whatsoever.

What are the particular issues you are running into?


---
The fact that there's a Highway to Hell but only a Stairway to Heaven says a 
lot about anticipated traffic volume.




___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Core dump with python multi thread programming using sqlite3 DB

2019-03-01 Thread Richard Hipp 
On 2/28/19, Rishi Mutnuru  wrote:
> Hi Experts,
> I am developing a Multi threaded solution using sqlite3 DB. I know there are
> some issues
> With SQlite3 and python multithreads.
>
> 1. I compiled libsqlite3 with SQLITE_THREADSAFE=2 so that multiple threads
> can use it.
>  2. I used the APIs threading.Lock(), and lock.acquire(), lock.release() to
> serialize the DB read/update operations.

Segfaults in sqlite3_free() are almost always the result of heap
corruption caused by some other component of the application.  SQLite
just happened to be the unlucky library that first stumbled over the
heap corruption.


-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Core dump with python multi thread programming using sqlite3 DB

2019-03-01 Thread Rishi Mutnuru 
Hi Experts,
I am developing a Multi threaded solution using sqlite3 DB. I know there are 
some issues
With SQlite3 and python multithreads.

1. I compiled libsqlite3 with SQLITE_THREADSAFE=2 so that multiple threads can 
use it.
 2. I used the APIs threading.Lock(), and lock.acquire(), lock.release() to 
serialize the DB read/update operations.

After this, I am always getting following segmentation fault. Can you please 
let me know what is going wrong and
Potential fix to this.



*** Error in `python': double free or corruption (!prev): 0x7f5c30b20d30 ***
=== Backtrace: =
/lib64/libc.so.6(+0x814e9)[0x7f5c44ed04e9]
/usr/local/lib/libsqlite3.so.0(sqlite3_free+0x4f)[0x7f5c43165baf]
/usr/local/lib/libsqlite3.so.0(+0x2c1f1)[0x7f5c431711f1]
/usr/local/lib/libsqlite3.so.0(+0x2c287)[0x7f5c43171287]
/usr/local/lib/libsqlite3.so.0(+0x78dc6)[0x7f5c431bddc6]
/usr/local/lib/libsqlite3.so.0(sqlite3_finalize+0x3c)[0x7f5c431bde1c]
/root/.pyenv/versions/2.7.12/lib/python2.7/lib-dynload/_sqlite3.so(pysqlite_statement_finalize+0x21)[0x7f5c4345e7e1]
/root/.pyenv/versions/2.7.12/lib/python2.7/lib-dynload/_sqlite3.so(pysqlite_do_all_statements+0x45)[0x7f5c43459205]
/root/.pyenv/versions/2.7.12/lib/python2.7/lib-dynload/_sqlite3.so(pysqlite_connection_close+0x2e)[0x7f5c43459d3e]
python(PyEval_EvalFrameEx+0x5cab)[0x4b63ab]
python(PyEval_EvalFrameEx+0x6334)[0x4b6a34]
python(PyEval_EvalCodeEx+0x7f8)[0x4b9d78]
python[0x52dec0]
python(PyObject_Call+0x3a)[0x422daa]
python[0x42728d]
python(PyObject_Call+0x3a)[0x422daa]
python[0x48136f]
python[0x47ea9a]
python(PyObject_Call+0x3a)[0x422daa]
python(PyEval_EvalFrameEx+0x179a)[0x4b1e9a]
python(PyEval_EvalFrameEx+0x6334)[0x4b6a34]
python(PyEval_EvalFrameEx+0x6334)[0x4b6a34]
python(PyEval_EvalFrameEx+0x6334)[0x4b6a34]
python(PyEval_EvalFrameEx+0x6334)[0x4b6a34]
python(PyEval_EvalFrameEx+0x6334)[0x4b6a34]
python(PyEval_EvalFrameEx+0x6334)[0x4b6a34]
python(PyEval_EvalFrameEx+0x6334)[0x4b6a34]
python(PyEval_EvalFrameEx+0x6334)[0x4b6a34]
python(PyEval_EvalCodeEx+0x7f8)[0x4b9d78]
python[0x52df8f]
python(PyObject_Call+0x3a)[0x422daa]
python(PyEval_EvalFrameEx+0x1e33)[0x4b2533]
python(PyEval_EvalFrameEx+0x6334)[0x4b6a34]
python(PyEval_EvalFrameEx+0x6334)[0x4b6a34]
python(PyEval_EvalCodeEx+0x7f8)[0x4b9d78]
python[0x52dec0]
python(PyObject_Call+0x3a)[0x422daa]
python[0x42728d]
python(PyObject_Call+0x3a)[0x422daa]
python(PyEval_CallObjectWithKeywords+0x47)[0x4b0657]
python[0x4fed42]
/lib64/libpthread.so.0(+0x7dd5)[0x7f5c4592cdd5]
/lib64/libc.so.6(clone+0x6d)[0x7f5c44f4dbfd]
=== Memory map: 
0040-005a6000 r-xp  08:01 668539 
/root/.pyenv/versions/2.7.12/bin/python2.7
007a5000-007a6000 r--p 001a5000 08:01 668539 
/root/.pyenv/versions/2.7.12/bin/python2.7
007a6000-007e2000 rw-p 001a6000 08:01 668539 
/root/.pyenv/versions/2.7.12/bin/python2.7
007e2000-00804000 rw-p  00:00 0 
021d4000-0285f000 rw-p  00:00 0  [heap]
7f5c2800-7f5c29112000 rw-p  00:00 0 
7f5c29112000-7f5c2c00 ---p  00:00 0 
7f5c2c00-7f5c2c021000 rw-p  00:00 0 
7f5c2c021000-7f5c3000 ---p  00:00 0 
7f5c3000-7f5c310fd000 rw-p  00:00 0 
7f5c310fd000-7f5c3400 ---p  00:00 0 
7f5c3400-7f5c350f9000 rw-p  00:00 0 
7f5c350f9000-7f5c3800 ---p  00:00 0 
7f5c3800-7f5c39104000 rw-p  00:00 0 
7f5c39104000-7f5c3c00 ---p  00:00 0 
7f5c3c216000-7f5c3c22b000 r-xp  08:01 133728 
/usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f5c3c22b000-7f5c3c42a000 ---p 00015000 08:01 133728 
/usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f5c3c42a000-7f5c3c42b000 r--p 00014000 08:01 133728 
/usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f5c3c42b000-7f5c3c42c000 rw-p 00015000 08:01 133728 
/usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f5c3c42c000-7f5c3c52c000 rw-p  00:00 0 
7f5c3c52c000-7f5c3c52d000 ---p  00:00 0 
7f5c3c52d000-7f5c3cd2d000 rw-p  00:00 0 
7f5c3cd2d000-7f5c3cd2e000 ---p  00:00 0 
7f5c3cd2e000-7f5c3d52e000 rw-p  00:00 0 
7f5c3d52e000-7f5c3d52f000 ---p  00:00 0 
7f5c3d52f000-7f5c3dd2f000 rw-p  00:00 0 
7f5c3dd2f000-7f5c3dd3 ---p  00:00 0 
7f5c3dd3-7f5c3e63 rw-p  00:00 0 
7f5c3e63-7f5c3e644000 r-xp  08:01 671144 
/root/.pyenv/versions/2.7.12/lib/python2.7/lib-dynload/_ssl.so
7f5c3e644000-7f5c3e843000 ---p 00014000 08:01 671144 
/root/.pyenv/versions/2.7.12/lib/python2.7/lib-dynload/_ssl.so
7f5c3e843000-7f5c3e844000 r--p 00013000 08:01 671144 
/root/.pyenv/versions/2.7.12/lib/python2.7/lib-dynload/_ssl.so
7f5c3e844000-7f5c3e848000 rw-p 00014000 08:01 671144 
/root/.pyenv/versions/2.7.12/lib/python2.7/lib-dynload/_ssl.so

Re: [sqlite] Is there a tool to convert `where`s to equivalent `join`s?

2019-03-01 Thread Don V Nielsen 
>   So for learning sake, is there a tool that converts a query using
`WHERE`
to a query (that yields identical results) using JOINs?

I'm not aware of a tool, but I would assert that not using a tool is for
learning sake. Recoding by hand is going to be your best learning
experience.

Note: there was a lot of good reading in the links you provided. One went
right down in logical order how Select statement is evaluated by SQL
engines. That there is the greatest tool for learning sake.


On Fri, Mar 1, 2019 at 2:15 AM Rocky Ji  wrote:

> In SQL world, generally
> ,
> not just SQLite
> <
> https://www.mail-archive.com/sqlite-users@mailinglists.sqlite.org/msg114165.html
> >,
> a lot  of
> pros say that the constructs `from...inner join...on` and `from...where`
> are equivalent
>  >,
> and that query optimizer should build identical plans for either variant.
>
> Understanding `WHERE` is simple / straight-forward / intutive, to those
> 1. without mathematics background, or
> 2. used to imperative code and for-loops
>
> So for learning sake, is there a tool that converts a query using `WHERE`
> to a query (that yields identical results) using JOINs? Like a English ->
>  side-by-side translator.
>
> Cheers
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Site search bug - https://sqlite.org/

2019-03-01 Thread Dianne Dunn 
Please remove me ASAP thx

Sent from my iPad

> On Feb 27, 2019, at 5:28 AM, Chris Locke  wrote:
> 
> The link you quote mentions SQLite 3.24 though... ?
> 
> 
> Thanks,
> Chris
> 
>> On Wed, Feb 27, 2019 at 12:44 PM niki  wrote:
>> 
>> This link demonstrates the problem:
>> 
>> https://sqlite.org/search?s=c=3.24
>> 
>> Best regards,
>> 
>> Niki
>> 
>> ___
>> sqlite-users mailing list
>> sqlite-users@mailinglists.sqlite.org
>> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>> 
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Deserialize a WAL database file

2019-03-01 Thread Lloyd 
On Fri, Mar 1, 2019 at 4:09 PM Simon Slavin  wrote:

> On 1 Mar 2019, at 10:27am, Lloyd  wrote:
>
> > I am able to serialize, deserialize and prepare a SQL query against the
> rollback database. When I do the same against the WAL database file, the
> 'prepare' statement fails with code '1'.
>
> It's possible that some of the data for the WAL database is in the journal
> and hasn't been written back to the database properly.  Was that database
> correctly closed when last used ?
>
> Using the SQLite command-line tool,
> open the database
> run "PRAGMA integrity_check;" on it
> close the database
>
> Then try serializing it again.  If still fails, do the above again but
> "VACUUM;" it instead of the integrity_check.
>
> Simon.
>


I forgot to mention that the database is in a good state. No temp WAL/SHM
files exist in the directory. I tried the steps you suggested. Integrity
check returned 'ok'. Executed vacuum command also. Still, the prepare
statement fails with 1. This is a small test database created. I closed the
database from shell using '.quit' command. I believe that it closes the
connection properly.



>
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Deserialize a WAL database file

2019-03-01 Thread Simon Slavin 
On 1 Mar 2019, at 10:27am, Lloyd  wrote:

> I am able to serialize, deserialize and prepare a SQL query against the 
> rollback database. When I do the same against the WAL database file, the 
> 'prepare' statement fails with code '1'.

It's possible that some of the data for the WAL database is in the journal and 
hasn't been written back to the database properly.  Was that database correctly 
closed when last used ?

Using the SQLite command-line tool,
open the database
run "PRAGMA integrity_check;" on it
close the database

Then try serializing it again.  If still fails, do the above again but 
"VACUUM;" it instead of the integrity_check.

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Deserialize a WAL database file

2019-03-01 Thread Lloyd 
I have two database files. One in Rollback mode and the other in WAL mode.
I am able to serialize, deserialize and prepare a SQL query against the
rollback database. When I do the same against the WAL database file, the
'prepare' statement fails with code '1'. Is it not possible to do this on
WAL based database file? A sample code fragment is given below-

sqlite3 *dbHandle=nullptr;
if (sqlite3_open_v2("db_filename", , SQLITE_OPEN_READONLY, NULL)
!= SQLITE_OK){//error}

sqlite3_int64 sz=0;
unsigned char* mem=sqlite3_serialize(dbHandle,"main",,0);

if(sqlite3_deserialize(dbHandle, "main", mem, sz,
sz,SQLITE_DESERIALIZE_READONLY) != SQLITE_OK){//error}

char* Query = "select * from test";
sqlite3_stmt *statement = nullptr;
int res=sqlite3_prepare_v2(dbHandle, Query, strlen(Query), , 0);
//res is 1 for WAL


Thanks,
   Lloyd
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Is there a tool to convert `where`s to equivalent `join`s?

2019-03-01 Thread Rocky Ji 
In SQL world, generally
,
not just SQLite
,
a lot  of
pros say that the constructs `from...inner join...on` and `from...where`
are equivalent
,
and that query optimizer should build identical plans for either variant.

Understanding `WHERE` is simple / straight-forward / intutive, to those
1. without mathematics background, or
2. used to imperative code and for-loops

So for learning sake, is there a tool that converts a query using `WHERE`
to a query (that yields identical results) using JOINs? Like a English ->
 side-by-side translator.

Cheers
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users