Re: [sqlite] Sqlite3 WAL (file is encrypted or is not a database, database table is locked, database disk image is malformed') at high load
I am sorry I won't able to share the complete program. Also it is quite some work to extract that piece alone. *Some good news. As a last resort I downloaded latest sqlite source, compiled and ran the tests. It never failed. The problem seems **to be fixed in 3.7.6.3.* But I am still curious to know the root cause in 3.7.0.1. Are there things which should be taken care by the applications while doing BULK inserts and concurrent SELECT/INSERTS/UPDATES to avoid such 'errors' from sqlite3 ? Thanks Anoop On Thu, Jun 23, 2011 at 7:12 PM, Dan Kennedy wrote: > On 06/23/2011 12:11 PM, Anoop K wrote: > > I am using sqlite3(sqlite-3.7.0.1) with WAL enabled as storage for a > > multiprocessing daemon(python). On doing a BULK insert of .5 million rows > > each of size 230 bytes in batches of 500 where each batch is a > transaction, > > following errors happen in other processes which perform (<10) SELECTS > and > > INSERTS. > > > > These errors does not happen always and database do function even after > > these errors. SELECTS, INSERTS queries after the BULK insert operation do > > succeed after the batch operation is completed. > > > > - file is encrypted or is not a database > > - database table is locked > > > > Sqlite3 configuration > > > > - 'PRAGMA synchronous=OFF' > > - 'PRAGMA journal_mode=wal' > > - 'PRAGMA wal_checkpoint=FULL' Rest of the configuration parameters > have > > have default values > > > > If I continue BULK inserts with a sleep of 30 sec, after 10+ runs I > > see *'database > > disk image is malformed' . *After that the table to which BULK inserts > were > > made does not work. They fail with same error '*database disk image is > > malformed'.*. Other tables do respond to queries. > > > > Any clues on the root cause ? Is this a Sqlite3 WAL bug ? > > Do you have a program you can share that exhibits this bug? > > Dan. > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Sqlite3 WAL (file is encrypted or is not a database, database table is locked, database disk image is malformed') at high load
On 06/23/2011 12:11 PM, Anoop K wrote: > I am using sqlite3(sqlite-3.7.0.1) with WAL enabled as storage for a > multiprocessing daemon(python). On doing a BULK insert of .5 million rows > each of size 230 bytes in batches of 500 where each batch is a transaction, > following errors happen in other processes which perform (<10) SELECTS and > INSERTS. > > These errors does not happen always and database do function even after > these errors. SELECTS, INSERTS queries after the BULK insert operation do > succeed after the batch operation is completed. > > - file is encrypted or is not a database > - database table is locked > > Sqlite3 configuration > > - 'PRAGMA synchronous=OFF' > - 'PRAGMA journal_mode=wal' > - 'PRAGMA wal_checkpoint=FULL' Rest of the configuration parameters have > have default values > > If I continue BULK inserts with a sleep of 30 sec, after 10+ runs I > see *'database > disk image is malformed' . *After that the table to which BULK inserts were > made does not work. They fail with same error '*database disk image is > malformed'.*. Other tables do respond to queries. > > Any clues on the root cause ? Is this a Sqlite3 WAL bug ? Do you have a program you can share that exhibits this bug? Dan. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Sqlite3 WAL (file is encrypted or is not a database, database table is locked, database disk image is malformed') at high load
On 23 Jun 2011, at 2:22pm, Anoop K wrote: > By the way I > did change it to 'NORMAL' and tried. The issues mentioned in the mail still > happened Okay, that's useful information. Simon. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Sqlite3 WAL (file is encrypted or is not a database, database table is locked, database disk image is malformed') at high load
I had to turn synchronous=OFF to achieve required performance. By the way I did change it to 'NORMAL' and tried. The issues mentioned in the mail still happened Some more info regarding the problem. At this high load IO Wait was about 50-60%. Thanks Anoop On Thu, Jun 23, 2011 at 5:42 PM, Simon Slavin wrote: > > On 23 Jun 2011, at 6:11am, Anoop K wrote: > > > Sqlite3 configuration > > > > - 'PRAGMA synchronous=OFF' > > Remove that and try it again. > > Simon. > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Sqlite3 WAL (file is encrypted or is not a database, database table is locked, database disk image is malformed') at high load
On 23 Jun 2011, at 6:11am, Anoop K wrote: > Sqlite3 configuration > > - 'PRAGMA synchronous=OFF' Remove that and try it again. Simon. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Sqlite3 WAL (file is encrypted or is not a database, database table is locked, database disk image is malformed') at high load
I am using sqlite3(sqlite-3.7.0.1) with WAL enabled as storage for a multiprocessing daemon(python). On doing a BULK insert of .5 million rows each of size 230 bytes in batches of 500 where each batch is a transaction, following errors happen in other processes which perform (<10) SELECTS and INSERTS. These errors does not happen always and database do function even after these errors. SELECTS, INSERTS queries after the BULK insert operation do succeed after the batch operation is completed. - file is encrypted or is not a database - database table is locked Sqlite3 configuration - 'PRAGMA synchronous=OFF' - 'PRAGMA journal_mode=wal' - 'PRAGMA wal_checkpoint=FULL' Rest of the configuration parameters have have default values If I continue BULK inserts with a sleep of 30 sec, after 10+ runs I see *'database disk image is malformed' . *After that the table to which BULK inserts were made does not work. They fail with same error '*database disk image is malformed'.*. Other tables do respond to queries. Any clues on the root cause ? Is this a Sqlite3 WAL bug ? Thanks Anoop ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
