[sqlite] SQLite and (high) concurrency

2008-03-12 Thread Tore Austraatt
Ken & John, thank you very much indeed! That did the trick - so we could perhaps say that IGNORE safely can be ignored... Thanks also for heading me in the right direction regarding transactions, and guess what - all this cut loose several kilos of code. But perhaps the most importent effect

Re: [sqlite] SQLite and (high) concurrency

2008-03-11 Thread John Stanton
It looks as if you are not syncing correctly. Do you check for SQLITE_BUSY? If you are using transactions are you certain that you COMMIT? Are you using mutexes for synchronization or using the Sqlite BUSY checks? Tore Austraatt wrote: > Thanks, but I'm afraid this don't add up. > I have test

Re: [sqlite] SQLite and (high) concurrency

2008-03-11 Thread Ken
Tore, A few things: 1. Remove the "or IGNORE" from your insert. 2. You only need to test for BUSY on the BEGIN IMMEDIATE command. 3. Transactions (begin immediate/commit) are beneficial for multi row operations. Like inserting a group of records then commit. Your not going to see any gain if yo

Re: [sqlite] SQLite and (high) concurrency

2008-03-11 Thread Dan
On Mar 11, 2008, at 10:57 PM, Tore Austraatt wrote: > Ken, > sorry, it didn't make a difference, including _LOCKED in my test. > None of the dropped records are involved in lock situations. > > an example... > _exec("BEGIN IMMEDIATE..." > sqlite3_mprintf("INSERT OR IGNORE INTO ..."...

[sqlite] SQLite and (high) concurrency

2008-03-11 Thread Tore Austraatt
Ken, sorry, it didn't make a difference, including _LOCKED in my test. None of the dropped records are involved in lock situations. an example... _exec("BEGIN IMMEDIATE..." sqlite3_mprintf("INSERT OR IGNORE INTO ..."... _prepare if _OK _step if _DONE _exec("COMMIT"... else if _BUSY || LOCKED

[sqlite] SQLite and (high) concurrency

2008-03-11 Thread Tore Austraatt
Ken, A. multi process B. receive all return codes, eg. rc = sqlite3_..(); and handle accordingly. C. Yes. sqlite3_exec(db, "BEGIN IMMEDIATE", NULL,NULL,NULL); and if _step OK, ...exec("COMMIT"); D. Good question. In the latest testing, only two processes "compete", but at a rather slow rate

Re: [sqlite] SQLite and (high) concurrency

2008-03-11 Thread Ken
Tore, a couple of questions: A. How are your concurrent inserts implemented, in a multi threaded application or multi process? B. What type of error handling do you have in place for failures ? C. Do you use transactions? If so what how are they initiated? D. What do you mean by "high concurre

[sqlite] SQLite and (high) concurrency

2008-03-11 Thread Tore Austraatt
Thanks, but I'm afraid this don't add up. I have tested this in numerous examples. Concurrent INSERT's disappears into thin air, they leave no trace what so ever. It seems very strange if some of you haven't experienced simular problems...? John, PRAGMA sync = whatever does not remedy this eithe

Re: [sqlite] SQLite and (high) concurrency

2008-03-10 Thread John Stanton
If your synchronization logic is sound Sqlite will not ignore inserts. Tore Austraatt wrote: > I find sqlite3 as the perfect choice for our embedded application - if it > hasn't been for the > issues concering concurrency. > My questions are; > 1. is the "default behaviour" from sqlite3 to si

Re: [sqlite] SQLite and (high) concurrency

2008-03-10 Thread Ken
1. Inserts are not ignored. You can ignore inserts in your app if the db is busy/locked etc.. 2. Test the return code for success/failure. Tore Austraatt <[EMAIL PROTECTED]> wrote: I find sqlite3 as the perfect choice for our embedded application - if it hasn't been for the issues concering c

[sqlite] SQLite and (high) concurrency

2008-03-10 Thread Tore Austraatt
I find sqlite3 as the perfect choice for our embedded application - if it hasn't been for the issues concering concurrency. My questions are; 1. is the "default behaviour" from sqlite3 to simply ignore inserts if the concurrency gets to high? 2. is there anything (C API) I can do to at le