Re: [sqlite] non-blocking concurrent read questions

2010-03-09 Thread Pavel Ivanov
I see your point and your algorithm looks pretty reasonable. Except that again it can be reasonable for you but not for general case which is SQLite for. You have one flaw: SQLite doesn't "read page list in cache", it reads every page that is needed during query execution. So if one applied your

Re: [sqlite] non-blocking concurrent read questions

2010-03-09 Thread Pavel Ivanov
> You said that only references are changed, right? That means, during appends > the page content is still valid even if B-trees structure is changed because > of references. If B-tree is implemented with concurrency in mind then yes, but SQLite wasn't implemented this way. First of all when page

Re: [sqlite] non-blocking concurrent read questions

2010-03-09 Thread Max Vlasov
> The writer application must be failsafe, as much as possible (acoustic > emission recording devices); I simply can not afford that a reader makes a > select and because of a programming error the acquisition be blocked. I had > this just by opening sqliteman. > > The recording rate is variable;

Re: [sqlite] non-blocking concurrent read questions

2010-03-09 Thread Pavel Ivanov
Probably I will express just my opinion but still... Gabriel, what you described is clearly not a good or anywhere intended use of SQLite. If you need writing at the rate of 800,000 records per second you can't afford using database engine for this. Much better option for you will be to have some

[sqlite] non-blocking concurrent read questions

2010-03-09 Thread Gabriel Corneanu
Hello, Thanks for your attention, > Although speaking generally such method could be used in some situations, I > don't think it's good to allow to use it even with a "i know what I'm > doing" > pragma. Any structured file (sqlite is an example) have internal > dependencies. One of the reasons

[sqlite] non-blocking concurrent read questions

2010-03-09 Thread Gabriel Corneanu
Hello again, I start with your final words, "it's a general database engine". On the main page it writes: "Think of SQLite not as a replacement for Oracle but as a replacement for fopen()" That's why I try sqlite and not other database (I actually tried embedded innodb but sqlite was muuuch

[sqlite] non-blocking concurrent read questions

2010-03-09 Thread Gabriel Corneanu
Sure I could have some kind of intermediate storage, but that would mean unnecessary data moving / copying. I really hope that I'll find some time and try to study the source and eventually implement my ideas (maybe others find it interesting and/or useful too). You said that only references are

Re: [sqlite] non-blocking concurrent read questions

2010-03-09 Thread Max Vlasov
> Then I tried in a loop with 2 programs to write / read in parallel and it > seems to work without problems. Can anyone advise if this has any chance to > work (or say it would definitely NOT work)? > > As a short summary: would it be interesting for anyone to enable read-only > open with a

[sqlite] non-blocking concurrent read questions

2010-03-09 Thread Gabriel Corneanu
Hello everybody, I have the following situation: 1. a writer needs to continuously append some data in 1 or 2 tables, * without* any possibility to be blocked. 2. one (or eventually more) reader needs to read the data for analysis. Pt 1 is very important; therefore I use a "PRAGMA locking_mode =