[sqlite] WAL file growth concern

2010-10-25 Thread Bob Smith
On Sat Oct 23 20:57:56 GMT 2010, H. Phil Duby wrote: > Give the described conditions, I do not think you need to worry about the > WAL file growing without bound. I think that each increase should be > smaller than the previous one, and will stop growing all together [for your > stress test]

Re: [sqlite] WAL file growth concern

2010-10-25 Thread Nicolas Williams
On Fri, Oct 22, 2010 at 09:56:22PM -0400, Richard Hipp wrote: > On many (most?) filesystems, it is faster to overwrite an existing area of a > file than it is to extend the file by writing past the end. That's why > SQLite doesn't truncate the WAL file on each checkpoint - so that subsequent >

Re: [sqlite] WAL file growth concern

2010-10-23 Thread H. Phil Duby
On Sat, Oct 23, 2010 at 9:26 AM, Bob Smith wrote: > On Sat, Oct 23, 2010 at 5:56 AM, Richard Hipp wrote: > > > > Is that what you are seeing? > > > > Which filesystem are you using? > > > > > > Thanks Richard, this does seem to be what I am seeing. I am definitely > seeing

[sqlite] WAL file growth concern

2010-10-23 Thread Bob Smith
On Sat, Oct 23, 2010 at 5:56 AM, Richard Hipp wrote: > > Is that what you are seeing? > > Which filesystem are you using? > > Thanks Richard, this does seem to be what I am seeing. I am definitely seeing  more writes during the subsequent cycles and an explanation that this is likely due to the

Re: [sqlite] WAL file growth concern

2010-10-23 Thread Max Vlasov
On Sat, Oct 23, 2010 at 5:56 AM, Richard Hipp wrote: > > Is that what you are seeing? > > Which filesystem are you using? > > > > I think the misfeature even easier to reproduce than with the scenario Bob wrote. If your admin has a "repeated query" feature, you even don't need

Re: [sqlite] WAL file growth concern

2010-10-22 Thread Richard Hipp
On Fri, Oct 22, 2010 at 2:54 PM, Bob Smith wrote: > What I have determined (via using sqlite3_wal_hook, catching some > statistics > in the callback as well as calling sqlite3_wal_checkpoint from the callback > myself) is that each time I am writing during a read operation

Re: [sqlite] WAL file growth concern

2010-10-22 Thread Andrey Cherezov
From: Bob Smith Date: 2010-10-21 22:55:05 GMT > I might possibly have found an issue/concern with the way sqlite handles > doing new writes to the WAL file during a time that checkpoints are unable > to checkpoint data from the WAL file back into the database due to a reader >

[sqlite] WAL file growth concern

2010-10-22 Thread Bob Smith
I have been able to do some more investigation here. What I was seeing originally made me suspicious that a writer was skipping over parts of the WAL file during the time that a checkpoint is unable to proceed due to a reader. After further investigation, however, I have determined that this is

[sqlite] WAL file growth concern

2010-10-21 Thread Bob Smith
> Are you *sure* you aren't accidentally holding a read transaction open > somewhere? > Do you have any other clues on how we can isolate the problem? A test case > that will we can run here, perhaps? A am pretty certain that I am not accidentally holding a read transaction open. I have my

Re: [sqlite] WAL file growth concern

2010-10-21 Thread Richard Hipp
On Thu, Oct 21, 2010 at 6:55 PM, Bob Smith wrote: > I have been using WAL mode for a few months and have been quite happy with > the write performance increases. > > I might possibly have found an issue/concern with the way sqlite handles > doing new writes to the WAL file

[sqlite] WAL file growth concern

2010-10-21 Thread Bob Smith
I have been using WAL mode for a few months and have been quite happy with the write performance increases. I might possibly have found an issue/concern with the way sqlite handles doing new writes to the WAL file during a time that checkpoints are unable to checkpoint data from the WAL file