[sqlite] Slow real world performance - Any suggestions please (warning long)

2015-07-04 Thread Rob Willett
Thanks for the comments. Yes we did look at this before we posted our original plea for help. We have a BEGIN/COMMIT around all the relevant code. Rob > On 4 Jul 2015, at 20:28, droedel wrote: > > Rob Willett writes: > > [snip] >> The headline figures are we have gone from 213 secs to

[sqlite] Slow real world performance - Any suggestions please (warning long)

2015-07-04 Thread droedel
Rob Willett writes: [snip] > The headline figures are we have gone from 213 secs to process 20 files down to 90 secs to process 20 files. We > are running approx 2.5x faster. To get this improvement the biggest change was simply adding COLLATE > NOCASE to the table schema. This saved around

[sqlite] Slow real world performance - Any suggestions please (warning long)

2015-07-04 Thread Simon Slavin
On 4 Jul 2015, at 5:46pm, William Drago wrote: > Clearly, in this case, using COLLATE NOCASE in the table definition is the > right thing to do. Under what conditions would using it in the index instead > be the right > thing to do? It's rare. Sometimes you have a column where case normally

[sqlite] SQLite in multi-thread application

2015-07-04 Thread Simon Slavin
On 3 Jul 2015, at 4:15pm, ALBERT Aur?lien wrote: > - Using a mutex, only a single thread can write to the database (but > reads can happen during this time) > > But I have sometimes "Database is locked" errors. > > Did I miss something in my configuration ? > Did I miss something in

[sqlite] Slow real world performance - Any suggestions please (warning long)

2015-07-04 Thread Keith Medcalf
> Clearly, in this case, using COLLATE NOCASE in the table > definition is the right thing to do. Under what conditions > would using it in the index instead be the right > thing to do? When you want the default to be case sensitive, and only use collate nocase when you specifically specify

[sqlite] Slow real world performance - Any suggestions please (warning long)

2015-07-04 Thread William Drago
On 7/3/2015 8:39 AM, Rob Willett wrote: > Simon, > > We had missed the incorrect defn of Calculate in the index. We?ve been > changing around stuff and suspect we omitted to check as carefully as we we > should when changing :( > > We?ll also update BayesAttribute as well. > > We?re actually

[sqlite] SQLite in multi-thread application

2015-07-04 Thread Keith Medcalf
> I'm using SQLite v3.8.8.3 in my muli-threaded application. > > SQLite is configured so these asserts are satisfied : > > assert(sqlite3_threadsafe() > 0); > assert(sqlite3_config(SQLITE_CONFIG_MULTITHREAD) == SQLITE_OK); > > I have multiple connections to the same database

[sqlite] Reader.GetBytes() - when is a byte not a byte?

2015-07-04 Thread William Drago
Jean, Thanks for the reply. I understand this very well, and I have read this page many times over the past few years: http://www.sqlite.org/datatype3.html My argument is that regardless of a column's type or type affinity, a method called ReadBytes() should read the bytes as stored in the

[sqlite] Ensuring the WAL doesn't grow too big

2015-07-04 Thread Mikael
Hi, Some of this can be read out of the manual but addressing for complete clarity, is slightly related to the fragmentation conversation that just was (+ I saw a GB-size WAL file the other day, perhaps because of someone playing with the wal_autocheckpoint pragma setting): a) How do you ensure