Re: [sqlite] SQlite 3 - bottleneck with rbuFindMaindb

2018-09-21 Thread Simon Slavin
On 20 Sep 2018, at 10:31pm, Roger Cuypers wrote: > rbuFindMaindb > rbuVfsAccess > sqlite3OsAccess > hasHotJournal > sqlite3PagerSharedLock > zipvfsLockFile Thanks. That's very useful. Your stack includes both zipvfsLockFile and rbuVfsAccess, and I'm not familiar with either of these. So I

Re: [sqlite] SQlite 3 - bottleneck with rbuFindMaindb

2018-09-21 Thread Roger Cuypers
Ok, I have more info now. The database consists of multiple individual database files which are opened and closed individually each with their own connection, multiple at at time. There is a root file but its just another database file whose only purpose is to tell the application where to find

Re: [sqlite] SQlite 3 - bottleneck with rbuFindMaindb

2018-09-20 Thread Dan Kennedy
On 09/20/2018 03:05 AM, Roger Cuypers wrote: I think it does at some point. I’m at home right now so I have to check this again tomorrow when I have access to the source. Should there be rbu calls if the application is only _reading_ fro the database and not updating? Maybe. If the app

Re: [sqlite] SQlite 3 - bottleneck with rbuFindMaindb

2018-09-19 Thread Simon Slavin
On 19 Sep 2018, at 8:47pm, Roger Cuypers wrote: > the database has a root file. The subfiles are all loaded via separate > connections as far as I know. Sorry, but this makes no sense. Each database file can have only one WAL file. You say that the program is looking through lots of WAL

Re: [sqlite] SQlite 3 - bottleneck with rbuFindMaindb

2018-09-19 Thread Roger Cuypers
Hello, the database has a root file. The subfiles are all loaded via separate connections as far as I know. Another idea of mine: If I know the database will be only written to very rarely, can I prevent sqlite from using the WAL files at all in the meantime? > Am 19.09.2018 um 21:36 schrieb

Re: [sqlite] SQlite 3 - bottleneck with rbuFindMaindb

2018-09-19 Thread Roger Cuypers
I think it does at some point. I’m at home right now so I have to check this again tomorrow when I have access to the source. Should there be rbu calls if the application is only _reading_ fro the database and not updating? > Am 19.09.2018 um 21:48 schrieb Dan Kennedy : > > On 09/20/2018

Re: [sqlite] SQlite 3 - bottleneck with rbuFindMaindb

2018-09-19 Thread Dan Kennedy
On 09/20/2018 01:49 AM, Roger Cuypers wrote: I'm trying to optimize a C++ application that uses sqlite 3 for database access. As far as I know it uses journaling with WAL and has a lot of files/tables (about 400). Profiling this application with Linux perf, I found that it spends about 30% of

Re: [sqlite] SQlite 3 - bottleneck with rbuFindMaindb

2018-09-19 Thread Simon Slavin
On 19 Sep 2018, at 7:49pm, Roger Cuypers wrote: > As far as I know it uses journaling with WAL and has a lot of files/tables > (about 400). Excuse the low-end questions, but they might help save us a lot of silly suggestions. Does SQLite have lots of these open at one time ? If so, does it

[sqlite] SQlite 3 - bottleneck with rbuFindMaindb

2018-09-19 Thread Roger Cuypers
I'm trying to optimize a C++ application that uses sqlite 3 for database access. As far as I know it uses journaling with WAL and has a lot of files/tables (about 400). Profiling this application with Linux perf, I found that it spends about 30% of its time inside the rbuFindMaindb function of