Re: [sqlite] sqlite3_open_v2 performance degrades as number of opens increase

2011-09-04 Thread Terry Cumaranatunge
e to improve write speeds. 70% of the operations we have are inserts + updates.   Any thoughts on this approach?   Thanks From: Simon Slavin >To: Terry Cumaranatunge ; General Discussion of SQLite >Database >Sent: Friday, September 2, 2011 12:41 PM >Subject: Re: [sqlite] sqlite3_open_

Re: [sqlite] sqlite3_open_v2 performance degrades as number of opens increase

2011-09-02 Thread Simon Slavin
On 2 Sep 2011, at 6:34pm, Terry Cumaranatunge wrote: > This does suggest that we should re-evaluate our design and not open as many > databases from a single process. The original motivation was to limit the > size the database for performance reasons, which resulted in a large number > of dat

Re: [sqlite] sqlite3_open_v2 performance degrades as number of opens increase

2011-09-02 Thread Terry Cumaranatunge
ecommended size for a database file? From: Pavel Ivanov >To: General Discussion of SQLite Database >Sent: Friday, September 2, 2011 9:28 AM >Subject: Re: [sqlite] sqlite3_open_v2 performance degrades as number of opens >increase > >> Does anyone know why the performance de

Re: [sqlite] sqlite3_open_v2 performance degrades as number of opens increase

2011-09-02 Thread Jay A. Kreibich
On Fri, Sep 02, 2011 at 06:30:57AM -0500, Terry Cumaranatunge scratched on the wall: > Hello, > > We have an application that creates many small databases (over 100K) to be > able to control the size of the database and provide more deterministic > performance. At process startup, it opens many o

Re: [sqlite] sqlite3_open_v2 performance degrades as number of opens increase

2011-09-02 Thread Pavel Ivanov
rg [sqlite-users-boun...@sqlite.org] on > behalf of Black, Michael (IS) [michael.bla...@ngc.com] > Sent: Friday, September 02, 2011 8:00 AM > To: General Discussion of SQLite Database > Subject: EXT :Re: [sqlite] sqlite3_open_v2 performance degrades as number of > opens increase > >

Re: [sqlite] sqlite3_open_v2 performance degrades as number of opens increase

2011-09-02 Thread Black, Michael (IS)
] on behalf of Black, Michael (IS) [michael.bla...@ngc.com] Sent: Friday, September 02, 2011 8:00 AM To: General Discussion of SQLite Database Subject: EXT :Re: [sqlite] sqlite3_open_v2 performance degrades as number of opens increase I assume you've overridden the system default for 1024 fil

Re: [sqlite] sqlite3_open_v2 performance degrades as number of opens increase

2011-09-02 Thread Black, Michael (IS)
I assume you've overridden the system default for 1024 files in ulimit for # of open files? I don't see the times you're seeing using this program...though my databases are empty which probably makes a difference. I do see the gradual increase in time...I think this is almost all due to the OS

Re: [sqlite] sqlite3_open_v2 performance degrades as number of opens increase

2011-09-02 Thread Eric Minbiole
My first thought would be to check the amount of memory being used by your many connections. Each connection will consume a non-trivial amount of resources (page cache, file handles, OS file cache, etc.) It's certainly plausible that your overall system performance is reduced as you run out of phy

Re: [sqlite] sqlite3_open_v2 performance degrades as number of opens increase

2011-09-02 Thread Simon Slavin
On 2 Sep 2011, at 12:30pm, Terry Cumaranatunge wrote: > We have an application that creates many small databases (over 100K) to be > able to control the size of the database and provide more deterministic > performance. At process startup, it opens many of these databases in a loop > to keep them

[sqlite] sqlite3_open_v2 performance degrades as number of opens increase

2011-09-02 Thread Terry Cumaranatunge
Hello, We have an application that creates many small databases (over 100K) to be able to control the size of the database and provide more deterministic performance. At process startup, it opens many of these databases in a loop to keep them open for faster transaction response times. The behavio