Re: [sqlite] questions about performance

2006-04-13 Thread Miha Vrhovnik
>This problem has been coming up with more and more frequency. >I need to either figure out a solution or at least write up >some official documentation on it. I think that table partitioning as I purposed about a month ago would be a solution if data is of nature, that it can be partitioned

Re: [sqlite] questions about performance

2006-04-12 Thread Jerome Alet
Hello, On Wed, Apr 12, 2006 at 09:02:34PM -0400, [EMAIL PROTECTED] wrote: > Jerome Alet <[EMAIL PROTECTED]> wrote: > > > > SQLite is way faster than the two other databases at least when the > > number of records is reasonable, but when the database reaches > > around 300 Mb which is something

Re: [sqlite] questions about performance

2006-04-12 Thread Joe Wilson
> Thanks for the link. But InnoDB uses the same basic architecture > as SQLite. So this does not explain why performance drops off > on SQLite but not on InnoDB. > > Still looking Does SQLite perform this InnoDB optimization? "Dirty (changed) database pages are not immediately sent to the

Re: [sqlite] questions about performance

2006-04-12 Thread drh
Joe Wilson <[EMAIL PROTECTED]> wrote: > http://en.wikipedia.org/wiki/InnoDB > > --- [EMAIL PROTECTED] wrote: > > I'm really curious to know how PostgreSQL and MySQL avoid this > > thrashing problem. Does anybody have any insight on this? > Thanks for the link. But InnoDB uses the same basic

RE: [sqlite] questions about performance

2006-04-12 Thread Richard Dale
> I'm really curious to know how PostgreSQL and MySQL avoid this > thrashing problem. Does anybody have any insight on this? MySQL (in the InnoDB engine) reduces thrashing through clustering of data in (roughly) primary key order. This helps if you obtain data with similar primary keys. It

Re: [sqlite] questions about performance

2006-04-12 Thread Joe Wilson
http://en.wikipedia.org/wiki/InnoDB --- [EMAIL PROTECTED] wrote: > I'm really curious to know how PostgreSQL and MySQL avoid this > thrashing problem. Does anybody have any insight on this? __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the

Re: [sqlite] questions about performance

2006-04-12 Thread Jay Sprenkle
On 4/12/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > > I'm really curious to know how PostgreSQL and MySQL avoid this > thrashing problem. Does anybody have any insight on this? The big databases do it by moving the database code to a separate server machine. It has it's own cache and

Re: [sqlite] questions about performance

2006-04-12 Thread spaminos-sqlite
<[EMAIL PROTECTED]> wrote: > The problem (I suspect) is that you have an index on Table3. > As you insert to records into Table3, those record go at the > end, which is very efficient. But the index entries have to > be inserted in index order, which means they get scattered > out all through

Re: [sqlite] questions about performance

2006-04-12 Thread drh
Jerome Alet <[EMAIL PROTECTED]> wrote: > > SQLite is way faster than the two other databases at least when the > number of records is reasonable, but when the database reaches > around 300 Mb which is something like 400 records in Table3, > it slows down dramatically (in fact the slowdown is

Re: [sqlite] questions about performance

2006-04-12 Thread drh
Jerome Alet <[EMAIL PROTECTED]> wrote: > > SQLite is way faster than the two other databases at least when the > number of records is reasonable, but when the database reaches > around 300 Mb which is something like 400 records in Table3, > it slows down dramatically (in fact the slowdown is

Re: [sqlite] questions about performance

2006-04-12 Thread Will Leshner
On 4/12/06, Jerome Alet <[EMAIL PROTECTED]> wrote: > However any of these command line tools MUST be interruptible > with Ctrl+C (for example) by the user who launched them, for > example because of a typo in the command, or something like that. If you are using SQLite directly, you might want

RE: [sqlite] questions about performance

2006-04-12 Thread Richard Dale
What you're seeing is the effect of file system caching buffers running out on the operating system (used for write caching, caching of indices, caching of the transaction log file etc.) and the PC having to go to disk. We had this exact same issue with regards to our stock price databases. All

Re: [sqlite] questions about performance

2006-04-12 Thread Jerome Alet
On Wed, Apr 12, 2006 at 04:27:29PM -0700, Will Leshner wrote: > On 4/12/06, Jerome Alet <[EMAIL PROTECTED]> wrote: > > > All the test program does in the mainloop is a bunch of INSERT > > queries all on the same table, but all INSERTS (millions of them) > > are done within a single SQL

Re: [sqlite] questions about performance

2006-04-12 Thread Will Leshner
On 4/12/06, Jerome Alet <[EMAIL PROTECTED]> wrote: > All the test program does in the mainloop is a bunch of INSERT > queries all on the same table, but all INSERTS (millions of them) > are done within a single SQL transaction. Are you doing millions of inserts in one transaction?

[sqlite] questions about performance

2006-04-12 Thread Jerome Alet
Hi there, I've added support for SQLite 3 in my PyKota print accounting software. This software supports PostgreSQL, MySQL and SQLite as its database backend, and so it's easy to do some benchmarking, since in all cases the very same SQL queries are executed (I know this **may not** be the