Re: [sqlite] Sqlite insertion performance

2007-09-18 Thread drh
John Stanton <[EMAIL PROTECTED]> wrote: > I haven't looked at how Sqlite manages its cache, so my suggestion may > be irrelevant but we have had success in such a cacheing situation by > crafting the cache with pages organized in most recently used order. > The least recently used pages is the

Re: [sqlite] Sqlite insertion performance

2007-09-18 Thread John Stanton
[EMAIL PROTECTED] wrote: "Paul Harris" <[EMAIL PROTECTED]> wrote: A thought along the same lines, can sqlite create a unique index that is hash-based? this would provide the UNIQUE support, but it wouldn't provide a sorted index. That should resolve the massive-insert-too-slow problem, and

Re: [sqlite] Sqlite insertion performance

2007-09-18 Thread drh
"Paul Harris" <[EMAIL PROTECTED]> wrote: > > > A thought along the same lines, can sqlite create a unique index that > is hash-based? this would provide the UNIQUE support, but it wouldn't > provide a sorted index. > > That should resolve the massive-insert-too-slow problem, and > afterwards

Re: [sqlite] Sqlite insertion performance

2007-09-17 Thread Paul Harris
On 17/09/2007, John Machin <[EMAIL PROTECTED]> wrote: > On 17/09/2007 1:07 PM, Joe Wilson wrote: > > --- "D. Richard Hipp" <[EMAIL PROTECTED]> wrote: > > I have been struggling with the performance of insertion in sqlite. > > > > Here we have a very simple case : > > > > A

Re: [sqlite] Sqlite insertion performance

2007-09-17 Thread John Machin
On 17/09/2007 1:07 PM, Joe Wilson wrote: --- "D. Richard Hipp" <[EMAIL PROTECTED]> wrote: I have been struggling with the performance of insertion in sqlite. Here we have a very simple case : A table with an integer autoincrement primary key and a text field that is unique. CREATE TABLE

Re: [sqlite] Sqlite insertion performance

2007-09-16 Thread Joe Wilson
make that: pragma default_cache_size=[half your machine RAM in bytes / page_size]; --- Joe Wilson <[EMAIL PROTECTED]> wrote: > You don't have to recompile. > Just make a new database file with: > > pragma page_size=32768; > pragma default_cache_size=[your machine RAM in bytes / 32768]; > >

Re: [sqlite] Sqlite insertion performance

2007-09-16 Thread Joe Wilson
You don't have to recompile. Just make a new database file with: pragma page_size=32768; pragma default_cache_size=[your machine RAM in bytes / 32768]; --- RaghavendraK 70574 <[EMAIL PROTECTED]> wrote: > But there is a limit (3.4.0) which stops at 32KB. > the compile macro would do?

Re: [sqlite] Sqlite insertion performance

2007-09-16 Thread RaghavendraK 70574
, please notify the sender by phone or email immediately and delete it! * - Original Message - From: Joe Wilson <[EMAIL PROTECTED]> Date: Monday, September 17, 2007 12:11 pm Subject: Re: [sqlite]

Re: [sqlite] Sqlite insertion performance

2007-09-16 Thread Joe Wilson
--- RaghavendraK 70574 <[EMAIL PROTECTED]> wrote: > But with large cache there is too much fragmentation observered (pgin/pgout) > over a period of time. Do u see this behaviour? Fragmentation is not a function of page cache size, but yes, I also see this fragmentation if the secondary index

Re: [sqlite] Sqlite insertion performance

2007-09-16 Thread RaghavendraK 70574
Wilson <[EMAIL PROTECTED]> Date: Monday, September 17, 2007 11:07 am Subject: Re: [sqlite] Sqlite insertion performance > --- "D. Richard Hipp" <[EMAIL PROTECTED]> wrote: > > >>> > > >>> I have been struggling with the performance of insertion in

Re: [sqlite] Sqlite insertion performance

2007-09-16 Thread Joe Wilson
--- "D. Richard Hipp" <[EMAIL PROTECTED]> wrote: > >>> > >>> I have been struggling with the performance of insertion in sqlite. > >>> > >>> Here we have a very simple case : > >>> > >>> A table with an integer autoincrement primary key and a text > >>> field that is > >>> unique. > >>> > >>>

Re: [sqlite] Sqlite insertion performance

2007-09-15 Thread Kefah T. Issa
Dear Kees and Richard, Much appreciated. I tried the ordered-urls-insert the results were better, but it is still taking progressively longer time as the number of records increases. A fundamental question to be asked here : Shouldn't the time complexity (Big-O) of the insert operation be

Re: [sqlite] Sqlite insertion performance

2007-09-15 Thread D. Richard Hipp
I have been struggling with the performance of insertion in sqlite. Here we have a very simple case : A table with an integer autoincrement primary key and a text field that is unique. CREATE TABLE my (id PRIMARY KEY, url); CREATE UNIQUE INDEX myurl ON my(url); My application requires

Re: [sqlite] Sqlite insertion performance

2007-09-15 Thread Kees Nuyt
Hi Kefah, On Sat, 15 Sep 2007 04:43:46 +0300, you wrote: >Thanks Kees, > >In fact using integer instead of string gives very similar results. > >Dropping the unique index constraint and then creating it again when needed >sounds interesting, as insertion without the unique index is satisfactory

Re: [sqlite] Sqlite insertion performance

2007-09-14 Thread Kefah T. Issa
Thanks Kees, In fact using integer instead of string gives very similar results. Dropping the unique index constraint and then creating it again when needed sounds interesting, as insertion without the unique index is satisfactory and constact over time. I tried this, but got a trivial

Re: [sqlite] Sqlite insertion performance

2007-09-14 Thread Kees Nuyt
On Fri, 14 Sep 2007 23:20:53 +0300, you wrote: > >Dear All, > >I have been struggling with the performance of insertion in sqlite. > >Here we have a very simple case : > >A table with an integer autoincrement primary key and a text field that is >unique. > >CREATE TABLE my (id PRIMARY KEY,