Re: [sqlite] populating a table as quickly as possible

2013-02-05 Thread Paul Mathieu
This is a fast way to load data. sqlite3.exe C:\\Your Dir\\dbname.db

Re: [sqlite] populating a table as quickly as possible

2013-02-05 Thread Paul Sanderson
Currently ading 5000 at a time, will try increasing to 20 and 50K and see what happens. Just one table - but interesting On 4 February 2013 18:24, Dominique Pellé wrote: > Paul Sanderson wrote: > > > I want to populate a large table (millions of rows) as quickly as

Re: [sqlite] populating a table as quickly as possible

2013-02-04 Thread Dominique Pellé
Paul Sanderson wrote: > I want to populate a large table (millions of rows) as quickly as possible, > > The data set will not be operated on until the table is fully populated and > if the operation fails I will be starting again. the operation is a one off > and the table will not be added to at

Re: [sqlite] populating a table as quickly as possible

2013-02-04 Thread Nigel Verity
Hi I had a similar issue a few months ago, only it was several hundred thousand records rather than millions. I found that dividing the job into smaller transactions did definitely speed things up considerably. In my case I think I limited each transaction to 20,000 records. I would

Re: [sqlite] populating a table as quickly as possible

2013-02-03 Thread Simon Slavin
On 4 Feb 2013, at 12:02am, Paul Sanderson wrote: > currently using : > journal_mode = off > page_size=16386 > cache_size = 1 > synchronous = off Reasonable. > I load lots of similar data sets (each into a separate db) and load time is > definitely an issue.

Re: [sqlite] populating a table as quickly as possible

2013-02-03 Thread Paul Sanderson
currently using : journal_mode = off page_size=16386 cache_size = 1 synchronous = off I load lots of similar data sets (each into a separate db) and load time is definitely an issue. There are processing delays and loading a db can take 30+ minutes, if I can shave off even a few minutes on

Re: [sqlite] populating a table as quickly as possible

2013-02-03 Thread Dominique Pellé
Simon Slavin wrote: > > On 3 Feb 2013, at 6:34pm, Paul Sanderson wrote: > >> I want to populate a large table (millions of rows) as quickly as possible, >> >> The data set will not be operated on until the table is fully populated and >> if