[sqlite] Sqlite Insert Speed Optimization

2010-07-13 Thread Werner Smit
Hi there. I've been playing around with sqlite. Very impressed so far. Using 3.5.6 in windows developing with Clarion. My question(s) If I want to use the "insert or replace" to populate my database of around 1 million records. And I want to do it as fast as possible. What are all the tricks I

Re: [sqlite] Sqlite Insert Speed Optimization

2010-07-14 Thread Werner Smit
-Original Message- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of John Drescher Sent: Tuesday, July 13, 2010 12:37 PM To: General Discussion of SQLite Database Subject: Re: [sqlite] Sqlite Insert Speed Optimization > I also wrap my statements

Re: [sqlite] Sqlite Insert Speed Optimization

2010-07-14 Thread Werner Smit
Thanks for all the feedback! It helped a lot. 1. I'm going to try and see what happen if I leave the "end transaction" until 5 insert was done. 2. I'm going to increase cache_size from 8192 to 16384 Extra info, 1. This program saved data to a clarion file before and in sqlite it's about

Re: [sqlite] Sqlite Insert Speed Optimization

2010-07-14 Thread Werner Smit
-Original Message- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Paul Corke Sent: 14 July 2010 01:03 PM To: General Discussion of SQLite Database Subject: Re: [sqlite] Sqlite Insert Speed Optimization On 14 July 2010 11:56, Werner Smit wrote

Re: [sqlite] Sqlite Insert Speed Optimization

2010-07-14 Thread Werner Smit
>>On 14 Jul 2010, at 11:55am, Werner Smit wrote: >> ps. When I started with sqlite it took 500 minutes to save the 1 million >> records. >> I've got it down to just less than 200 minutes with current settings. >> Clarion does it in between 100 and 200 minutes. &g

Re: [sqlite] EXTERNAL:Re: Sqlite Insert Speed Optimization

2010-07-14 Thread Werner Smit
>According to my math your final database size should be on the order of 100Meg? > >That means at 200 minutes and 1,000,000 records: >83 inserts per second >8333 bytes per second >Both of these values are terrible. >#1 What kind of network connection do you have? 100BaseT? >#2 What kind of

Re: [sqlite] Sqlite Insert Speed Optimization

2010-07-14 Thread Werner Smit
>>According to my math your final database size should be on the order of 100Meg? >> >>That means at 200 minutes and 1,000,000 records: >>83 inserts per second >>8333 bytes per second >>Both of these values are terrible. >>#1 What kind of network connection do you have? 100BaseT? >>#2 What

Re: [sqlite] EXTERNAL:Re: Sqlite Insert Speed Optimization

2010-07-14 Thread Werner Smit
On 14 July 2010 17:00, Werner Smit wrote: > It DOES sound terrible since 90%? of the time is spend in retrieving > data from a remote oracle server over a slow line. I think you're trying to optimise the wrong thing :) Assuming you can't upgrade that slow line, how about running a comp

Re: [sqlite] Sqlite Insert Speed Optimization

2010-07-14 Thread Werner Smit
If you don't know why it's slow you'll be shooting in the dark. And doing compression on a local network link isn't likely to buy you much. Might even hurt. In other words, is it latency or bandwidth? Give 8K/sec I'm guessing it's latency unless you're running a 64KBit line. Are you THAT

[sqlite] (no subject)

2012-06-07 Thread Werner Smit
___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] List manager webpage unavailible.

2012-06-07 Thread Werner Smit
The List manager webpage cannot be loaded at the moment. http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users (as specified at the bottom of every mail) How can I change my subscription parameters without going there? In the old days you could send a blank email to a mailing list, and