My guess would be the OS slowing things down with write caching. The system will hold so much data in memory as a cache to write to the disk, and when the cache gets full, the OS slows down and waits on the HDD. Try doing a [dd] to a few gig worth of random data and see if you get the same kind of slow down.
On Fri, Dec 28, 2012 at 4:42 PM, Michael Black <mdblac...@yahoo.com> wrote: > Perhaps the rowid index cache gets too big? I assume you don't have any > indexes of your own? > > Does the knee change if you say, double your cache_size? > > Default should be 2000; > > pragma cache_size=4000; > > > -----Original Message----- > From: sqlite-users-boun...@sqlite.org > [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Dan Frankowski > Sent: Friday, December 28, 2012 3:34 PM > To: sqlite-users@sqlite.org > Subject: [sqlite] Fwd: Write performance question for 3.7.15 > > I am running a benchmark of inserting 100 million (100M) items into a > table. I am seeing performance I don't understand. Graph: > http://imgur.com/hH1Jr. Can anyone explain: > > 1. Why does write speed (writes/second) slow down dramatically around 28M > items? > 2. Are there parameters (perhaps related to table size) that would change > this write performance? > > ======================= > > Create and insert statements: > > create table if not exists t_foo ( > key binary(16) primary key, > value binary(16)); > > insert or replace into t_foo (key, value) values (?, ?) > > key and value are each 16-byte arrays. > > I turn auto-commit off and commit every 1000 inserts. > I set synchronous mode to OFF and journaling mode to WAL (write-ahead log). > > I am using sqlite 3.7.15 through the Xerial JDBC driver (see > https://bitbucket.org/xerial/sqlite-jdbc). I built it myself, due to a > glibc incompatibility (see > https://groups.google.com/d/msg/Xerial/F9roGuUjH6c/6RuxqmG6UK4J). > > I am running on Gentoo. Output of uname -a: > > Linux mymachine 3.2.1-c42.31 #1 SMP Mon Apr 30 10:55:12 CDT 2012 x86_64 > Quad-Core AMD Opteron(tm) Processor 1381 AuthenticAMD GNU/Linux > > It has 8G of memory. > _______________________________________________ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > > _______________________________________________ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
