No indexes... create db and table structure - process tick tapes inserting records - than create index. Windows 2008 R2 Servers ... We can throw hardware at this problem.
We need to be able to run 1000's of extractors concurrently processing different tick tapes and symbol sets. aka service bureau. The Daily tick tapes are approx 20gb each.. 30TB repository and growing. An extraction run take 1 - 5 minutes for small symbol sets per tape. An example would be to concurrently extract 5 years of a particular stock's tick data. 1500 days x 2 minute extraction jobs across 15 VMs each running 100 extractors. I'll try journal mode off and increasing page size. > Response to: So you want to minimize the cpu usage? It looks like your os is running on the last core, it has work to do too ;) Apart form using directly a hard disk instead a vdmk virtual hard disk, i don't know other tricks on vmware sorry. Did you create the indexs before the mass inserts? If yes, sqlite is reindexing after your 10.000 inserts. Create the tables without any indexes, and recreate them after the bulk insert. Set pragma automatic_index=FALSE too to disable automatic index creation on primary key on the not indexed table. Calculate how much database size increase happens when you insert the 10.000 rows. If you are using wal mode, set wal file size to this size plus a bit more. This way only one wal file will be used. Other journal modes (try off mode) may be faster but riskier. 9MB of memory, perhaps the system is copying data between kernel structures and sqlite, like disk cache entries (I don't know what os are you using, it's a blind shoot) try to increase the pragma cache page. Increase page size to 8K-32K, this way sqlite will reduce the amount of administrative internal work on pages. Of course, read the documentation, and if you have any cuestion on it ask. _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
