At 06:48 05/06/2012, you wrote:
Have a market data extractor running tick tapes and saving data to sqlite db.
All coded in C++ and previously ran fine and fast on bare metal.
Processes tick tapes from NAS and saved to local sqlite db's on c drive.
We moved program onto VMWare VM's... same nas tick tapes. Writing
db's to a share.
On a 4 core vm we can run 3 extractors concurrently at 25 - 75% CPU
utilization. The 4th pegs the CPU at 100%
Same on an 8 core VM ... seems we can run Cores - 1 without pegging the CPU.
The memory footprint for the program is a tiny 9mb... On bare metal
we can run 20 extractors concurrently. Looking for suggestions on
how to optimize for VMware.
So you want to minimize the cpu usage? It looks like your os is
running on the last core, it has work to do too ;) Apart form using
directly a hard disk instead a vdmk virtual hard disk, i don't know
other tricks on vmware sorry.
Did you create the indexs before the mass inserts? If yes, sqlite is
reindexing after your 10.000 inserts. Create the tables without any
indexes, and recreate them after the bulk insert. Set pragma
automatic_index=FALSE too to disable automatic index creation on
primary key on the not indexed table.
Calculate how much database size increase happens when you insert the
10.000 rows. If you are using wal mode, set wal file size to this
size plus a bit more. This way only one wal file will be used. Other
journal modes (try off mode) may be faster but riskier.
9MB of memory, perhaps the system is copying data between kernel
structures and sqlite, like disk cache entries (I don't know what os
are you using, it's a blind shoot) try to increase the pragma cache page.
Increase page size to 8K-32K, this way sqlite will reduce the amount
of administrative internal work on pages.
Of course, read the documentation, and if you have any cuestion on it ask.
_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
