On 7 Jun 2012, at 12:59am, IQuant <sql...@iquant.co.cc> wrote: > Running a single extractor on a VM runs near bare metal speeds. I can > run concurrent extractors up to the # of cores allocated to the VM and > sustain decent throughput. Once I go above this count the VM's get > crippled.. Pegged at 100% - Disk, RAM and LAN loads are miniscule. > > For a test we ran 50 extractors across 2 VM's (8 core and 4 core) and > it took 14 hours to extract a single symbol. A single exctractor > running on one 4 core VM can do the job in 50 hours. The same run > limited to running 12 extractors finished in 4 hours.
That's entirely in line with how VMWare works. One core = one piece of work. Your problem, if it is a problem, is with VMWare, not SQLite. Maybe an expert on VMWare can suggest a change in configuration which would make it do process-switching, but that still won't reduce the time taken below the 14 hour count. > There is obviously an issue with the virtualization but no clear fix > or workaround were aware of. Our objective is to extract and process > all tick tapes in one shot. the task you're doing requires more hardware than you have available. Get more cores or, as you've already found out, limit your current extractions to the number of cores you have available. Or both. Simon. _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
