On 06/06/2012 7:45 AM, Simon Slavin wrote:
On 6 Jun 2012, at 12:00pm, IQuant<sql...@iquant.co.cc> wrote:
We need to be able to run 1000's of extractors concurrently processing
different tick tapes and symbol sets. aka service bureau. The Daily
tick tapes are approx 20gb each.. 30TB repository and growing. An
extraction run take 1 - 5 minutes for small symbol sets per tape.
An example would be to concurrently extract 5 years of a particular
stock's tick data. 1500 days x 2 minute extraction jobs across 15 VMs
each running 100 extractors.
I'll try journal mode off and increasing page size.
Your problem appears to relate to your virtualisation layer. You yourself
reported
"previously ran fine and fast on bare metal."
I see no indication that SQLite is doing anything weird here. The only problem
you're reporting is 100% CPU utilisation which is not, of itself, a problem.
It just means that CPUs are being utilised very thoroughly which is good in a
process that munches through lots of data.
Unless pegging the CPU causes the hypervisor to preempt the guest's
synchronization primitives and cause priority inversion, in which case
OP's throughput would plummet. If throughput remains reasonably stable
in the VM, though, I would agree that a pegged CPU isn't necessarily a
problem. Another possibility: the guest OS idle loop will probably count
as CPU usage on the host, and with heavy load the guest won't
necessarily attempt to park the cores it was using in between scheduling
decisions; on bare metal, the spinning idle loop doesn't get reported as
CPU usage, even though it is.
Ryan
_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
