On Fri, 3 Mar 2017 20:13:52 +0000 Andrew Brown <andrew.br...@economicmodeling.com> wrote:
> once we start running them on a large server with a lot of > multithreading going on, I find that we spend a lot of time in > __raw_spin_lock - perhaps 75%+ of the time (In one case, 87% of 350 > seconds x 72 cores was spent in __raw_spin_lock). This is being run > on 64 and 72 core machines, and the more cores I run it on, the > slower it ends up going. My initial reaction is that your 72 cores are apt to block on I/O contention. Your description is consistent with that hypothesis. Whatever else you do, I recommend SQLightning. It's a version of SQLite that uses a memory-mapped key-value store, LMDB. It's designed for problems like yours, and demonstrated to sometimes be 10-100x faster. You don't mention RAM size. Even if your data fit in memory, though, you still have to pass through the kernel to retrieve each row. Memory-mapped files bring the data into userspace. https://github.com/LMDB/sqlightning --jkl _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
