Hi, just my few cents: if you are using the RTree module I made very bad experiences running SQLite in parallel, even when running only two or three threads in parallel. In this case I use a single thread-safe queue that is handling all SQLite access.
Regards, Hartwig > Am 2017-03-04 um 16:54 schrieb James K. Lowden <jklow...@schemamania.org>: > > On Fri, 3 Mar 2017 20:13:52 +0000 > Andrew Brown <andrew.br...@economicmodeling.com> wrote: > >> once we start running them on a large server with a lot of >> multithreading going on, I find that we spend a lot of time in >> __raw_spin_lock - perhaps 75%+ of the time (In one case, 87% of 350 >> seconds x 72 cores was spent in __raw_spin_lock). This is being run >> on 64 and 72 core machines, and the more cores I run it on, the >> slower it ends up going. > > My initial reaction is that your 72 cores are apt to block on I/O > contention. Your description is consistent with that hypothesis. > > Whatever else you do, I recommend SQLightning. It's a version of > SQLite that uses a memory-mapped key-value store, LMDB. It's designed > for problems like yours, and demonstrated to sometimes be 10-100x > faster. > > You don't mention RAM size. Even if your data fit in memory, though, > you still have to pass through the kernel to retrieve each row. > Memory-mapped files bring the data into userspace. > > https://github.com/LMDB/sqlightning > > --jkl > _______________________________________________ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
