As an aside, you may likely find that the OS will schedule multiple processes far more efficiently than it schedules multiple threads (especially from the perspective of scheduling the cores independently). You may find that it is far more efficient to perhaps schedule a dozen processes with six to a dozen worker threads each (or whatever number seems to be optimal) and use some kind of IPC to submit workload to the individual processes.
Also, make sure the database is in WAL journal mode -- this will help even if you are only running read transactions + if you have enough memory make sure the page cache and temp space is big enough on each connection and force temp work into memory - but do not overcommit memory. > -----Original Message----- > From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] > On Behalf Of Andrew Brown > Sent: Friday, 3 March, 2017 18:24 > To: SQLite mailing list > Subject: Re: [sqlite] Massively multithreaded SQLite queries > > Now this is a interesting question. As it happens I /am/ blindly assuming > the os would schedule it properly. I will look at my options this evening > and see about addressing that. > > On Mar 3, 2017 4:52 PM, Keith Medcalf <kmedc...@dessus.com> wrote: > > No, the good rule of thumb is to allocate one thread per CPU. Depending > on the particular multi-core CPU you "may" be able to use all the cores > simultaneously but in many cases diminishing returns will set in long > before you can execute one thread per core. If this is an Intel processor > that claims it has more than one thread per core be *extremely* careful as > that will give you one thread and one half-assed thread per core. > Sometimes, half-assed cores are presented as real cores when they lack a > separate execution unit. Be vary wary. > > I presume you are forcing separate threads to separate cores by setting > processor affinity on the threads and not just blindly hoping that the OS > scheduler does "the right thing"? > > > -----Original Message----- > > From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] > > On Behalf Of Jens Alfke > > Sent: Friday, 3 March, 2017 17:19 > > To: SQLite mailing list > > Subject: Re: [sqlite] Massively multithreaded SQLite queries > > > > > > > On Mar 3, 2017, at 3:51 PM, Simon Slavin <slav...@bigfraud.org> wrote: > > > > > > Then mess with the '5' until you find a good value. > > > > A common rule of thumb with thread pools is to allocate one thread per > CPU > > core. > > > > —Jens > > _______________________________________________ > > sqlite-users mailing list > > sqlite-users@mailinglists.sqlite.org > > > https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailinglis > ts.sqlite.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fsqlite- > users&data=01%7C01%7Candrew.brown%40economicmodeling.com%7Ca4729ccfb1754b9 > a534008d46298b2d9%7C7cc1677566a34e8b80fd5b1f1db15061%7C0&sdata=C6yNkbM7p0u > oib5OgXXYS7%2BEy8cJR1I6N8JjaQ4iMCU%3D&reserved=0 > > > > _______________________________________________ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailinglis > ts.sqlite.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fsqlite- > users&data=01%7C01%7Candrew.brown%40economicmodeling.com%7Ca4729ccfb1754b9 > a534008d46298b2d9%7C7cc1677566a34e8b80fd5b1f1db15061%7C0&sdata=C6yNkbM7p0u > oib5OgXXYS7%2BEy8cJR1I6N8JjaQ4iMCU%3D&reserved=0 > > _______________________________________________ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
