Right - if we make sqlite works with LocalExecutor, there is no reason to keep Sequential Executor :).
J. On Wed, Oct 7, 2020 at 11:26 AM Ash Berlin-Taylor <[email protected]> wrote: > Oh good point. > > I'll take a look -- I think our "don't use SQLite from more than one > process" is over-zealous, as SQLite has built in locking and can be used by > multiple processes at the same time, with a few caveats. > > http://www.sqlite.org/draft/faq.html#q5 > > Multiple processes can have the same database open at the same time. > Multiple processes can be doing a SELECT at the same time. But only one > process can be making changes to the database at any moment in time, > however. > > SQLite uses reader/writer locks to control access to the database. (Under > Win95/98/ME which lacks support for reader/writer locks, a probabilistic > simulation is used instead.) But use caution: this locking mechanism might > not work correctly if the database file is kept on an NFS filesystem. This > is because fcntl() file locking is broken on many NFS implementations. You > should avoid putting SQLite database files on NFS if multiple processes > might try to access the file at the same time. On Windows, Microsoft's > documentation says that locking may not work under FAT filesystems if you > are not running the Share.exe daemon. People who have a lot of experience > with Windows tell me that file locking of network files is very buggy and > is not dependable. If what they say is true, sharing an SQLite database > between two or more Windows machines might cause unexpected problems. > > We are aware of no other *embedded* SQL database engine that supports as > much concurrency as SQLite. SQLite allows multiple processes to have the > database file open at once, and for multiple processes to read the database > at once. When any process wants to write, it must lock the entire database > file for the duration of its update. But that normally only takes a few > milliseconds. Other processes just wait on the writer to finish then > continue about their business. Other embedded SQL database engines > typically only allow a single process to connect to the database at once. > > > So it's doable, we've just been overly cautious in the past. > > You're right that the change isn't just removing the executor though! I > think worth it overall though. > > -ash > > On Oct 7 2020, at 10:07 am, Jarek Potiuk <[email protected]> wrote: > > How about sqlite? I believe it only runs with Sequential Executor? > > On Wed, Oct 7, 2020 at 10:59 AM Ash Berlin-Taylor <[email protected]> wrote: > > Hi everyone, > > I've just had a thought: the sequential executor is gives an all around > pretty bad experience (it blocks the scheduler, you'll see "scheduler > stopped heartbeating" messages if your task run takes a while. > > So I'd like to propose we change the default executor to LocalExecutor -- > to do this we should probably change the default number of slots/processes > from 16 to num cpus. > > Thoughts? > > None of this has to happen for 2.0 (I don't have time to do it), but just > wanted to suggest it. > > -ash > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
