Oh good point. I'll take a look -- I think our "don't use SQLite from more than one process" is over-zealous, as SQLite has built in locking and can be used by multiple processes at the same time, with a few caveats. http://www.sqlite.org/draft/faq.html#q5 > Multiple processes can have the same database open at the same time. Multiple > processes can be doing a SELECT at the same time. But only one process can be > making changes to the database at any moment in time, however. > SQLite uses reader/writer locks to control access to the database. (Under > Win95/98/ME which lacks support for reader/writer locks, a probabilistic > simulation is used instead.) But use caution: this locking mechanism might > not work correctly if the database file is kept on an NFS filesystem. This is > because fcntl() file locking is broken on many NFS implementations. You > should avoid putting SQLite database files on NFS if multiple processes might > try to access the file at the same time. On Windows, Microsoft's > documentation says that locking may not work under FAT filesystems if you are > not running the Share.exe daemon. People who have a lot of experience with > Windows tell me that file locking of network files is very buggy and is not > dependable. If what they say is true, sharing an SQLite database between two > or more Windows machines might cause unexpected problems. > We are aware of no other embedded SQL database engine that supports as much > concurrency as SQLite. SQLite allows multiple processes to have the database > file open at once, and for multiple processes to read the database at once. > When any process wants to write, it must lock the entire database file for > the duration of its update. But that normally only takes a few milliseconds. > Other processes just wait on the writer to finish then continue about their > business. Other embedded SQL database engines typically only allow a single > process to connect to the database at once.
So it's doable, we've just been overly cautious in the past. You're right that the change isn't just removing the executor though! I think worth it overall though. -ash On Oct 7 2020, at 10:07 am, Jarek Potiuk <[email protected]> wrote: > How about sqlite? I believe it only runs with Sequential Executor? > > On Wed, Oct 7, 2020 at 10:59 AM Ash Berlin-Taylor <[email protected] > (mailto:[email protected])> wrote: > > Hi everyone, > > > > I've just had a thought: the sequential executor is gives an all around > > pretty bad experience (it blocks the scheduler, you'll see "scheduler > > stopped heartbeating" messages if your task run takes a while. > > So I'd like to propose we change the default executor to LocalExecutor -- > > to do this we should probably change the default number of slots/processes > > from 16 to num cpus. > > Thoughts? > > None of this has to happen for 2.0 (I don't have time to do it), but just > > wanted to suggest it. > > -ash > > -- > > > Jarek Potiuk > Polidea (https://www.polidea.com/) | Principal Software Engineer > > > > > > > > M: +48 660 796 129 (tel:+48660796129) > > > > > > > > > > > > > > > > >
