Hi Adam, hi all, Adam Kennedy wrote: > There is an expected but never before observed bug to do with locking > and threads.
Yummy. > ORLite in default mode runs every SQL statement as a separate > connection. The connectionless nature means we don't have to care > about the problem of DBI being incompatible with threads. > > Unfortunately, now we have this great locking system. > > And the locking system is going to run persistant connections to the > database, which might end up surviving across the thread fork, and > thus exploding in the manner you see. DBI isn't really considered thread safe by Tim Bunce, either. > Because Padre only uses transactions as a convenience to speed up > statements and control when the disk blocking will occur, we can use > the following solution. > > 1. Make the Task Manager aware that the locking system exists. > > 2. Add special code to the lock manager so that when the Task Manager > is about to fork off a thread, any current database connection is > committed, and then disconnected. Note: New threads should be spawned fairly rarely: o(tens/hundreds of seconds). And doing so is already hideously expensive. > 3. After the thread is spawned, the Task Manager should return the DB > lock to the state it was before the thread spawn (either connected, or > not connected). > > 4. The above should have very very tight so there's no chance for > anything other than the thread spawn to occur (other code might screw > up the lock state). I sense danger. Let me get a few bits straight: 1) Can threads other than the main thread access the DB? I think yes. Barring them from using the configuration would be a problem. => Therefore, your point 4. isn't enough. It's asking for strange race conditions. 2) Can ORLite have multiple connections to the same DB? => I guess not. 3) Can SQLite handle multiple connections (i.e. from different processes) to the same DB? => I guess yes. 4) Can DBD::SQLite handle multiple connections from the same process to the same DB? => No idea. I was thinking that maybe each thread should/could handle its own DB connection. ORLite is pure Perl, so by just spawning off a new ithread, it should be safe unless it holds a db handle. The db handle itself is an opaque XS object (DBI, I presume). Now, let me go on a wild rampage: If we create a class that trivially inherits from DBI and use that class in place of DBI everywhere. Then we can use XS to add a thread cloning hook to our new class. When a new thread is spawned, this hook simply creates a new DB connection for the new thread -- an entirely new DBI handle. Now, there are a few drawbacks to this plan: a) It's a lot of work and potentially fragile. b) DBI docs IIRC say something like "should be fine in threads, but having handles to the same DB in different threads might be weird", i.e. warn about this scenario specifically. c) DBD's are usually not thread-safe. This could manifest in that they use some C-level storage for the db handles. This in turn would make it impossible to have separate connections to the same per thread. I have no idea about DBD::SQLite, but if in doubt, it's probably not up to par with the likes of DBD::Pg/mysql/Oracle. Okay, so let's consider this a thought experiment for now and move on. Back to Adam' suggestion One could make this thread-safe-ish by using a global semaphore (Thread::Semaphore or whatever) in the Locking system to synchronize all interaction with the database. This may be a problem for applications that hit the DB a lot, but feels safe for us. Suppose I want to access the config DB. What's the way this works? Is it: a) I get a lock on it b) I use the DB c) lock goes out of scope d) other bits of code could use the DB? If so, we're in luck. Just add the afortmentioned semaphore to the DB locking. AND, of course, we need to implement Adam's plan. > 5. The database operations should occur WITHOUT changing the > underlying state (lock counts etc) of the locking system. Of course. I won't be able to do any hacking before the week-end and even then, it looks shaky. Best regards, Steffen _______________________________________________ Padre-dev mailing list [email protected] http://mail.perlide.org/mailman/listinfo/padre-dev
