Re: [sqlite] open db cx to fork(2)d children
On Mon, Jul 12, 2010 at 11:25:07PM -0700, Roger Binns wrote: > > No, just entry points into the library. What makes you think that I > > meant that every function in the library should check this? > > I also meant entry points but wasn't specific. There are a lot of them that > acquire/release mutexes and every single one would need to be modified. That > was my original approach until I hit on the idea of swizzling the mutexes > instead. There's about ~170 entry points, depending on which version of SQLite3. There's about ~110 calls to sqlite3_mutex_enter(), and about as many to drop mutexes. Some mutex enter/leave calls might occur in loops (haven't checked), but already ~170 < ~220 :) Not that such a static analysis says much. Most apps probably use a small sub-set of entry points, and estimating the number of mutex calls on average per-call to one of those entry points is non-trivial. But the point is that I would generally expect #-of-entry-points < #-of-mutex-calls -- that needn't always be true, but it's probably a good rule of thumb. Your approach does allow you to implement fork-safety without modifying SQLite3 source itself, which is pretty cool. But that's not necessarily how the SQLite3 maintainers might do it. > > but when it comes to > > POSIX file byte range locking (at least for Solaris anyways, which is > > what I know best, implementation-wise) you can assume that the only > > broken thing about them is their semantics, not their implementations. > > Having spent a lot of my commercial programming life with many different > Unix implementations, including Solaris, I do not trust them, especially for > little used functionality. And Solaris had many bugs in the past, but I > haven't touched it in the last few years to form a recent opinion. (Like > this one time when I found Sun's NFS implementation was broken based on log > messages on an HPUX machine and then doing packet capture, and Sun invented > NFS ...) > > In any event this roughly comes down to me saying to assume they are > untrustworthy until it can be substantially be proven otherwise, and you > seeming to believe that they are all correct seemingly based on > extrapolating from a recent version of Solaris. Or being a pessimist and > optimist respectively :-) > > This doesn't really matter. [...] I can't argue with detail-free anecdotes. And I'm sure there have been bugs. I do have access to the Solaris bug database, so I could search it, but by then I'd be spending an enormous amount of time just to argue a rather minor point. If you fill in some details I will do a search for the bugs in question though (since details will help narrow down the search). Nico -- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] open db cx to fork(2)d children
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 07/12/2010 08:53 PM, Nicolas Williams wrote: > On Mon, Jul 12, 2010 at 08:43:32PM -0700, Roger Binns wrote: >> Earlier you were trying to optimise out calls to getpid() and now you want >> every SQLite function source to be changed? > > No, just entry points into the library. What makes you think that I > meant that every function in the library should check this? I also meant entry points but wasn't specific. There are a lot of them that acquire/release mutexes and every single one would need to be modified. That was my original approach until I hit on the idea of swizzling the mutexes instead. > but when it comes to > POSIX file byte range locking (at least for Solaris anyways, which is > what I know best, implementation-wise) you can assume that the only > broken thing about them is their semantics, not their implementations. Having spent a lot of my commercial programming life with many different Unix implementations, including Solaris, I do not trust them, especially for little used functionality. And Solaris had many bugs in the past, but I haven't touched it in the last few years to form a recent opinion. (Like this one time when I found Sun's NFS implementation was broken based on log messages on an HPUX machine and then doing packet capture, and Sun invented NFS ...) In any event this roughly comes down to me saying to assume they are untrustworthy until it can be substantially be proven otherwise, and you seeming to believe that they are all correct seemingly based on extrapolating from a recent version of Solaris. Or being a pessimist and optimist respectively :-) This doesn't really matter. It is easy for anyone to add this functionality to SQLite by swizzling mutex pointers, does not require any changes or recompilation of SQLite, and can have whatever error behaviour desired, plus whatever micro-optimisations will make them happy. A straight forward implementation with an application that spends all its time in SQLite and with synchronous turned off (ie no disk waiting) will have a 1% performance penalty. Consider that a worst case. If enough people find this useful then there is a case for modifying SQLite itself. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkw8Br4ACgkQmOOfHg372QR9dQCeM0D2j/KxC85yvVagFpdvxVro V3QAn3eu1HQTf0/ItPiDNe/AJmytVIy1 =Cbzq -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] open db cx to fork(2)d children
On Mon, Jul 12, 2010 at 08:43:32PM -0700, Roger Binns wrote: > > They're insane (the _first_, not last, fildes close(2) in a process > > drops all locks on the underlying file), but the child won't clobber the > > parent's locks. > > That is assuming all components (C libraries, threading, compatibility > layers, operating system etc) are perfect. Given how rarely these locks are The point of support contracts is that you can assume crazy bugs like you're supposing will be fixed (and, in this case, because of conformance testing, won't happen in the first place). If we not only don't assume "perfect" components but also go so far the other way as to assume these components are uselessly broken then we might as well go home. Surely the SQLite community sells support based on this very same idea. > used, and how many different platforms SQLite runs on as well as different > vintages of those platforms, that is indeed a big gamble. ie I would want > to see evidence that they are all substantially right, ideally a way to > establish (non-destructively) at runtime if some aren't etc. The semantics of POSIX file byte range locking are well-defined. The operating systems supporting this feature have had it for very long periods of time. There are conformance test suites. There's a lot of things you might be able to suspect are broken, but when it comes to POSIX file byte range locking (at least for Solaris anyways, which is what I know best, implementation-wise) you can assume that the only broken thing about them is their semantics, not their implementations. Of course, well-tested-implementation-but-broken-design might not be very comforting :( Nico -- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] open db cx to fork(2)d children
On Mon, Jul 12, 2010 at 08:43:32PM -0700, Roger Binns wrote: > Earlier you were trying to optimise out calls to getpid() and now you want > every SQLite function source to be changed? No, just entry points into the library. What makes you think that I meant that every function in the library should check this? ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] open db cx to fork(2)d children
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 07/12/2010 02:59 PM, Nicolas Williams wrote: > You can return an error to the caller too. That would require substantial surgery to SQLite. Almost every routine acquires and releases locks so they would have to be changed along with SQLite mutex functions to deal with this (void to some sort of error code plus returning that code). Plus the changes would all need to be tested. > But if this were integrated into libsqlite3 then the entry > points could return an error. Earlier you were trying to optimise out calls to getpid() and now you want every SQLite function source to be changed? > That said, it's better to abort() Indeed. Another unmentioned constraint is that you don't want to compile two versions of SQLite - one with the checks and one without. That is why swizzling the mutexes which are already runtime changeable is an attractive solution, and doing an abort doesn't require code changes. > They're insane (the _first_, not last, fildes close(2) in a process > drops all locks on the underlying file), but the child won't clobber the > parent's locks. That is assuming all components (C libraries, threading, compatibility layers, operating system etc) are perfect. Given how rarely these locks are used, and how many different platforms SQLite runs on as well as different vintages of those platforms, that is indeed a big gamble. ie I would want to see evidence that they are all substantially right, ideally a way to establish (non-destructively) at runtime if some aren't etc. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkw74OAACgkQmOOfHg372QQ/awCePpvhLvHJL9HLRFHg8YF+rwhZ mBIAn1ceQRyx8n7d/ek7+FzzMDp/y0dH =7eLH -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] open db cx to fork(2)d children
On Mon, Jul 12, 2010 at 02:47:19PM -0700, Roger Binns wrote: > About the only correct thing to do in a process using SQLite through a fork > is to terminate the process. You can return an error to the caller too. Of course, if you're using the mutex functions to do this then it's too late, so you have to abort(). But if this were integrated into libsqlite3 then the entry points could return an error. That said, it's better to abort() as the error could be ignored by the caller and this kind of error is so serious that it should be discovered as soon as possible (it's hard to ignore an abort()). > (Even then that could mess with locks > depending on how the operating systems deals with it - POSIX locks are > insane.) They're insane (the _first_, not last, fildes close(2) in a process drops all locks on the underlying file), but the child won't clobber the parent's locks. Nico -- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] open db cx to fork(2)d children
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 07/12/2010 12:07 PM, Eric Smith wrote: > I don't think the devs would complain about that. "This code has too > few lines!" They also have to have signed paper copies dedicating the code to public domain and acknowledging the right to do so. That is far more work than just reimplementing it in the same coding style as the rest of SQLite, not to mention testing etc. (My test code is in Python, theirs is in TCL.) >> and it is an open issue as to how you report the cross fork usage should >> it happen. (In my case I know Python is being used.) > > I didn't look at it deeply, but out of curiosity: what's wrong with just > returning (or causing SQLite to return) a standard unhappy return code? Because that isn't sufficient. For example sqlite3_mutex_leave returns void and that makes sense normally. But a mutex could be acquired then a fork happens in which case you want the release in the child process to be VERY upset. Similarly sqlite3_mutex_enter returns void for similar reasons - it cannot fail, just block. About the only correct thing to do in a process using SQLite through a fork is to terminate the process. (Even then that could mess with locks depending on how the operating systems deals with it - POSIX locks are insane.) Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkw7jWAACgkQmOOfHg372QTEHgCgyrVFNqEojYCJBQogujIpGX3B ecoAn14F0afCCkuEpeybbvqlX1J86X3I =JHN4 -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] open db cx to fork(2)d children
Roger Binns wrote: > I'd have no problem contributing the code to SQLite, but there isn't > very much of it I don't think the devs would complain about that. "This code has too few lines!" > and it is an open issue as to how you report the cross fork usage should > it happen. (In my case I know Python is being used.) I didn't look at it deeply, but out of curiosity: what's wrong with just returning (or causing SQLite to return) a standard unhappy return code? Eric -- Eric A. Smith Usisahau ubaharia kwa sababu ya unahodha. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] open db cx to fork(2)d children
On Mon, Jul 12, 2010 at 01:59:30PM -0500, Nicolas Williams wrote: > The result is that you end up with a tiny penalty for fork > detection: two loads, a compare and a likely-not-taken branch. Actually three loads, two compares and two likely-not-taken branch, unless you know you always have pthread_atfork() available at run-time (as in Solaris 10 and up, whih is what I had in mind when I wrote the above), in which case you can make this two loads, one compare and one likely-not-taken branch. Nico -- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] open db cx to fork(2)d children
On Sat, Jul 10, 2010 at 11:27:41AM -0700, Roger Binns wrote: > On 07/10/2010 07:12 AM, Eric Smith wrote: > > Your wrapper is nice -- have you considered folding something like it > > into the core (disabled by default, enabled by a compile-time flag) and > > submitting it to drh for official adoption? > > DRH and the developers are aware of it as my approach was discussed on one > of the mailing lists, and I did the performance measurements in response to > their queries. > > I'd have no problem contributing the code to SQLite, but there isn't very > much of it and it is an open issue as to how you report the cross fork usage > should it happen. (In my case I know Python is being used.) Fast fork-detection logic to make the library a little more fork-safe is a very good thing to do in all otherwise-fork-unsafe libraries. getpid() might not be fast enough on all operating systems. What I'd do is make getpid() a macro like so: #if defined(HAVE_PTHREAD_ATFORK) && defined(HAVE_WEAK_SYMBOLS) extern pid_t my_saved_pid; /* set from a pthread_atfork(3C) child handler */ #define my_getpid() \ (pthread_atfork == NULL ? getpid() : my_saved_pid) #else #ifdef NO_FORK_DETECTION #define my_getpid() ((pid_t)-1) #else #define my_getpid() getpid() #endif #endif /* * Library entry points should check whether we're forked: * * if (FORKED(handle)) * return (); */ #define FORKED(handle) ((handle->owner_pid == my_getpid()) ? 0 : 1) This guarantees a fast getpid() no matter how slow the actual getpid() might be, at least on modern operating systems. Where getpid() is slow and pthread_atfork() is not available you end up disabling fork detection. The result is that you end up with a tiny penalty for fork detection: two loads, a compare and a likely-not-taken branch. Also, where the library gets initialized, it has to set a fork child handler with pthread_atfork(3C) if defined(HAVE_PTHREAD_ATFORK) && defined(HAVE_WEAK_SYMBOLS) and pthread_atfork != NULL. The handler would do nothing much more than my_saved_pid = getpid(). Where handles are created, handle->owner_pid has to be set to my_getpid(). I strongly recommend this approach for all library developers. Nico -- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] open db cx to fork(2)d children
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 07/10/2010 07:12 AM, Eric Smith wrote: > Your wrapper is nice -- have you considered folding something like it > into the core (disabled by default, enabled by a compile-time flag) and > submitting it to drh for official adoption? DRH and the developers are aware of it as my approach was discussed on one of the mailing lists, and I did the performance measurements in response to their queries. I'd have no problem contributing the code to SQLite, but there isn't very much of it and it is an open issue as to how you report the cross fork usage should it happen. (In my case I know Python is being used.) Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkw4u5kACgkQmOOfHg372QTcOACfcWCMm4xnBWegCUDhCihTH6XO YG0AoK3yw381ea7O0kS4/KZdXCjNFrra =nVfb -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] open db cx to fork(2)d children
Roger Binns wrote: > I'll bet you are actually getting exit(3) which means anything > registered with atexit will be run. (SQLite does not register with > atexit.) I knew what Nico meant :) just repeated him for expositional clarity (deeming the distinction to be unimportant for my particular question, and figuring that most people mean exit(3) when they say exit(2)). > In my wrapper I provide functionality that can check SQLite objects are > not being used across processes. > > I'd recommend doing something like this My fork call is from Tcl -- you've convinced me to add a line in the child unsetting the Tcl variable containing the name of the parent's database connection. :) Your wrapper is nice -- have you considered folding something like it into the core (disabled by default, enabled by a compile-time flag) and submitting it to drh for official adoption? Thanks much to both of you! Eric -- Eric A. Smith More people would come here if it weren't so crowded. -- Yogi Berra ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] open db cx to fork(2)d children
On Fri, Jul 09, 2010 at 02:38:08PM -0700, Roger Binns wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 07/09/2010 02:31 PM, Nicolas Williams wrote: > > The trick to making that go fast is to use pthread_atfork() to get the > > new PID on the child side of fork() and store the PID in a global > > variable so that you don't need to call getpid(). > > That assumes that pthreads is in use, that getpid is poorly implemented/slow > and that enough of the 1% slowdown was due to getpid calls that changing it > would make a difference :-) I'm spoiled by Solaris, where (since Solaris 10) there's a unified process model and every process has pthreads (not necessarily more than one thread, mind you), with pthreads folded into libc. :) ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] open db cx to fork(2)d children
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 07/09/2010 02:31 PM, Nicolas Williams wrote: > The trick to making that go fast is to use pthread_atfork() to get the > new PID on the child side of fork() and store the PID in a global > variable so that you don't need to call getpid(). That assumes that pthreads is in use, that getpid is poorly implemented/slow and that enough of the 1% slowdown was due to getpid calls that changing it would make a difference :-) Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkw3lrsACgkQmOOfHg372QRlvACePV+jbjnvVxC6BsLZq0mvJOiG AHAAn0yjpqdhxd/MaYnk/lRbtIeMcb/K =S41l -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] open db cx to fork(2)d children
On Fri, Jul 09, 2010 at 02:22:37PM -0700, Roger Binns wrote: > On 07/09/2010 01:52 PM, Eric Smith wrote: > > What do you mean, "immediately"? As I said, my child comes to life, > > does some work without touching (its copy of) existing SQLite strucures, > > and then calls exit(2). > > I'll bet you are actually getting exit(3) which means anything registered > with atexit will be run. (SQLite does not register with atexit.) Oh, duh. I forgot that distinction. Yes, exit(3), not exit(2). (Any library with atexit handlers should check the process' PID if fork- safety is an issue, and do nothing when called on the child side of a fork().) > In my wrapper I provide functionality that can check SQLite objects are not > being used across processes. The way it does this is by providing an > alternate mutex implementation (almost every SQLite operation acquires and > releases mutexes) and verifies the mutex is used in the same process id it > was allocated in. In a benchmark doing only SQLite operations I found a 1% > performance hit. The trick to making that go fast is to use pthread_atfork() to get the new PID on the child side of fork() and store the PID in a global variable so that you don't need to call getpid(). Nico -- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] open db cx to fork(2)d children
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 07/09/2010 01:52 PM, Eric Smith wrote: > What do you mean, "immediately"? As I said, my child comes to life, > does some work without touching (its copy of) existing SQLite strucures, > and then calls exit(2). I'll bet you are actually getting exit(3) which means anything registered with atexit will be run. (SQLite does not register with atexit.) In my wrapper I provide functionality that can check SQLite objects are not being used across processes. The way it does this is by providing an alternate mutex implementation (almost every SQLite operation acquires and releases mutexes) and verifies the mutex is used in the same process id it was allocated in. In a benchmark doing only SQLite operations I found a 1% performance hit. A little tricky thing is that you can't change mutex implementations after SQLite has initialised. Consequently the code ensures SQLite is initialised to get the current mutex implementation, then does a shutdown, installs the new implementation and initialises SQLite again. You should not shutdown SQLite unless all items are fully closed and freed. Therefore this code is best run as close to the beginning of main() as possible. I'd recommend doing something like this, even if it is only enabled during testing because other people may write code, or after time has passed you may forget about the whole forking issue. You can see example code at: http://code.google.com/p/apsw/source/browse/src/apsw.c#858 Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkw3kxkACgkQmOOfHg372QTomgCg1g8LUOQnJK7z3B0j2tsax6FN 2OsAn1K6eO1zHZx+eR66PVasjaugm9tp =Hm6K -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] open db cx to fork(2)d children
On Fri, Jul 09, 2010 at 04:52:35PM -0400, Eric Smith wrote: > > > I strongly recommend that you always make the child side of fork(2) > > either exit(2) or exec(2) immediately. > > Sorry Nico, I never saw this response -- I appreciate it! > > What do you mean, "immediately"? Good question. One answer: "before ever utilizing state inherited from the parent for libraries such as libsqlite3". A much more conservative answer: "the exit(2) or exec(2) call should be lexically close to the fork(2) call, and it should be obvious from lexical context that the child will not ever utilize state inherited from the parent for libraries such as libsqlite3". (What a mouthful. If there's a good English-language word to describe the above without superfluously referring to 'time', as 'immediately' does, I'd love to hear what it is.) > As I said, my child comes to life, > does some work without touching (its copy of) existing SQLite strucures, > and then calls exit(2). The lifetime of the child is small wrt the > lifetime of the parent. That's fine. The child could run forever then. Of course, if you design an architecture like that you end up creating the temptation (for subsequent developers) to do more in the child process later, so I'd not recommend it. > Let's assume for the moment that I don't care about safety wrt > non-sqlite libraries (except of course any libraries on which > sqlite depends). Good luck! :) > > With respect to SQLite3, there are two sets of fork-safety issues: file > > descriptor offsets (use USE_PREAD to avoid this issue), > > I take you to mean that the child and parent's fds point to the same > underlying file description, and if the child changes the file > description then it will have a side effect in the parent. Exactly. > But I have assumed that the child does not make any sqlite api calls > against existing sqlite structures. I believe this assumption allows > me to conclude that sqlite will not touch any existing fd, and hence > will not bear such an impact on the parent (even if the child makes > sqlite api calls against structures the child creates on its own). > Am I right? Correct. > > and POSIX file byte range locks. > > I'm not using POSIX locks, so I'm good to go there. But even if I were, > I believe my above reasoning applies equally well here, since I believe > your reason for being concerned about it is similar. The fds that were > duplicated across the fork refer to the same underlying file > description, so we are technically in a "dangerous" state: the child > *could*, at its whim, release the parent's lock (for example). But if > it guarantees not to do so (by guaranteeing to make no sqlite calls > against existing structures), then no harm will result. No, I meant that if you stepped, in the child, a statement inherited from the parent then the child would go ahead believing it has a lock when in fact it does not (because the parent got the lock). Nico -- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] open db cx to fork(2)d children
> I strongly recommend that you always make the child side of fork(2) > either exit(2) or exec(2) immediately. Sorry Nico, I never saw this response -- I appreciate it! What do you mean, "immediately"? As I said, my child comes to life, does some work without touching (its copy of) existing SQLite strucures, and then calls exit(2). The lifetime of the child is small wrt the lifetime of the parent. Let's assume for the moment that I don't care about safety wrt non-sqlite libraries (except of course any libraries on which sqlite depends). > With respect to SQLite3, there are two sets of fork-safety issues: file > descriptor offsets (use USE_PREAD to avoid this issue), I take you to mean that the child and parent's fds point to the same underlying file description, and if the child changes the file description then it will have a side effect in the parent. But I have assumed that the child does not make any sqlite api calls against existing sqlite structures. I believe this assumption allows me to conclude that sqlite will not touch any existing fd, and hence will not bear such an impact on the parent (even if the child makes sqlite api calls against structures the child creates on its own). Am I right? > and POSIX file byte range locks. I'm not using POSIX locks, so I'm good to go there. But even if I were, I believe my above reasoning applies equally well here, since I believe your reason for being concerned about it is similar. The fds that were duplicated across the fork refer to the same underlying file description, so we are technically in a "dangerous" state: the child *could*, at its whim, release the parent's lock (for example). But if it guarantees not to do so (by guaranteeing to make no sqlite calls against existing structures), then no harm will result. Thanks, Eric -- Eric A. Smith Impartial, adj.: Unable to perceive any promise of personal advantage from espousing either side of a controversy or adopting either of two conflicting opinions. -- Ambrose Bierce, "The Devil's Dictionary" ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] open db cx to fork(2)d children
On Mon, Jun 28, 2010 at 11:30:49AM -0400, Eric Smith wrote: > From the docs: > > > Under Unix, you should not carry an open SQLite database across a > > fork() system call into the child process. Problems will result if you > > do. > > What if I fork a process that promises not to use the handle, and > furthermore the child process certainly dies before the parent does? Libraries could well (and many do) make use of pthread_atfork(3C) to render "handles" unusable in a child process. If an API is described as caller "should not" or "must not" follow fork(2) then you need to be prepared for it to use pthread_atfork(3C) as described, either now or in the future. > Will this still cause problems? > > What if I add an additional assumption that the parent process is the > only process that ever accesses the database? I strongly recommend that you always make the child side of fork(2) either exit(2) or exec(2) immediately. Self re-exec()ing is fine, if you can pass in whatever state you need preserved on the command-line and/or pipe and/or temp file. If you're going to fork(2) and neither exit nor exec then you should be careful to do whatever cleanup the various libraries demand, and stick to whatever fork-safety protocol they insist on, if any. Not all fork-unsafe libraries will give you a way to re-initialize them on the child-side of fork(2); the only fool-proof way forward is to exec(2) in the child of fork(2). With respect to SQLite3, there are two sets of fork-safety issues: file descriptor offsets (use USE_PREAD to avoid this issue), and POSIX file byte range locks. There may be other fork-safety issues too that don't leap to my mind at this time. Basically, fork(2) must be handled with extreme care. The easiest thing for library developers to do is to declare their libraries to be fork-unsafe and make the matter a problem for their callers, which in a sufficiently layered system means many libraries become inherently fork-unsafe, which leads one to conclude even more strongly that one must always exec(2) or exit(2) the child of fork(2). What you really need from libraries is a way to recover whatever state you'll want to re-establish after exec()ing in the child. SQLite3 has no such state (what state it has the caller already knows how to re-create). (Think of crypto APIs, where the state of a cipher might not be feasible to reproduce with ease; there you need the library to give you a way to export the cipher's state.) Nico -- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] open db cx to fork(2)d children
>From the docs: > Under Unix, you should not carry an open SQLite database across a > fork() system call into the child process. Problems will result if you > do. What if I fork a process that promises not to use the handle, and furthermore the child process certainly dies before the parent does? Will this still cause problems? What if I add an additional assumption that the parent process is the only process that ever accesses the database? -- Eric A. Smith Slurm, n.: The slime that accumulates on the underside of a soap bar when it sits in the dish too long. -- Rich Hall, "Sniglets" ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
