Re: [sqlite] Problems with threadsafe opt correction #2623
On 9/5/05, Ian Monroe <[EMAIL PROTECTED]> wrote: > > > > > SQLite has *never* supported the ability of a handle to be used > > by more than one thread. By luck, such use would sometimes work on > > some operating systems. But it would fail on others. Such a > > situation is very dangerous since if a developer is working on > > a system where the misuse of SQLite just happened to work, they > > might not detect their design error and then ship non-working code > > to a customer where it would fail. A minor change in check-in [2517] > > detects the misuse of SQLite on all systems and prevents it from > > working even by chance. This allows the problem to be detected > > early and corrected before it reaches a customer. > > But it has already reached our users. amaroK has never had a reported > problem with this. In this case "early" is a year or so too late. > > What about an --incorrect-behavior configuration option? Or a warning > to stderr instead? Perhaps the gentoo default emerge build script should have the sqlite version specified instead of just assuming the default version. This cures the problem immediately while the authors consider their code changes.
Re: [sqlite] Problems with threadsafe opt correction #2623
Ian Monroe wrote: I see your just ignorant of how open source software gets released. Ian, You are right, I hope you will forgive my ignorance. Beyond that, it wasn't my place to criticize those projects even if I had fully understood the situation. --kiel
RE: [sqlite] Problems with threadsafe opt correction #2623
I hope that this restriction is not enforced on windows, ever! I have an application that will be completely broken by this: it is an application server, that serves multiple clients, pools db connections, and hands them out on demand, protecting the pool with a number of mechanisms that have never failed. so, the application is quite thread-safe. on windows, the option to use processes instead of threds,is not a very good one, as you know.so, any scheme that uses threads, AND resource pooling, will fail if you enforce above restriction, which is meaningless in windows, anyway. so, please don't! > -Original Message- > From: D. Richard Hipp [mailto:[EMAIL PROTECTED] > Sent: Monday, September 05, 2005 9:19 PM > To: sqlite-users@sqlite.org > Subject: Re: [sqlite] Problems with threadsafe opt correction #2623 > > On Mon, 2005-09-05 at 19:58 +0200, René Tegel wrote: > > > As far as i can tell now the windows version of 3.2.5 seems > not affected. > > > > The restriction that a DB handle can only be used from a > single thread is currently only enforced on Unix, because > Unix boxes are the only place where using DB handles in > multiple threads is a problem. > > -- > D. Richard Hipp <[EMAIL PROTECTED]> > > > >
Re: [sqlite] Problems with threadsafe opt correction #2623
On Mon, 2005-09-05 at 19:58 +0200, René Tegel wrote: > As far as i can tell now the windows version of 3.2.5 seems not affected. > The restriction that a DB handle can only be used from a single thread is currently only enforced on Unix, because Unix boxes are the only place where using DB handles in multiple threads is a problem. -- D. Richard Hipp <[EMAIL PROTECTED]>
Re: [sqlite] Problems with threadsafe opt correction #2623
Miha Vrhovnik wrote: So what does that mean. On windows I'm protecting database handle with MREWS. Should I rewrite about 10.000 lines of code. And Check other 30.000. To fix that? As far as i can tell now the windows version of 3.2.5 seems not affected. For other os maybe you can extend your wrapper (if you have one) and add class that implements a helper thread to open/close/query everything in a single thread, at least that is what i plan to do. Shouldn't be too hard and also not really affect the application, at the cost of some performance. regards, rene
Re: [sqlite] Problems with threadsafe opt correction #2623
On 9/5/05, Tobias Rundström <[EMAIL PROTECTED]> wrote: > Kiel W. wrote: > > Ian Monroe wrote: > > > >> I do not see how such a major change can be justified in a minor point > >> release. For instance, currently amaroK does not work when using a > >> sqlite database on Debian Sid since they package it with sqlite 3.2.5. > >> > >> > > Just my two cents, but if this change is breaking things in amaroK, > > XMMS2 and others then more testing should've been done on _those_ > > products before they were released to the wild. Regression/ unit tests > > should be run on most any change but definately when you upgrade a major > > sub-component of your system. > > Well in this case it broke all the currently installed applications that > was installed and where MISSUSING the API. We have corrected this now > but it will take a while before we can make a release out of it. In the > meantime distributions that ship a newer sqlite3 will not be able to use > amarok and xmms2. Well, amaroK bundles sqlite. Its not that fatal. Its just that distributions can't use the system sqlite. > But it was our fault for not reading the docs correctly. > > -- Tobias >
Re: [sqlite] Problems with threadsafe opt correction #2623
On Mon, 2005-09-05 at 19:23 +0200, Tobias Rundström wrote: > Well in this case it broke all the currently installed applications that > was installed and where MISSUSING the API. We have corrected this now > but it will take a while before we can make a release out of it. In the > meantime distributions that ship a newer sqlite3 will not be able to use > amarok and xmms2. If you were using the same handle in multiple threads, then your applications never have worked on RedHat 9. They might seem to worked, but that would only be because you have not tested them sufficiently to find the subtle failures. Normally the problem only appears under high load. You can avoid these kinds of problems in the future by statically linking against SQLite so that you are not beholden to whatever version of SQLite is shipped with the distribution. > > But it was our fault for not reading the docs correctly. > Not really. The documentation did not bring this point out at all - it was ambiguously worded. (That has since been changed.) We did not realize how bad the situation was until people started reporting problems on RedHat 9. See http://www.sqlite.org/cvstrac/tktview?tn=1272 and http://www.sqlite.org/cvstrac/tktview?tn=1285. -- D. Richard Hipp <[EMAIL PROTECTED]>
Re: [sqlite] Problems with threadsafe opt correction #2623
On 9/5/05, Kiel W. <[EMAIL PROTECTED]> wrote: > Ian Monroe wrote: > > >I do not see how such a major change can be justified in a minor point > >release. For instance, currently amaroK does not work when using a > >sqlite database on Debian Sid since they package it with sqlite 3.2.5. > > > > > Just my two cents, but if this change is breaking things in amaroK, > XMMS2 and others then more testing should've been done on _those_ > products before they were released to the wild. Regression/ unit tests > should be run on most any change but definately when you upgrade a major > sub-component of your system. I was going to make a scarcastic comment involving time machines, but I see your just ignorant of how open source software gets released. We didn't upgrade a major sub-component of our system. Debian Sid did (they're fixing this now). And regardless, it means amaroK can no longer use newer (unpatched) sqlite versions. Thats the actual issue. > --K >
Re: [sqlite] Problems with threadsafe opt correction #2623
"D. Richard Hipp" <[EMAIL PROTECTED]> je ob 5.9.2005 18:45:42 napisal(a): >SQLite has *never* supported the ability of a handle to be used >by more than one thread. By luck, such use would sometimes work on >some operating systems. But it would fail on others. Such a >situation is very dangerous since if a developer is working on >a system where the misuse of SQLite just happened to work, they >might not detect their design error and then ship non-working code >to a customer where it would fail. A minor change in check-in [2517] >detects the misuse of SQLite on all systems and prevents it from >working even by chance. This allows the problem to be detected >early and corrected before it reaches a customer. > >This is not a design change. This is not an API change. >It does not break backwards compatibility. All that changed >is that certain errors in the way SQLite is used are now >detected sooner rather than later. > So what does that mean. On windows I'm protecting database handle with MREWS. Should I rewrite about 10.000 lines of code. And Check other 30.000. To fix that? Regards, Miha
Re: [sqlite] Problems with threadsafe opt correction #2623
Kiel W. wrote: Ian Monroe wrote: I do not see how such a major change can be justified in a minor point release. For instance, currently amaroK does not work when using a sqlite database on Debian Sid since they package it with sqlite 3.2.5. Just my two cents, but if this change is breaking things in amaroK, XMMS2 and others then more testing should've been done on _those_ products before they were released to the wild. Regression/ unit tests should be run on most any change but definately when you upgrade a major sub-component of your system. Well in this case it broke all the currently installed applications that was installed and where MISSUSING the API. We have corrected this now but it will take a while before we can make a release out of it. In the meantime distributions that ship a newer sqlite3 will not be able to use amarok and xmms2. But it was our fault for not reading the docs correctly. -- Tobias
Re: [sqlite] Problems with threadsafe opt correction #2623
Mon, Sep 05, 2005 at 12:45:42PM -0400: D. Richard Hipp wrote: > On Mon, 2005-09-05 at 10:57 -0500, Ian Monroe wrote: > > I do not see how such a major change can be justified in a minor point > > release. For instance, currently amaroK does not work when using a > > sqlite database on Debian Sid since they package it with sqlite 3.2.5. > > > > Shouldn't this have waited for 4.0? > > > > There are no plans at this time to ever release version 4.0. > > SQLite has *never* supported the ability of a handle to be used > by more than one thread. By luck, such use would sometimes work on > some operating systems. But it would fail on others. Such a > situation is very dangerous since if a developer is working on > a system where the misuse of SQLite just happened to work, they > might not detect their design error and then ship non-working code > to a customer where it would fail. A minor change in check-in [2517] > detects the misuse of SQLite on all systems and prevents it from > working even by chance. This allows the problem to be detected > early and corrected before it reaches a customer. In this case, the ability of a handle to be used by more than one thread should be always denied to avoid portability problems. -- Guillaume FOUGNIES Eulerian Technologies
Re: [sqlite] Problems with threadsafe opt correction #2623
René Tegel wrote: this does only affect linux? Forgive me that i said linux where i ment unix...
Re: [sqlite] Problems with threadsafe opt correction #2623
On 9/5/05, D. Richard Hipp <[EMAIL PROTECTED]> wrote: > On Mon, 2005-09-05 at 10:57 -0500, Ian Monroe wrote: > > I do not see how such a major change can be justified in a minor point > > release. For instance, currently amaroK does not work when using a > > sqlite database on Debian Sid since they package it with sqlite 3.2.5. > > > > Shouldn't this have waited for 4.0? > > > > There are no plans at this time to ever release version 4.0. > > SQLite has *never* supported the ability of a handle to be used > by more than one thread. By luck, such use would sometimes work on > some operating systems. But it would fail on others. Such a > situation is very dangerous since if a developer is working on > a system where the misuse of SQLite just happened to work, they > might not detect their design error and then ship non-working code > to a customer where it would fail. A minor change in check-in [2517] > detects the misuse of SQLite on all systems and prevents it from > working even by chance. This allows the problem to be detected > early and corrected before it reaches a customer. But it has already reached our users. amaroK has never had a reported problem with this. In this case "early" is a year or so too late. What about an --incorrect-behavior configuration option? Or a warning to stderr instead? > This is not a design change. This is not an API change. > It does not break backwards compatibility. Um, yes it does. amaroK works with one version of sqlite but not the next. What is the actual definition of backwards compatable? This reminds me of the scene in Office Space where they lay off Milton by "correcting" the accounting error that lead to him continuing to get his checks. :P > All that changed > is that certain errors in the way SQLite is used are now > detected sooner rather than later. > > -- > D. Richard Hipp <[EMAIL PROTECTED]> > >
Re: [sqlite] Problems with threadsafe opt correction #2623
D. Richard Hipp wrote: This is not a design change. This is not an API change. It does not break backwards compatibility. All that changed is that certain errors in the way SQLite is used are now detected sooner rather than later. May be you can add some pragma to disable this check? Also, if i read this source right: http://www.sqlite.org/cvstrac/chngview?cn=2517 this does only affect linux? regards, rene
Re: [sqlite] Problems with threadsafe opt correction #2623
Ian Monroe wrote: I do not see how such a major change can be justified in a minor point release. For instance, currently amaroK does not work when using a sqlite database on Debian Sid since they package it with sqlite 3.2.5. Just my two cents, but if this change is breaking things in amaroK, XMMS2 and others then more testing should've been done on _those_ products before they were released to the wild. Regression/ unit tests should be run on most any change but definately when you upgrade a major sub-component of your system. --K
Re: [sqlite] Problems with threadsafe opt correction #2623
On Mon, 2005-09-05 at 18:36 +0200, Tobias Rundström wrote: > How "expensive" is it to open a new sqlite3 handler? The schema has to be read and parsed. Usually that is very quick - SQLite has an efficient parser and can process tens of thousands of SQL statements per second. > This seems to occur when this happens: > > thread1: begin > thread2: begin > thread1: insert () > thread2: select () -> EBUSY > thread1: commit -> EBUSY > > to solve this you have to rollback thread2 before you can commit > thread1. It was a strange behaviour that didn't fit my code at all so I > reverted to not using transactions, this works well but probably hurt my > preformance. > > Is there a better way to handle this? > The BEGIN statement defers locks until they are needed. If you do BEGIN EXCLUSIVE instead of just BEGIN, it acquires the locks immediately and would thus cause the EBUSY error when thread2 did its BEGIN. This simplifies recovery at the price of some concurrency. -- D. Richard Hipp <[EMAIL PROTECTED]>
Re: [sqlite] Problems with threadsafe opt correction #2623
On Mon, 2005-09-05 at 10:57 -0500, Ian Monroe wrote: > I do not see how such a major change can be justified in a minor point > release. For instance, currently amaroK does not work when using a > sqlite database on Debian Sid since they package it with sqlite 3.2.5. > > Shouldn't this have waited for 4.0? > There are no plans at this time to ever release version 4.0. SQLite has *never* supported the ability of a handle to be used by more than one thread. By luck, such use would sometimes work on some operating systems. But it would fail on others. Such a situation is very dangerous since if a developer is working on a system where the misuse of SQLite just happened to work, they might not detect their design error and then ship non-working code to a customer where it would fail. A minor change in check-in [2517] detects the misuse of SQLite on all systems and prevents it from working even by chance. This allows the problem to be detected early and corrected before it reaches a customer. This is not a design change. This is not an API change. It does not break backwards compatibility. All that changed is that certain errors in the way SQLite is used are now detected sooner rather than later. -- D. Richard Hipp <[EMAIL PROTECTED]>
Re: [sqlite] Problems with threadsafe opt correction #2623
I do not see how such a major change can be justified in a minor point release. For instance, currently amaroK does not work when using a sqlite database on Debian Sid since they package it with sqlite 3.2.5. Shouldn't this have waited for 4.0? On Wed, 31 Aug 2005 05:38:05 -0700 D. Richard Hipp wrote: > On Wed, 2005-08-31 at 12:53 +0200, Guillaume Fougnies wrote: > > My code is sharing a pool of SQLite connections on > > multiple databases between a bunch of treatment > > threads. > > Each thread pops a connection from the pool safely > > and push it back once finished. > > > > This works on some systems but not on others. On some > versions of Linux, a thread is not able to override locks > created by a different thread in the same process. When > that happens, a database connection created on one thread > will not be usable by a different thread. > > Additional information: > > http://www.sqlite.org/cvstrac/tktview?tn=1272 > http://www.sqlite.org/cvstrac/chngview?cn=2521
Re: [sqlite] Problems with threadsafe opt correction #2623
> This works on some systems but not on others. On some > versions of Linux, a thread is not able to override locks > created by a different thread in the same process. When > that happens, a database connection created on one thread > will not be usable by a different thread. This is not the point here because the file lock should be released to allow the previous OsFile->tid to be reset (this is my case). (i forgot this from the enhancement...) 1: - "sqlite3_release" function. It checks the lock state of files. If all is clear, set OsFile->tid to 0. return the state. 1-bis: - "sqlite3_release" function. It checks the lock state of files. If something is locked, it releases transactions/file locks and finally set OsFile->tid to 0. 2: - Atomically, a thread try to 'CHECK_THREADID' in sqlite3OsLock and if the OsFile->tid eq 0, just call SET_THREADID. Best regards, -- Guillaume FOUGNIES Eulerian Technologies
Re: [sqlite] Problems with threadsafe opt correction #2623
Hello, Is this the only reason for the... "database handle can only be used on the same thread that opened it" ...rule? Does Windows have that issue? If so, can we convert this to a compile time option? Eg. OS_SUPPORTS_THREADSAFE_FILE_LOCKS or similar flag which can be set to true for OSes which do not have this issue? My application is Windows only. It's also a component that can be passed around between threads. Hence I have to do a 'thread-id check' on *every* method call into the component. This rule also forces us to pass around the database filename and path rather than the handle. Since on any call, if the 'thread-id check' fails, the database has to be re-opened. Regards, Kervin D. Richard Hipp wrote: On Wed, 2005-08-31 at 12:53 +0200, Guillaume Fougnies wrote: My code is sharing a pool of SQLite connections on multiple databases between a bunch of treatment threads. Each thread pops a connection from the pool safely and push it back once finished. This works on some systems but not on others. On some versions of Linux, a thread is not able to override locks created by a different thread in the same process. When that happens, a database connection created on one thread will not be usable by a different thread. Additional information: http://www.sqlite.org/cvstrac/tktview?tn=1272 http://www.sqlite.org/cvstrac/chngview?cn=2521
Re: [sqlite] Problems with threadsafe opt correction #2623
On Wed, 2005-08-31 at 12:53 +0200, Guillaume Fougnies wrote: > My code is sharing a pool of SQLite connections on > multiple databases between a bunch of treatment > threads. > Each thread pops a connection from the pool safely > and push it back once finished. > This works on some systems but not on others. On some versions of Linux, a thread is not able to override locks created by a different thread in the same process. When that happens, a database connection created on one thread will not be usable by a different thread. Additional information: http://www.sqlite.org/cvstrac/tktview?tn=1272 http://www.sqlite.org/cvstrac/chngview?cn=2521 -- D. Richard Hipp <[EMAIL PROTECTED]>
