Re: [sqlite] Is sqlite thread-safety sufficient for use with "Go" language ?
Hello, Maxim, I use your excellent driver, indeed ;-) About my post, the poison of doubt has just crept into my mind when I read this: http://www.sqlite.org/cvstrac/wiki?p=MultiThreading There is a bug in some Linux implementations (RedHat9 is the canonical example) that prevents fcntl() locks created by one thread from being modified in a different thread. If you are running on one of those systems, then you should always use an SQLite database connection in the same thread in which it was originally created. It is not safe to open the database in one thread and then pass the connection off to a different thread for processing. The restriction of the previous paragraph has been relaxed somewhat as of SQLite version 3.3.1. Beginning with version 3.3.1, you should be able to move database connections from one thread to another as long as there are no locks outstanding at the time you move the thread. If you are not running on one of the systems effected by the fcntl() locking bug, then you can move your database connections at any time you want. But for portability, you probably should assume your system has the bug. It is an old issue, 8 years ago, and has certainly been resolved. As I have a quite paranoid personality, I prefer to ask to be sure that there is no similar problem I am not aware of. But I have found nothing so far, so I think you are right and there is no problem. 2014-11-06 3:06 GMT+01:00 Maxim Khitrov <m...@mxcrypt.com>: > On Wed, Nov 5, 2014 at 7:10 PM, nicolas riesch <nicolas.rie...@gmail.com> > wrote: > > Pardon me, I will try to reformulate my question more clearly. > > > > My scenario: > > > > - sqlite is set to Multi-thread mode (SQLITE_THREADSAFE=2), or > Serialized > > mode (SQLITE_THREADSAFE=1) > > - I create N logical threads in my "Go" program. > > - Each logical thread creates a database connection, for its > "exclusive" > > usage. > > Logical thread LT1 creates connection C1, logical thread LT2 creates > > connection C2, etc. > > Logical thread LT1 only makes call to connection C1, never to > > connection C2, C3, etc. Same for other threads. > > > > Normally, in any mainstream language (C, PHP, etc), the same OS thread > > makes the successive calls to sqlite3_prepare(), sqlite3_step(), > > sqlite3_column(), sqlite3_finalize(), etc. > > In the loop to retrieve all records in a table, there is no reason to > call > > sqlite3_step() on a different OS thread each time. > > > > But in Go, it is possible that each call to sqlite3_step() is scheduled > to > > run on a different OS thread. > > Indeed, the execution of a logical Go thread (called a Goroutine) can > > switch from one OS thread to another one, without the user being aware of > > it, at each function call. > > I'm the author of https://github.com/mxk/go-sqlite. You can't use a > single connection and its derived objects from multiple goroutines > without external synchronization, but as far as I know, there are no > problems with the same goroutine being scheduled to different OS > threads. I'm pretty sure that this applies to all other Go SQLite > drivers as well. > > You actually can lock the current goroutine to a specific OS thread by > calling runtime.LockOSThread(). Some things, like OpenGL, need this > for thread-local storage and the other reasons that you mentioned. I > don't think SQLite falls into this category. > > - Max > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Is sqlite thread-safety sufficient for use with "Go" language ?
Pardon me, I will try to reformulate my question more clearly. My scenario: - sqlite is set to Multi-thread mode (SQLITE_THREADSAFE=2), or Serialized mode (SQLITE_THREADSAFE=1) - I create N logical threads in my "Go" program. - Each logical thread creates a database connection, for its "exclusive" usage. Logical thread LT1 creates connection C1, logical thread LT2 creates connection C2, etc. Logical thread LT1 only makes call to connection C1, never to connection C2, C3, etc. Same for other threads. Normally, in any mainstream language (C, PHP, etc), the same OS thread makes the successive calls to sqlite3_prepare(), sqlite3_step(), sqlite3_column(), sqlite3_finalize(), etc. In the loop to retrieve all records in a table, there is no reason to call sqlite3_step() on a different OS thread each time. But in Go, it is possible that each call to sqlite3_step() is scheduled to run on a different OS thread. Indeed, the execution of a logical Go thread (called a Goroutine) can switch from one OS thread to another one, without the user being aware of it, at each function call. E.g. logical thread LT1 can dispatch function calls on connection C1 like this: OS thread a --sqlite3_prepare(C1)-- --sqlite3_column(C1)-- OS thread b --sqlite3_step(C1)-- --sqlite3_column(C1)-- OS thread c --sqlite3_step(C1)-- --sqlite3_finalize(C1)-- For each connection, function calls always occur sequentially, but possibly on a different OS thread each time. Logical thread LT2 executes simultaneously, but calling functions only on connection C2. Logical thread LT3 executes simultaneously, but calling functions only on connection C3. etc... So, in this scenario, I imagine that with SQLITE_THREADSAFE=1 or SQLITE_THREADSAFE=2, there should be no problem ? Is it correct to say that each function of the C API doesn't care on which OS thread it is run, as long as the sequence of calls is correct ? I know that in www.sqlite.org/threadsafe.html, it is written that "In serialized mode, SQLite can be safely used by multiple threads with no restriction.", but I just wanted to have a confirmation that it really applies in the particular scenario above. 2014-11-05 23:13 GMT+01:00 Simon Slavin <slav...@bigfraud.org>: > > On 5 Nov 2014, at 10:05pm, nicolas riesch <nicolas.rie...@gmail.com> > wrote: > > > Even if the user writes a Go program with only one logical thread, he has > > no control about which OS thread will process a function call. > > > > This means that EACH SUCCESSIVE function in the sequence above can be > > processed on a DIFFERENT OS THREAD. > > > > It means that to run safely, sqlite source code should not depend in any > > way on the identity of the threads, which must be fully interchangeable. > > So, the following conditions should be true. Are these sentences correct > ? > > > > 1) no local-thread-storage is used in sqlite code. > > 2) thread id (gettid()) are not used. > > 3) when a function of the API enters a mutex, it leaves it before the > > function returns. > > Between two API function calls, no mutex should be locked (else, it > > would be impossible to ensure that the mutex is unlocked by the same > thread > > that locked it). > > 4) all file locking information is attached to connections, and not to > > threads. > > Since you don't already refer to it, can I ask that you read this page > > <https://www.sqlite.org/threadsafe.html> > > and then ask any questions which remain, plus any new ones ? You should > probably tell us which threading mode you intend to use based on the needs > you outline above. > > Simon. > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Is sqlite thread-safety sufficient for use with "Go" language ?
I would like to use sqlite with "Go" (golang) language. There are already several drivers available in this language. But before using them, I would like to ensure it is really safe. The normal sequence to access a database is: sqlite3_prepare() loop sqlite3_step() sqlite3_column() sqlite3_finalize() Normally, all functions in this sequence are called from the same OS thread. But Go is inherently a multithreaded language, with many threads running inside. Even if the user writes a Go program with only one logical thread, he has no control about which OS thread will process a function call. This means that EACH SUCCESSIVE function in the sequence above can be processed on a DIFFERENT OS THREAD. It means that to run safely, sqlite source code should not depend in any way on the identity of the threads, which must be fully interchangeable. So, the following conditions should be true. Are these sentences correct ? 1) no local-thread-storage is used in sqlite code. 2) thread id (gettid()) are not used. 3) when a function of the API enters a mutex, it leaves it before the function returns. Between two API function calls, no mutex should be locked (else, it would be impossible to ensure that the mutex is unlocked by the same thread that locked it). 4) all file locking information is attached to connections, and not to threads. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
