-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Gerry Snyder wrote:
> The wiki is there and open to all.
>
> I look forward to reading your additions to it.
To be fair, only some of the documentation is in the wiki. The
remainder is generated. For example you can't edit any of the pages
listed
Emerson Clarke wrote:
Fix the out of date documentation
The wiki is there and open to all.
I look forward to reading your additions to it.
Gerry
-
To unsubscribe, send email to [EMAIL PROTECTED]
For anyone who is interested i have created a standalone test case
which demonstrates the threading behaviour that i had, or as close as
i can get it. Feel free to use the code for whatever purposes you see
fit.
It will compile on linux and windows, and comes with 4 versions of the
sqlite
sqlite-users@sqlite.org
> Subject: Re: [sqlite] sqlite performance, locking & threading
>
>
> Roger Binns wrote:
>
> >>Im sorry for being so harsh, and i know im not winning any friends
> >>here,
> >>
> >>
> >
> >So far noo
Bill King wrote:
Roger Binns wrote:
Im sorry for being so harsh, and i know im not winning any friends
here,
So far noone has agreed with you :-)
This would be incorrect. The correct statement is "so far no one has
vocally agreed with you".
If people didn't agree, this whole once
Roger Binns wrote:
Im sorry for being so harsh, and i know im not winning any friends
here,
So far noone has agreed with you :-)
This would be incorrect. The correct statement is "so far no one has
vocally agreed with you".
If people didn't agree, this whole once a month people
Emerson,
I agree with you somewhat. Not 100% convinced but, I like you am a little
dissapointed how sqlite handles "threadsafe" and multiple connections. Even in
the "test_server.c" module is not "concurrent" As it serializes all processing
to a single thread, this is not concurrent
Roger,
Of course you can test threading behaviour, yes its not exactly
repeatable but under most circumstances and with enough test cases you
can catch the problems.
I don't think sqlite is such a large and complicated piece of software
that it would be impossible to reproduce such errors.
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Emerson Clarke wrote:
> I have to admit i am a little dissapointed. As the primary author of
> the software i would have thought that you would have a good
> understanding of what the thread safety characteristics of your own
> api were.
He does!
Richard,
I have to admit i am a little dissapointed. As the primary author of
the software i would have thought that you would have a good
understanding of what the thread safety characteristics of your own
api were.
Suggesting that suppressing the safety checks will result in random
and non
"Emerson Clarke" <[EMAIL PROTECTED]> wrote:
>
> The problem i had was with sqlite not being compatible with the simple
> design that i wanted. I did try several alternate designs, but only
> as a way of working around the problem i had with sqlite. It took a
> long time but eventually i managed
Bill,
Thanks for the description, thats pretty much how i designed the
index, but with a few modifications. The filesystem becomes the tree
structure which is indexed by a hash of the original document url. It
works like a big hashtable so its quite scalable.
>
Sorry if this has been
Emerson Clarke wrote:
The indexing process works like this.
1.) Open a document and parse its contents.
2.) Look up records in the first database based on the contents of the
document, updating records where appropriate and inserting new ones.
3.) Transforming the document based on what was
On Thu, Jan 04, 2007 at 12:50:01AM +, Emerson Clarke wrote:
> My oppologies, your right that explanation had been given.
OK.
> But i didnt actually take it seriously, i guess i found it hard to
> believe that it being the easier option was the only reason why this
> limitation was in place.
Nicholas,
My oppologies, your right that explanation had been given.
But i didnt actually take it seriously, i guess i found it hard to
believe that it being the easier option was the only reason why this
limitation was in place.
If this is the case, then surely the fix is simple. Given that
sendet: Mittwoch, 3. Januar 2007 15:14
An: sqlite-users@sqlite.org
Betreff: Re: [sqlite] sqlite performance, locking & threading
Michael,
Im not sure that atomic operations would be a suitable alternative.
The reason why im using events/conditions is so that the client thread
blocks until the ser
On Tue, Jan 02, 2007 at 11:56:42PM +, Emerson Clarke wrote:
> The single connection multiple thread alternative apparently has
> problems with sqlite3_step being active on more than one thread at the
> same moment, so cannot easily be used in a safe way. But it is by far
> the fastest and
ad has ownership of the
queue, except may be the main thread.
Michael
-Ursprüngliche Nachricht-
Von: Emerson Clarke [mailto:[EMAIL PROTECTED]
Gesendet: Mittwoch, 3. Januar 2007 00:57
An: sqlite-users@sqlite.org
Betreff: Re: [sqlite] sqlite performance, locking & threading
Nico,
I have i
"Emerson Clarke" <[EMAIL PROTECTED]> wrote:
>
> Firstly can i clarify what you mean regarding the same moment. Do you
> mean that no two threads can be executing the call, or that no two
> threads can be in the middle of stepping through a series of results
> using the step function (assuming
Nico,
I have implemented all three strategies (thead specific connections,
single connection multiple threads, and single thread server with
multiple client threads).
The problem with using thread specific contexts is that you cant have
a single global transaction which wraps all of those
On Sat, Dec 30, 2006 at 03:34:01PM +, Emerson Clarke wrote:
> Technically sqlite is not thread safe. [...]
Solaris man pages describe APIs with requirements like SQLite's as
"MT-Safe with exceptions" and the exceptions are listed in the man page.
That's still MT-Safe, but the caller has to
"Emerson Clarke" <[EMAIL PROTECTED]> wrote:
>
> "With that and subsequent versions, it is safe to move a connection
> handle across threads as long as the connection is not holding any
> fcntl() locks. You can safely assume that no locks are being held if
> no transaction is pending and all
Richard,
Im getting some mixed signals here, on the one hand i am being told
that it should be ok to use a single transaction and connection across
multiple threads (assuming that it is correctly synchronised). But on
the other hand i am reading statements like that as part of a list of
things
Joe Wilson <[EMAIL PROTECTED]> wrote:
> --- [EMAIL PROTECTED] wrote:
> > SQLite allows you to have multiple queries running at the
> > same time in multiple threads, as long as no two threads are
> > running sqlite3_step() at the same moment.
>
> Do you mean "as long as no two threads are running
--- [EMAIL PROTECTED] wrote:
> SQLite allows you to have multiple queries running at the
> same time in multiple threads, as long as no two threads are
> running sqlite3_step() at the same moment.
Do you mean "as long as no two threads are running sqlite3_step()
_for the same sqlite3*
Good advice. If you want to experiment with a certain architecture,
write the programs yourself rather than trying to get others to do it
for you. Sqlite is a well thought through solution for an embedded
database with a wide range of applications, but if you want something
more specialised,
"Michael Ruck" <[EMAIL PROTECTED]> wrote:
> Richard,
>
> I believe his problem is this:
>
> "Each query is allowed to complete before the other one starts, but each
> thread may have multiple statements or result sets open."
>
> The open resultsets/multiple started statements are causing him =
"Emerson Clarke" <[EMAIL PROTECTED]> wrote:
> Richard,
>
> Are you sure we are not just getting into semantic knots here ?
>
> Do we have the same definition of "at the same time". I mean
> concurrently, so that both threads use the same sqlite3 * structure,
> within mutexes. Each query is
"Emerson Clarke" <[EMAIL PROTECTED]> wrote:
>
> I have code which creates a transaction on a connection in the parent
> thread, then creates several child threads which attempt to use the
> same connection and transaction in a synchronised mannor. It does not
> work, and by all the documentation
> Sent: Saturday, December 30, 2006 9:34 AM
> To: sqlite-users@sqlite.org
> Subject: Re: [sqlite] sqlite performance, locking & threading
>
>
> Roger,
>
> I think sqlite suffers somewhat from a bit of an identity crisis.
> Whilst it is both a library and a piece of code wh
Richard,
Are you sure we are not just getting into semantic knots here ?
Do we have the same definition of "at the same time". I mean
concurrently, so that both threads use the same sqlite3 * structure,
within mutexes. Each query is allowed to complete before the other one
starts, but each
"Emerson Clarke" <[EMAIL PROTECTED]> wrote:
> But why then can i not have a single transaction wrapping a single
> connection which is used within multiple threads, obvioulsy not at the
> same time.
You can. What makes you think you can't?
--
D. Richard Hipp <[EMAIL PROTECTED]>
Richard,
Ok, im pretty clear on the file locking being the cause of the
problems with the sqlite3 * structures, but thanks for confirming it.
I understand that on platforms that dont have this issue its not a
problem.
But why then can i not have a single transaction wrapping a single
connection
"Emerson Clarke" <[EMAIL PROTECTED]> wrote:
> Even on the
> platforms where a single sqlite3 * structure can be used on multiple
> threads (provided it is not at the same time), it is not possible to
> have a transaction which works across these threads.
I beg to differ. What makes you think
Roger,
I think sqlite suffers somewhat from a bit of an identity crisis.
Whilst it is both a library and a piece of code which you embed in a
project it is often talked about as though it is some external
component.
Technically sqlite is not thread safe. Just because the library has
explicitly
"Emerson Clarke" <[EMAIL PROTECTED]> wrote:
> Richard,
>
> My complaint, if you want to call it that. Was simply that there are
> seemingly artificial constraints on what you can and cant do accross
> threads.
>
> If i have a linked list, i can use it across threads if i want to,
> provided
Emerson Clarke wrote:
If i have a linked list, i can use it across threads if i want to,
provided that i synchronise operations in such a way that the list
does not get corrupted.
And of course you also have to know about memory barriers and compiler
re-ordering. That is highly dependent on
Richard,
My complaint, if you want to call it that. Was simply that there are
seemingly artificial constraints on what you can and cant do accross
threads.
If i have a linked list, i can use it across threads if i want to,
provided that i synchronise operations in such a way that the list
does
"Emerson Clarke" <[EMAIL PROTECTED]> wrote:
>>
> It seemed to me that making a library which only functioned on a per
> thread basis was something that you would have to do deliberately and
> by design.
I'm still trying to understand what your complaint is.
--
D. Richard Hipp <[EMAIL
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Emerson Clarke wrote:
| I am left to assume that all other locking mechanisms like ipc and
| files have already been tried and been found wanting. I also assume
| that priority has been given to making sqlite operate across network
| boundaries
Roger,
My original question was in fact not a statement. I did not want
sqlite to work differently. Rather the opposite, sqlite already works
differently to the way i, and probably a lot of users assume that it
would. So all i wanted to know was why that is the case.
It seemed to me that
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Emerson Clarke wrote:
| I have deliberately tried to avoid giving too much detail on the
| architecture of the index since that was not the point and i didnt
| want to end up debating it.
I don't want to debate your index architecture either :-).
Roger,
I have deliberately tried to avoid giving too much detail on the
architecture of the index since that was not the point and i didnt
want to end up debating it.
The design of the index is not the issue, suffice to say that i think
you are over complicating things. It is a desceptively
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
You never answered the bit about whether indexing A can be done at the
same time as B. (I get the feeling you have a certain design in mind
and insist that SQLite changes to meet that design, rather than change
your own design around the constraints
This old mailing list thread better describes this proposed algorithm
to contain all open/close/lock/unlock activity in a single work thread:
http://www.mail-archive.com/sqlite-users@sqlite.org/msg15852.html
--- Joe Wilson <[EMAIL PROTECTED]> wrote:
> As you already know, it's not just Linux -
Richard,
First let me say thank you for your earilier suggestion regarding the
compile time options and overlaoding of the os routines. It was
exactly the kind of information i was searching to for and i am
looking into using that strategy at the moment.
Sorry if ive managed to offend you,
--- [EMAIL PROTECTED] wrote:
> On some older versions of Linux, SQLite is unable to pass
> database connections from one thread to another. But this
> is a problem with the threading libraries used in those older
> linux versions and is outside the control of SQLite. I do not
> think this issue
"Emerson Clarke" <[EMAIL PROTECTED]> wrote:
> Im not sure that the current situation of forced thread
> incompatibility is better than leaving it up to users to manage the
> threading. Usually it is assumed that a library is thread unsafe
> unless otherwise specified.
>
> Developing
--- Emerson Clarke <[EMAIL PROTECTED]> wrote:
> Developing multithreaded applications is difficult, i wouldnt dispute
> that. But i do dispute the wisdom of actively making a library
> incompatible with threads.
"Actively"? That's a bit much.
There are constraints on the ability to pass
In general worker threads is not an efficient solution to the problem
even if parallelism is available. There is nothing to be gained by
having one thread handing off A to another set of worker threads
because you have to synchronise on whatever queue/list/pipe you use to
transfer the
Im not sure that the current situation of forced thread
incompatibility is better than leaving it up to users to manage the
threading. Usually it is assumed that a library is thread unsafe
unless otherwise specified.
Developing multithreaded applications is difficult, i wouldnt dispute
that.
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Emerson Clarke wrote:
| For architectural reasons the above steps must be performed in that
| order. This means that operations cant be separated or queued up in
| the way that you suggested. Each step is dependent on the previous
| step.
I was
On 12/27/06, Emerson Clarke <[EMAIL PROTECTED]> wrote:
The first question is why database locking has been enabled on a per thread
basis instead of per process so that the normal methods of thread
synchronisation (mutexes, ciritcal sections) could not be used for
maintaining consistency within
The indexing process works like this.
1.) Open a document and parse its contents.
2.) Look up records in the first database based on the contents of the
document, updating records where appropriate and inserting new ones.
3.) Transforming the document based on what was obtained from the
first
Emerson,
Is the Database on the same disk as the rest of the file operations? If so is
it possible that you are I/O bound and causing seek issues due to i/o access
patterns?
Take a look at the test_server.c code in the sqlite/src directory. I used that
as a basis to build a custom library
Emerson Clarke wrote:
The idea is that because i am accessing two databases, and doing
several file system operations per document, there should be a large
gain by using many threads. There is no actual indexing process, the
whole structure is the index, but if anything the database operations
I am curious as to how multiple threads would perform faster inserts
into an Sqlite database, which is a single file plus the journal.
Are you using a multiple processor machine?
Emerson Clarke wrote:
Roger,
Thanks for the suggestions. I think using a worker thread and a queue
would be
Roger,
Thanks for the suggestions. I think using a worker thread and a queue
would be equivalent to just running a single thread since it
effectively makes the database operations synchronous. Although i can
see what your driving at regarding the transactions every n records.
The idea is that
"Emerson Clarke" <[EMAIL PROTECTED]> wrote:
>
> The first question is why database locking has been enabled on a per thread
> basis instead of per process so that the normal methods of thread
> synchronisation (mutexes, ciritcal sections) could not be used for
> maintaining consistency within the
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Emerson Clarke wrote:
| modified the api to ensure that each thread was given its own sqlite3 *
| structure.
I would assume that the actual indexing is the expensive part since it
involves a lot of I/O (SQLite page size is 1KB). Why don't you do
60 matches
Mail list logo
