Re: [sqlite] mutex and begin/end transaction

[Douglas E. Fajardo]

Gentlefolk:

Sorry to be a 'wet blanket' here, but while this thread is an interesting 
debate, *this* list is about SQLITE, not the Scientific Method... As an 
observer on the sidelines, I would suggest that this discussion has strayed a 
little bit off topic :-)

 Uh... does anyone remember what this thread was *really* about? Oh yes, 
something about Mutexes and transactions...

*** Doug Fajardo


-Original Message-
From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] 
On Behalf Of James Gregurich
Sent: Friday, May 01, 2009 10:02 AM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] mutex and begin/end transaction


I describe reality.

Someone has to be the arbiter of "better." Generally, that arbiter is  
the guy handing out the research grants.

On May 1, 2009, at 5:33 AM, John Stanton wrote:

>
> Science is the Scientific Method - observation, hypothesis and
> skepticism.  The antithesis of politics.  There are no facts in  
> science,
> only observations and any hypothesis is only valid until a better one
> replaces it.
>
> You describe bad, politicized science.
>
> James Gregurich wrote:
>> With all due respect,  science itself is a set of
>> "positions" (opinions) which are endorsed by small group of people as
>> official doctrine after appropriate study. Saying "A 'position' is
>> politics, not science" is not a particularly meaningful statement.   
>> If
>> you want to argue that point, feel free to send me a private email.
>>
>> My threaded application works pretty darn well. I can process
>> thousands of print industry files on an 8-core system keeping the
>> cores busy without lagging the GUI for other applications. Just
>> because many people create ill conceived programs doesn't mean
>> threaded programs are inherently doomed to be ill-conceived. The
>> development tools and techniques for building concurrent systems are
>> advancing and making concurrency quite feasible.
>>
>> James Gregurich
>> Engineering Manager
>> Markzware
>>
>> On Apr 30, 2009, at 5:01 AM, John Stanton wrote:
>>
>>> A "position" is politics, not science.  Warnings about the use of
>>> threads are based on science, and advise you to avoid them if  
>>> possible
>>> for your own protection.
>>>
>>> I see ill conceived programs using threads which go to complex
>>> synchronization to achieve the equivalent of single stream execution
>>> but
>>> with much greater overhead.  A KISS situation.
>>>
>>> James Gregurich wrote:
>>>> thanks for the info. That should work for me.
>>>>
>>>> Given the industry is going multicore and 16-core macintoshes for
>>>> your
>>>> grand-mother are  just a few years away, I recommend you rethink  
>>>> your
>>>> position on the use of threading. Apple is heavily pushing
>>>> parallelism
>>>> on its developers.  NSOperation is a major part of that effort.  
>>>> As I
>>>> understand it, MS is developing their copy of NSOperation for  
>>>> VS2010.
>>>> The development landscape is only going to get more threaded as  
>>>> time
>>>> goes on.
>>>>
>>>> -James
>>>>
>>>>
>>>>
>>>>> On Apr 29, 2009, at 10:03 PM, James Gregurich wrote:
>>>>>
>>>>>
>>>>>> howdy!
>>>>>>
>>>>>> question:
>>>>>>
>>>>>> for an in-memory db with the threading mode set to serialized, is
>>>>>>
>>>>> the
>>>>>
>>>>>> internal mutex held for an entire transaction so that one thread
>>>>>>
>>>>> won't
>>>>>
>>>>>> access the db while another one is in the middle of a transaction
>>>>>>
>>>>> with
>>>>>
>>>>>> multiple insert statements?
>>>>>>
>>>>> No.  But the mutex is recursive.  So you can get a copy of it  
>>>>> using
>>>>> sqlite3_db_mutex() then lock it yourself using
>>>>> sqlite3_mutex_enter()/
>>>>> leave().
>>>>>
>>>>> Also remember:  You should not be using threads.  Threads will  
>>>>> bring
>>>>> only grief and woe.  On your own head be it.
>>>>>
>>>>>
>>>>>
>>>>> D. Richard Hipp
>>>>> drh at hwaci.com
>>>>>
>>>>>
>>>> ___
>>>> sqlite-users mailing list
>>>> sqlite-users@sqlite.org
>>>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>>>
>>> ___
>>> sqlite-users mailing list
>>> sqlite-users@sqlite.org
>>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>
>> ___
>> sqlite-users mailing list
>> sqlite-users@sqlite.org
>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] mutex and begin/end transaction

[John Stanton]

The better sales pitch and better political connections persuade the 
granter of research grants.  Better design and better science proves 
itself by performance and success in the marketplace and bad design and 
bad science consumes the grant without productivity.

This has strayed from the topic.  It should get back to Sqlite.

James Gregurich wrote:
> I describe reality.
> 
> Someone has to be the arbiter of "better." Generally, that arbiter is  
> the guy handing out the research grants.
> 
> On May 1, 2009, at 5:33 AM, John Stanton wrote:
> 
>> Science is the Scientific Method - observation, hypothesis and
>> skepticism.  The antithesis of politics.  There are no facts in  
>> science,
>> only observations and any hypothesis is only valid until a better one
>> replaces it.
>>
>> You describe bad, politicized science.
>>
>> James Gregurich wrote:
>>> With all due respect,  science itself is a set of
>>> "positions" (opinions) which are endorsed by small group of people as
>>> official doctrine after appropriate study. Saying "A 'position' is
>>> politics, not science" is not a particularly meaningful statement.   
>>> If
>>> you want to argue that point, feel free to send me a private email.
>>>
>>> My threaded application works pretty darn well. I can process
>>> thousands of print industry files on an 8-core system keeping the
>>> cores busy without lagging the GUI for other applications. Just
>>> because many people create ill conceived programs doesn't mean
>>> threaded programs are inherently doomed to be ill-conceived. The
>>> development tools and techniques for building concurrent systems are
>>> advancing and making concurrency quite feasible.
>>>
>>> James Gregurich
>>> Engineering Manager
>>> Markzware
>>>
>>> On Apr 30, 2009, at 5:01 AM, John Stanton wrote:
>>>
 A "position" is politics, not science.  Warnings about the use of
 threads are based on science, and advise you to avoid them if  
 possible
 for your own protection.

 I see ill conceived programs using threads which go to complex
 synchronization to achieve the equivalent of single stream execution
 but
 with much greater overhead.  A KISS situation.

 James Gregurich wrote:
> thanks for the info. That should work for me.
>
> Given the industry is going multicore and 16-core macintoshes for
> your
> grand-mother are  just a few years away, I recommend you rethink  
> your
> position on the use of threading. Apple is heavily pushing
> parallelism
> on its developers.  NSOperation is a major part of that effort.  
> As I
> understand it, MS is developing their copy of NSOperation for  
> VS2010.
> The development landscape is only going to get more threaded as  
> time
> goes on.
>
> -James
>
>
>
>> On Apr 29, 2009, at 10:03 PM, James Gregurich wrote:
>>
>>
>>> howdy!
>>>
>>> question:
>>>
>>> for an in-memory db with the threading mode set to serialized, is
>>>
>> the
>>
>>> internal mutex held for an entire transaction so that one thread
>>>
>> won't
>>
>>> access the db while another one is in the middle of a transaction
>>>
>> with
>>
>>> multiple insert statements?
>>>
>> No.  But the mutex is recursive.  So you can get a copy of it  
>> using
>> sqlite3_db_mutex() then lock it yourself using
>> sqlite3_mutex_enter()/
>> leave().
>>
>> Also remember:  You should not be using threads.  Threads will  
>> bring
>> only grief and woe.  On your own head be it.
>>
>>
>>
>> D. Richard Hipp
>> drh at hwaci.com
>>
>>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
 ___
 sqlite-users mailing list
 sqlite-users@sqlite.org
 http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>> ___
>>> sqlite-users mailing list
>>> sqlite-users@sqlite.org
>>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>> ___
>> sqlite-users mailing list
>> sqlite-users@sqlite.org
>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
> 
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] mutex and begin/end transaction

[James Gregurich]

I describe reality.

Someone has to be the arbiter of "better." Generally, that arbiter is  
the guy handing out the research grants.

On May 1, 2009, at 5:33 AM, John Stanton wrote:

>
> Science is the Scientific Method - observation, hypothesis and
> skepticism.  The antithesis of politics.  There are no facts in  
> science,
> only observations and any hypothesis is only valid until a better one
> replaces it.
>
> You describe bad, politicized science.
>
> James Gregurich wrote:
>> With all due respect,  science itself is a set of
>> "positions" (opinions) which are endorsed by small group of people as
>> official doctrine after appropriate study. Saying "A 'position' is
>> politics, not science" is not a particularly meaningful statement.   
>> If
>> you want to argue that point, feel free to send me a private email.
>>
>> My threaded application works pretty darn well. I can process
>> thousands of print industry files on an 8-core system keeping the
>> cores busy without lagging the GUI for other applications. Just
>> because many people create ill conceived programs doesn't mean
>> threaded programs are inherently doomed to be ill-conceived. The
>> development tools and techniques for building concurrent systems are
>> advancing and making concurrency quite feasible.
>>
>> James Gregurich
>> Engineering Manager
>> Markzware
>>
>> On Apr 30, 2009, at 5:01 AM, John Stanton wrote:
>>
>>> A "position" is politics, not science.  Warnings about the use of
>>> threads are based on science, and advise you to avoid them if  
>>> possible
>>> for your own protection.
>>>
>>> I see ill conceived programs using threads which go to complex
>>> synchronization to achieve the equivalent of single stream execution
>>> but
>>> with much greater overhead.  A KISS situation.
>>>
>>> James Gregurich wrote:
 thanks for the info. That should work for me.

 Given the industry is going multicore and 16-core macintoshes for
 your
 grand-mother are  just a few years away, I recommend you rethink  
 your
 position on the use of threading. Apple is heavily pushing
 parallelism
 on its developers.  NSOperation is a major part of that effort.  
 As I
 understand it, MS is developing their copy of NSOperation for  
 VS2010.
 The development landscape is only going to get more threaded as  
 time
 goes on.

 -James



> On Apr 29, 2009, at 10:03 PM, James Gregurich wrote:
>
>
>> howdy!
>>
>> question:
>>
>> for an in-memory db with the threading mode set to serialized, is
>>
> the
>
>> internal mutex held for an entire transaction so that one thread
>>
> won't
>
>> access the db while another one is in the middle of a transaction
>>
> with
>
>> multiple insert statements?
>>
> No.  But the mutex is recursive.  So you can get a copy of it  
> using
> sqlite3_db_mutex() then lock it yourself using
> sqlite3_mutex_enter()/
> leave().
>
> Also remember:  You should not be using threads.  Threads will  
> bring
> only grief and woe.  On your own head be it.
>
>
>
> D. Richard Hipp
> drh at hwaci.com
>
>
 ___
 sqlite-users mailing list
 sqlite-users@sqlite.org
 http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

>>> ___
>>> sqlite-users mailing list
>>> sqlite-users@sqlite.org
>>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>
>> ___
>> sqlite-users mailing list
>> sqlite-users@sqlite.org
>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] mutex and begin/end transaction

[John Stanton]

Science is the Scientific Method - observation, hypothesis and 
skepticism.  The antithesis of politics.  There are no facts in science, 
only observations and any hypothesis is only valid until a better one 
replaces it.

You describe bad, politicized science.

James Gregurich wrote:
> With all due respect,  science itself is a set of  
> "positions" (opinions) which are endorsed by small group of people as  
> official doctrine after appropriate study. Saying "A 'position' is  
> politics, not science" is not a particularly meaningful statement.  If  
> you want to argue that point, feel free to send me a private email.
> 
> My threaded application works pretty darn well. I can process  
> thousands of print industry files on an 8-core system keeping the  
> cores busy without lagging the GUI for other applications. Just  
> because many people create ill conceived programs doesn't mean  
> threaded programs are inherently doomed to be ill-conceived. The  
> development tools and techniques for building concurrent systems are  
> advancing and making concurrency quite feasible.
> 
> James Gregurich
> Engineering Manager
> Markzware
> 
> On Apr 30, 2009, at 5:01 AM, John Stanton wrote:
> 
>> A "position" is politics, not science.  Warnings about the use of
>> threads are based on science, and advise you to avoid them if possible
>> for your own protection.
>>
>> I see ill conceived programs using threads which go to complex
>> synchronization to achieve the equivalent of single stream execution  
>> but
>> with much greater overhead.  A KISS situation.
>>
>> James Gregurich wrote:
>>> thanks for the info. That should work for me.
>>>
>>> Given the industry is going multicore and 16-core macintoshes for  
>>> your
>>> grand-mother are  just a few years away, I recommend you rethink your
>>> position on the use of threading. Apple is heavily pushing  
>>> parallelism
>>> on its developers.  NSOperation is a major part of that effort. As I
>>> understand it, MS is developing their copy of NSOperation for VS2010.
>>> The development landscape is only going to get more threaded as time
>>> goes on.
>>>
>>> -James
>>>
>>>
>>>
 On Apr 29, 2009, at 10:03 PM, James Gregurich wrote:


> howdy!
>
> question:
>
> for an in-memory db with the threading mode set to serialized, is
>
 the

> internal mutex held for an entire transaction so that one thread
>
 won't

> access the db while another one is in the middle of a transaction
>
 with

> multiple insert statements?
>
 No.  But the mutex is recursive.  So you can get a copy of it using
 sqlite3_db_mutex() then lock it yourself using  
 sqlite3_mutex_enter()/
 leave().

 Also remember:  You should not be using threads.  Threads will bring
 only grief and woe.  On your own head be it.



 D. Richard Hipp
 drh at hwaci.com


>>> ___
>>> sqlite-users mailing list
>>> sqlite-users@sqlite.org
>>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>>
>> ___
>> sqlite-users mailing list
>> sqlite-users@sqlite.org
>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
> 
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] mutex and begin/end transaction

[John Stanton]

I suggest that you peruse the source code of Pthreads to get some idea 
of what you do when you create a thread.  They are not, as some people 
appear to believe, a magic way of making the machine do more and go faster.

Threads are of course invaluable when you want one resource, such as a 
user, to have access to multiple resources.

When you have multiple threads accessing one resource, such as a 
database, then you must have a robust synchronization method, such as a 
mutex, queue or similar.  There is a danger that programmers will 
implement a busy wait to perform their synchronization and get poor 
performance. In some low level, low latency cases however the use of the 
ultimate busy wait, a spin lock, is the best approach.

Pavel Ivanov wrote:
> So you don't even try to argue about database-servers-like
> applications? Apache maybe indeed not very good example because of
> good degree of independence between threads, though I think in some
> cases caching across different connections and thus across different
> threads could be useful.
> 
>> The danger with threading is in concurrent access to data.
> 
> I understand that. That's why concurrent access should be made very
> wisely. But if this concurrent access is to some cache which allows to
> avoid huge amount of disk reads - it's worth the effort. And my
> overall point here is that there's no universal taboo "threads are
> evil in all cases". It totally depends on the type of application.
> 
> Pavel
> 
> On Thu, Apr 30, 2009 at 2:54 PM, Roger Binns  wrote:
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA1
>>
>> Pavel Ivanov wrote:
>>> I'm curious: with all these "warnings based on science" how would you
>>> implement application similar to apache web-server
>> The danger with threading is in concurrent access to data.  Apache has
>> several different modes of operation (forking etc) but in the one that
>> uses threading, each thread handles a connection and does not handle or
>> share any data with other threads during its operation on that
>> connection.  For the non-Windows worker (MPM) it also uses multiple
>> processes with multiple threads and the processes exit after handling a
>> certain number of requests.
>>
>> As an example you can read about mod_wsgi which provides for running
>> Python code as though it was CGI - see http://code.google.com/p/modwsgi/
>>
>> Roger
>> -BEGIN PGP SIGNATURE-
>> Version: GnuPG v1.4.9 (GNU/Linux)
>>
>> iEYEARECAAYFAkn58/MACgkQmOOfHg372QSoOQCfXvz8ef7vS0HP/Uc9hZ/1BQSO
>> Cw0AoMXzgpNwO0PA5uMBvG/DB2Y3lKHG
>> =Cage
>> -END PGP SIGNATURE-
>> ___
>> sqlite-users mailing list
>> sqlite-users@sqlite.org
>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] mutex and begin/end transaction

[James Gregurich]

So, you suggest I should build a commercial desktop application (for  
processing print-industry files and presenting them in a UI)  in such  
a way that it spawns multiple processes and communicates with them via  
the filesystem or IPC APIs?

Why would I want to go to that level of complexity in an  
uncontrollable environment (i.e. a consumer desktop computer) when I  
can just use NSOperation, boost::thread, and boost::mutex to build a  
single-process solution that shares data in a normal way between tasks?

James Gregurich
Engineering Manager
Markzware


On Apr 29, 2009, at 11:23 PM, Roger Binns wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> James Gregurich wrote:
>> Given the industry is going multicore and 16-core macintoshes for  
>> your
>> grand-mother are  just a few years away, I recommend you rethink your
>> position on the use of threading.
>
> Threading is the worst solution to many cpu/core and large memory.
> Having the same memory addressed (which is what threading does) across
> multiple concurrently executing cpus/cores causes cache thrashing,
> memory contention, frequent use of memory barriers for synchronization
> (which also slows things down) and as memory becomes attached to
> individual cpus leads to access being over hypertransport/csi.
>
> Far better is using multiple processes which don't have those  
> issues.  A
> good example application to study is Google Chrome which uses multiple
> processes prolifically - each tab is a separate process, as well as  
> the
> various plugins, languages etc.  Each process can be appropriately
> locked down using the principle of least privilege.  If you use  
> threads
> then typically they all have permission to do anything the process  
> could do.
>
> (Also the multi-process approach is way easier to test, record/replay
> and is more deterministic)
>
> Erlang is also worth studying.  It only has single threaded processes
> (although the processes are lighter weight than operating system  
> processes).
>
>> NSOperation is a major part of that effort.
>
> If the "single encapsulated task" doesn't go around concurrently
> touching bits of memory then it could be shunted to a separate process
> anyway.
>
> The danger from threads is not the threads themselves, but the
> concurrency.  It is virtually impossible to prove that a threaded
> process does the concurrency correctly under all circumstances.  If  
> you
> eliminate the concurrency then you can use multiple processes, and can
> usually even make it scale over multiple machines!
>
> Roger
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.9 (GNU/Linux)
>
> iEYEARECAAYFAkn5Q9MACgkQmOOfHg372QSS/QCfSje/tyX0hmidHyubVKqrXlHt
> Sq0AoKEbmcwx/fmAFtcVeMjbcUgN8dr3
> =8lUQ
> -END PGP SIGNATURE-
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] mutex and begin/end transaction

[Roger Binns]

James Gregurich wrote:
> I want everything in a  
> nice, tidy black-box that the average joe is incredibly unlikely to  
> screw up.

Err, you don't have seventy five executables and expect the user to
manually launch and click them.  It is indeed all hidden.  They see one
window to one program, and the right thing just happens behind the scenes.

> I have no idea what google chrome does and can't comment on
> it. I don't use the app.

I'd strongly recommend looking up articles about the internal
architecture (using Chrome won't tell you much).  Even if you reject it
as a design pattern you should at least be aware of it.  Some starters:

http://arstechnica.com/open-source/news/2008/09/hands-on-with-chrome-googles-browser-shines-mostly.ars
(overview in internals section)
http://arstechnica.com/security/news/2008/09/chrome-antics-did-google-reverse-engineer.ars
(down and dirty nuts and bolts of it)
http://cr.yp.to/qmail/qmailsec-20071101.pdf (ideally read all of it, but
especially section 5)

> Beyond that, I don't see how that approach solves the problem you  
> point out. You still have concurrency going on with shared data  
> structures. 

The point is to send the data and all other information that needs to be
worked on to that "helper" process.  For example a web browser needs the
HTML content, CSS styling and Javascript files all applied to each other
to produce the final rendered document.  That data is not shared or used
concurrently.

> You still have to implement serialization on the shared  
> data structures.

There will still be some of that.  For example in a web browser the
master process would be keeping track of the child processes (per tab),
but the goal is to have the absolute least amount possible.  And by
having the separate processes you know there is less sharing even in the
face of programming mistakes.

> The only thing you gain from that design over a
> threaded design is an extra degree of resiliency in that a crashed  
> task won't bring down the app. On the downside, you have the extra  
> hassle and complication of IPC.

Or if you read the links above and suitably drop permissions from the
processes then you also gain a far greater degree of immunity from other
issues - for example a security bug in an image library in a process
that can only decode images would not be able to touch your file system,
allocate too much memory, use too much cpu, do networking or anything
else inappropriate to decoding an image.

Things like testing get considerably easier since you can test single
task helpers in isolation.  You can also test other parts of your
application with other cooperating processes being "mocks".  You can do
things like upgrading substantial parts of your application while it is
running (probably not applicable to your application).

Any suitably useful program these days will find that it gets data in
increasingly diverse ways.  Consider something like a simple document
viewer which in the olden days pretty much only got fed trusted
documents.  Now the documents could come from email attachments and over
the web.  Embedded images and clipart would previously have come from
Microsoft Office or similar, but now could have been copy and pasted
from anywhere.  (I assume you disallow images in your system or have
code that cannot possibly be compromised by hostile image data :-)

And as you mentioned, SQLite is a good library to use for data
structures both within the process and between processes.  But how many
people even noticed that when used in a threaded fashion, it was
possible for error messages to be deallocated if another thread caused
an error on the same connection?  That could lead to a crash or lots of
string garbage.  I was actually the first to notice and that was long
after people started using sqlite in a multi-threaded way.  That is why
the third paragraph is in http://www.sqlite.org/c3ref/errcode.html and
why sqlite3_mutex api was added in the first place!  Threads are very
hard to get right and require very good programmers, ideally ones who
never have a bad day :-)  Or as DRH simply put it - "evil".

Roger
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] mutex and begin/end transaction

[James Gregurich]

I can't agree that such a thing would be a good approach in a  
commercial desktop application environment. I'd never deploy something  
like that to millions of graphic designers. I want everything in a  
nice, tidy black-box that the average joe is incredibly unlikely to  
screw up. I have no idea what google chrome does and can't comment on  
it. I don't use the app.

Beyond that, I don't see how that approach solves the problem you  
point out. You still have concurrency going on with shared data  
structures. You still have to implement serialization on the shared  
data structures. The only thing you gain from that design over a  
threaded design is an extra degree of resiliency in that a crashed  
task won't bring down the app. On the downside, you have the extra  
hassle and complication of IPC.

The way I guard against a single task bringing the app down is that I  
religiously keep code exception safe, check for NULLs, and use  
shared_ptr's...I expect the same from my staff. Another way to guard  
the app is to minimize the use  of mutexes. Instead of blocking  
threads, you keep your tasks very small and focused, and you set up  
execution dependencies between tasks. Task B can be made not to run  
until task A is completed. Finally, the primary shared data structure  
is a SQLite in-memory store which is wrapped in C++ code that handles  
the dirty details of serializing transactions on the DB.


Handling the limited 32 bit VM space was indeed a challenge. I had to  
come up with a scheme to throttle the task queue once memory  
consumption reached a certain level.


As for the quality of staff members, that is always a challenge. All I  
can do about that is recruit and retain people who are talented and  
can write solid code.

-James


On Apr 30, 2009, at 4:37 PM, Roger Binns wrote:

> James Gregurich wrote:
>> So, you suggest I should build a commercial desktop application (for
>> processing print-industry files and presenting them in a UI)  in such
>> a way that it spawns multiple processes and communicates with them  
>> via
>> the filesystem or IPC APIs?
>
> You obviously know more about your application, APIs, libraries etc  
> but
> it does sound like it would actually be a very good approach.  And of
> course you can also spawn processes on other machines too should the
> need arise.  The description sounds not too different than what Google
> Chrome does.
>
>> Why would I want to go to that level of complexity in an
>> uncontrollable environment (i.e. a consumer desktop computer) when I
>> can just use NSOperation, boost::thread, and boost::mutex to build a
>> single-process solution that shares data in a normal way between  
>> tasks?
>
> Because while you are a perfect programming machine, not everyone else
> who will touch the code in the future is.  As an example if one mutex
> call is left out or the wrong acquired by programming accident, how  
> long
> would it take to know about it and fix it?
>
> If you have to run in a 32 bit address space then that also limits how
> much you can do in one process.  Do you even know how large the  
> maximum
> stack size is per thread and will other coders never exceed that?
> [Don't answer here - its your application, architecture and team :]
>
> Roger
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] mutex and begin/end transaction

[Roger Binns]

James Gregurich wrote:
> So, you suggest I should build a commercial desktop application (for  
> processing print-industry files and presenting them in a UI)  in such  
> a way that it spawns multiple processes and communicates with them via  
> the filesystem or IPC APIs?

You obviously know more about your application, APIs, libraries etc but
it does sound like it would actually be a very good approach.  And of
course you can also spawn processes on other machines too should the
need arise.  The description sounds not too different than what Google
Chrome does.

> Why would I want to go to that level of complexity in an  
> uncontrollable environment (i.e. a consumer desktop computer) when I  
> can just use NSOperation, boost::thread, and boost::mutex to build a  
> single-process solution that shares data in a normal way between tasks?

Because while you are a perfect programming machine, not everyone else
who will touch the code in the future is.  As an example if one mutex
call is left out or the wrong acquired by programming accident, how long
would it take to know about it and fix it?

If you have to run in a 32 bit address space then that also limits how
much you can do in one process.  Do you even know how large the maximum
stack size is per thread and will other coders never exceed that?
[Don't answer here - its your application, architecture and team :]

Roger

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] mutex and begin/end transaction

[Roger Binns]

Pavel Ivanov wrote:
> I understand that. That's why concurrent access should be made very
> wisely. But if this concurrent access is to some cache which allows to
> avoid huge amount of disk reads - it's worth the effort.

The operating system also just happens to have a cache that allows
avoiding disk reads :-)  It will even grow and shrink based on what else
is going on on the machine.  It is even automatically shared amongst all
processes without them having to be specifically coded for it.  [Null
and void for Windows XP]

> And my
> overall point here is that there's no universal taboo "threads are
> evil in all cases". It totally depends on the type of application.

It also depends on the programmer.  And the future.  I am perfectly
willing to grant that you and others are in the top 95th percentile of
programmer ability and talent.  You are unlikely to have too many
threading issues.  But in the future others will be working on it too,
and they won't be the perfect specimens you are.  They'll take
shortcuts, they'll misunderstand the perfect architecture documentation
you created, they'll write the shortest amount of code necessary, they
will test on machines unlikely to cause threading problems etc.

Roger
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] mutex and begin/end transaction

[James Gregurich]

So, you suggest I should build a commercial desktop application (for  
processing print-industry files and presenting them in a UI)  in such  
a way that it spawns multiple processes and communicates with them via  
the filesystem or IPC APIs?

Why would I want to go to that level of complexity in an  
uncontrollable environment (i.e. a consumer desktop computer) when I  
can just use NSOperation, boost::thread, and boost::mutex to build a  
single-process solution that shares data in a normal way between tasks?

James Gregurich
Engineering Manager
Markzware


On Apr 29, 2009, at 11:23 PM, Roger Binns wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> James Gregurich wrote:
>> Given the industry is going multicore and 16-core macintoshes for  
>> your
>> grand-mother are  just a few years away, I recommend you rethink your
>> position on the use of threading.
>
> Threading is the worst solution to many cpu/core and large memory.
> Having the same memory addressed (which is what threading does) across
> multiple concurrently executing cpus/cores causes cache thrashing,
> memory contention, frequent use of memory barriers for synchronization
> (which also slows things down) and as memory becomes attached to
> individual cpus leads to access being over hypertransport/csi.
>
> Far better is using multiple processes which don't have those  
> issues.  A
> good example application to study is Google Chrome which uses multiple
> processes prolifically - each tab is a separate process, as well as  
> the
> various plugins, languages etc.  Each process can be appropriately
> locked down using the principle of least privilege.  If you use  
> threads
> then typically they all have permission to do anything the process  
> could do.
>
> (Also the multi-process approach is way easier to test, record/replay
> and is more deterministic)
>
> Erlang is also worth studying.  It only has single threaded processes
> (although the processes are lighter weight than operating system  
> processes).
>
>> NSOperation is a major part of that effort.
>
> If the "single encapsulated task" doesn't go around concurrently
> touching bits of memory then it could be shunted to a separate process
> anyway.
>
> The danger from threads is not the threads themselves, but the
> concurrency.  It is virtually impossible to prove that a threaded
> process does the concurrency correctly under all circumstances.  If  
> you
> eliminate the concurrency then you can use multiple processes, and can
> usually even make it scale over multiple machines!
>
> Roger
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.9 (GNU/Linux)
>
> iEYEARECAAYFAkn5Q9MACgkQmOOfHg372QSS/QCfSje/tyX0hmidHyubVKqrXlHt
> Sq0AoKEbmcwx/fmAFtcVeMjbcUgN8dr3
> =8lUQ
> -END PGP SIGNATURE-
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] mutex and begin/end transaction

[James Gregurich]

With all due respect,  science itself is a set of  
"positions" (opinions) which are endorsed by small group of people as  
official doctrine after appropriate study. Saying "A 'position' is  
politics, not science" is not a particularly meaningful statement.  If  
you want to argue that point, feel free to send me a private email.

My threaded application works pretty darn well. I can process  
thousands of print industry files on an 8-core system keeping the  
cores busy without lagging the GUI for other applications. Just  
because many people create ill conceived programs doesn't mean  
threaded programs are inherently doomed to be ill-conceived. The  
development tools and techniques for building concurrent systems are  
advancing and making concurrency quite feasible.

James Gregurich
Engineering Manager
Markzware

On Apr 30, 2009, at 5:01 AM, John Stanton wrote:

> A "position" is politics, not science.  Warnings about the use of
> threads are based on science, and advise you to avoid them if possible
> for your own protection.
>
> I see ill conceived programs using threads which go to complex
> synchronization to achieve the equivalent of single stream execution  
> but
> with much greater overhead.  A KISS situation.
>
> James Gregurich wrote:
>> thanks for the info. That should work for me.
>>
>> Given the industry is going multicore and 16-core macintoshes for  
>> your
>> grand-mother are  just a few years away, I recommend you rethink your
>> position on the use of threading. Apple is heavily pushing  
>> parallelism
>> on its developers.  NSOperation is a major part of that effort. As I
>> understand it, MS is developing their copy of NSOperation for VS2010.
>> The development landscape is only going to get more threaded as time
>> goes on.
>>
>> -James
>>
>>
>>
>>> On Apr 29, 2009, at 10:03 PM, James Gregurich wrote:
>>>
>>>
 howdy!

 question:

 for an in-memory db with the threading mode set to serialized, is

>>> the
>>>
 internal mutex held for an entire transaction so that one thread

>>> won't
>>>
 access the db while another one is in the middle of a transaction

>>> with
>>>
 multiple insert statements?

>>> No.  But the mutex is recursive.  So you can get a copy of it using
>>> sqlite3_db_mutex() then lock it yourself using  
>>> sqlite3_mutex_enter()/
>>> leave().
>>>
>>> Also remember:  You should not be using threads.  Threads will bring
>>> only grief and woe.  On your own head be it.
>>>
>>>
>>>
>>> D. Richard Hipp
>>> drh at hwaci.com
>>>
>>>
>> ___
>> sqlite-users mailing list
>> sqlite-users@sqlite.org
>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] mutex and begin/end transaction

[Pavel Ivanov]

So you don't even try to argue about database-servers-like
applications? Apache maybe indeed not very good example because of
good degree of independence between threads, though I think in some
cases caching across different connections and thus across different
threads could be useful.

> The danger with threading is in concurrent access to data.

I understand that. That's why concurrent access should be made very
wisely. But if this concurrent access is to some cache which allows to
avoid huge amount of disk reads - it's worth the effort. And my
overall point here is that there's no universal taboo "threads are
evil in all cases". It totally depends on the type of application.

Pavel

On Thu, Apr 30, 2009 at 2:54 PM, Roger Binns  wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Pavel Ivanov wrote:
>> I'm curious: with all these "warnings based on science" how would you
>> implement application similar to apache web-server
>
> The danger with threading is in concurrent access to data.  Apache has
> several different modes of operation (forking etc) but in the one that
> uses threading, each thread handles a connection and does not handle or
> share any data with other threads during its operation on that
> connection.  For the non-Windows worker (MPM) it also uses multiple
> processes with multiple threads and the processes exit after handling a
> certain number of requests.
>
> As an example you can read about mod_wsgi which provides for running
> Python code as though it was CGI - see http://code.google.com/p/modwsgi/
>
> Roger
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.9 (GNU/Linux)
>
> iEYEARECAAYFAkn58/MACgkQmOOfHg372QSoOQCfXvz8ef7vS0HP/Uc9hZ/1BQSO
> Cw0AoMXzgpNwO0PA5uMBvG/DB2Y3lKHG
> =Cage
> -END PGP SIGNATURE-
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] mutex and begin/end transaction

[Roger Binns]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Pavel Ivanov wrote:
> I'm curious: with all these "warnings based on science" how would you
> implement application similar to apache web-server 

The danger with threading is in concurrent access to data.  Apache has
several different modes of operation (forking etc) but in the one that
uses threading, each thread handles a connection and does not handle or
share any data with other threads during its operation on that
connection.  For the non-Windows worker (MPM) it also uses multiple
processes with multiple threads and the processes exit after handling a
certain number of requests.

As an example you can read about mod_wsgi which provides for running
Python code as though it was CGI - see http://code.google.com/p/modwsgi/

Roger
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkn58/MACgkQmOOfHg372QSoOQCfXvz8ef7vS0HP/Uc9hZ/1BQSO
Cw0AoMXzgpNwO0PA5uMBvG/DB2Y3lKHG
=Cage
-END PGP SIGNATURE-
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] mutex and begin/end transaction

[Pavel Ivanov]

I'm curious: with all these "warnings based on science" how would you
implement application similar to apache web-server (forget about cgis
- just plain HTML) or something like database server? I understand
that you always can write something like cgi application which run on
every web-page hit but even cgis have tendency to move to fast-cgis to
avoid running start-up and tear-down code all the time and thus
significantly improve performance and number of simultaneouse clients
these applications can accept.

Pavel

On Thu, Apr 30, 2009 at 8:01 AM, John Stanton  wrote:
> A "position" is politics, not science.  Warnings about the use of
> threads are based on science, and advise you to avoid them if possible
> for your own protection.
>
> I see ill conceived programs using threads which go to complex
> synchronization to achieve the equivalent of single stream execution but
> with much greater overhead.  A KISS situation.
>
> James Gregurich wrote:
>> thanks for the info. That should work for me.
>>
>> Given the industry is going multicore and 16-core macintoshes for your
>> grand-mother are  just a few years away, I recommend you rethink your
>> position on the use of threading. Apple is heavily pushing parallelism
>> on its developers.  NSOperation is a major part of that effort. As I
>> understand it, MS is developing their copy of NSOperation for VS2010.
>> The development landscape is only going to get more threaded as time
>> goes on.
>>
>> -James
>>
>>
>>
>>> On Apr 29, 2009, at 10:03 PM, James Gregurich wrote:
>>>
>>>
 howdy!

 question:

 for an in-memory db with the threading mode set to serialized, is

>>> the
>>>
 internal mutex held for an entire transaction so that one thread

>>> won't
>>>
 access the db while another one is in the middle of a transaction

>>> with
>>>
 multiple insert statements?

>>> No.  But the mutex is recursive.  So you can get a copy of it using
>>> sqlite3_db_mutex() then lock it yourself using sqlite3_mutex_enter()/
>>> leave().
>>>
>>> Also remember:  You should not be using threads.  Threads will bring
>>> only grief and woe.  On your own head be it.
>>>
>>>
>>>
>>> D. Richard Hipp
>>> drh at hwaci.com
>>>
>>>
>> ___
>> sqlite-users mailing list
>> sqlite-users@sqlite.org
>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] mutex and begin/end transaction

[John Stanton]

A "position" is politics, not science.  Warnings about the use of 
threads are based on science, and advise you to avoid them if possible 
for your own protection.

I see ill conceived programs using threads which go to complex 
synchronization to achieve the equivalent of single stream execution but 
with much greater overhead.  A KISS situation.

James Gregurich wrote:
> thanks for the info. That should work for me.
>
> Given the industry is going multicore and 16-core macintoshes for your  
> grand-mother are  just a few years away, I recommend you rethink your  
> position on the use of threading. Apple is heavily pushing parallelism  
> on its developers.  NSOperation is a major part of that effort. As I  
> understand it, MS is developing their copy of NSOperation for VS2010.  
> The development landscape is only going to get more threaded as time  
> goes on.
>
> -James
>
>
>   
>> On Apr 29, 2009, at 10:03 PM, James Gregurich wrote:
>>
>> 
>>> howdy!
>>>
>>> question:
>>>
>>> for an in-memory db with the threading mode set to serialized, is  
>>>   
>> the
>> 
>>> internal mutex held for an entire transaction so that one thread  
>>>   
>> won't
>> 
>>> access the db while another one is in the middle of a transaction  
>>>   
>> with
>> 
>>> multiple insert statements?
>>>   
>> No.  But the mutex is recursive.  So you can get a copy of it using
>> sqlite3_db_mutex() then lock it yourself using sqlite3_mutex_enter()/
>> leave().
>>
>> Also remember:  You should not be using threads.  Threads will bring
>> only grief and woe.  On your own head be it.
>>
>>
>>
>> D. Richard Hipp
>> drh at hwaci.com
>>
>> 
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>   

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] mutex and begin/end transaction

[Roger Binns]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

James Gregurich wrote:
> Given the industry is going multicore and 16-core macintoshes for your  
> grand-mother are  just a few years away, I recommend you rethink your  
> position on the use of threading. 

Threading is the worst solution to many cpu/core and large memory.
Having the same memory addressed (which is what threading does) across
multiple concurrently executing cpus/cores causes cache thrashing,
memory contention, frequent use of memory barriers for synchronization
(which also slows things down) and as memory becomes attached to
individual cpus leads to access being over hypertransport/csi.

Far better is using multiple processes which don't have those issues.  A
good example application to study is Google Chrome which uses multiple
processes prolifically - each tab is a separate process, as well as the
various plugins, languages etc.  Each process can be appropriately
locked down using the principle of least privilege.  If you use threads
then typically they all have permission to do anything the process could do.

(Also the multi-process approach is way easier to test, record/replay
and is more deterministic)

Erlang is also worth studying.  It only has single threaded processes
(although the processes are lighter weight than operating system processes).

> NSOperation is a major part of that effort.

If the "single encapsulated task" doesn't go around concurrently
touching bits of memory then it could be shunted to a separate process
anyway.

The danger from threads is not the threads themselves, but the
concurrency.  It is virtually impossible to prove that a threaded
process does the concurrency correctly under all circumstances.  If you
eliminate the concurrency then you can use multiple processes, and can
usually even make it scale over multiple machines!

Roger
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkn5Q9MACgkQmOOfHg372QSS/QCfSje/tyX0hmidHyubVKqrXlHt
Sq0AoKEbmcwx/fmAFtcVeMjbcUgN8dr3
=8lUQ
-END PGP SIGNATURE-
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] mutex and begin/end transaction

[James Gregurich]

thanks for the info. That should work for me.

Given the industry is going multicore and 16-core macintoshes for your  
grand-mother are  just a few years away, I recommend you rethink your  
position on the use of threading. Apple is heavily pushing parallelism  
on its developers.  NSOperation is a major part of that effort. As I  
understand it, MS is developing their copy of NSOperation for VS2010.  
The development landscape is only going to get more threaded as time  
goes on.

-James


> On Apr 29, 2009, at 10:03 PM, James Gregurich wrote:
>
> > howdy!
> >
> > question:
> >
> > for an in-memory db with the threading mode set to serialized, is  
> the
> > internal mutex held for an entire transaction so that one thread  
> won't
> > access the db while another one is in the middle of a transaction  
> with
> > multiple insert statements?
>
>
> No.  But the mutex is recursive.  So you can get a copy of it using
> sqlite3_db_mutex() then lock it yourself using sqlite3_mutex_enter()/
> leave().
>
> Also remember:  You should not be using threads.  Threads will bring
> only grief and woe.  On your own head be it.
>
>
>
> D. Richard Hipp
> drh at hwaci.com
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] mutex and begin/end transaction

[D. Richard Hipp]

On Apr 29, 2009, at 10:03 PM, James Gregurich wrote:

> howdy!
>
> question:
>
> for an in-memory db with the threading mode set to serialized, is the
> internal mutex held for an entire transaction so that one thread won't
> access the db while another one is in the middle of a transaction with
> multiple insert statements?


No.  But the mutex is recursive.  So you can get a copy of it using  
sqlite3_db_mutex() then lock it yourself using sqlite3_mutex_enter()/ 
leave().

Also remember:  You should not be using threads.  Threads will bring  
only grief and woe.  On your own head be it.



D. Richard Hipp
d...@hwaci.com



___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users