Re: [sqlite] sqlite3_mutex_enter(db->mutex) problem
On May 1, 2009, at 7:58 AM, Joanne Pham wrote: > Hi All, > I have the application and it is crashed on at ../src/vdbeapi.c:538 > (sqlite3_mutex_enter(db->mutex); by sqlite3_step. > I couldn't nail down what was the root cause of this problem. Why it > crashed on sqlite3_mutex_enter API. > Would you please shed some light on this? > Thank in advance, > JP > > Below is core file: > sqlite3_step (pStmt=0x4012d0c3) at ../src/vdbeapi.c:538 > > at ../src/vdbeapi.c:538 is > sqlite3_mutex_enter(db->mutex); > while( (rc = sqlite3Step(v))==SQLITE_SCHEMA There is something wrong with the statement handle that you are passing to sqlite3_step(). Possibly the handle has already been passed to sqlite3_finalize() already. Dan. > > > > > > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] mutex and begin/end transaction
James Gregurich wrote: > I want everything in a > nice, tidy black-box that the average joe is incredibly unlikely to > screw up. Err, you don't have seventy five executables and expect the user to manually launch and click them. It is indeed all hidden. They see one window to one program, and the right thing just happens behind the scenes. > I have no idea what google chrome does and can't comment on > it. I don't use the app. I'd strongly recommend looking up articles about the internal architecture (using Chrome won't tell you much). Even if you reject it as a design pattern you should at least be aware of it. Some starters: http://arstechnica.com/open-source/news/2008/09/hands-on-with-chrome-googles-browser-shines-mostly.ars (overview in internals section) http://arstechnica.com/security/news/2008/09/chrome-antics-did-google-reverse-engineer.ars (down and dirty nuts and bolts of it) http://cr.yp.to/qmail/qmailsec-20071101.pdf (ideally read all of it, but especially section 5) > Beyond that, I don't see how that approach solves the problem you > point out. You still have concurrency going on with shared data > structures. The point is to send the data and all other information that needs to be worked on to that "helper" process. For example a web browser needs the HTML content, CSS styling and Javascript files all applied to each other to produce the final rendered document. That data is not shared or used concurrently. > You still have to implement serialization on the shared > data structures. There will still be some of that. For example in a web browser the master process would be keeping track of the child processes (per tab), but the goal is to have the absolute least amount possible. And by having the separate processes you know there is less sharing even in the face of programming mistakes. > The only thing you gain from that design over a > threaded design is an extra degree of resiliency in that a crashed > task won't bring down the app. On the downside, you have the extra > hassle and complication of IPC. Or if you read the links above and suitably drop permissions from the processes then you also gain a far greater degree of immunity from other issues - for example a security bug in an image library in a process that can only decode images would not be able to touch your file system, allocate too much memory, use too much cpu, do networking or anything else inappropriate to decoding an image. Things like testing get considerably easier since you can test single task helpers in isolation. You can also test other parts of your application with other cooperating processes being "mocks". You can do things like upgrading substantial parts of your application while it is running (probably not applicable to your application). Any suitably useful program these days will find that it gets data in increasingly diverse ways. Consider something like a simple document viewer which in the olden days pretty much only got fed trusted documents. Now the documents could come from email attachments and over the web. Embedded images and clipart would previously have come from Microsoft Office or similar, but now could have been copy and pasted from anywhere. (I assume you disallow images in your system or have code that cannot possibly be compromised by hostile image data :-) And as you mentioned, SQLite is a good library to use for data structures both within the process and between processes. But how many people even noticed that when used in a threaded fashion, it was possible for error messages to be deallocated if another thread caused an error on the same connection? That could lead to a crash or lots of string garbage. I was actually the first to notice and that was long after people started using sqlite in a multi-threaded way. That is why the third paragraph is in http://www.sqlite.org/c3ref/errcode.html and why sqlite3_mutex api was added in the first place! Threads are very hard to get right and require very good programmers, ideally ones who never have a bad day :-) Or as DRH simply put it - "evil". Roger ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] sqlite3_mutex_enter(db->mutex) problem
Hi All, I have the application and it is crashed on at ../src/vdbeapi.c:538 (sqlite3_mutex_enter(db->mutex); by sqlite3_step. I couldn't nail down what was the root cause of this problem. Why it crashed on sqlite3_mutex_enter API. Would you please shed some light on this? Thank in advance, JP Below is core file: sqlite3_step (pStmt=0x4012d0c3) at ../src/vdbeapi.c:538 at ../src/vdbeapi.c:538 is sqlite3_mutex_enter(db->mutex); while( (rc = sqlite3Step(v))==SQLITE_SCHEMA ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] mutex and begin/end transaction
I can't agree that such a thing would be a good approach in a commercial desktop application environment. I'd never deploy something like that to millions of graphic designers. I want everything in a nice, tidy black-box that the average joe is incredibly unlikely to screw up. I have no idea what google chrome does and can't comment on it. I don't use the app. Beyond that, I don't see how that approach solves the problem you point out. You still have concurrency going on with shared data structures. You still have to implement serialization on the shared data structures. The only thing you gain from that design over a threaded design is an extra degree of resiliency in that a crashed task won't bring down the app. On the downside, you have the extra hassle and complication of IPC. The way I guard against a single task bringing the app down is that I religiously keep code exception safe, check for NULLs, and use shared_ptr's...I expect the same from my staff. Another way to guard the app is to minimize the use of mutexes. Instead of blocking threads, you keep your tasks very small and focused, and you set up execution dependencies between tasks. Task B can be made not to run until task A is completed. Finally, the primary shared data structure is a SQLite in-memory store which is wrapped in C++ code that handles the dirty details of serializing transactions on the DB. Handling the limited 32 bit VM space was indeed a challenge. I had to come up with a scheme to throttle the task queue once memory consumption reached a certain level. As for the quality of staff members, that is always a challenge. All I can do about that is recruit and retain people who are talented and can write solid code. -James On Apr 30, 2009, at 4:37 PM, Roger Binns wrote: > James Gregurich wrote: >> So, you suggest I should build a commercial desktop application (for >> processing print-industry files and presenting them in a UI) in such >> a way that it spawns multiple processes and communicates with them >> via >> the filesystem or IPC APIs? > > You obviously know more about your application, APIs, libraries etc > but > it does sound like it would actually be a very good approach. And of > course you can also spawn processes on other machines too should the > need arise. The description sounds not too different than what Google > Chrome does. > >> Why would I want to go to that level of complexity in an >> uncontrollable environment (i.e. a consumer desktop computer) when I >> can just use NSOperation, boost::thread, and boost::mutex to build a >> single-process solution that shares data in a normal way between >> tasks? > > Because while you are a perfect programming machine, not everyone else > who will touch the code in the future is. As an example if one mutex > call is left out or the wrong acquired by programming accident, how > long > would it take to know about it and fix it? > > If you have to run in a 32 bit address space then that also limits how > much you can do in one process. Do you even know how large the > maximum > stack size is per thread and will other coders never exceed that? > [Don't answer here - its your application, architecture and team :] > > Roger > > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] mutex and begin/end transaction
James Gregurich wrote: > So, you suggest I should build a commercial desktop application (for > processing print-industry files and presenting them in a UI) in such > a way that it spawns multiple processes and communicates with them via > the filesystem or IPC APIs? You obviously know more about your application, APIs, libraries etc but it does sound like it would actually be a very good approach. And of course you can also spawn processes on other machines too should the need arise. The description sounds not too different than what Google Chrome does. > Why would I want to go to that level of complexity in an > uncontrollable environment (i.e. a consumer desktop computer) when I > can just use NSOperation, boost::thread, and boost::mutex to build a > single-process solution that shares data in a normal way between tasks? Because while you are a perfect programming machine, not everyone else who will touch the code in the future is. As an example if one mutex call is left out or the wrong acquired by programming accident, how long would it take to know about it and fix it? If you have to run in a 32 bit address space then that also limits how much you can do in one process. Do you even know how large the maximum stack size is per thread and will other coders never exceed that? [Don't answer here - its your application, architecture and team :] Roger ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] mutex and begin/end transaction
Pavel Ivanov wrote: > I understand that. That's why concurrent access should be made very > wisely. But if this concurrent access is to some cache which allows to > avoid huge amount of disk reads - it's worth the effort. The operating system also just happens to have a cache that allows avoiding disk reads :-) It will even grow and shrink based on what else is going on on the machine. It is even automatically shared amongst all processes without them having to be specifically coded for it. [Null and void for Windows XP] > And my > overall point here is that there's no universal taboo "threads are > evil in all cases". It totally depends on the type of application. It also depends on the programmer. And the future. I am perfectly willing to grant that you and others are in the top 95th percentile of programmer ability and talent. You are unlikely to have too many threading issues. But in the future others will be working on it too, and they won't be the perfect specimens you are. They'll take shortcuts, they'll misunderstand the perfect architecture documentation you created, they'll write the shortest amount of code necessary, they will test on machines unlikely to cause threading problems etc. Roger ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] mutex and begin/end transaction
So, you suggest I should build a commercial desktop application (for processing print-industry files and presenting them in a UI) in such a way that it spawns multiple processes and communicates with them via the filesystem or IPC APIs? Why would I want to go to that level of complexity in an uncontrollable environment (i.e. a consumer desktop computer) when I can just use NSOperation, boost::thread, and boost::mutex to build a single-process solution that shares data in a normal way between tasks? James Gregurich Engineering Manager Markzware On Apr 29, 2009, at 11:23 PM, Roger Binns wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > James Gregurich wrote: >> Given the industry is going multicore and 16-core macintoshes for >> your >> grand-mother are just a few years away, I recommend you rethink your >> position on the use of threading. > > Threading is the worst solution to many cpu/core and large memory. > Having the same memory addressed (which is what threading does) across > multiple concurrently executing cpus/cores causes cache thrashing, > memory contention, frequent use of memory barriers for synchronization > (which also slows things down) and as memory becomes attached to > individual cpus leads to access being over hypertransport/csi. > > Far better is using multiple processes which don't have those > issues. A > good example application to study is Google Chrome which uses multiple > processes prolifically - each tab is a separate process, as well as > the > various plugins, languages etc. Each process can be appropriately > locked down using the principle of least privilege. If you use > threads > then typically they all have permission to do anything the process > could do. > > (Also the multi-process approach is way easier to test, record/replay > and is more deterministic) > > Erlang is also worth studying. It only has single threaded processes > (although the processes are lighter weight than operating system > processes). > >> NSOperation is a major part of that effort. > > If the "single encapsulated task" doesn't go around concurrently > touching bits of memory then it could be shunted to a separate process > anyway. > > The danger from threads is not the threads themselves, but the > concurrency. It is virtually impossible to prove that a threaded > process does the concurrency correctly under all circumstances. If > you > eliminate the concurrency then you can use multiple processes, and can > usually even make it scale over multiple machines! > > Roger > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.9 (GNU/Linux) > > iEYEARECAAYFAkn5Q9MACgkQmOOfHg372QSS/QCfSje/tyX0hmidHyubVKqrXlHt > Sq0AoKEbmcwx/fmAFtcVeMjbcUgN8dr3 > =8lUQ > -END PGP SIGNATURE- > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] mutex and begin/end transaction
With all due respect, science itself is a set of "positions" (opinions) which are endorsed by small group of people as official doctrine after appropriate study. Saying "A 'position' is politics, not science" is not a particularly meaningful statement. If you want to argue that point, feel free to send me a private email. My threaded application works pretty darn well. I can process thousands of print industry files on an 8-core system keeping the cores busy without lagging the GUI for other applications. Just because many people create ill conceived programs doesn't mean threaded programs are inherently doomed to be ill-conceived. The development tools and techniques for building concurrent systems are advancing and making concurrency quite feasible. James Gregurich Engineering Manager Markzware On Apr 30, 2009, at 5:01 AM, John Stanton wrote: > A "position" is politics, not science. Warnings about the use of > threads are based on science, and advise you to avoid them if possible > for your own protection. > > I see ill conceived programs using threads which go to complex > synchronization to achieve the equivalent of single stream execution > but > with much greater overhead. A KISS situation. > > James Gregurich wrote: >> thanks for the info. That should work for me. >> >> Given the industry is going multicore and 16-core macintoshes for >> your >> grand-mother are just a few years away, I recommend you rethink your >> position on the use of threading. Apple is heavily pushing >> parallelism >> on its developers. NSOperation is a major part of that effort. As I >> understand it, MS is developing their copy of NSOperation for VS2010. >> The development landscape is only going to get more threaded as time >> goes on. >> >> -James >> >> >> >>> On Apr 29, 2009, at 10:03 PM, James Gregurich wrote: >>> >>> howdy! question: for an in-memory db with the threading mode set to serialized, is >>> the >>> internal mutex held for an entire transaction so that one thread >>> won't >>> access the db while another one is in the middle of a transaction >>> with >>> multiple insert statements? >>> No. But the mutex is recursive. So you can get a copy of it using >>> sqlite3_db_mutex() then lock it yourself using >>> sqlite3_mutex_enter()/ >>> leave(). >>> >>> Also remember: You should not be using threads. Threads will bring >>> only grief and woe. On your own head be it. >>> >>> >>> >>> D. Richard Hipp >>> drh at hwaci.com >>> >>> >> ___ >> sqlite-users mailing list >> sqlite-users@sqlite.org >> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users >> > > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] sqlite performance tuning and optimization
I am using SQLIte3 database and my query is running slow sometimes,sometimes it runs fast. Is there any sqlite Profiler available just like SQL Profiler for SQL server? My query looks like "SELECT ID, EventClassName, EventClassRef, TransitionTime, Message, MonitoredRef, EventRef,ToState,Priority,Acked from Events WHERE Events.NotificationTime >= "&GetDateTimeAsString(LastEvent)&" " and ID > "&lastID & " Order By ID DESC LIMIT 100" Recordset CursorLocation = 2,CursorType = 2,LockType = 3 My Events table has 2259205 records. ID is my primary key so index is on that field. NOtification time also have index on it. The page running this query is executing in 312 ms -2.5 seconds on Windows XP machine. The page running this query is executing in 366 ms -2.8 minutes on Windows Web server 2008 machine. IS there any way I can figure out why its taking so long to run?? Thanks, -KPH ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] mutex and begin/end transaction
So you don't even try to argue about database-servers-like applications? Apache maybe indeed not very good example because of good degree of independence between threads, though I think in some cases caching across different connections and thus across different threads could be useful. > The danger with threading is in concurrent access to data. I understand that. That's why concurrent access should be made very wisely. But if this concurrent access is to some cache which allows to avoid huge amount of disk reads - it's worth the effort. And my overall point here is that there's no universal taboo "threads are evil in all cases". It totally depends on the type of application. Pavel On Thu, Apr 30, 2009 at 2:54 PM, Roger Binns wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > Pavel Ivanov wrote: >> I'm curious: with all these "warnings based on science" how would you >> implement application similar to apache web-server > > The danger with threading is in concurrent access to data. Apache has > several different modes of operation (forking etc) but in the one that > uses threading, each thread handles a connection and does not handle or > share any data with other threads during its operation on that > connection. For the non-Windows worker (MPM) it also uses multiple > processes with multiple threads and the processes exit after handling a > certain number of requests. > > As an example you can read about mod_wsgi which provides for running > Python code as though it was CGI - see http://code.google.com/p/modwsgi/ > > Roger > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.9 (GNU/Linux) > > iEYEARECAAYFAkn58/MACgkQmOOfHg372QSoOQCfXvz8ef7vS0HP/Uc9hZ/1BQSO > Cw0AoMXzgpNwO0PA5uMBvG/DB2Y3lKHG > =Cage > -END PGP SIGNATURE- > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] mutex and begin/end transaction
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Pavel Ivanov wrote: > I'm curious: with all these "warnings based on science" how would you > implement application similar to apache web-server The danger with threading is in concurrent access to data. Apache has several different modes of operation (forking etc) but in the one that uses threading, each thread handles a connection and does not handle or share any data with other threads during its operation on that connection. For the non-Windows worker (MPM) it also uses multiple processes with multiple threads and the processes exit after handling a certain number of requests. As an example you can read about mod_wsgi which provides for running Python code as though it was CGI - see http://code.google.com/p/modwsgi/ Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAkn58/MACgkQmOOfHg372QSoOQCfXvz8ef7vS0HP/Uc9hZ/1BQSO Cw0AoMXzgpNwO0PA5uMBvG/DB2Y3lKHG =Cage -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Transaction control At the SQL Level
Again thanks Igor for detail information about autocommit for the sqlite database connection. Thanks, JP From: Igor Tandetnik To: sqlite-users@sqlite.org Sent: Thursday, April 30, 2009 10:24:16 AM Subject: Re: [sqlite] Transaction control At the SQL Level Joanne Pham wrote: > Just want to make sure that I am fully understand about the single > database connection with multiple database statement handle here. > For example I have one database connection and 2 database statement > handle using the same connection. Using the first database statement > handle I use to select the data from database but not yet > sqlite3_reset nor sqlite3_finalize. The second database statement > handle I use to delete the data from database and use sqlite3_reset > or sqlite3_finalize for the second database handle statement. So > "delete" statement of the second database statement handle doesn't > commit to the database until the first database statement handle > sqlite3_reset or sqlite3_finalize right. Correct. Igor Tandetnik ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Transaction control At the SQL Level
Joanne Pham wrote: > Just want to make sure that I am fully understand about the single > database connection with multiple database statement handle here. > For example I have one database connection and 2 database statement > handle using the same connection. Using the first database statement > handle I use to select the data from database but not yet > sqlite3_reset nor sqlite3_finalize. The second database statement > handle I use to delete the data from database and use sqlite3_reset > or sqlite3_finalize for the second database handle statement. So > "delete" statement of the second database statement handle doesn't > commit to the database until the first database statement handle > sqlite3_reset or sqlite3_finalize right. Correct. Igor Tandetnik ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Transaction control At the SQL Level
Thanks a lot Igor respond my email. Just want to make sure that I am fully understand about the single database connection with multiple database statement handle here. For example I have one database connection and 2 database statement handle using the same connection. Using the first database statement handle I use to select the data from database but not yet sqlite3_reset nor sqlite3_finalize. The second database statement handle I use to delete the data from database and use sqlite3_reset or sqlite3_finalize for the second database handle statement. So "delete" statement of the second database statement handle doesn't commit to the database until the first database statement handle sqlite3_reset or sqlite3_finalize right. Once again thank for respond my email JP From: Igor Tandetnik To: sqlite-users@sqlite.org Sent: Wednesday, April 29, 2009 7:48:41 PM Subject: Re: [sqlite] Transaction control At the SQL Level "Joanne Pham" wrote in message news:464293.67815...@web90308.mail.mud.yahoo.com > 1) : If I have mulitple commands which are used the same SQL database > connection then all commands after the first won't commit to the > database if the first one is not completed" Correct. > 2) Is that sqlite3_reset will be the command to completed the > statement? That, or sqlite3_finalize. > 3) From document above it seems like the "Autocommited" is for each > SQLite database connection - So if I have serveral commands are using > the same connection then is there any command that I can use to > commit each individual command but not wait until the first statement > finishes. No. Igor Tandetnik ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] in memory database and jdbc
2009/4/29 Tom van Ees :
> Hi,
>
> my java application uses a 80Mb reference database (read-only) that needs to
> be consulted app. 4M times during a batch run. I would like to use the
> in-memory capabilities of sqlite3 to improve the performance of my app.
>
> In order to do so, I setup a Connection to the -in-memory- sqlite database
> and next I would like to load the file-version of the reference database as
> efficiently as possible into the in-memory version. What would be the
> practical approach? The .restore/.backup commands do not seem to agree with
> the jdbc-driver.
Perhaps jdbc-driver needs update to implement .restore/.backup?
Anyway, the following tcl script should give an idea of what would work...
# create memory db connection
#
sqlite3 db :memory:
# attach to the source file
#
db eval "attach 'mySource.db' as srcDb;"
# copy schema tables
#
db eval "
select sql from srcDb.sqlite_master
where type = 'table';
" {
db eval "$sql"
}
# copy the data for all tables
#
db eval "
select name from srcDb.sqlite_master
where type = 'table';
" {
db eval "insert into $name select * from srcDb.$name;"
}
# copy schema views, indexes and triggers (+any other non-tables that
I have not thought of)
#
db eval "
select sql from srcDb.sqlite_master
where type <> 'table';
" {
db eval "$sql"
}
# tidy up
#
db eval "detach srcDb;"
>
> regards
>
> Tom
>
Rgds,
Simon
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] mutex and begin/end transaction
I'm curious: with all these "warnings based on science" how would you implement application similar to apache web-server (forget about cgis - just plain HTML) or something like database server? I understand that you always can write something like cgi application which run on every web-page hit but even cgis have tendency to move to fast-cgis to avoid running start-up and tear-down code all the time and thus significantly improve performance and number of simultaneouse clients these applications can accept. Pavel On Thu, Apr 30, 2009 at 8:01 AM, John Stanton wrote: > A "position" is politics, not science. Warnings about the use of > threads are based on science, and advise you to avoid them if possible > for your own protection. > > I see ill conceived programs using threads which go to complex > synchronization to achieve the equivalent of single stream execution but > with much greater overhead. A KISS situation. > > James Gregurich wrote: >> thanks for the info. That should work for me. >> >> Given the industry is going multicore and 16-core macintoshes for your >> grand-mother are just a few years away, I recommend you rethink your >> position on the use of threading. Apple is heavily pushing parallelism >> on its developers. NSOperation is a major part of that effort. As I >> understand it, MS is developing their copy of NSOperation for VS2010. >> The development landscape is only going to get more threaded as time >> goes on. >> >> -James >> >> >> >>> On Apr 29, 2009, at 10:03 PM, James Gregurich wrote: >>> >>> howdy! question: for an in-memory db with the threading mode set to serialized, is >>> the >>> internal mutex held for an entire transaction so that one thread >>> won't >>> access the db while another one is in the middle of a transaction >>> with >>> multiple insert statements? >>> No. But the mutex is recursive. So you can get a copy of it using >>> sqlite3_db_mutex() then lock it yourself using sqlite3_mutex_enter()/ >>> leave(). >>> >>> Also remember: You should not be using threads. Threads will bring >>> only grief and woe. On your own head be it. >>> >>> >>> >>> D. Richard Hipp >>> drh at hwaci.com >>> >>> >> ___ >> sqlite-users mailing list >> sqlite-users@sqlite.org >> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users >> > > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] mutex and begin/end transaction
A "position" is politics, not science. Warnings about the use of threads are based on science, and advise you to avoid them if possible for your own protection. I see ill conceived programs using threads which go to complex synchronization to achieve the equivalent of single stream execution but with much greater overhead. A KISS situation. James Gregurich wrote: > thanks for the info. That should work for me. > > Given the industry is going multicore and 16-core macintoshes for your > grand-mother are just a few years away, I recommend you rethink your > position on the use of threading. Apple is heavily pushing parallelism > on its developers. NSOperation is a major part of that effort. As I > understand it, MS is developing their copy of NSOperation for VS2010. > The development landscape is only going to get more threaded as time > goes on. > > -James > > > >> On Apr 29, 2009, at 10:03 PM, James Gregurich wrote: >> >> >>> howdy! >>> >>> question: >>> >>> for an in-memory db with the threading mode set to serialized, is >>> >> the >> >>> internal mutex held for an entire transaction so that one thread >>> >> won't >> >>> access the db while another one is in the middle of a transaction >>> >> with >> >>> multiple insert statements? >>> >> No. But the mutex is recursive. So you can get a copy of it using >> sqlite3_db_mutex() then lock it yourself using sqlite3_mutex_enter()/ >> leave(). >> >> Also remember: You should not be using threads. Threads will bring >> only grief and woe. On your own head be it. >> >> >> >> D. Richard Hipp >> drh at hwaci.com >> >> > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] in memory database and jdbc
Hi, my java application uses a 80Mb reference database (read-only) that needs to be consulted app. 4M times during a batch run. I would like to use the in-memory capabilities of sqlite3 to improve the performance of my app. In order to do so, I setup a Connection to the -in-memory- sqlite database and next I would like to load the file-version of the reference database as efficiently as possible into the in-memory version. What would be the practical approach? The .restore/.backup commands do not seem to agree with the jdbc-driver. regards Tom ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] the speed of embedded database engines, sqlite3 vs berkeley db, I'm confused
"Matthew L. Creech" wrote in message news:5ee96a840904271946o315df05dxb45024d5c0474...@mail.gmail.com... > On Mon, Apr 27, 2009 at 10:25 PM, liubin liu > <7101227-k+ct0dcb...@public.gmane.org> wrote: >> >> thanks >> >> I'm not sure of the real reason. >> It's true that the speed of inserting with transaction is very fast. But >> my >> project is using SQLite mainly in selecting something. I don't know how >> to >> use transaction in the situation. May some friends give me some codes on >> that? >> > > As someone else mentioned, you probably don't want to open/close the > database every time you do a single SELECT. In the example you're > doing 100,000 iterations of open/exec/close, which will give pretty > terrible performance. > > Instead, try creating an initialization call, which opens the > database, prepare()s your SELECT stmt (with a '?' placeholder for the > ID in this example), and returns a handle to the caller. The caller > can then loop 100,000 times calling getdata() with this handle. > getdata() can just do sqlite3_bind_XXX() followed by sqlite3_step() > (and sqlite3_reset()), which will be _much_ faster than > sqlite3_exec(). > > -- > Matthew L. Creech > If he wants go faster he can use BEGIN TRANSACTION (EXCLUSIVE or other) before the 100,000 iterations. In this way sqlite doesn't lock the database file 100.000 times, but only 1 I don't know if he can use this trick, but maybe he can group 10, 50 or 100 select into a TRANSACTION. Giacomo Mussati ___ > sqlite-users mailing list > sqlite-users-czdrofg0bjidnm+yrof...@public.gmane.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] the speed of embedded database engines, sqlite3 vs berkeley db, I'm confused
"Matthew L. Creech" wrote in message news:5ee96a840904271946o315df05dxb45024d5c0474...@mail.gmail.com... > On Mon, Apr 27, 2009 at 10:25 PM, liubin liu > <7101227-k+ct0dcb...@public.gmane.org> wrote: >> >> thanks >> >> I'm not sure of the real reason. >> It's true that the speed of inserting with transaction is very fast. But >> my >> project is using SQLite mainly in selecting something. I don't know how >> to >> use transaction in the situation. May some friends give me some codes on >> that? >> > > As someone else mentioned, you probably don't want to open/close the > database every time you do a single SELECT. In the example you're > doing 100,000 iterations of open/exec/close, which will give pretty > terrible performance. > > Instead, try creating an initialization call, which opens the > database, prepare()s your SELECT stmt (with a '?' placeholder for the > ID in this example), and returns a handle to the caller. The caller > can then loop 100,000 times calling getdata() with this handle. > getdata() can just do sqlite3_bind_XXX() followed by sqlite3_step() > (and sqlite3_reset()), which will be _much_ faster than > sqlite3_exec(). > > -- > Matthew L. Creech > If he wants go faster he can use BEGIN TRANSACTION (EXCLUSIVE or other) before the 100,000 iterations. In this way sqlite doesn't lock the database file 100.000 times, but only 1 I don't know if he can use this trick, but maybe he can group 10, 50 or 100 select into a TRANSACTION. Giacomo Mussati ___ > sqlite-users mailing list > sqlite-users-czdrofg0bjidnm+yrof...@public.gmane.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] ResultSetMetaData with javasqlite
Justin Deoliveira wrote: Hi Justin, > Thanks for the quick response. I am not sure if this will be of any help > or not but I wrote this (very crude) patch to get around the problem: > > http://files.opengeo.org/sqlite_jni.c.patch > > ... Based on your patch there's now a improved version in http://www.ch-werner.de/javasqlite Best regards, Christian ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Database disk image is malformed
Hi, I have an application running on Windows CE4.1 PDAs with SD cards. Sometimes I have errors executing commands on database that are: 11-database disk image is malformed This happens for example on table MSTBP executing "Delete From MSTBP". Or inserting into it. The strange thing is, that after I restart my application (hence the connection) the error disappears. Everything continues working normally and the previous SQL statements, that before generated this error, start working normally. Any ideas how I can prevent this from happening? I looked into pragma and specifically into: PRAGMA journal_mode = /PERSIST /Do you thing this may help? Do you think that maybe this problem is because at a given time, the sqlite engine could not "access" the journal file? Some strange hardware or OS problem that did not created/deleted the file immediately, mainly because it is on a SD Card? Thanks -- Cumprimentos / Best Regards Filipe Madureira - SYSDEV, LDA - Mobile Solutions (www.sysdevsolutions.com) Tel: +351 234098066 Fax: +351 234188400 - ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
