Re: [sqlite] Use sqlite3_update_hook() to watch for changes
One method you can use to see changes is to hook up triggers on update/insert. Since you can add functions to be called from SQL, you just have to add whatever IPC system you desire as an SQLite function (see sqlite_create_function). This function can then be invoked through your triggers, and away you go! Frosstoise. On Fri, Mar 20, 2009 at 10:22 PM, Ricky Huangwrote: > Here's my scenario, I have two applications, reader and a writer, and > a shared database. Writer will be writing changes to the database and > reader will be reading. Originally my idea was for the reader to use > sqlite3_update_hook() to watch for database changes. But after I got > the code written, the hooked function was never called. A little > inter-tubes research revealed this: http://tinyurl.com/dc279r. It > basically says sqlite3_update_hook() is not designed for that. > > My question is, does SQLite offer ways of hooking up a second > application to changes in a shared DB without polling? > > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Transaction behavior with large numbers of inserts
So thinking about this, what would be the impediment (apart from current implementation, of course,) to implementing a disk-based temp file page cache, that could grow arbitrarily large? Operations against the page cache in memory are already lost if your process dies before commission. Are there complications that keep this cache from being backed by the file system? Also, I've seen hints of a new pluggable page-cache system being implemented. Would this type of solution be appropriate for implementation using this new methodology? Cheers, Ian On Wed, Jan 28, 2009 at 12:54 PM,wrote: > Hi, > > Does your reader threads access to the same table as your writer ? > > Maybe (I'm not 100% sure it works) you could try : > on a single file database, with the PRAGMA read_uncommited=1 on the > readers connections (this is not a good idea if the tables are the same > between readers and writers, it depends on what degree of reliability you > need on your data), and with shared_cache_enable(1). > > Maybe it won't work because it's in the same file. Then another idea would > be : > create 2 database files, for_writer.sqlite, and for_reader.sqlite > if writer and / or reader thread need to access the other db (for_reader > resp. for_writer) then it's possible trough a attach command (attach > "for_reader.sqlite" as forreader" > > these are suggestions, think about it and perhaps it would bring you to a > solution ! > > regards, > > Vincent > > > I have an application which on a single thread writes data to a database. > > This writing can result in millions of writes to the database in a single > > transaction (started with BEGIN TRANSACTION.) I also have other threads > > and > > processes which read the same database, but never write to it, and never > > start transactions (they just issue select commands against it.) In the > > writing application, I have one thread which originally opens the > > database, > > but then hands it off to the writer thread, and never touches it again. > > > > The issue that I am seeing is that when the transaction is started, > > everything behaves as expected; the writer happily calls inserts within > > his > > connection, and they go to wherever the isolated modification data goes, > > which I presume is the journal file. Initially, other processes and > > threads > > can freely read the database, and all is good. Fast forward a bit, to > > when > > a large number of inserts have occurred. It seems that when the page > > cache > > gets filled up, SQLite tries to find some free pages, which causes an > > exclusive lock to be obtained within the context of the transaction, that > > is > > not relinquished until the completion of the transaction. This causes > > some > > pain, because my readers that were happily able to read from the original > > database are now stopped up until the potentially long running > transaction > > completes (they either get database is locked errors, or in my case, > their > > infintely waiting busy handlers do just that.) > > > > My question is; am I assessing the situation correctly, and if so, is > > there > > anything I can do to avoid this rather nasty situation? I would rather > > not > > take the periodic-commit approach, and increasing the page cache size > > would > > potentially cause the use of more memory than I can spare. Disk usage, > > however, is not a problem; I can use as much as is necessary. > > > > Thanks very much. > > ___ > > sqlite-users mailing list > > sqlite-users@sqlite.org > > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > > > > > > > > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Transaction behavior with large numbers of inserts
I have an application which on a single thread writes data to a database. This writing can result in millions of writes to the database in a single transaction (started with BEGIN TRANSACTION.) I also have other threads and processes which read the same database, but never write to it, and never start transactions (they just issue select commands against it.) In the writing application, I have one thread which originally opens the database, but then hands it off to the writer thread, and never touches it again. The issue that I am seeing is that when the transaction is started, everything behaves as expected; the writer happily calls inserts within his connection, and they go to wherever the isolated modification data goes, which I presume is the journal file. Initially, other processes and threads can freely read the database, and all is good. Fast forward a bit, to when a large number of inserts have occurred. It seems that when the page cache gets filled up, SQLite tries to find some free pages, which causes an exclusive lock to be obtained within the context of the transaction, that is not relinquished until the completion of the transaction. This causes some pain, because my readers that were happily able to read from the original database are now stopped up until the potentially long running transaction completes (they either get database is locked errors, or in my case, their infintely waiting busy handlers do just that.) My question is; am I assessing the situation correctly, and if so, is there anything I can do to avoid this rather nasty situation? I would rather not take the periodic-commit approach, and increasing the page cache size would potentially cause the use of more memory than I can spare. Disk usage, however, is not a problem; I can use as much as is necessary. Thanks very much. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] How to get older source
I'm trying to access the 3.4.2 (and previous) sources from the website by changing the version numbers in the source download link to various values, which has worked for me in the past. If I use the following links, things work and I can get both the long form and amalgamated sources: http://sqlite.org/sqlite-amalgamation-3_5_3.zip http://sqlite.org/sqlite-amalgamation-3_5_2.zip http://www.sqlite.org/sqlite-source-3_5_3.zip http://www.sqlite.org/sqlite-source-3_5_2.zip However, versions 3.5.1 and back don't work (file not found.) Note that I don't try to use the amalgamation stuff for 3.4.2 back, just the normal source links. Have historical versions been removed, or is there another place I should be looking for this source? Thanks, Ian
Re: [sqlite] why select distinct does not use index?
In the call to create index, it looks like you're creating a duplicate index on [id], which I believe references the default btree id (also called rowid.) If you change that line to: CREATE INDEX [by_id] ON [profile_data] ([profile_id]); your index should be used (though whether indexes get used with select distinct clauses, I'm not sure.) Ian On Nov 8, 2007 4:30 AM, Maxim V. Shiyanovsky <[EMAIL PROTECTED]> wrote: > Suppose I have simple table: > > CREATE TABLE [profile_data] ( > > [profile_id] INTEGER, > > [version] INTEGER); > > CREATE INDEX [by_id] ON [profile_data] ([id]); > > > > Why > > sqlite> explain query plan > > ...> select distinct(profile_id) from profile_data; > > 0|0|TABLE profile_data > > > > Does not use index? > > - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Proposed incompatible change to SQLite
A most welcome modification. On 8/9/07, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > > We are considering a change to the SQLite API which is > technically not backwards compatible. On the other hand, > it is difficult to imagine a situation where the change > would cause anyone any serious hardship. We suspect that > most people will consider this change a vast improvement. > > The proposed changes is to these APIs: > > sqlite3_soft_heap_limit(); > sqlite3_enable_shared_cache(); > sqlite3_release_memory(); > > As currently implemented (and documented), these > routines only work across database connections in > the same thread. We propose to modify this so > that these routines work across all database > connections in the same process. > > If you think such a change will cause problems for > you, please let me know. Tnx. > > -- > D. Richard Hipp <[EMAIL PROTECTED]> > > > > - > To unsubscribe, send email to [EMAIL PROTECTED] > > - > >
Re: [sqlite] SQLite 64-Bit
I build SQLite's code using Visual Studio 2005 using the 64 bit tool chain, and it runs without a hitch. I haven't seen a pre-compiled library for it, so you may have to get your hands a bit dirty. Building it is a snap though; add the sqlite3.c and sqlite3.h files to a project, define NO_TCL and THREADSAFE in the preprocessor for your various configurations, and voila. On 7/4/07, Ahmed Sulaiman <[EMAIL PROTECTED]> wrote: Hi all, We are interested in using the SQLite in our project. I wonder if there a version that works in 64-Bit machine? I have looked in the download page and noticed that only Win32 (x86) builds are available. Regards Ahmed
Re: [sqlite] Database File size not exceeding 2GB
Is the file system holding your file Fat32, or NTFS? If it's Fat32, it may be the source of your problem, as it doesn't support very large files. Ian On 7/4/07, Krishnamoorthy, Priya (IE10) <[EMAIL PROTECTED]> wrote: Hi all, I am using SQLite3 database in my application. My application runs on Windows XP (32 bit) platform. I am not able to store more than 2GB of data in my database. Is it not possible to store more than 2gb data in windows XP? I used SQlite3 in Linux and could store more than 2GB. Please help me in this regard Best regards, Priya
Re: [sqlite] Concurrency
For Windows, this is not the case with Automatic Reset events. The system guarantees that only one thread waiting on the event is woken up (it keeps a queue): the others happily keep sleeping until the next setting of the event. On 6/1/07, Doug Currie <[EMAIL PROTECTED]> wrote: On Friday, June 01, 2007 Ian Frosst wrote: > On the topic of a more efficient busy handler, one approach I considered was > to implement an event which was signalled when a database unlock occurred. > That way, the busy handler could just wait on the event (which is an > efficient wait state), and be guaranteed of a wake up when the lock is > released (the event would be signalled at this time.) However, I wasn't at > the time familiar enough with SQLite's innards to implement such a beast. > Can anyone see any pitfalls to such an approach? The problems occur when multiple threads are waiting on the event; they all wake up and compete for the resource again. For better solutions, see: http://world.std.com/~jmhart/batons.htm e -- Doug Currie Londonderry, NH, USA - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Concurrency
On the topic of a more efficient busy handler, one approach I considered was to implement an event which was signalled when a database unlock occurred. That way, the busy handler could just wait on the event (which is an efficient wait state), and be guaranteed of a wake up when the lock is released (the event would be signalled at this time.) However, I wasn't at the time familiar enough with SQLite's innards to implement such a beast. Can anyone see any pitfalls to such an approach? Ian On 6/1/07, John Stanton <[EMAIL PROTECTED]> wrote: Tom Briggs wrote: > > >>I don't want to use >>other database, because I think Sqlite is great for an >>embedded system that I >>am using. > > >I think that your own questions about concurrency prove this > incorrect. If you need high concurrency and you don't like retries, > SQLite is not the database for you. > >-T > If you require ACID type data integrity and have a single disk there is no such thing as a "high concurrency database". They all share a single disk resource in some way. With Sqlite it is up to the designer to build in concurrency and that cna be done by single streaming. Better performance is achieved by using a single database connection so that cache hits are maximized. If your design is such that you never get a busy then you have an effective allocation of your disk resource. Using mutexes between threads and semaphores between processes gives you that capability. To my mind the only time you should use the busy logic is when you are working across a network with shared files. - To unsubscribe, send email to [EMAIL PROTECTED] -
[sqlite] Clarification of bound parameter usage
Hi all, I'm looking for some clarification on the usage of bound parameters in queries. I have one SQL statement that is going to be executed numerous times (thousands of times, in a fairly tight loop.) Right now I am using a string, "INSERT INTO tableX (col1, col2) VALUES ('%s', '%s')". and calling printf to substitute in the parameters (I know, it's seceptible to injection, and doesn't escape, but that's not of concern to me right now.) The problem here though, is that I'm doing the prepare/step/finalize each time I want to execute the query, even though the only thing that is going to change are the values. Can I write a loop which prepares the SQL, using named parameters, then in a loop just call sqlite3_bind_*/step/reset, finalizing after all of my calls have been done? Thanks, Ian
Re: [sqlite] Causes of SQLITE_SCHEMA errors
Aha. I shall give it a try, and see how I fare. Many thanks, Ian On 3/6/07, Marco Bambini <[EMAIL PROTECTED]> wrote: Are you using sqlite_prepare or sqlite_prepare_v2? If you use the v2 version I think that your problem will be solved... Regards, --- Marco Bambini http://www.sqlabs.net http://www.sqlabs.net/blog/ http://www.sqlabs.net/realsqlserver/ On Mar 6, 2007, at 2:22 PM, Ian Frosst wrote: > Hey all, I've got a bit of an odd situation going on. I have a fairly > heavily threaded app making use of SQLite (which is working > beautifully for > the most part.) However, I've run into a bit of a situation. It > seems that > I have a periodic occurrence of SQLITE_SCHEMA occurring when > executing some > queries. I've pretty much narrowed this down to the interaction of > two > threads working with the database. One is a simple insert, and the > other is > a fairly complicated db processing, consisting of three parts: > > 1) Create a temporary table with the results of a query > 2) Create an index on the temporary table > 3) Use the indexed temporary table in another query > > The reason I'm doing the above is that the query executed in step 3 > contains > a sub-query, which can potentially return a large amount of data. > I was > finding that without any type of indexing on the data coming back > from this > sub-query, my performance was unacceptable. > > My theory regarding the SQLITE_SCHEMA error is that the creation of > the > index on the temporary table is causing a change to the schema > revision > number, but I'm not quite familiar enough with the code to verify > this. I > was hoping someone on the list could enlighten me as to: > > - Whether creations of temporary tables, or the creation of indexes > on these > tables incur schema updates > - If the above is the case, whether there are any strategies for > avoiding > this situation. > > Thanks, > Ian - To unsubscribe, send email to [EMAIL PROTECTED] -
[sqlite] Causes of SQLITE_SCHEMA errors
Hey all, I've got a bit of an odd situation going on. I have a fairly heavily threaded app making use of SQLite (which is working beautifully for the most part.) However, I've run into a bit of a situation. It seems that I have a periodic occurrence of SQLITE_SCHEMA occurring when executing some queries. I've pretty much narrowed this down to the interaction of two threads working with the database. One is a simple insert, and the other is a fairly complicated db processing, consisting of three parts: 1) Create a temporary table with the results of a query 2) Create an index on the temporary table 3) Use the indexed temporary table in another query The reason I'm doing the above is that the query executed in step 3 contains a sub-query, which can potentially return a large amount of data. I was finding that without any type of indexing on the data coming back from this sub-query, my performance was unacceptable. My theory regarding the SQLITE_SCHEMA error is that the creation of the index on the temporary table is causing a change to the schema revision number, but I'm not quite familiar enough with the code to verify this. I was hoping someone on the list could enlighten me as to: - Whether creations of temporary tables, or the creation of indexes on these tables incur schema updates - If the above is the case, whether there are any strategies for avoiding this situation. Thanks, Ian
Re: [sqlite] sqlite / gac
It looks to be, yes (I haven't used that particular piece of kit, but from the page, it seems it will fit the bill nicely.) Ian On 2/4/07, Andrew <[EMAIL PROTECTED]> wrote: Hi Ian. Thanks for the info. P/Invoke is unsafe in the C# / common language infrastructure sense. Anything unmanaged is unsafe. I had been planning to use the System.Data.SQLite implementation from http://sourceforge.net/projects/sqlite-dotnet2 . After I wrote the first email to this list, I was able to install it in the global assembly cache manually using "gacutil /i". I still don't know whether that is all that is required. Is System.Data.SQLite, then, a managed wrapper around the unmanaged part of SQLite? Andrew On Sun, Feb 04, 2007 at 08:16:50AM -0400, Ian Frosst wrote: > SQLite isn't managed code: it's unmanaged. As such, unless you write the > p/invoke code yourself, I'd recommend finding a good SQLite .Net wrapper, > especially one of them that use ADO.Net. You can then follow normal rules > for GAC registration with the wrapper. > ... > Again, SQLite's DLL is unmanaged code, not managed, so it can't go into > the GAC. > > I don't think using the p/invoke services in C# causes you to drop into > unsafe mode, but I'm not 100% sure of that. > > If you find yourself a wrapper that makes use of ADO.Net, you can find > numerous examples of it's use. > > With the right wrappers, it should work splendidly (I use it from C# through > a COM wrapper myself.) > Cheers, > Ian - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] sqlite / gac
On 2/4/07, Andrew <[EMAIL PROTECTED]> wrote: Hi.. I'm writing a small application for my personal use and would like to try SQLite. I have a few basic questions that I didn't see answers for at sqlite.org and am hoping that someone on this list can help. First, I'd like to get SQLite installed in the global assembly cache (for use on my machine with .NET 2.0). How can I do this? SQLite isn't managed code: it's unmanaged. As such, unless you write the p/invoke code yourself, I'd recommend finding a good SQLite .Net wrapper, especially one of them that use ADO.Net. You can then follow normal rules for GAC registration with the wrapper. I remember reading that SQLite uses strong names so it looks like this scenario is one that has been planned for. But, I installed SQLite-1.0.40.0 on my machine using the .exe installer and didn't see any obvious SQLite DLL in the global assembly cache. Again, SQLite's DLL is unmanaged code, not managed, so it can't go into the GAC. Second, can SQLite be used in safe C#, or does it require going into unsafe mode? I don't think using the p/invoke services in C# causes you to drop into unsafe mode, but I'm not 100% sure of that. Finally, is there a simple example of using SQLite in C# somewhere, especially one that does not assume prior knowledge of SQL? If you find yourself a wrapper that makes use of ADO.Net, you can find numerous examples of it's use. I compile C# programs from the command line and am not using Visual Studio or other IDEs I have on my machine. I am using Windows XP (with Cygwin). SQLite looks really nice - I hope I can use it. Andrew With the right wrappers, it should work splendidly (I use it from C# through a COM wrapper myself.) Cheers, Ian - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] SQLite last inserted id
My issue with that is I need the rowid of the existing row back. Insert or ignore doesn't seem to update the database's last_insert_rowid in the ignore case, from what I've seen. Insert or Replace almost does it, but it re-allocates the rowid, which breaks my referential integrity. Should the Insert or Replace operation fill in the last insert ID with the row causing the constraint violation? Ian On 1/22/07, Nicolas Williams <[EMAIL PROTECTED]> wrote: If you'd have a unique index on that column then you could just use "INSERT OR IGNORE ..." Nico -- - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] SQLite last inserted id
I would like to solicit ideas on this as well. An operation similar to insert or replace, but that doesn't allocate a new row ID on replace (just returns the old one through the last insert row ID.) Having to do the search operation, and if it fails do an insert (or vice versa) doesn't seem optimal. Ian On 1/22/07, Weston Weems <[EMAIL PROTECTED]> wrote: What I'd ultimately like to do, and what I used to do with a sproc, is the following Pass a string to a sproc, it'll select from table where field = string value, and if no records insert it and grab last inserted id. That way with one call, I can ensure one string in db, and always get the an id back. Is there a way i can do this with sqlite 3 latest? Looks like there is SOME sort of concept of this with views (xp_proc) with parameters etc, but I cant really find any documentation etc. Anyone have any advice for a sqlite newb? Thanks in advance.
Re: [sqlite] Performance test on windows
It could be that there is no transaction block wrapped around the inserts (a BEGIN TRANSACTION before the start of the insert loop, and a COMMIT TRANSACTION at the end.) If there is no explicit transaction, then every insert has an implied transaction, which considerably slows down the database engine. Ian On 1/18/07, Gaurav Arora <[EMAIL PROTECTED]> wrote: Hi All, I am a newbie to SQlite, just saw that the performance numbers on www.sqlite.org are not reliable (as per the notce on website, http://www.sqlite.org/speed.html http://www.sqlite.org/speed.html> ) So, I thought of profiling SQlite operations, on linux platform the performance time is quite good. I build sqlite on Windows too, and saw discouraging numbers coming out of it, following are some of them. e.g. RH9: inserting 1000 records ~.07 seconds. WindowsXP: inserting 100 records ~11 seconds. inserting 1000 records ~113 seconds. May be I did something wrong for building, or the parameters passed for building sqlite werent correct. I am willing to update the test results on the site, could anyone here guide me for the things which I might be doing wrong. Thanks in advance. //Gaurav
[sqlite] Re: SQLite 3.3.10 compilation with VC++ for Win64
My first post sounded *way* too alarmist, I appologise. I should have chosen my words more carefully. the two suspects I see in the warnings are: static int winFileHandle(OsFile *id) { return (int)((winFile*)id)->h; } The HANDLE will be64 bit, then get trunc'd when cast (unless there is some compiler magic afoot that I'm not aware of.) And, in vtab.c(573) int sqlite3VtabCommit(sqlite3 *db) { callFinaliser(db, (int)(&((sqlite3_module *)0)->xCommit)); return SQLITE_OK; } the explicit cast of the commit function pointer to int could be a truncation operation (though looking deeper into this one, it looks like that's not used as a pointer directly, but as an offset, so I could be off base.) Those are the only pointer truncation threats I've found, but looking through the archives for builds on 64 bit machines, I'm still a bit fuzzy on the effect of the Microsoft compiler on the final 64 bit build output. It seems that Microsoft's compiler maintains int as 32 bit, while from what I'm reading, gcc and it's brethren compile it as 64 bit. I'm really just looking for some guidance on 64 bit builds of SQLite on Windows boxes (I know it runs perfectly fine in the Wow64 subsystem, but we're trying to do a native 64 bit version of the app I'm working on, which requires a native 64 bit build of SQLite for windows.) If someone has tried a build using the Microsoft 64 bit compiler, and had it work successfully, I'd be happy to hear it. On 1/15/07, Ian Frosst <[EMAIL PROTECTED]> wrote: Has anyone here gotten 3.3.10 (or any subsequent version) to compile and run successfully through Visual C++ (I'm using 2005, but again, any version would do.) Looking at the code, there are pointer to int truncations everywhere, and I've found no magic bullet for declalring int's as 64 bit (__int64 and longlong is how I normally do it.) If there really is no way to get VC++ to work, would it be fair to assume that I have to get a 64 bit version of gcc, and use that to do the compilation (I've read that it will compile int to 64 bit)? Thanks, Frosstoise
[sqlite] SQLite 3.3.10 compilation with VC++ for Win64
Has anyone here gotten 3.3.10 (or any subsequent version) to compile and run successfully through Visual C++ (I'm using 2005, but again, any version would do.) Looking at the code, there are pointer to int truncations everywhere, and I've found no magic bullet for declalring int's as 64 bit (__int64 and longlong is how I normally do it.) If there really is no way to get VC++ to work, would it be fair to assume that I have to get a 64 bit version of gcc, and use that to do the compilation (I've read that it will compile int to 64 bit)? Thanks, Frosstoise