Re: [sqlite] bug report: data race in multi-threaded execution ofsqlite3

2010-07-19 Thread Dave Toll
Hi Heechul I have used this test code as well, and I found that you have to treat SQLITE_SCHEMA the same way as SQLITE_LOCKED - you should retry the operation that returned this error. I believe it indicates that the master table is locked by another thread due to a DDL command (e.g. CREATE

Re: [sqlite] 64 bit soft heap limit feature request

2010-09-04 Thread Dave Toll
Hi Roger While I broadly agree that SQLite is 32-bit software, I do not understand why there is an assert towards the end of sqlite3_initialize() that fails on a 32-bit platform if SQLITE_DEBUG is defined (code taken from SQLite 3.6.23.1): /* The following is just a sanity check to make sure

Re: [sqlite] 64 bit soft heap limit feature request

2010-09-05 Thread Dave Toll
heap limit feature request -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 09/05/2010 12:15 AM, Dave Toll wrote: > I do not > understand why there is an assert towards the end of > sqlite3_initialize() that fails on a 32-bit platform if SQLITE_DEBUG is > defined (code taken from SQL

[sqlite] Index memory usage in SQLite

2008-08-27 Thread Dave Toll
reported by sqlite3_memory_used(). Dropping the index does not return any of the memory used, and does not reduce the DB file size. Are these results normal, or is some optimisation possible? Cheers, Dave Toll. ___ sqlite-users mailing list sqlite-users

Re: [sqlite] Index memory usage in SQLite

2008-08-27 Thread Dave Toll
, Aug 26, 2008 at 06:13:29PM +0100, Dave Toll scratched on the wall: > I'm running some general performance tests on SQLite 3.5.9 (embedded C > platform), and I noticed that creating an index seems to use a lot more > memory than I expected. Creating an index requires reading an

Re: [sqlite] Index memory usage in SQLite

2008-08-28 Thread Dave Toll
Hipp [mailto:[EMAIL PROTECTED] Sent: 27 August 2008 06:55 To: General Discussion of SQLite Database Subject: Re: [sqlite] Index memory usage in SQLite On Aug 26, 2008, at 1:13 PM, Dave Toll wrote: > Hello all > > > > I'm running some general performance tests on SQLite 3.5.9 (embedd

[sqlite] memsys3 vs memsys5

2008-09-09 Thread Dave Toll
argument), otherwise the page-cache would not be used as the page-cache units were too small. Is there any reliable method to determine the minimum page-cache allocation size needed for a given page_size? Cheers, Dave Toll. ___ sqlite-users mailing

Re: [sqlite] memsys3 vs memsys5

2008-09-10 Thread Dave Toll
ver double the DB file size for any page size value. On an embedded system with limited memory, I don't see the advantages of using memsys5 over memsys3. One last point - I wasn't able to find a link to the malloc page from anywhere in the SQLite site. Cheers, Dave Toll. -Original Message-

[sqlite] calculating cached page size for SQLITE_CONFIG_PAGECACHE

2008-10-06 Thread Dave Toll
Hello list I wrote a function that attempts to calculate the cached page size for a given page size, to be passed as the sz parameter for sqlite3_config(SQLITE_CONFIG_PAGECACHE, buf, sz, N). It appears to work so far (with 3.6.3) but I'd appreciate if anyone could go over it and see if they

[sqlite] assert() in sqlite3_initialize() when SQLITE_DEBUG is defined

2008-10-09 Thread Dave Toll
Hello list I'd like to define SQLITE_DEBUG to help trace some problems I am seeing, but this causes sqlite3_initialize() to assert on a 32-bit system. Does SQLite support 32-bit systems? /* The following is just a sanity check to make sure SQLite has ** been compiled correctly. It is

Re: [sqlite] SQLite version 3.6.4

2008-10-15 Thread Dave Toll
I found a small compile error when SQLITE_MUTEX_NOOP is defined with the amalgamation source: sh4gcc -ansi -fsigned-char -mruntime=os21 -ml -c -O0 -DSQLITE_OS_OTHER=1 -DSQLITE_32BIT_ROWID=1 -DSQLITE_INT64_TYPE=long -DSQLITE_OMIT_AUTHORIZATION -DSQLITE_OMIT_AUTOINCREMENT -DSQLITE_OMIT_AUTOINIT

Re: [sqlite] SQLite version 3.6.4

2008-10-15 Thread Dave Toll
, Dave Toll wrote: > I found a small compile error when SQLITE_MUTEX_NOOP is defined with > the > amalgamation source: > SQLITE_MUTEX_NOOP is an internal symbol that gets set if and only if you define SQLITE_THREADSAFE=0. You should not set SQLITE_MUTEX_NOOP yourself.

[sqlite] reading beyond end of file

2009-01-16 Thread Dave Toll
Hello list I recently upgraded from SQLite 3.6.7 to 3.6.10 and I'm now noticing some apparently undesirable behaviour. I'm running on an embedded system with my own VFS implementation, and I see in my tests that SQLite is now trying to read journal files at an offset beyond the end of the

Re: [sqlite] reading beyond end of file

2009-01-16 Thread Dave Toll
Database Subject: Re: [sqlite] reading beyond end of file On Jan 16, 2009, at 5:38 PM, Dave Toll wrote: > Hello list > > > > I recently upgraded from SQLite 3.6.7 to 3.6.10 and I'm now noticing > some apparently undesirable behaviour. I'm running on an embedded > sys

Re: [sqlite] reading beyond end of file

2009-01-19 Thread Dave Toll
Returning SQLITE_IOERR_SHORT_READ in this case solves my problem. Many thanks, Dave. -Original Message- From: D. Richard Hipp [mailto:d...@hwaci.com] Sent: 16 January 2009 15:57 To: General Discussion of SQLite Database Subject: Re: [sqlite] reading beyond end of file On Jan 16,

Re: [sqlite] SQLITEBUSYTIMEOUT

2009-01-21 Thread Dave Toll
Some of your writes may be failing with SQLITE_BUSY or SQLITE_LOCKED if others take too long to complete. If this is happening, you should retry the write until it succeeds or increase your busy timeout (or both). Cheers, Dave. -Original Message- From: SATISH

[sqlite] shared cache and journal files

2009-01-28 Thread Dave Toll
Hello list I have been using a modified version of threadtest1.c from the SQLite test code to test the robustness of multi-threaded read/write connections (SQLite 3.6.10) on an embedded platform (with my own VFS implementation). The test code spawns 10 threads, each of which creates, populates

Re: [sqlite] shared cache and journal files

2009-01-29 Thread Dave Toll
database - so a shared lock is assumed and the function is a no-op. Does this imply some misuse by my test code? Cheers, Dave. -Original Message- From: Dave Toll Sent: 28 January 2009 17:06 To: sqlite-users@sqlite.org Subject: shared cache and journal files Hello list I have been using

Re: [sqlite] shared cache and journal files

2009-01-29 Thread Dave Toll
. -Original Message- From: Dave Toll Sent: 29 January 2009 12:27 To: sqlite-users@sqlite.org Subject: RE: shared cache and journal files A little more detail on this issue: It seems that my reading thread is not getting a shared lock on the database before accessing the journal file

Re: [sqlite] sqlite_open_v2() vs sqlite_open16()

2009-02-10 Thread Dave Toll
Hi John You can execute the following SQL after opening your DB (but before inserting any rows): "PRAGMA encoding=UTF16" http://www.sqlite.org/pragma.html Cheers, Dave. -Original Message- From: John Horton [mailto:john_horto...@hotmail.com] Sent: 10 February 2009 01:30 To:

Re: [sqlite] sqlite cache question

2009-02-23 Thread Dave Toll
Hi Marcus My understanding of SQLite caching is that in your scenario, 40MB is the *maximum* cache memory that will be used per connection - this memory is not immediately pre-allocated when you open a new connection. Using the default memory allocator (memsys1) SQLite will allocate from the heap

Re: [sqlite] PRAGMA page_cache=x doesnt seem to have an effect

2009-02-25 Thread Dave Toll
Try sqlite3_soft_heap_limit() or use memsys3/memsys5 - http://www.sqlite.org/malloc.html Cheers, Dave. -Original Message- From: ed [mailto:epdm...@gmail.com] Sent: 25 February 2009 10:50 To: General Discussion of SQLite Database Subject: Re: [sqlite] PRAGMA page_cache=x doesnt seem to

Re: [sqlite] shared cache and SQLITE_MISUSE on sqlite3_step()

2009-03-06 Thread Dave Toll
Without wishing to complicate the subject, I have found that retrying a sqlite3_step() after receiving SQLITE_LOCKED works fine for me without needing sqlite3_reset(). I do this in a multi-threaded test app (based on some old SQLite test code) that I have left running with 10 threads for a long

[sqlite] question about shared cache

2009-03-24 Thread Dave Toll
Hello list I have been trying to understand how shared cache works under the covers, and how a custom VFS should behave when shared cache is enabled - can anyone confirm this assumption: Journal file handles (opened with SQLITE_OPEN_EXCLUSIVE) are shared between database connections opened

[sqlite] SQLITE_CORRUPT - database disk image is malformed

2009-04-10 Thread Dave Toll
Hello list I am sometimes seeing the error SQLITE_CORRUPT with message "database disk image is malformed", when I attempt CREATE TEMP TRIGGER. My test spawns 5 threads, each of which modifies and queries its own table in a common database (each thread uses a separate connection from

[sqlite] Concurrent sqlite3OsAccess() and sqlite3OsDelete() on journal file

2009-05-08 Thread Dave Toll
Hello list I'm seeing an issue (on Linux build) where 2 different shared-cache DB connections accessing the same database get confused about the status of the journal file. One thread checks for the existence of the journal with sqlite3OsAccess(), and at the same time the other deletes the

Re: [sqlite] TOCTOU pager bug in 3.6.14.2

2009-05-29 Thread Dave Toll
Could this be the same issue as this bug I reported a while ago? http://www.sqlite.org/cvstrac/tktview?tn=3833 Cheers, Dave. -Original Message- From: Brian Dantes [mailto:bdan...@coverity.com] Sent: 28 May 2009 16:01 To: Discussion of SQLite Database Subject: [sqlite] TOCTOU pager bug

[sqlite] minimum cache_size for a database connection?

2009-06-16 Thread Dave Toll
Hello list I've noticed that if I call "PRAGMA cache_size=0", my database rows are still cached (very little disk access observed). If I call "PRAGMA cache_size=1" I can see that there is very little caching (disk access observed). Is there a minimum allowed cache_size setting? Does

Re: [sqlite] minimum cache_size for a database connection?

2009-06-17 Thread Dave Toll
...@hwaci.com] Sent: 16 June 2009 17:33 To: General Discussion of SQLite Database Subject: Re: [sqlite] minimum cache_size for a database connection? On Jun 16, 2009, at 8:28 PM, Dave Toll wrote: > Hello list > > > > I've noticed that if I call "PRAGMA cache_size=

Re: [sqlite] minimum cache_size for a database connection?

2009-06-17 Thread Dave Toll
it's a case I need to prevent. Many thanks for the replies, Dave. -Original Message- From: D. Richard Hipp [mailto:d...@hwaci.com] Sent: 17 June 2009 10:08 To: General Discussion of SQLite Database Subject: Re: [sqlite] minimum cache_size for a database connection? On Jun 17, 2009, at 12:5

[sqlite] Determining which DB a journal or temp file belongs to

2009-09-02 Thread Dave Toll
Hello list I'm using SQLite with a custom VFS on an embedded platform. I'd like to place an upper (soft) limit on the amount of disk space used by a database - i.e. the sum of the main database file size, journal file size, and any associated temp file sizes. At the VFS level, is there any way

Re: [sqlite] SQLite Porting: Adding new VFS

2009-09-02 Thread Dave Toll
Hi Mwnn If you don't require persistent storage and always call sqlite3_open[_v2]() with the filename ":memory:" for an in-memory database, then yes you should be able to assign pMethods to NULL. In fact if you only use ":memory:", you should be able to compile with SQLITE_OMIT_DISKIO and you

Re: [sqlite] I'm surprised what the author of web2py on web2py mailing list just said about sqlite

2009-09-09 Thread Dave Toll
> >I apologize and I conclude my statement is not correct. > >I have experienced this problem once (out of disk space and sqlite >database corruption) but I made the mistake of confusing correlation >with causality. Except in that occasion I never had any problem with >sqlite and I do use it on

Re: [sqlite] SQLite database on a certain high-performance "SSD"

2009-09-21 Thread Dave Toll
Hi Mark I've had a little experience working with flash-based filesystems - I'd recommend playing with the page_size and temp_store PRAGMAs (and of course make sure you are using transactions to minimise the number of file writes) to improve performance. Cheers, Dave. -Original

[sqlite] Ticket 3810: SQLITE_ERROR on concurrent CREATE TEMP TRIGGER and sqlite3_close()

2009-10-19 Thread Dave Toll
Hello list I'm posting here as my ticket was recently closed. The race condition described in http://www.sqlite.org/cvstrac/tktview?tn=3810 is still present in SQLite 3.6.19 (amalgamation version running on Linux) - the source code used to reproduce this issue has not changed and is still

Re: [sqlite] Ticket 3810: SQLITE_ERROR on concurrent CREATE TEMP TRIGGER and sqlite3_close()

2009-10-20 Thread Dave Toll
on concurrent CREATE TEMP TRIGGER and sqlite3_close() -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Dave Toll wrote: > The race condition described in http://www.sqlite.org/cvstrac/tktview?tn=3810 > is > still present in SQLite 3.6.19 (amalgamation version running on Linux) My bigges

Re: [sqlite] Ticket 3810: SQLITE_ERROR on concurrent CREATE TEMP TRIGGER and sqlite3_close()

2009-10-21 Thread Dave Toll
te] Ticket 3810: SQLITE_ERROR on concurrent CREATE TEMP TRIGGER and sqlite3_close() -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Dave Toll wrote: > Could you point me to the test case you referred to? Look near the bottom of the ticket where it lists the various checkins. [6955] is the te

Re: [sqlite] sqlite-users Digest, Vol 25, Issue 14

2010-01-14 Thread Dave Toll
Hi Jamie Someone posted a problem recently related to SQLITE_CANTOPEN - I believe the issue was that too many file descriptors were opened. Perhaps you are using a file descriptor for every path you insert, and at some point the OS returns an error when SQLite tries to get a file descriptor for

Re: [sqlite] Sqlite without 64-bit integers?

2010-03-12 Thread Dave Toll
I have been successfully using SQLite on a 32-bit embedded system for some time without any problems - you can add -DSQLITE_INT64_TYPE=long to the compile line, no need to hack the source. The only issue I have seen is that it is not possible to use -DSQLITE_DEBUG as it enables some assert() calls