[sqlite] Question regarding I/O under the new VFS model

2007-10-11 Thread Mark Spiegel
I'm working with implementing my own VFS. If the developer(s) could answer a couple of questions, it would be helpful. In section 2.1.4 of the SQLite 3.4.2 to 3.5.0 document, reference is made to "a database file will be doing page-aligned sector reads and writes in random order" regarding

Re: [sqlite] Question regarding I/O under the new VFS model

2007-10-11 Thread Mark Spiegel
(with the exception of the header). That is a useful hint for these types of files. It sounds like journal reads and writes won't be sector sized and/or sector aligned. I can accommodate that with little effort as well. [EMAIL PROTECTED] wrote: Mark Spiegel <[EMAIL PROTECTED]> wrote

[sqlite] winLock() in SQLITE 3.5.1...

2007-10-17 Thread Mark Spiegel
While working on a VFS for use in 3.5.1, I was looking at the winLock() and have a question. Is it possible for the lock on a winFile object to progress from SHARED_LOCK to EXCLUSIVE_LOCK without first acquiring a RESERVED_LOCK? Assuming that it is, it seems that the comments at the start of

[sqlite] winLock() in SQLITE 3.5.1...

2007-10-22 Thread Mark Spiegel
I posted this to the list last Wednesday and haven't seen a reply. D0n't want to create traffic in the defect database if I am in error. Anyone want to take a crack at this? While working on a VFS for use in 3.5.1, I was looking at the winLock() and have a question. Is it possible for the

Re: [sqlite] winLock() in SQLITE 3.5.1...

2007-10-22 Thread Mark Spiegel
Thanks Dan. How about the second part. Should the PENDING_LOCK be taken en route from the SHARD_LOCK to EXCLUSIVE_LOCK? Which is right, the code or the function header? Dan Kennedy wrote: On Mon, 2007-10-22 at 10:56 -0700, Mark Spiegel wrote: I posted this to the list last Wednesday

Re: [sqlite] Re: Some Questions Regarding Access To a SQLite Database By More Than One Process

2007-10-25 Thread Mark Spiegel
Here's a bit more locking info that I found useful to help understand it all out of the archives: http://www.mail-archive.com/sqlite-users@sqlite.org/msg02845.html If you are writing a Windows app, you can use a named mutex which can be shared across processes. I have need for a blocking

Re: [sqlite] Mechanism for pre-allocating contiguous files for Sqlite?

2007-10-29 Thread Mark Spiegel
I'm writing a VFS right now. One of the parameters to the open is the file type (various types of DBs and journals). You should be able to use that info. Also, look for a took called config on the sysinternals site. It allows you to pre-allocate contiguous files. Better, it allows you to

Re: [sqlite] Proposed sqlite3_initialize() interface

2007-10-30 Thread Mark Spiegel
be called after all other calls have completed and there are no more resources in use. While it may be a noop at this time, it may not be at some point. It is also a good place for debug code to ensure that all resources have been released. Mark Spiegel [EMAIL PROTECTED] wrote: As currently

Re: [sqlite] Disk caching impacts performance.

2007-11-08 Thread Mark Spiegel
As Trevor cautioned, you run the risk of really impacting the other applications running on the system and the system itself. As you fill up the Windows cache manager with your file, you will push out other relevant data including data that the OS may be using. This will cause applications

Re: [sqlite] SQLITE_OMIT_MEMORY_ALLOCATION

2007-11-08 Thread Mark Spiegel
you are trying to accomplish with the memory management changes. [EMAIL PROTECTED] wrote: Mark Spiegel <[EMAIL PROTECTED]> wrote: I just started looking at 3.5.2 and notice that support for SQLITE_OMIT_MEMORY_ALLOCATION was removed. This is a problem for me. Is there some rea

Re: [sqlite] Memory Usage

2007-11-19 Thread Mark Spiegel
Well said. While it may be true that some memory allocators are lacking, the ones I use are quite good. I view with great suspicion developers who thinks they can outsmart the pool allocator. These folks usually add great complexity while having at best a neutral impact on performance and

[sqlite] Moving port from 3.5.1 to 3.5.7...

2008-03-21 Thread Mark Spiegel
I'm looking to jump my code port forward from 3.5.1 to 3.5.7. Clearly I have some memory management work to do since SQLITE_OMIT_MEMORY_ALLOCATION support has been dropped. None of the existing allocation implementations look acceptable so I'll have to roll my own, but that looks pretty

Re: [sqlite] Moving port from 3.5.1 to 3.5.7...

2008-03-21 Thread Mark Spiegel
. Thanks for your help. [EMAIL PROTECTED] wrote: > Mark Spiegel <[EMAIL PROTECTED]> wrote: > >> I'm looking to jump my code port forward from 3.5.1 to 3.5.7. >> >> Clearly I have some memory management work to do since >> SQLITE_OMIT_MEMORY_ALLOCA

Re: [sqlite] SQLITE_OMIT_xxx

2008-04-21 Thread Mark Spiegel
That's what I do. Once your makefile is set up, make the "sqlite3.c" target if you want an amalgamated source file. Be sure to carefully coordinate the defined values between the preprocessing step (to generate your source file(s)) and the build of your application/dll. If you are using

Re: [sqlite] SQLITE_OMIT_xxx

2008-04-21 Thread Mark Spiegel
> warnings, and even some errors. When fixing these problems, > it is simply easier to edit many smaller files rather than one > huge, unwieldy file. > > - Richard > > Mark Spiegel wrote: > >> That's what I do. Once your makefile is set up, make the "sqlite3.c

Re: [sqlite] OMIT_VIEW / TRIGGER bug?

2008-04-22 Thread Mark Spiegel
This was a problem for me too. I just chalked it up to deleting options and using the amalgamated source. (Seem to recall reading that this is not recommended.) Replace: SQLITE_PRIVATE void sqlite3MaterializeView(Parse*, Select*, Expr*, u32, int); with: #if !defined(SQLITE_OMIT_VIEW) &&

[sqlite] Perf improvements in 3.5.9

2008-05-15 Thread Mark Spiegel
The release notes for 3.5.9 indicate that performance improvements have been made around the way integers are stored. "Performance enhancement: Reengineer the internal routines used to interpret and render variable-length integers." Can someone in dev add some color to this statement? What

Re: [sqlite] Vista frustrations

2008-09-18 Thread Mark Spiegel
FILE_FLAG_RANDOM_ACCESS and FILE_FLAG_SEQUENTIAL_SCAN are hints to the cache manager (CC) in Windows and the underlying file system(s). With respect to the cache manager, it is going to affect whether or not there is read ahead, how much read ahead will be used, and how long data will remain

Re: [sqlite] Vista frustrations

2008-09-18 Thread Mark Spiegel
ll open a ticket. > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Mark Spiegel > Sent: Thursday, September 18, 2008 7:56 AM > To: [EMAIL PROTECTED]; General Discussion of SQLite Database > Subject: Re: [sqlite] Vista frustrations &

Re: [sqlite] Vista frustrations

2008-09-18 Thread Mark Spiegel
EMAIL PROTECTED] Behalf Of Mark Spiegel > Sent: Thursday, September 18, 2008 11:25 AM > To: General Discussion of SQLite Database > Subject: Re: [sqlite] Vista frustrations > > > I'm sorry, I have to take issue with that statement. The design of the > file system/cache m

[sqlite] Syntax sanity question...

2008-10-01 Thread Mark Spiegel
Doing some experimenting in my application and having trouble with an "INSERT OR REPLACE" statement. SQLite version 3.5.9 Table schema: CREATE TABLE DDS (FileId INTEGER PRIMARY KEY, _ModTime INTEGER, _CreTime INTEGER , _vtresult INTEGER, _md5 BINARY, _sha1 BINARY, _sha256 BINARY,

Re: [sqlite] Syntax sanity question...

2008-10-01 Thread Mark Spiegel
the source, the #ifdef in mkkeywordhas.c is the one that beat me? I'll remove this #define when I move forward to the more recent version of SQLite and try again. D. Richard Hipp wrote: > On Oct 1, 2008, at 6:25 PM, Mark Spiegel wrote: > >> -DSQLITE_OMIT_CONFL

Re: [sqlite] Compile SQLite3 for MS Windows Driver Kit user-mode application

2008-10-07 Thread Mark Spiegel
Not sure why the Win32 DLL is not compatible. I would think it should be. You might want to work that out first. Can you elaborate? As for compiling with the WDK, it can be done. The amalgamated source is best. The flood of warnings is a pain. SQLite dev claims they are all spurious, but

Re: [sqlite] VFS implementation guidance

2008-11-04 Thread Mark Spiegel
I would start with an existing VFS implementation and modify it to use the primitives available to you. I started with the OS_WIN and modified it. That will probably be easier than starting from scratch. You can switch VFSs with each open. You can use a different one for each DB open if you

Re: [sqlite] virtual file system

2009-03-26 Thread Mark Spiegel
As far as I'm aware, sqlite3_vfs_register() is the only way to register your own VFS. Martin Pfeifle wrote: > Dear all, > > I have a question regarding virtual file systems. > I assume I can load my own virtual file system by calling > the c-function sqlite3_vfs_register(...). > > Am I right

Re: [sqlite] PRAGMA doesn't support parameter binds?

2009-04-13 Thread Mark Spiegel
As an alternative, you can use the WinDgb tool from MSFT. It is not the nicest debugger in the world, but understands all the MSFT PDB formats. It is free for download from the MSDN site. While it is a bit slow with large source files like the SQLite amalgamation, it does handle them fine.

Re: [sqlite] SQLite crashing on iPhone (or so says Apple)

2009-04-21 Thread Mark Spiegel
I'm a bit confused by the following: "The assign 100K or so to each database connection's lookaside memory allocator using sqlite3_db_config(SQLITE_DBCONFIG_LOOKASIDE, ...) immediately after it is opened." If memory is at a premium, why would you reserve a large amount of it for SQLite's

Re: [sqlite] Run-Time Check Failure

2009-04-22 Thread Mark Spiegel
The compiler is not broken. If this behavior in the compiler bugs you, (pun intended) you can disable the warning locally or globally. This change is not obfuscating the code. It is not a work-around. It is making your intentions clear. When I read the line as written, I have to research

Re: [sqlite] Validating a file is a SQLite DB

2009-04-23 Thread Mark Spiegel
I would use the sqlite3_open_V2() call. It won't read read the whole database, just enough to get started, including the header. It will save you having to write and debug your own mechanism. Tommy Ocel wrote: > Hi, > > Other than using sqlite3_open_V2() with a SQLITE_OPEN_READONLY flag,

Re: [sqlite] SQLite version 3.6.14

2009-05-08 Thread Mark Spiegel
Based on what I've read, it certainly could. As I understand it, there is a single queue for all the writes so the writes for multiple databases, journals, etc. are going to that single queue. Based on that, it is certainly "possible" that your overall throughput and throughput for any

Re: [sqlite] 64 bit compatibility of sqlite 3.5.7

2009-06-04 Thread Mark Spiegel
For 32 bits it should be a long, for 64 bites, a 64 bit integer type. For example __int64 in Windows. sqlite3_intptr_t has to be large enough to hold the larger of an integer or a pointer on each platform. We had to fix this too... Gopala Surya wrote: > Hi All > We have been using sqlite

Re: [sqlite] Reducing the size of executable linked with sqlite3.c

2009-06-09 Thread Mark Spiegel
http://www.sqlite.org/compile.html There are options you can use to disable/remove unused features. Some can give significant size savings. (I did this on Windows, YMMV.) Unfortunately, you can't just use these with the amalgamated source. You will probably have to rebuild it. It's not

Re: [sqlite] Repost: Accessing a DB while copying it causes Windows to eat virtual memory

2009-07-07 Thread Mark Spiegel
Stan Bielski wrote: > In the course of copying a largish (20 GB) database file while > accessing it via sqlite3, the machine became very unresponsive. I > opened task manager and found that the system was using a huge amount > of virtual memory, causing it to thrash. Per-process memory usage >

Re: [sqlite] Repost: Accessing a DB while copying it causes Windows to eat virtual memory

2009-07-08 Thread Mark Spiegel
> Just to be clear, I think Windows is really the one that is at fault > here; the behavior seems very broken to me. Unfortunately, I need the > app to work on Windows without this happening, so I need to figure out > some kind of workaround in a.) Windows settings b.) the sqlite source > or c.)

Re: [sqlite] Repost: Accessing a DB while copying it causes Windows to eat virtual memory

2009-07-08 Thread Mark Spiegel
Stan Bielski wrote: > Hello again, > > Copying the database with Explorer and accessing it via sqlite is just > a contrived example that exhibits the same problem I'm having in my > application. The app does make a copy of the DB, but it has app-layer > locking that will prevent modifications

Re: [sqlite] Booleans in SQLite

2009-09-03 Thread Mark Spiegel
Jay A. Kreibich wrote: > Integer values between -128 and 127 use only a single byte of storage above > and beyond the header size that all values have. > Not quite. Values between 0 & 127 use 1 byte of storage. Negative values use the full 9 bytes in my experience. (I'm setting aside

Re: [sqlite] Booleans in SQLite

2009-09-03 Thread Mark Spiegel
D. Richard Hipp wrote: > You are both right and both wrong. There are two different integer > representations used in SQLite. > > (1) "varint" or variable length integer is an encoding of 64-bit > signed integers into between 1 and 9 bytes. Negative values use the > full 9 bytes as do

Re: [sqlite] Booleans in SQLite

2009-09-03 Thread Mark Spiegel
D. Richard Hipp wrote: > Most varints are "type varints" and type varints are almost always a > single byte (the only exceptions being for large blobs or strings). > Varints are also used to store the total number of bytes in a row > (also usually one byte). Most varints are a single byte.

Re: [sqlite] Booleans in SQLite

2009-09-03 Thread Mark Spiegel
D. Richard Hipp wrote: > Most varints are "type varints" and type varints are almost always a > single byte (the only exceptions being for large blobs or strings). > Varints are also used to store the total number of bytes in a row > (also usually one byte). Most varints are a single byte.

Re: [sqlite] Q. about core SQLite library

2010-01-13 Thread Mark Spiegel
Jay A. Kreibich wrote: > On Tue, Jan 12, 2010 at 02:53:55PM -0500, Ray Gold scratched on the wall: > >> Hello. >> >> My name is Ray Gold with Berliner, Corcoran & Rowe, LLP in Washington, DC. >> >> I am writing to ask if the core SQLite library (not the proprietary >> SQLite Extensions) has

Re: [sqlite] Q. about core SQLite library

2010-01-13 Thread Mark Spiegel
> I was under the impression that is how SEE works. Hooking the SQLite > Virtual File System interface seems like a very straight forward method. > All you really need to do is splice your code between the VFS that > the SQLite engine sees and the native VFS layer that comes with the >

Re: [sqlite] Q. about core SQLite library

2010-01-13 Thread Mark Spiegel
Sorry, Mr. Gold should contact DRH and ask about the encryption extension. My error. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SQLite on Windows 2003

2010-04-07 Thread Mark Spiegel
Burnett, Joe wrote: > Hi Teg, > > UTF-8, no special characters in the file name WorkData.s3db. > > Thanks, > > Joe > > Joe Burnett | Principal Software Engineer | FIDELITY INSTITUTIONAL > TECHNOLOGY > 2 Contra Way > Merrimack, NH 03054 > 603.791.5113 > cell: 603.289.0481 > If you can debug