Re: [sqlite] SQLite and Large Databases

2007-11-09 Thread drh
Teg <[EMAIL PROTECTED]> wrote: > > dhc> Recompile version 3.5.2 with -DSQLITE_MEMORY_SIZE=1000 > dhc> or however much memory you want SQLite to use. This will > dhc> create a static array of char[] of size 1000 (or whatever > dhc> other size you give it) and use that instead of malloc()

Re: [sqlite] SQLite and Large Databases

2007-11-08 Thread Nuno Lucas
On Nov 8, 2007 5:04 PM, PokerAce <[EMAIL PROTECTED]> wrote: > I am using Windows Task Manager to determine memory usage. I minimize my > application (which causes the task manager to show a more accurate > measurement of memory used) and still the memory usage is enormous. Windows Task Manager

Re: [sqlite] SQLite and Large Databases

2007-11-08 Thread drh
PokerAce <[EMAIL PROTECTED]> wrote: > I am using Windows Task Manager to determine memory usage. I minimize my > application (which causes the task manager to show a more accurate > measurement of memory used) and still the memory usage is enormous. > > Recompile version 3.5.2 with

Re: [sqlite] SQLite and Large Databases

2007-11-08 Thread PokerAce
Actually, I am. I abstract the database communication in my application, but I access each database using their native API. On Nov 8, 2007 12:44 PM, Joe Wilson <[EMAIL PROTECTED]> wrote: > --- PokerAce <[EMAIL PROTECTED]> wrote: > > "Are you certain it's sqlite RAM, and not your application?"

Re: [sqlite] SQLite and Large Databases

2007-11-08 Thread Joe Wilson
--- PokerAce <[EMAIL PROTECTED]> wrote: > "Are you certain it's sqlite RAM, and not your application?" > > Yes, we are also testing PostgreSQL and MySQL and the application memory > with those stays < 20 mb. You're not using the sqlite3 API directly, are you? If that's the case, I think your

Re: [sqlite] SQLite and Large Databases

2007-11-08 Thread PokerAce
"Are you certain it's sqlite RAM, and not your application?" Yes, we are also testing PostgreSQL and MySQL and the application memory with those stays < 20 mb. "Is your temp_store set to memory or file?" I have not changed this, so it's whatever the default is. "How are you performing your

Re: [sqlite] SQLite and Large Databases

2007-11-08 Thread PokerAce
I am using Windows Task Manager to determine memory usage. I minimize my application (which causes the task manager to show a more accurate measurement of memory used) and still the memory usage is enormous. On Nov 8, 2007 11:55 AM, Roger Binns <[EMAIL PROTECTED]> wrote: > -BEGIN PGP

Re: [sqlite] SQLite and Large Databases

2007-11-08 Thread PokerAce
If you have to set that manually, it was not set. I am using version 3.5.2. I am doing tens of thousands of transactions, each without ~30 separate INSERTs. On Nov 8, 2007 11:40 AM, <[EMAIL PROTECTED]> wrote: > PokerAce <[EMAIL PROTECTED]> wrote: > > I'm trying to see if SQLite is suitable

Re: [sqlite] SQLite and Large Databases

2007-11-08 Thread Joe Wilson
--- PokerAce <[EMAIL PROTECTED]> wrote: > I'm trying to see if SQLite is suitable for large databases ( > 1 gb, > millions of rows in each of several tables). Initially, the memory usage > was outrageous (~ 500 mb for a 1.3 gb db), but I got that down to < 30 mb by > setting the cache size to 0

Re: [sqlite] SQLite and Large Databases

2007-11-08 Thread Roger Binns
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 PokerAce wrote: > Initially, the memory usage > was outrageous (~ 500 mb for a 1.3 gb db), How are you measuring memory usage? You have to be very careful as most operating system tools will report a number greater than the size of all current

Re: [sqlite] SQLite and Large Databases

2007-11-08 Thread drh
PokerAce <[EMAIL PROTECTED]> wrote: > I'm trying to see if SQLite is suitable for large databases ( > 1 gb, > millions of rows in each of several tables). Initially, the memory usage > was outrageous (~ 500 mb for a 1.3 gb db), but I got that down to < 30 mb by > setting the cache size to 0 and