[sqlite] Memory Usage

2014-10-03 Thread David Muchene
Hi, I was wondering what options I can tune to make sqlite use more memory. We are currently using the memsys5 allocator and giving it a 2G buffer, but it doesn't seem to be using any more than 32MB. Thanks, Dave M ___ sqlite-users mailing list

Re: [sqlite] Memory usage of sqlite3

2013-07-17 Thread Paolo Bolzoni
On Tue, Jul 16, 2013 at 8:13 PM, Eduardo emorr...@yahoo.es wrote: Can you show us the query and/or schemas? If not: Sure, I appended everything in the bottom of this email. Unfortunately gmail will mess-up the layout, I hope it will be readable. (See here, it seems google does not know the mean

Re: [sqlite] Memory usage of sqlite3

2013-07-17 Thread Eduardo Morras
On Wed, 17 Jul 2013 12:04:52 +0200 Paolo Bolzoni paolo.bolzoni.br...@gmail.com wrote: On Tue, Jul 16, 2013 at 8:13 PM, Eduardo emorr...@yahoo.es wrote: Can you show us the query and/or schemas? If not: Sure, I appended everything in the bottom of this email. Unfortunately gmail will

Re: [sqlite] Memory usage of sqlite3

2013-07-17 Thread Paolo Bolzoni
Some changes, if blob is bigger than a few bytes, you should normalize them. If 2 blobs are equal, their id must be equal and you don't waste time comparing nor memory joining blob content. So you get: They are quite small (max ~70 bytes...) DROP TABLE IF EXISTS tour_blob; CREATE TABLE

Re: [sqlite] Memory usage of sqlite3

2013-07-16 Thread Paolo Bolzoni
The test ended sometime during the night and setting temp_store to 0 the result is exactly the same. I suspect it was the default anyway. On Mon, Jul 15, 2013 at 9:20 PM, Paolo Bolzoni paolo.bolzoni.br...@gmail.com wrote: On Mon, Jul 15, 2013 at 9:08 PM, Eduardo Morras emorr...@yahoo.es wrote:

Re: [sqlite] Memory usage of sqlite3

2013-07-16 Thread Dan Kennedy
On 07/16/2013 01:49 AM, Paolo Bolzoni wrote: From 35-40MB to 940MB; I would put massif result but I think the list deletes attachments. A very large blob or string result? Code allocates (or leaks) tremendous numbers of sqlite3_stmt* handles? SQLite has various APIs for querying memory

Re: [sqlite] Memory usage of sqlite3

2013-07-16 Thread Paolo Bolzoni
On Tue, Jul 16, 2013 at 1:00 PM, Dan Kennedy danielk1...@gmail.com wrote: On 07/16/2013 01:49 AM, Paolo Bolzoni wrote: A very large blob or string result? I would exclude this, I do use blobs... but they are at most few dozen of bytes... Code allocates (or leaks) tremendous numbers of

Re: [sqlite] Memory usage of sqlite3

2013-07-16 Thread Paolo Bolzoni
I tried the experiment again with -g3 -O0, I got less information than expected (there are still many unknown symbols in libsqlite3.so), but the function requiring all this memory is sqlite3_step. So maybe it is one complex query? I would like to avoid excessive swapping on the production server,

Re: [sqlite] Memory usage of sqlite3

2013-07-16 Thread Simon Slavin
On 16 Jul 2013, at 5:17pm, Paolo Bolzoni paolo.bolzoni.br...@gmail.com wrote: the function requiring all this memory is sqlite3_step. So maybe it is one complex query? Possibly a query for which no good index exists, so SQLite decides to make up its own temporary index. If you consider

Re: [sqlite] Memory usage of sqlite3

2013-07-16 Thread Eduardo
On Tue, 16 Jul 2013 18:17:41 +0200 Paolo Bolzoni paolo.bolzoni.br...@gmail.com wrote: I tried the experiment again with -g3 -O0, I got less information than expected (there are still many unknown symbols in libsqlite3.so), but the function requiring all this memory is sqlite3_step. Can you

[sqlite] Memory usage of sqlite3

2013-07-15 Thread Paolo Bolzoni
I wrote an C++ application that uses sqlite3 to save part of the data when it become larger than a known threshold. The idea is to use at most a known quantity of memory; to check if it is working I executed a relevant test using valgrind's massif. It worked fairly well most of the time, but in

Re: [sqlite] Memory usage of sqlite3

2013-07-15 Thread Stephan Beal
On Mon, Jul 15, 2013 at 8:39 PM, Paolo Bolzoni paolo.bolzoni.br...@gmail.com wrote: So, sorry if the question sounds very vague. But what can cause high memory usage in sqlite? A large transaction maybe? What is high? In my apps sqlite tends to use 200-400kb or so, which i don't consider

Re: [sqlite] Memory usage of sqlite3

2013-07-15 Thread Paolo Bolzoni
From 35-40MB to 940MB; I would put massif result but I think the list deletes attachments. On Mon, Jul 15, 2013 at 8:41 PM, Stephan Beal sgb...@googlemail.com wrote: On Mon, Jul 15, 2013 at 8:39 PM, Paolo Bolzoni paolo.bolzoni.br...@gmail.com wrote: So, sorry if the question sounds very

Re: [sqlite] Memory usage of sqlite3

2013-07-15 Thread Simon Slavin
On 15 Jul 2013, at 7:49pm, Paolo Bolzoni paolo.bolzoni.br...@gmail.com wrote: From 35-40MB to 940MB; I would put massif result but I think the list deletes attachments. Do you have in-memory tables ? Do you use sqlite3_exec() ? Do you have SELECTs for which there is no good index, forcing

Re: [sqlite] Memory usage of sqlite3

2013-07-15 Thread Jay A. Kreibich
On Mon, Jul 15, 2013 at 08:49:52PM +0200, Paolo Bolzoni scratched on the wall: From 35-40MB to 940MB; I would put massif result but I think the list deletes attachments. By default, the page-cache is 2000. Pages are typically 1KB, but have some minor overhead in the cache. Assuming you

Re: [sqlite] Memory usage of sqlite3

2013-07-15 Thread Paolo Bolzoni
On Mon, Jul 15, 2013 at 8:59 PM, Simon Slavin slav...@bigfraud.org wrote: On 15 Jul 2013, at 7:49pm, Paolo Bolzoni paolo.bolzoni.br...@gmail.com wrote: From 35-40MB to 940MB; I would put massif result but I think the list deletes attachments. Do you have in-memory tables ? No. Do you

Re: [sqlite] Memory usage of sqlite3

2013-07-15 Thread Eduardo Morras
On Mon, 15 Jul 2013 20:49:52 +0200 Paolo Bolzoni paolo.bolzoni.br...@gmail.com wrote: From 35-40MB to 940MB; I would put massif result but I think the list deletes attachments. What does PRAGMA temp_store show? Set it to 0 and recheck. Did you compile with SQLITE_TEMP_STORE set to 3? ---

Re: [sqlite] Memory usage of sqlite3

2013-07-15 Thread Paolo Bolzoni
On Mon, Jul 15, 2013 at 9:08 PM, Eduardo Morras emorr...@yahoo.es wrote: On Mon, 15 Jul 2013 20:49:52 +0200 Paolo Bolzoni paolo.bolzoni.br...@gmail.com wrote: From 35-40MB to 940MB; I would put massif result but I think the list deletes attachments. What does PRAGMA temp_store show? Set it

[sqlite] Track SQLite memory usage from inside Java code

2012-06-26 Thread mira
SQL's memory usage. I found a solution for C++ but I use Java. Any help would be greatly appreciated! Mira -- View this message in context: http://sqlite.1065341.n5.nabble.com/Track-SQLite-memory-usage-from-inside-Java-code-tp62756.html Sent from the SQLite mailing list archive at Nabble.com

Re: [sqlite] Memory Usage/ Drawbacks of Statements

2012-04-24 Thread Eric Minbiole
Similar to Pavel's suggestion, our implementation maintains a simple cache of prepared statements, keyed by the SQL query that created them. For example: pStatement = Cache.GetQuery(SELECT * FROM xyz); would return the cached statement if the query had been seen before, or would auto-create

Re: [sqlite] Memory Usage/ Drawbacks of Statements

2012-04-24 Thread Mohit Sindhwani
Hi Eric, On 24/4/2012 10:03 PM, Eric Minbiole wrote: Similar to Pavel's suggestion, our implementation maintains a simple cache of prepared statements, keyed by the SQL query that created them. For example: pStatement = Cache.GetQuery(SELECT * FROM xyz); would return the cached statement

[sqlite] Memory Usage/ Drawbacks of Statements

2012-04-23 Thread Mohit Sindhwani
Hi, our system does fairly predictable queries when it runs. A number of modules all access data using a handful of queries of each. We open the database at the start and close it at the end of the program. Each query follows the usual pattern of prepare - bind - step - reset - (eventually)

Re: [sqlite] Memory Usage/ Drawbacks of Statements

2012-04-23 Thread Pavel Ivanov
1. Do statements do any thing that would require a lot of memory to be maintained? No, they don't need a lot of memory, but still some memory is used. So if you have like thousands of statements you should worry about this. If you have 20 or 30 statements your database cache will likely consume

Re: [sqlite] Memory Usage/ Drawbacks of Statements

2012-04-23 Thread Mohit Sindhwani
Thanks Pavel, That gives me something new to do with SQLite over the next few weeks. On 23/4/2012 8:47 PM, Pavel Ivanov wrote: 1. Do statements do any thing that would require a lot of memory to be maintained? No, they don't need a lot of memory, but still some memory is used. So if you have

Re: [sqlite] memory usage after VACUUM

2011-03-10 Thread Philip Graham Willoughby
On 9 Mar 2011, at 15:23, Nick Hodapp wrote: I'm using sqlite in an iOS app, via the popular FMDB wrapper. My profiling tool is showing me that the app is using 2.5 MB of memory before a VACUUM, and nearly 6MB after. The tool shows that the extra memory was allocated by sqlite3MemMalloc().

[sqlite] memory usage after VACUUM

2011-03-09 Thread Nick Hodapp
I'm using sqlite in an iOS app, via the popular FMDB wrapper. My profiling tool is showing me that the app is using 2.5 MB of memory before a VACUUM, and nearly 6MB after. The tool shows that the extra memory was allocated by sqlite3MemMalloc(). If I close and re-open the database then the

Re: [sqlite] memory usage after VACUUM

2011-03-09 Thread Enrico Thierbach
On 09.03.2011, at 16:23, Nick Hodapp wrote: I'm using sqlite in an iOS app, via the popular FMDB wrapper. My profiling tool is showing me that the app is using 2.5 MB of memory before a VACUUM, and nearly 6MB after. The tool shows that the extra memory was allocated by sqlite3MemMalloc().

Re: [sqlite] memory usage after VACUUM

2011-03-09 Thread Pavel Ivanov
Is there any sqlite function I can call, or some other technique, to reduce the memory allocated and hung-onto by sqlite, particularly during a VACUUM? Yes, execute pragma cache_size = 100 for example, or put other number of your liking into there. If closing and re-opening of the database

[sqlite] SQLite Memory Usage

2010-11-08 Thread Sachin . 2 . Gupta
Hi, We are trying to Integrate SQLite in our Application and are trying to populate as a Cache. We are planning to use it as a In Memory Database. Using it for the first time. Our Application is C++ based. Our Application interacts with the Master Database to fetch data and performs numerous

Re: [sqlite] SQLite Memory Usage

2010-11-08 Thread Black, Michael (IS)
From: sqlite-users-boun...@sqlite.org on behalf of Sachin.2.Gupta Sent: Mon 11/8/2010 4:47 AM To: sqlite-users@sqlite.org Subject: EXTERNAL:[sqlite] SQLite Memory Usage Hi, We are trying to Integrate SQLite in our Application and are trying to populate as a Cache. We are planning

Re: [sqlite] Memory usage – one data base versus tw o smaller ones

2010-02-19 Thread Israel Lins Albuquerque
: Quarta-feira, 17 de Fevereiro de 2010 16:38:27 (GMT-0300) Auto-Detected Assunto: [sqlite] Memory usage – one data base versus two smaller ones For some reasons it is more convenient for the project to have a few smaller databases with unrelated data than one containing everything. My only

Re: [sqlite] Memory usage ??? one data base versus two smaller ones

2010-02-19 Thread Jay A. Kreibich
On Fri, Feb 19, 2010 at 09:39:08AM -0300, Israel Lins Albuquerque scratched on the wall: Samuel, Each one attached database has its own page cache with 2000 (default number of pages in cache) * 1024 (default size in bytes of a page), On many Windows systems it will default to 4096. It

[sqlite] Memory usage – one data base versus tw o smaller ones

2010-02-17 Thread a1rex
For some reasons it is more convenient for the project to have a few smaller databases with unrelated data than one containing everything. My only concern is RAM memory. How much burden/memory overhead an additional database would introduce? Thank you for your input, Samuel

[sqlite] memory usage and queries

2009-05-12 Thread Christopher Taylor
I am running into issues where I am running out of memory on my embedded app. I have stored 10K records in a table. There is an index on a key field and a sort field. An external program needs to reconcile its data with the data in this table. It does so 200 records at a time. The code I am

Re: [sqlite] memory usage and queries

2009-05-12 Thread Christopher Taylor
I fixed this by reducing the cache size from 2000 to 200. I may want to adjust some more but will continue to test. Chris ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Memory Usage

2009-04-21 Thread Marco Bambini
Hello guys, I am trying to write you again about a simple question... how can I limit sqlite memory usage during insert commands? It seems that the amount of memory usage increases when the number of objects inserted into the database is increased and memory is never freed. I tried to set

Re: [sqlite] Memory Usage

2009-04-21 Thread D. Richard Hipp
On Apr 21, 2009, at 8:08 AM, Marco Bambini wrote: Hello guys, I am trying to write you again about a simple question... how can I limit sqlite memory usage during insert commands? It seems that the amount of memory usage increases when the number of objects inserted into the database

Re: [sqlite] Memory Usage

2009-04-21 Thread Marco Bambini
http://www.creolabs.com/payshield/ On Apr 21, 2009, at 2:11 PM, D. Richard Hipp wrote: On Apr 21, 2009, at 8:08 AM, Marco Bambini wrote: Hello guys, I am trying to write you again about a simple question... how can I limit sqlite memory usage during insert commands? It seems

Re: [sqlite] Memory Usage

2009-04-21 Thread D. Richard Hipp
On Apr 21, 2009, at 8:22 AM, Marco Bambini wrote: Yes, executing sqlite3_memory_used () after 183,000 INSERT statement returns: 106,766,848. Database is never closed during application lifetime. Each statement is prepared, stepped and properly finalized. We do that kind of test all the

Re: [sqlite] Memory Usage

2009-04-21 Thread Marco Bambini
The database is on-disk ... does huge not committed transactions uses memory? -- Marco Bambini http://www.sqlabs.com http://www.creolabs.com/payshield/ On Apr 21, 2009, at 2:27 PM, D. Richard Hipp wrote: On Apr 21, 2009, at 8:22 AM, Marco Bambini wrote: Yes, executing

Re: [sqlite] Memory Usage

2009-04-21 Thread D. Richard Hipp
On Apr 21, 2009, at 9:07 AM, Marco Bambini wrote: The database is on-disk ... does huge not committed transactions uses memory? It should do writes to disk periodically to free up memory, once you hit your cache_size limit. What is cache_size set to. What are the output from

Re: [sqlite] Memory Usage

2009-04-21 Thread Marco Bambini
cache_size is set to default 2000, page size is 1K... here you go the output of sqlite3_status: 2009-04-21 15:24:25 SQLITE_STATUS_MEMORY_USED current: 106704136 high: 109873952 2009-04-21 15:24:25 SQLITE_STATUS_PAGECACHE_USED current: 0 high: 0 2009-04-21 15:24:25

Re: [sqlite] Memory Usage

2009-04-21 Thread D. Richard Hipp
On Apr 21, 2009, at 9:25 AM, Marco Bambini wrote: cache_size is set to default 2000, page size is 1K... here you go the output of sqlite3_status: 2009-04-21 15:24:25 SQLITE_STATUS_MEMORY_USED current: 106704136 high: 109873952 2009-04-21 15:24:25 SQLITE_STATUS_PAGECACHE_USED current: 0

Re: [sqlite] Memory Usage

2009-04-21 Thread D. Richard Hipp
On Apr 21, 2009, at 9:25 AM, Marco Bambini wrote: cache_size is set to default 2000, page size is 1K... here you go the output of sqlite3_status: 2009-04-21 15:24:25 SQLITE_STATUS_MEMORY_USED current: 106704136 high: 109873952 2009-04-21 15:24:25 SQLITE_STATUS_PAGECACHE_USED current: 0

Re: [sqlite] Memory Usage

2009-04-21 Thread D. Richard Hipp
On Apr 21, 2009, at 10:54 AM, Marco Bambini wrote: and the read lock is released when the virtual machine is finalized right? ... even if the writer is inside a BEGIN IMMEDIATE transaction? Correct. D. Richard Hipp d...@hwaci.com ___

Re: [sqlite] Memory Usage

2009-04-21 Thread D . Richard Hipp
On Apr 21, 2009, at 7:12 PM, Marco Bambini wrote: Hello Dr. Hipp, I was finally able to track down the issue ... the problem is due to the fact that after each write operation the client executes a query like: SELECT 123 AS changes; (the number 123 changes all the time) this select

[sqlite] Memory usage

2009-04-17 Thread Marco Bambini
Hello guys, I need your help in order to solve a very annoying issue with sqlite 3.6.11. I have two opened db inside my application and when I insert 180,000 rows inside a transaction I can see heap memory usage that exceeds 100MB (data is written twice so I have 2 transactions inside two

[sqlite] memory usage

2008-12-30 Thread #
Hello, My multi-threaded application has various sqlite db's open simultaneously, in memory using the :memory: keyword, disk based db's and at times, tmpfs (ram) db's. Is there a way to view each individual database's memory usage? I found the functions sqlite3_memory_used() and

[sqlite] memory usage

2008-12-30 Thread ed
Hello, My multi-threaded application has various sqlite db's open simultaneously, in memory using the :memory: keyword, disk based db's and at times, tmpfs (ram) db's. Is there a way to view each individual database's memory usage? I found the functions sqlite3_memory_used() and

[sqlite] Fwd: sqlite memory usage

2008-12-22 Thread #
Hello, My multi-threaded application has various sqlite db's open simultaneously, in memory using the :memory: keyword, disk based db's and at times, tmpfs (ram) db's. Is there a way to view each individual database's memory usage? I found the functions sqlite3_memory_used() and

[sqlite] sqlite memory usage

2008-12-18 Thread #
Hello,My first mail attempt appears not to have gotten through yesterday due to the spam storm. My multi-threaded application has various sqlite db's open simultaneously, in memory using the :memory: keyword, disk based db's and at times, tmpfs based db's. Is there a way to view each individual

[sqlite] memory usage

2008-12-18 Thread #
Hello,My application has several sqlite db's open simultaneously, in memory using the :memory: keyword, disk based db's and at times, tmpfs based db's. Is there a way to view each individual database's memory usage? I found the functions sqlite3_memory_used() and

Re: [sqlite] Memory Usage

2007-11-20 Thread Joe Wilson
Once again you're missing the point. Of course you can get a malloc/free implementation that performs garbage collection, such as Boehm's conservative GC. But C garbage collection and malloc/free memory fragmentation are quite different things. You can still get heavily fragmented memory with

Re: [sqlite] Memory Usage

2007-11-19 Thread John Stanton
Not only applicable to real time systems. If you want a program to run with stability over a long time the first step it to eliminate frees and if malloc is used confine it to the intialization. Jim Dodgen wrote: One other note, just about all real-time systems limit the dynamic allocation

RE: [sqlite] Memory Usage

2007-11-19 Thread James Dennett
-Original Message- From: John Stanton [mailto:[EMAIL PROTECTED] Sent: Monday, November 19, 2007 7:36 AM To: sqlite-users@sqlite.org Subject: Re: [sqlite] Memory Usage Not only applicable to real time systems. If you want a program to run with stability over a long time the first

Re: [sqlite] Memory Usage

2007-11-19 Thread D. Richard Hipp
On Nov 19, 2007, at 12:36 PM, James Dennett wrote: -Original Message- From: John Stanton [mailto:[EMAIL PROTECTED] Sent: Monday, November 19, 2007 7:36 AM To: sqlite-users@sqlite.org Subject: Re: [sqlite] Memory Usage Not only applicable to real time systems. If you want a program

Re: [sqlite] Memory Usage

2007-11-19 Thread John Stanton
: Re: [sqlite] Memory Usage Not only applicable to real time systems. If you want a program to run with stability over a long time the first step it to eliminate frees and if malloc is used confine it to the intialization. I have to challenge this, not because it's entirely wrong (it's

RE: [sqlite] Memory Usage

2007-11-19 Thread James Dennett
-Original Message- From: John Stanton [mailto:[EMAIL PROTECTED] Sent: Monday, November 19, 2007 12:14 PM To: sqlite-users@sqlite.org Subject: Re: [sqlite] Memory Usage Dynamic allocation is not the problem, it is malloc and free. there is a difference between being certain

Re: [sqlite] Memory Usage

2007-11-19 Thread John Stanton
To: sqlite-users@sqlite.org Subject: Re: [sqlite] Memory Usage Not only applicable to real time systems. If you want a program to run with stability over a long time the first step it to eliminate frees and if malloc is used confine it to the intialization. I have to challenge

Re: [sqlite] Memory Usage

2007-11-19 Thread John Stanton
D. Richard Hipp wrote: On Nov 19, 2007, at 12:36 PM, James Dennett wrote: -Original Message- From: John Stanton [mailto:[EMAIL PROTECTED] Sent: Monday, November 19, 2007 7:36 AM To: sqlite-users@sqlite.org Subject: Re: [sqlite] Memory Usage Not only applicable to real time systems

Re: [sqlite] Memory Usage

2007-11-19 Thread John Stanton
James Dennett wrote: -Original Message- From: John Stanton [mailto:[EMAIL PROTECTED] Sent: Monday, November 19, 2007 12:14 PM To: sqlite-users@sqlite.org Subject: Re: [sqlite] Memory Usage Dynamic allocation is not the problem, it is malloc and free. there is a difference between

Re: [sqlite] Memory Usage

2007-11-19 Thread Mark Spiegel
Well said. While it may be true that some memory allocators are lacking, the ones I use are quite good. I view with great suspicion developers who thinks they can outsmart the pool allocator. These folks usually add great complexity while having at best a neutral impact on performance and

Re: [sqlite] Memory Usage

2007-11-19 Thread Trevor Talbot
On 11/19/07, John Stanton [EMAIL PROTECTED] wrote: Malloc is a concept implemented in various ways, some more successful than others but all of them hidden from the programmer. Free tries to give back memory but as you can appreciate unless you use some garbage collection scheme with

Re: [sqlite] Memory Usage

2007-11-19 Thread John Stanton
You confused my point which is that your usual malloc/free definitely does no garbage collection. That does not mean that a C language program cannot perform garbage collection, just look at a Java run time package for an example. If you never execute a free your dynamic memory is

Re: [sqlite] Memory Usage

2007-11-19 Thread Trevor Talbot
On 11/19/07, John Stanton [EMAIL PROTECTED] wrote: Joe Wilson wrote: If a C program employs perfect 1:1 malloc'ing to free'ing, i.e., has no memory leaks, then garbage collection is irrelevant to the topic of memory fragmentation. It's not like C can employ a copying garbage collector

Re: [sqlite] Memory Usage

2007-11-18 Thread Russell Leighton
On Nov 17, 2007, at 4:56 PM, [EMAIL PROTECTED] wrote: If you compile with -DSQLITE_MEMORY_SIZE= then SQLite will *never* call malloc(). Instead, it uses a static array that is bytes in size for all of its memory needs. You can get by with as little as 100K or so of memory, though

Re: [sqlite] Memory Usage

2007-11-18 Thread D. Richard Hipp
On Nov 18, 2007, at 8:12 AM, Russell Leighton wrote: On Nov 17, 2007, at 4:56 PM, [EMAIL PROTECTED] wrote: If you compile with -DSQLITE_MEMORY_SIZE= then SQLite will *never* call malloc(). Instead, it uses a static array that is bytes in size for all of its memory needs. You can

Re: [sqlite] Memory Usage

2007-11-18 Thread Jim Dodgen
One other note, just about all real-time systems limit the dynamic allocation of memory because you lose the deterministic behavior, typically all memory is allocated when the task starts, memory is usually managed internally in standard sized chunks. Also for long running tasks

Re: [sqlite] Memory Usage

2007-11-17 Thread Raymond Hurst
Hi Scott, My initial evaluation of this database was that it allocates memory for each operation on the database. It returns the memory only when the database is committed. So the behavior you see is normal. Ray Hurst ScottDerrick wrote: I am using sqlite3 in a DAQ device. Data can be

Re: [sqlite] Memory Usage

2007-11-17 Thread Raymond Hurst
Hi Scott, Ooops..meant to say the following. My initial evaluation of this database was that it allocates memory for each operation on the database. It returns the memory only when the database is CLOSED. So the behavior you see is normal. Ray Hurst ScottDerrick wrote: I am using sqlite3

Re: [sqlite] Memory Usage

2007-11-17 Thread ScottDerrick
There must be a way to flush what ever is being cached. Help It's hard to believe I'm the only guy that wants to keep the database open and just do inserts, to save CPU time. Scott rhurst2 wrote: Hi Scott, Ooops..meant to say the following. My initial evaluation of this

Re: [sqlite] Memory Usage

2007-11-17 Thread drh
Raymond Hurst [EMAIL PROTECTED] wrote: Hi Scott, Ooops..meant to say the following. My initial evaluation of this database was that it allocates memory for each operation on the database. It returns the memory only when the database is CLOSED. So the behavior you see is normal. Ray

Re: [sqlite] Memory Usage

2007-11-17 Thread ScottDerrick
Dr. Hipp, thanks, I'm sure I can use one or more of your solutions below to solve my problem... Scott SQLite does maintain a cache of the database file. It will hold up to 2000 pages by default. You can change the cache size by using the PRAGMA cache_size=N pragma. You can set N as small as

[sqlite] Memory Usage

2007-11-16 Thread ScottDerrick
I am using sqlite3 in a DAQ device. Data can be viewed on the unit using a Rails enabled web server. The data is being stored to the database every 1 to 5 seconds. I wanted to leave the the database open for as long as teh application is running and then use a IMMEDIATE, PREPARE-INSERT(x),

Re: [sqlite] Memory Usage

2006-10-30 Thread Nuno Lucas
On 10/30/06, Ben Clewett [EMAIL PROTECTED] wrote: Hi Numo and others, I am very glad to hear the consensus is that there is nothing wrong with libsqlite3.so.0.8.6. However the fact is that the 'open' still acquires 16MB of memory. Immediately Before: VmSize: 8572 kB VmLck:

Re: [sqlite] Memory Usage

2006-10-30 Thread Ben Clewett
Nuno, Thanks for the excelent description of my error. I have learnt a little more about Linux virtual memory model. Very glad to hear Sqlite is as perfect as ever :) My problem, which is definitely my problem, is that 90 x 16MB of reserved memory is still a loss of 1.4G. Especially as I

Re: [sqlite] Memory Usage 2

2006-10-30 Thread drh
Ben Clewett [EMAIL PROTECTED] wrote: Is the allocation of 4108 KB normal for Sqlite? No. Not for me. If you open the same database using the command-line client, how much memory does it use? Why, then run from xined, is the memory allocation four times as much? Why is just Sqlite effected

Re: [sqlite] Memory Usage

2006-10-30 Thread Nuno Lucas
On 10/30/06, Ben Clewett [EMAIL PROTECTED] wrote: Nuno, Thanks for the excelent description of my error. I have learnt a little more about Linux virtual memory model. Very glad to hear Sqlite is as perfect as ever :) My problem, which is definitely my problem, is that 90 x 16MB of reserved

Re: [sqlite] Memory Usage

2006-10-30 Thread Lloyd
I don't know whether I am right in this perspective. Just to know whether sqlite is causing the high memory usage, comment the commands (statements) related to sqlite and check the memory status. Thanks, Lloyd. On Mon, 2006-10-30 at 12:45 +, Nuno Lucas wrote: On 10/30/06, Ben Clewett

Re: [sqlite] Memory Usage

2006-10-30 Thread Ben Clewett
Hi Nuno, Sqlite is one mailing list I have consistently found absolutely excelent knowledge, thanks again for your information. I don't know whether this should be off-thread now, but I don't have your email address. I'll have to research memory allocation further. But I'm glad to know

Re: [sqlite] Memory Usage

2006-10-30 Thread Martin Jenkins
Ben Clewett wrote: If you know a good URL on Linux virtual memory and allocation, I would be extremely interested. You could try: http://virtualthreads.blogspot.com/2006/02/understanding-memory-usage-on-linux.html The next two link to pages with links to a PDF of the gorman book

Re: [sqlite] Memory Usage

2006-10-30 Thread John Stanton
What happens when you write a simple test program to open the DB? You can certainly run Valgrind on that or even put in your own debug statements to examine the heap usage. This would be the first thing to do in indentifying the problem. It will tell you whether to look at Sqlite or your

Re: [sqlite] Memory Usage

2006-10-30 Thread Dennis Jenkins
If you are seeing different memory usage patterns for identical code based on if it is run from xinetd or on your command line, then I would check the process environment that xinetd creates. Maybe some component that sqlite uses is acting differently based on environment variables? I

Re: [sqlite] Memory Usage

2006-10-30 Thread Eduardo
At 14:40 30/10/2006, you wrote: I am suffering a 4GB memory 64-bit Zeon Linux box, which keeps crashing with 'No available memory'. I'm finding it quite hard to break down the memory into what processes are paged-in and using what's available. Sqlite seemed to be the smoking gun, so although

[sqlite] Memory Usage

2006-10-27 Thread Ben Clewett
Dear Sqlite, I very much enjoy using Sqlite, it is extremely useful. I have a memory usage query. I am linking to libsqlite3.so.0.8.6. After calling sqlite3_open(...) I find my programs data memory jumps by 16392 Kb. This seems a lot. The database I am opening is only 26K in size. I have

Re: [sqlite] Memory Usage

2006-10-27 Thread Nuno Lucas
On 10/27/06, Ben Clewett [EMAIL PROTECTED] wrote: I am linking to libsqlite3.so.0.8.6. After calling sqlite3_open(...) I find my programs data memory jumps by 16392 Kb. This seems a lot. The database I am opening is only 26K in size. There are many different ways of memory jump (like

Re: [sqlite] Memory Usage

2006-10-27 Thread Eduardo
At 18:00 27/10/2006, you wrote: Dear Sqlite, I very much enjoy using Sqlite, it is extremely useful. I have a memory usage query. I am linking to libsqlite3.so.0.8.6. After calling sqlite3_open(...) I find my programs data memory jumps by 16392 Kb. This seems a lot. The database I am

Re: [sqlite] Memory Usage

2006-10-27 Thread Lloyd
Most probably it will be a memory leak in your program. We must release the dynamically allocated memory ourselves. So check whether you are forgetting to do that. Most probably that leak will be happening inside some loops or repeatedly calling functions. On Fri, 2006-10-27 at 17:00 +0100, Ben

[sqlite] Memory Usage

2006-09-26 Thread Paul M
Hi all, I have a program that uses SQLite and it frequently updates and adds to the database. However, despite my freeing the result when done memory usage continues to increase over time. The database has currently has 940 rows and uses sever selects and a select with limit and offset. Also only

Re: [sqlite] Memory usage and multiple small databases

2005-07-14 Thread Christian Smith
On Wed, 13 Jul 2005, Joey Ekstrom wrote: Would sqlite perform well in a web application where each user got their own database to operate on? I know that the size of the sqlite library is small, but what is the memory footprint after opening the database? Would it be prohibitive to having a

Re: [sqlite] Memory usage and multiple small databases

2005-07-14 Thread Joey Ekstrom
On 7/14/05, Christian Smith [EMAIL PROTECTED] wrote: On Wed, 13 Jul 2005, Joey Ekstrom wrote: Would sqlite perform well in a web application where each user got their own database to operate on? I know that the size of the sqlite library is small, but what is the memory footprint after

RE: [sqlite] Memory usage and multiple small databases

2005-07-14 Thread Fred Williams
, as long as you promise not to start whining about multi user issues. :-) FOOTPRINT, FOOTPRINT, FOOTPRINT... Fred -Original Message- From: Christian Smith [mailto:[EMAIL PROTECTED] Sent: Thursday, July 14, 2005 9:23 AM To: sqlite-users@sqlite.org; Joey Ekstrom Subject: Re: [sqlite] Memory

Re: [sqlite] Memory usage and multiple small databases

2005-07-14 Thread D. Richard Hipp
On Thu, 2005-07-14 at 08:40 -0600, Joey Ekstrom wrote: On 7/14/05, Christian Smith [EMAIL PROTECTED] wrote: Each database would have it's own page cache, which will be the biggest memory user. So, with 1K pages, and up to 2000 pages by default, your cache footprint would be: 1024 * 2000 *

[sqlite] Memory usage and multiple small databases

2005-07-13 Thread Joey Ekstrom
Would sqlite perform well in a web application where each user got their own database to operate on? I know that the size of the sqlite library is small, but what is the memory footprint after opening the database? Would it be prohibitive to having a 500+ relatively small database open

[sqlite] Memory usage for queries containing a GROUP BY clause

2005-03-24 Thread Thomas Briggs
Is it possible to limit the amount of memory SQLite uses while processing an aggregate query? I have a 1GB database containing a single table. Simple queries against this table (SELECT COUNT(*), etc.) run without using more than a few MBs of memory; the amount used seems to correspond

Re: [sqlite] Memory usage for queries containing a GROUP BY clause

2005-03-24 Thread D. Richard Hipp
On Thu, 2005-03-24 at 10:09 -0500, Thomas Briggs wrote: I have a 1GB database containing a single table. Simple queries against this table (SELECT COUNT(*), etc.) run without using more than a few MBs of memory; the amount used seems to correspond directly with the size of the page cache,

RE: [sqlite] Memory usage for queries containing a GROUP BY clause

2005-03-24 Thread Thomas Briggs
am I oversimplifying this? :) Thanks -Tom -Original Message- From: D. Richard Hipp [mailto:[EMAIL PROTECTED] Sent: Thursday, March 24, 2005 10:32 AM To: sqlite-users@sqlite.org Subject: Re: [sqlite] Memory usage for queries containing a GROUP BY clause On Thu, 2005-03-24

RE: [sqlite] Memory usage for queries containing a GROUP BY clause

2005-03-24 Thread D. Richard Hipp
On Thu, 2005-03-24 at 10:57 -0500, Thomas Briggs wrote: After posting my question, I found the discussion of how aggregate operations are performed in the VDBE Tutorial; that implies that memory usage will correspond with the number of unique keys encountered by the query, but I appreciate

RE: [sqlite] Memory usage for queries containing a GROUP BY clause

2005-03-24 Thread Thomas Briggs
: Thursday, March 24, 2005 11:19 AM To: sqlite-users@sqlite.org Subject: RE: [sqlite] Memory usage for queries containing a GROUP BY clause On Thu, 2005-03-24 at 10:57 -0500, Thomas Briggs wrote: After posting my question, I found the discussion of how aggregate operations are performed

RE: [sqlite] Memory usage for queries containing a GROUP BY clause

2005-03-24 Thread Thomas Briggs
To: sqlite-users@sqlite.org Subject: RE: [sqlite] Memory usage for queries containing a GROUP BY clause On Thu, 2005-03-24 at 13:59 -0500, Thomas Briggs wrote: I feel like I'm missing something, but that didn't seem to help. I can see in the code why it should be behaving differently

  1   2   >