On Mon, 25 Aug 2008 12:26:18 -0700, you wrote:
>Interesting, I just tried that in my test application and Dennis's and I
>get access violations during the vacuum command execution when trying to
>resize the pages from 1k to 4k with my database or Dennis's test
>database.
I just used the command
Brown, Daniel wrote:
> Interesting, I just tried that in my test application and Dennis's and I
> get access violations during the vacuum command execution when trying to
> resize the pages from 1k to 4k with my database or Dennis's test
> database.
>
Daniel,
I have found that sqlite works
] On Behalf Of Kees Nuyt
Sent: Saturday, August 23, 2008 8:51 AM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Reducing SQLite Memory footprint(!)
On Thu, 21 Aug 2008 16:29:10 -0700, you wrote:
>How do I rebuild a database file for another page size
>or did the pra
On Thu, 21 Aug 2008 16:29:10 -0700, you wrote:
>How do I rebuild a database file for another page size
>or did the pragma do that already?
Use PRAGMA page_size={your_new_pagesize} immediately before
a vacuum. It will change the page size of the vacuumed
database. See:
Becker
> Sent: Friday, August 22, 2008 1:31 PM
> To: General Discussion of SQLite Database
> Subject: Re: [sqlite] Reducing SQLite Memory footprint(!)
>
> At one point I wrote some vbscript to generate a table declaration and
> insert statements for a csv. I might be able to dig it up
That script would be great :)
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Jeffrey Becker
Sent: Friday, August 22, 2008 1:31 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Reducing SQLite Memory footprint(!)
At one point I wrote
---Original Message-
> From: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED] On Behalf Of Dennis Cote
> Sent: Friday, August 22, 2008 7:10 AM
> To: General Discussion of SQLite Database
> Subject: Re: [sqlite] Reducing SQLite Memory footprint(!)
>
> Brown, Daniel wrote:
>>
uruOMeter where name="Ron Wilson";
2
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Brown, Daniel
Sent: Friday, August 22, 2008 1:59 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Reducing SQLite Memory footprint(!)
I jus
Brown, Daniel wrote:
> I just ran Dennis's test databases through the test application and
> we're getting similar results:
> 1k Pages (17.4 MB) used 18102 KB High 20416 KB
> 4k Pages (12.2 MB) used 18102 KB, High 26416 KB (not sure why
> the high is higher?)
> My test database however
-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Dennis Cote
Sent: Friday, August 22, 2008 7:10 AM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Reducing SQLite Memory footprint(!)
Brown, Daniel wrote:
> 2. And the other thing to try would be if anyone has a fai
Brown, Daniel wrote:
> Ok so after reading your feedback I tried:
> 1. "PRAGMA cache_size =10" no change in memory usage.
> 2. "PRAGMA page_size = 4096" no change in memory usage.
>
> I'm doing both those queries (in C++) after the 'sqlite3_open(
> ":memory:", _pDataBase );' in my test but before
Brown, Daniel wrote:
> 2. And the other thing to try would be if anyone has a fairly meaty test
> database they don't mind sharing that I could fling at my test
> application to try and rule out the data?
>
Daniel,
I can send you copies of the databases I am using for my testing, both
the
a 7645 KB database file turn into 22735
KB with a high of 25138 KB at runtime.
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of D. Richard Hipp
Sent: Thursday, August 21, 2008 5:24 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Reducing
On Aug 21, 2008, at 7:33 PM, Brown, Daniel wrote:
> if anyone has a fairly meaty test
> database they don't mind sharing that I could fling at my test
> application...
A project repository with "fossil" is a (meaty) SQLite database. You
could (for example) go clone one of the various fossil
mind sharing that I could fling at my test
application to try and rule out the data?
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Dennis Cote
Sent: Thursday, August 21, 2008 4:23 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Reducing
t for making
a temporary copy.
How do I rebuild a database file for another page size or did the pragma
do that already?
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Nicolas Williams
Sent: Thursday, August 21, 2008 3:13 PM
To: General Discussion of SQLite
Nicolas Williams wrote:
>
> I thought the DB was 9MB; forgive me for wasting your time then. If
> it's 17.4MB then the memory usage seems a lot more reasonable.
Daniel, the OP's, database is 9 MB. I don't have his database file, but
I do have his test code. I used a database of my own that is
On Thu, Aug 21, 2008 at 04:02:32PM -0600, Dennis Cote wrote:
> I built a copy of my test database using a 4096 byte page size and it
> reduced the database file size from 17.5 MB to 12.2 MB. When I repeat
> the tests using this database file I get the same 18102 KB of memory
> used, but a
Nicolas Williams wrote:
>
> I wonder too, what does the page cache do when doing full table scans?
> If the cache has an LRU/LFU page eviction algorithm then full table
> scans should not be a big deal. Ideally it should not allow pages read
> during a full table scan to push out other pages,
Jeffrey Becker wrote:
> Just out of curiosity what happens if you call
> "PRAGMA page_size=4096"
> before running the import?
>
As I expected, it has no effect. The page size pragma only effects the
:memory: database he is copying into. The page size of the database file
was set when it was
On Thu, Aug 21, 2008 at 10:32:23AM -0400, Jeffrey Becker wrote:
> Just out of curiosity what happens if you call
> "PRAGMA page_size=4096"
> before running the import?
I wonder too, what does the page cache do when doing full table scans?
If the cache has an LRU/LFU page eviction algorithm then
Brown, Daniel wrote:
>
> I just tried with rebuilt data from SQLite Analyzer in SQLite 3.6.1, I'm
> still seeing a memory usage that is roughly three times the size of the
> source database file, looking at your changes to my test there doesn't
> seem to be any fixes that would resolve that.
>
Brown, Daniel wrote:
Perhaps, when is the next release due? I'd be interested to see the
differences, if an upgrade reduces memory overhead that significantly it
would be most excellent :)
3.6.2 is (tentatively) due this Monday :)
___
sqlite-users
of SQLite Database
Subject: Re: [sqlite] Reducing SQLite Memory footprint(!)
Brown, Daniel wrote:
> I just upgraded to the latest version (3.6.1) of the pre-processed C
> source code from the website, running the test again gives me similar
> results of 22.2 MB used and 24.55 MB high water from
Brown, Daniel wrote:
> I just upgraded to the latest version (3.6.1) of the pre-processed C
> source code from the website, running the test again gives me similar
> results of 22.2 MB used and 24.55 MB high water from the same 9 MB file.
> Is there any way it could be the file that is causing the
, August 20, 2008 2:43 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Reducing SQLite Memory footprint(!)
On Aug 20, 2008, at 5:30 PM, Brown, Daniel wrote:
> I just upgraded to the latest version (3.6.1) of the pre-processed C
> source code from the website, running the test
On Aug 20, 2008, at 5:30 PM, Brown, Daniel wrote:
> I just upgraded to the latest version (3.6.1) of the pre-processed C
> source code from the website, running the test again gives me similar
> results of 22.2 MB used and 24.55 MB high water from the same 9 MB
> file.
> Is there any way it
try rebuilding it next.
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Brown, Daniel
Sent: Wednesday, August 20, 2008 1:50 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Reducing SQLite Memory footprint(!)
That is interesting, all
.
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of D. Richard Hipp
Sent: Wednesday, August 20, 2008 12:14 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Reducing SQLite Memory footprint(!)
On Aug 20, 2008, at 2:56 PM, Brown, Daniel wrote
On Aug 20, 2008, at 2:56 PM, Brown, Daniel wrote:
> Thank you, I imported our data from the source Excel file (.xls) using
> the third party SQLite Analyzer application
> (http://www.kraslabs.com/sqlite_analyzer.php) if that makes any
> difference?
>
> The size of the SQLite database on disc is
. Richard Hipp
Sent: Wednesday, August 20, 2008 11:44 AM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Reducing SQLite Memory footprint(!)
On Aug 20, 2008, at 2:22 PM, Brown, Daniel wrote:
> sqlite3_memory_highwater() ~ 25673060
> sqlite3_memory_used() ~ 23222709
>
On Aug 20, 2008, at 2:22 PM, Brown, Daniel wrote:
> sqlite3_memory_highwater() ~ 25673060
> sqlite3_memory_used() ~ 23222709
>
OK. I'll have a look
D. Richard Hipp
[EMAIL PROTECTED]
___
sqlite-users mailing list
sqlite-users@sqlite.org
More over, the memory usage reported by process explorer is subject to
the vagrancies of the windows memory allocator. Generally the memory
usage you see reported in tools is quite a bit higher than the actual
memory usage of your application. First off windows manages memory in
fixed size
(memHigh/1024));
sqlite3_close( pDataBase );
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of D. Richard Hipp
Sent: Wednesday, August 20, 2008 11:00 AM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Reducing SQLite Memory footp
On Aug 20, 2008, at 1:53 PM, Brown, Daniel wrote:
> Looking in process explorer on XP after the disc database detached
> should a memory size change of 28 MB of RAM in the test application, I
> assumed this was the size of the database in memory.
That would be the peak memory usage by the
Sent: Wednesday, August 20, 2008 10:44 AM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Reducing SQLite Memory footprint(!)
On Aug 19, 2008, at 6:24 PM, Brown, Daniel wrote:
> SQLite is very memory
> intensive compared to our current solution (although SQLite is
On Aug 19, 2008, at 6:24 PM, Brown, Daniel wrote:
> SQLite is very memory
> intensive compared to our current solution (although SQLite is faster
> and more feature rich), e.g. 9MB for our current solution versus 28 MB
> for SQLite with the same source data.
Where did you get the 28MB figure?
%
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of D. Richard Hipp
Sent: Tuesday, August 19, 2008 3:43 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Reducing SQLite Memory
On Aug 19, 2008, at 6:24 PM, Brown, Daniel wrote:
> Good afternoon list,
>
> Are there any known techniques for reducing SQLite's memory footprint?
Construct your database into a file (say "test.db"). Then run the
sqlite3_analyzer utility (available for download on the SQLite
website) over
39 matches
Mail list logo
