Nemanja Čorlija wrote:
On Jan 17, 2008 12:00 PM, Roger Binns <[EMAIL PROTECTED]> wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Nemanja Čorlija wrote:
My problem is that I am running out of memory when trying to load
1.6GB sqlite3 db from disk to :memmory: db on a computer with 4GB of
RAM.
Your RAM isn't the problem. You are running out of address space. For
a 32 bit process, the address space available is 4GB. The operating
system typically reserves 1 or 2GB of that for its own uses (eg Linux
typically reserves 1GB, Windows reserves 2GB, I don't know what OSX does).
That means that all the shared libraries, your stack, each stack's
thread if the process is multithreaded, memory mapped files as well as
dynamically allocated memory all have to fit within that remaining
space. In your case you ran out :-)
Here is an example of how to look at what is in a process' address space:
http://blog.mozilla.com/dolske/2007/10/16/os-x-and-virtual-bloat/
Thanks for clearing that up. That blog post was really useful in
helping me understand this issue better.
If you must take this :memory: approach then you will have to switch to
using a 64 bit process which will also require a 64 bit kernel and
processor. (Note that the amount of physical RAM only affects performance).
However wanting the entire database in memory implies you are having
performance issues. What were those?
I have a db with 27M rows and then I have 5 other dbs with exact same
schema that together have 100+M rows and for each of those 100M rows I
need to check if its primary key column has a match in 27M db and if
it does update that row to set a flag. There's of course a lot of ways
to go about this but I just assumed that it would be fastest if I can
perform all those lookups against memory db and already wrote a python
script to do all the work and then hit this problem. I first thought
that it is a python/APSW issue but then I got the exact same error
with sqlite3 command line program.
I'll try to take advantage of locality of reference on smaller
presorted sets of data and see if that can get the job done quick
enough.
Thanks.
You are unlikely to find a memory DB running faster than a disk based
one (because of caching).
-----------------------------------------------------------------------------
To unsubscribe, send email to [EMAIL PROTECTED]
-----------------------------------------------------------------------------
