On Tue, Oct 21, 2008 at 05:56:45AM -0700, Istvan Albert wrote:
-> On Oct 20, 10:39?pm, Christopher Lee <[EMAIL PROTECTED]> wrote:
->
-> > OK. ?I now understand the problem. ?The bsddb module btree index is ?
-> > screwing us over: when you simply ask for an iterator, it apparently ?
-> > loads the entire index into memory. ?
->
-> Is this really true? The bsddb module is very heavily used by lots of
-> people to store dictionaries that do not fit into memory. I have
-> never heard people mentioning this before.
It's a BsdDBShelf problem, not a bsddb issue -- see attached script.
The 'iter' call seems to be what's loading the index.
cheers,
--titus
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"pygr-dev" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/pygr-dev?hl=en
-~----------~----~----~----~------~----~------~--~---
from shelve import BsdDbShelf
import time
import bsddb
filename = 'test'
def create():
_db = bsddb.btopen(filename, 'n')
db = BsdDbShelf(_db)
for key in range(10**6):
key = str(key)
db[key] = key
db.close()
def read():
_db = bsddb.btopen(filename, 'r')
db = BsdDbShelf(_db)
start = time.time()
# iterating on the database
print db.first(), db.next(), db.next(), db.last()
end = time.time()
print 'Elapsed: %s' % (end-start)
start = time.time()
# with a custom iterator
it = iter(db)
print it.next(), it.next(), it.next()
end = time.time()
print 'Elapsed: %s' % (end-start)
db.close()
#create()
read()
