Hi list,
I'm not sure what this is... I've written a minimal script which shows
the following problem: fill up a table with 10 million rows, which costs
almost no memory. Then, do the following query:
r = data.root.events.col('event_id')
which brings up memory usage from 14 Mb to 99 Mb. Do it again, which
brings memory usage further up by tens of Mb's, which are freed after
the query finishes.
Instead, try the following query:
r = [x['event_id'] for x in data.root.events]
which brings memory usage from 14 Mb to 296 Mb. Do it again, which
brings memory usage up to 528 Mb.
I tried playing with the chunksize, but doesn't help much...
sys.getsizeof shows something funny as well. The second query results
takes up 40764028 bytes, whereas the first query result gives 40 bytes.
Huh?
Del-ing objects and imports doesn't clean up memory...
I tried with pytables 2.1.1 and 2.1.2 and python 2.6.
Can anyone understand this? Thanks!
Regards,
David
Script:
import tables
class Event(tables.IsDescription):
event_id = tables.UInt64Col()
ext_timestamp = tables.UInt64Col(dflt=9999)
other_value = tables.UInt64Col(dflt=9999)
def create_tables():
data = tables.openFile('test.h5', 'w', 'PyTables Test')
data.createTable('/', 'events', Event, 'Test Events',
expectedrows=50000)
# expectedrows=10000000)
# chunkshape=(10000,))
table = data.root.events
tablerow = table.row
for i in xrange(10000000):
tablerow['event_id'] = i
tablerow.append()
table.flush()
data.close()
def test_query():
data = tables.openFile('test.h5', 'r')
#r = [x['event_id'] for x in data.root.events]
r = data.root.events.col('event_id')
data.close()
return r
if __name__ == '__main__':
#create_tables()
x = test_query()
------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
