Hi Sorin, [I'm CC'ying this to the PyTables list so that other people can benefit from this discussion.]
I think you have found a memory leak in the numexpr subpackage. Fortunately this has already been addressed. See: http://www.pytables.org/trac/ticket/203 Please tell me if that cures your problem, Francesc A Monday 02 February 2009, escriguéreu: > Hi, > > > > My name is Sorin Pascu and I work as a developer for a software > company in Romania. > > > > I was wondering if you could spare a couple of minutes to look over a > piece of code involving PyTables that is generating a memory buildup > in my script. > > > > ##initialization > > h5file = > tables.openFile('c:\\Python25\\Scripts\\myScripts\\hdfFile.h5') #open > HDF DB > > tableGroup = h5file.root.MyGroup > > table = h5file.root. MyGroup._f_getChild('Table1') > > ## > > > > Def myFunc(#list of params): > > query = "(t2_date == %s) & (((t3_time / 100) + %s) < %s) & (((t3_time > / 100) + %s) >= %s )" % (currDate, timeAdj, currTime, timeAdj, > lastTime) > > > > ## call to objgraph > > for row in table.where(query,None,lastUsedIndexInH5 > ,lastUsedIndexInH5 + 10000): > > ## some processing on the values from the table, this does > not generate any memory increase > > lastRowIndex = row.nrow > > ##call to objgraph > > > > return lastRowIndex > > > > I am using objgraph library for debug purposes and this is its output > of live object count before and after the above function: > > -before > > tuple 7551 > > function 4853 > > dict 2552 > > wrapper_descriptor 1210 > > builtin_function_or_method 1173 > > instance 805 > > instancemethod 766 > > list 613 > > method_descriptor 578 > > weakref 542 > > > > -after > > tuple 7557 > > function 4853 > > dict 2557 > > wrapper_descriptor 1210 > > builtin_function_or_method 1173 > > instance 805 > > instancemethod 766 > > list 617 > > method_descriptor 578 > > weakref 542 > > > > > > My problem is that the above function is called about 900,000 times > for a medium sized table and builds up to a couple of gigs of memory > even though I call gc.collect() after each query. > > > > I am using Python 2.5, the latest stable release of PyTables and HDF5 > table format on a WinXp machine. > > > > I studied the PyTables manual and online documentation but I found no > reference to such an issue. > > > > Am I misusing the query mechanism or am I configuring something > incorrectly? > > > > Thank you for your time. > > > > Best regards, > > Sorin Pascu -- Francesc Alted ------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
