El dl 26 de 03 del 2007 a les 17:54 +0200, en/na Francesc Altet va
escriure:
> El dv 23 de 03 del 2007 a les 14:51 -0700, en/na Stefan Kuzminski va
> escriure:
> > Hi,
> >
> > In the code below, as it goes through the 100,000 loop appending rows, the
> > memory being used ( as measured by top on a linux box ) steadily increases.
> > How can I prevent this from happening? ( calling flush does not seem to
> > help ) This would seem to present a limit on how much data I can append to
> > a file? The behavior is with v1.4 and with the 2.0 beta.
> >
> > thanks!
> > Stefan Kuzminski
> >
> >
> > from tables import *
> > from time import sleep
> >
> > num_cols = 1500
> > fp = openFile( "foo", 'w' )
> >
> >
> > Float64Col( shape=(num_cols,))
> > table = fp.createTable( fp.root, 'title',
> > { 'var1' : Float64Col( shape=(num_cols,)) }, '' )
> > print 'appending...'
> > row = range( num_cols )
> > for i in range( 100000 ):
> >
> > table.append( [[row]])
> > if i % 1000 == 0:
> > print i
> > table.flush()
> > print 'done appending'
> > sleep( 20 )
>
> Stefan,
>
> After struggling a bit with this, the problem seems to be the use of the
> range() function. As you know, this function returns a list of elements,
> and that takes memory. Curiously enough, this memory is not consumed in
> one shot, but it grows steadily with time (sorry, but I don't know the
> interpreter of python enough for guessing the reason of this).
>
> Anyway, if you replace range() by xrange() everyting will go fine (i.e.
> the 'leak' will misteriously disappear).
Well, it is not that easy. I was fooled by the strange behaviour of
range() (in terms of memory consumption), but there exists a real
problem that (I think) I've traced down to the HDF5 library (H5Dwrite
function in particular). I'm going to study this more carefully and, if
appropriate, report the problem to the HDF5 maintainers.
I'll come back with more info about this issue.
--
Francesc Altet | Be careful about using the following code --
Carabos Coop. V. | I've only proven that it works,
www.carabos.com | I haven't tested it. -- Donald Knuth
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
