Thanks for looking into this. I have more complex code that displays the same
behavior which directly allocates the buffers getting passed to table.append
directly as numpy arrays, so I'm pretty sure that the memory is getting
allocated in pytables or below. If I comment out just the table.append line,
the memory usage is constant.
Thanks again, this is an important issue for us.
S
----- Original Message ----
From: Francesc Altet <[EMAIL PROTECTED]>
To: Stefan Kuzminski <[EMAIL PROTECTED]>
Cc: PyTables user list <pytables-users@lists.sourceforge.net>
Sent: Monday, March 26, 2007 12:17:38 PM
Subject: Re: [Pytables-users] memory usage while appending tables (with sample
code)
El dl 26 de 03 del 2007 a les 17:54 +0200, en/na Francesc Altet va
escriure:
> El dv 23 de 03 del 2007 a les 14:51 -0700, en/na Stefan Kuzminski va
> escriure:
> > Hi,
> >
> > In the code below, as it goes through the 100,000 loop appending rows, the
> > memory being used ( as measured by top on a linux box ) steadily increases.
> > How can I prevent this from happening? ( calling flush does not seem to
> > help ) This would seem to present a limit on how much data I can append to
> > a file? The behavior is with v1.4 and with the 2.0 beta.
> >
> > thanks!
> > Stefan Kuzminski
> >
> >
> > from tables import *
> > from time import sleep
> >
> > num_cols = 1500
> > fp = openFile( "foo", 'w' )
> >
> >
> > Float64Col( shape=(num_cols,))
> > table = fp.createTable( fp.root, 'title',
> > { 'var1' : Float64Col( shape=(num_cols,)) }, '' )
> > print 'appending...'
> > row = range( num_cols )
> > for i in range( 100000 ):
> >
> > table.append( [[row]])
> > if i % 1000 == 0:
> > print i
> > table.flush()
> > print 'done appending'
> > sleep( 20 )
>
> Stefan,
>
> After struggling a bit with this, the problem seems to be the use of the
> range() function. As you know, this function returns a list of elements,
> and that takes memory. Curiously enough, this memory is not consumed in
> one shot, but it grows steadily with time (sorry, but I don't know the
> interpreter of python enough for guessing the reason of this).
>
> Anyway, if you replace range() by xrange() everyting will go fine (i.e.
> the 'leak' will misteriously disappear).
Well, it is not that easy. I was fooled by the strange behaviour of
range() (in terms of memory consumption), but there exists a real
problem that (I think) I've traced down to the HDF5 library (H5Dwrite
function in particular). I'm going to study this more carefully and, if
appropriate, report the problem to the HDF5 maintainers.
I'll come back with more info about this issue.
--
Francesc Altet | Be careful about using the following code --
Carabos Coop. V. | I've only proven that it works,
www.carabos.com | I haven't tested it. -- Donald Knuth
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
____________________________________________________________________________________
The fish are biting.
Get more visitors on your site using Yahoo! Search Marketing.
http://searchmarketing.yahoo.com/arp/sponsoredsearch_v2.php
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
