Thanks for your feedback Ronald. The file is quite large: 4-5 GB pytables buffers the data and controlling the buffer size occasionally stops or slows the rot. The code has been tested under Linux without incident. [EMAIL PROTECTED], one of the developers of pytables wrote:
> A Tuesday 21 August 2007, David Worrall escrigué: > >> I lowered NODE_MAX_SLOTS from 256 to128 and that slowed the leak - >> enough to get some sort of DB happening. >> It eventually seg faulted, however. >> I've noticed that sometimes the seg fault causes the (non-python) >> heap to become corrupted, requiring a HW reboot before pytables >> becomes useful again. >> Who knows what memory it may be writing over! >> > > I don't think there is a leak in PyTables itself; it is just that it > takes a lot of memory to work with many nodes simultaneously. Perhaps > reducing the NODE_MAX_SLOTS (to 64, for example) could help a bit > more. > Perhaps it would be a good idea to reduce the memory requirements of > PyTables nodes (for example, avoid the creation of buffers when they > are not needed), but this should take quite a few of thought. > > Indeed it seems that there is a problem with the MacOSX platform, but > the fact that this is not reproducible on Linux is unfortunate. ... > I'm thinking that perhaps you can try with newer versions of Python > (2.5.1) and HDF5 (the 1.8.0 beta3 is out, and I know that it wears a > completely revamped cache system, so maybe it is worth a try). > Anyway I'll try with gdb as you suggest and report back. I need to put it on my task stack so it'll take some time. thanks again, David On 30/08/2007, at 3:48 PM, Ronald Oussoren wrote: > > On 30 Aug, 2007, at 1:56, David Worrall wrote: > >> Hi all, >> >> I've been using PyTables (www.pytables.org) with python2.4 on intel >> Mac OSX 10.4.10 >> and I'm running into a seg. fault when generating a large hdf5 file. >> >> Almost certainly something to do with relationship between OS and >> python. >> Has anyone had a similar (memory leak?) experience? > > How large a file? It might be a memory management bug in pytables > as well. > > If you have the developer tools (Xcode & friends) installed you can > find out where the crash occurs using gdb. > > That is, 'gdb python', then on the prompt for gdb: 'r > myscript.py' (adding arguments if needed). When the crash occurs > you can use 'where 20' to see the topmost 20 frames on the C stack, > or just 'where' to see the entire stack. > > Ronald >> >> >> >> >> _________________________________________________ >> experimental polymedia: www.avatar.com.au >> Sonic Communications Research Group, >> University of Canberra: www.canberra.edu.au/vc-forum/scrg >> vip=Verbal Interactivity Project >> >> >> >> _______________________________________________ >> Pythonmac-SIG maillist - Pythonmac-SIG@python.org >> http://mail.python.org/mailman/listinfo/pythonmac-sig > _________________________________________________ experimental polymedia: www.avatar.com.au Sonic Communications Research Group, University of Canberra: www.canberra.edu.au/vc-forum/scrg vip=Verbal Interactivity Project _______________________________________________ Pythonmac-SIG maillist - Pythonmac-SIG@python.org http://mail.python.org/mailman/listinfo/pythonmac-sig
