David, It might be stacksize related, you could try to to increase that to see if that helps. If you use bash you can use the following procedure to increase the stack size:
$ ulimit -Hs <some number> $ ulimit -s <some number> This increases the stacksize limit to the hard limit for the stack size. This wouldn't be real solution for the problem, but would help pinpoint the problem. Ronald On Friday, August 31, 2007, at 11:20AM, "David Worrall" <[EMAIL PROTECTED]> wrote: >Thanks for your feedback Ronald. >The file is quite large: 4-5 GB >pytables buffers the data and controlling the buffer size >occasionally stops or slows the rot. >The code has been tested under Linux without incident. >[EMAIL PROTECTED], one of the developers of pytables wrote: > >> A Tuesday 21 August 2007, David Worrall escrigué: >> >>> I lowered NODE_MAX_SLOTS from 256 to128 and that slowed the leak - >>> enough to get some sort of DB happening. >>> It eventually seg faulted, however. >>> I've noticed that sometimes the seg fault causes the (non-python) >>> heap to become corrupted, requiring a HW reboot before pytables >>> becomes useful again. >>> Who knows what memory it may be writing over! >>> >> >> I don't think there is a leak in PyTables itself; it is just that it >> takes a lot of memory to work with many nodes simultaneously. Perhaps >> reducing the NODE_MAX_SLOTS (to 64, for example) could help a bit >> more. >> Perhaps it would be a good idea to reduce the memory requirements of >> PyTables nodes (for example, avoid the creation of buffers when they >> are not needed), but this should take quite a few of thought. >> >> Indeed it seems that there is a problem with the MacOSX platform, but >> the fact that this is not reproducible on Linux is unfortunate. >... >> I'm thinking that perhaps you can try with newer versions of Python >> (2.5.1) and HDF5 (the 1.8.0 beta3 is out, and I know that it wears a >> completely revamped cache system, so maybe it is worth a try). >> > >Anyway I'll try with gdb as you suggest and report back. I need to >put it on my task stack so it'll take some time. > >thanks again, > >David >On 30/08/2007, at 3:48 PM, Ronald Oussoren wrote: > >> >> On 30 Aug, 2007, at 1:56, David Worrall wrote: >> >>> Hi all, >>> >>> I've been using PyTables (www.pytables.org) with python2.4 on intel >>> Mac OSX 10.4.10 >>> and I'm running into a seg. fault when generating a large hdf5 file. >>> >>> Almost certainly something to do with relationship between OS and >>> python. >>> Has anyone had a similar (memory leak?) experience? >> >> How large a file? It might be a memory management bug in pytables >> as well. >> >> If you have the developer tools (Xcode & friends) installed you can >> find out where the crash occurs using gdb. >> >> That is, 'gdb python', then on the prompt for gdb: 'r >> myscript.py' (adding arguments if needed). When the crash occurs >> you can use 'where 20' to see the topmost 20 frames on the C stack, >> or just 'where' to see the entire stack. >> >> Ronald >>> >>> >>> >>> >>> _________________________________________________ >>> experimental polymedia: www.avatar.com.au >>> Sonic Communications Research Group, >>> University of Canberra: www.canberra.edu.au/vc-forum/scrg >>> vip=Verbal Interactivity Project >>> >>> >>> >>> _______________________________________________ >>> Pythonmac-SIG maillist - Pythonmac-SIG@python.org >>> http://mail.python.org/mailman/listinfo/pythonmac-sig >> > >_________________________________________________ >experimental polymedia: www.avatar.com.au >Sonic Communications Research Group, >University of Canberra: www.canberra.edu.au/vc-forum/scrg >vip=Verbal Interactivity Project > > > >_______________________________________________ >Pythonmac-SIG maillist - Pythonmac-SIG@python.org >http://mail.python.org/mailman/listinfo/pythonmac-sig > > _______________________________________________ Pythonmac-SIG maillist - Pythonmac-SIG@python.org http://mail.python.org/mailman/listinfo/pythonmac-sig
