I am not sure if this is a numarray problem or a pytables
problem, but this is the simplest program I have been able to create
that illustrates the problem. The attached program causes my machine
to use all available memory (1G ram + 1G swap), it is eventually
killed by the kernel. Is there some other call I should make to
pytables to free the memory used by a file? My application actually
processes many large files in sequence, and after just a few the
machine runs out of memory.
The problem does not seem to happen for a program that creates a
large number of files, the files I am working on here are actually
converted from a different format, the conversion program runs fine.
Russel
import tables
import numarray.random_array as ra
from numarray import *
import sys
filename="test_%02d.h5"
nrows=2000
maxelements=15000
minelements=10000
times=20
def create_file(n=0):
name=filename%(n)
print "creating %s"%name
hf=tables.openFile(name,'w')
vl1 = hf.createVLArray("/", 'vlarray1', tables.Float32Atom(flavor="NumArray"), "test array 1")
for y in xrange(nrows):
rowlen=int(around(ra.random()*(maxelements-minelements)+minelements))
ar1=ra.random(rowlen)
vl1.append(ar1)
hf.close()
def sum_vla(n):
s=0
for r in n:
s+=sum(r)
return s
def read_file(n=0):
name=filename%(n)
print "reading %s"%name
print "summing",
for j in xrange(times):
hf=tables.openFile(name)
n1=hf.getNode('/vlarray1')
print ".",
sys.stdout.flush()
s=sum_vla(n1)
hf.close()
print " "
def test():
create_file()
print "creation done"
read_file()
if __name__=="__main__":
test()
uname -a
Linux omed 2.6.15-ck1 #1 SMP Wed Jan 11 07:22:22 PST 2006 i686 Intel
(R) Pentium(R) 4 CPU 2.40GHz GenuineIntel GNU/Linux
python
Python 2.4.2 (#1, Dec 24 2005, 13:38:58)
[GCC 3.3.6 (Gentoo 3.3.6, ssp-3.3.6-1.0, pie-8.7.8)] on linux2
Type help, copyright, credits or license for more information.
>>> import tables
>>> print tables.__version__
1.2.1
>>> import numarray
>>> print numarray.__version__
1.5.0
>>>
h5cc -showconfig
SUMMARY OF THE HDF5 CONFIGURATION
=================================
HDF5 Version: 1.6.5
Configured on: Tue Jan 24 07:47:44 PST 2006
Configured by: [EMAIL PROTECTED]
Configure mode: production
Host system: i686-pc-linux-gnu
Byte sex: little-endian
Libraries: static, shared
Parallel support: no
Installation point: /usr/local
Compiler: /usr/bin/gcc (gcc-3.3.6)
Compiler switches: -march=i686 -std=c99 -Wno-long-long -O -
fomit-frame-pointer -finline-functions -UH5_DEBUG_API -DNDEBUG -
D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -
D_POSIX_SOURCE -D_BSD_SOURCE
Extra libraries: -L/usr -lpthread -lssl -lcrypto -lz -lm
Archiver: ar
Ranlib: ranlib
Debugged Packages:
API Tracing: no
File addresses: large
Configure Summary
Compiling Options:
Compilation Mode: Production
C Compiler: gcc
CFLAGS: -march=i686 -std=c99 -Wno-long-
long -O -fomit-frame-pointer -finline-functions
CPPFLAGS: -UH5_DEBUG_API -DNDEBUG -
D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -
D_POSIX_SOURCE -D_BSD_SOURCE
LDFLAGS: -L/usr
Debug Mode: None
Shared Libraries: Yes
Static Libraries: Yes
Statically Linked Executables: No
Tracing: No
Optimization Instrumentation: No
Languages:
C++: No
Fortran: No
Features:
dmalloc: No
Function Stack Tracing: Disabled
GASS: No
GPFS: No
HDF5 v1.4 Compatibility: No
hsize_t: Large
I/O filters (external): deflate
I/O filters (internal): shuffle,fletcher32
Linux Large File Support (LFS): Enabled
MPE: No
Parallel HDF5: No
SRB: No
Stream VFD: Enabled
Threadsafety: Enabled
High Level library: Enabled
