Re: [Pytables-users] Full memory load of a hdf5-tree

2011-03-24 Thread daniele
I really don't know if this would change something, but... Would the performance be improved if you read/write a file which is stored on a RAM filesystem? (like tmpfs, http://en.wikipedia.org/wiki/Tmpfs) I imagine that anyway the data should be moved from the tmpfs to the RAM where the program effe

Re: [Pytables-users] Full memory load of a hdf5-tree

2011-03-24 Thread brunetto
2011/3/23 Francesc Alted : > 2011/3/23 brunetto >> >> For now the leaves max size is 300 (that is: leaves are Arrays >> containing 300 or less integers), the number of nodes dipends on the >> dimension of the dataset. Now I'm using dataset between 1000 and 10^8 >> elements. The tree is built split

Re: [Pytables-users] Full memory load of a hdf5-tree

2011-03-23 Thread Francesc Alted
2011/3/23 brunetto > For now the leaves max size is 300 (that is: leaves are Arrays > containing 300 or less integers), the number of nodes dipends on the > dimension of the dataset. Now I'm using dataset between 1000 and 10^8 > elements. The tree is built splitting every node (=PyTables Group) i

Re: [Pytables-users] Full memory load of a hdf5-tree

2011-03-23 Thread brunetto
>> I'm using pytables to buil a persistent binary tree containing from >> 10^3 to 10^8 (may be in future 10^10) integer numbers. The structure >> reflects a previus binary tree entirely built in memory. >> For now traversing the tree in the hdf5 file it's 10 time slower than >> traversing the tree

Re: [Pytables-users] Full memory load of a hdf5-tree

2011-03-23 Thread Francesc Alted
2011/3/23 brunetto > Hi All!! > > I'm quite new to pytables and this mailing list so I'm sorry if my > question is a bit stupid!:) > > I'm using pytables to buil a persistent binary tree containing from > 10^3 to 10^8 (may be in future 10^10) integer numbers. The structure > reflects a previus bi