I really don't know if this would change something, but... Would the
performance be improved if you read/write a file which is stored on a RAM
filesystem? (like tmpfs, http://en.wikipedia.org/wiki/Tmpfs)
I imagine that anyway the data should be moved from the tmpfs to the RAM
where the program effectively operate, but you would have to r/w from ram
instead of disk.
2011/3/24 brunetto <brunetto.zi...@gmail.com>
> 2011/3/23 Francesc Alted <fal...@pytables.org>:
> > 2011/3/23 brunetto <brunetto.zi...@gmail.com>
> >>
> >> For now the leaves max size is 300 (that is: leaves are Arrays
> >> containing 300 or less integers), the number of nodes dipends on the
> >> dimension of the dataset. Now I'm using dataset between 1000 and 10^8
> >> elements. The tree is built splitting every node (=PyTables Group) in
> >> two children until the number of elements is <=300.
> >
> > I see. May I ask why you use the 300 magic figure? >From a PyTables
> > performance point of view, perhaps it might make more sense to have
> larger
> > Arrays and lesser nodes.
>
> Because my tree comes from a previus in-memory implementation... At
> that time I did some tests and I found that on average the max speed
> correspond to leaves of <= 300 elements.
>
> I think this is because of what I do when I open a leaf, i.e.
> calculate distances and sorting arrays. Bigger leaves lead to longer
> operation on them!
>
> Also I can't use the walkNode method because I have a selective
> travers that automatically exclude all the nodes (and their subnodes)
> that are more distant than a certain quantity (spatially distant...
> every node has spatial coordinates).
>
> brunetto
>
>
> ------------------------------------------------------------------------------
> Enable your software for Intel(R) Active Management Technology to meet the
> growing manageability and security demands of your customers. Businesses
> are taking advantage of Intel(R) vPro (TM) technology - will your software
> be a part of the solution? Download the Intel(R) Manageability Checker
> today! http://p.sf.net/sfu/intel-dev2devmar
> _______________________________________________
> Pytables-users mailing list
> Pytables-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
------------------------------------------------------------------------------
Enable your software for Intel(R) Active Management Technology to meet the
growing manageability and security demands of your customers. Businesses
are taking advantage of Intel(R) vPro (TM) technology - will your software
be a part of the solution? Download the Intel(R) Manageability Checker
today! http://p.sf.net/sfu/intel-dev2devmar
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
