A Thursday 29 April 2010 09:20:16 Ernesto escrigué: > I'm sorry, I have a last question that it may be close related to what I'm > doing. In practice I need to extract a subset of rows from a tables and > then sort them according to specific criteria. Of course the best thing > should be a pre-sorting of the table before the extraction in order to get > a sorted list. Anyway I could extract my rows and sort them in a next > step. However, I don't know a priori if my extracted list can fit the > available memory. So the question or better my curiosity concerns the > possibility to extract a very big subset of rows also when I don't have > all available memory.
Ah, that clarifies a lot what you want to do. If you don't know a priori if your extracted list can fit in memory, then the only solution that I can think of is you to use the `Table.where()` iterator and extract the info you are interested in for each element that the iterator returns. You can see examples on how to use the `Table.where()` iterator in the chapter 3 of the User's Manual and, for an express guide, also here: http://pytables.org/moin/HowToUse#Selectingvalues [Please note that you can even make use of generators (`()`) instead of comprehension lists (`[]`) so as to avoid loading the results in-memory.] You also have a video explaining the basics of querying tables: http://showmedo.com/videotutorials/video?name=1780010&fromSeriesID=178 But still, you need a way to sort out your big tables first, and Pro is the only solution for doing this that *I* know for Python. There is also memmaped NumPy structured arrays, but there exists the limitation that the table size cannot exceed your available *virtual* memory. PyTables Pro does not have this limitation (the only practical limit is your available disk space). > Thank a lot and sorry if I'm taking your time with my basic (or stupid) > questions, Not at all. It is always a bit difficult to figure out all the parameters that people have when trying to solve their problems, but here you have my opinion. You may want to ask the NumPy list for yourself, though. -- Francesc Alted ------------------------------------------------------------------------------ _______________________________________________ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
