A Friday 15 April 2011 11:09:12 Leon Evers escrigué: > Hi Francesc, > > You are right, I am using lots of little files (I receive them over a > UDP link) and merge them into a few large ones. All small files > contain a single table (with same column structure), and it seems > that some or all of these column definitions are not freed up when > the small files are closed. I did some profiling and it seems that > no python object keeps references any more, so it probably has > something to do with the C interface. This problem seems to occur > only in multithreaded or multiprocess (with multiprocessing lib) > situations. In a single thread things are fine. > > I get by now by merging the small files in a separate process and > killing and restarting it periodically, but there seems to be a bug > here. Would you care for some code that exposes it?
Well, I'm not sure it is a bug, but rather a mis-optimization in PyTables/HDF5. But sure, please send the code so that I can see what's going on here. Cheers, -- Francesc Alted ------------------------------------------------------------------------------ Benefiting from Server Virtualization: Beyond Initial Workload Consolidation -- Increasing the use of server virtualization is a top priority.Virtualization can reduce costs, simplify management, and improve application availability and disaster protection. Learn more about boosting the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev _______________________________________________ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
