Hi Francesc,
You are right, I am using lots of little files (I receive them over a UDP link)
and merge them into a few large ones. All small files contain a single table
(with same column structure), and it seems that some or all of these column
definitions are not freed up when the small files are closed. I did some
profiling and it seems that no python object keeps references any more, so it
probably has something to do with the C interface. This problem seems to occur
only in multithreaded or multiprocess (with multiprocessing lib) situations. In
a single thread things are fine.
I get by now by merging the small files in a separate process and killing and
restarting it periodically, but there seems to be a bug here.
Would you care for some code that exposes it?
Leon
----------------------------------
A Thursday 14 April 2011 14:14:09 Leon Evers escrigu?:
> Hi all,
>
> I have just used pytables for a project the first time, with great
> success, except that I am running into a problem which I suspect may
> be a bug...
>
> I am reading in a large number of HDF5 files, in multiple threads,
> and I notice that my application's process is consuming more and
> more memory. Particularly, I see it is keeping hold of large amounts
> of tuples and dicts that store table definitions.
>
> This seems like a bug to me, is this a known issue? And if so, is
> there anything I can do about it?
How many tables do you have on each file? PyTables (and HDF5 in
general) is designed to deal mainly with few, large tables, not many,
small tables. Which is your situation?
--
Francesc Alted
------------------------------------------------------------------------------
Benefiting from Server Virtualization: Beyond Initial Workload
Consolidation -- Increasing the use of server virtualization is a top
priority.Virtualization can reduce costs, simplify management, and improve
application availability and disaster protection. Learn more about boosting
the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
