On Mon, Jan 03, 2011 at 05:48:50PM +0100, Francesc Alted wrote: > Array objects are non-chunked. In order to use compression, you need > to use a CArray: > http://www.pytables.org/docs/manual/ch04.html#CArrayClassDescr
Here's my chance to ask a question that I've had for a week or two: how are compressed arrays actually implemented? I gather that the array contents are compressed (using the chosen compressor) into the HDF5 data file, but are CArrays actually compressed in memory? My data set is larger than physical memory, but has a lot of repeated values that lead to ~90% compression. Thus, it should be possible to keep the whole array compressed in memory and decompress chunks of the array as necessary. Is this what PyTables does? Cheers, Ben ------------------------------------------------------------------------------ Learn how Oracle Real Application Clusters (RAC) One Node allows customers to consolidate database storage, standardize their database environment, and, should the need arise, upgrade to a full multi-node Oracle RAC database without downtime or disruption http://p.sf.net/sfu/oracle-sfdevnl _______________________________________________ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
