You would be suprised to see how convenient HDF5 can be in small scale data
:) There are cases where one may need to use binary serialization of a few
thousand items, but still needing metadata, indexing and other nice
features provided by HDF5/pyTables.
On Wed, Jun 5, 2013 at 2:29 AM, Tim Burgess <timburg...@mac.com> wrote:
> I was playing around with in-memory HDF5 prior to the 3.0 release. Here's
> an example based on what I was doing.
> I looked over the docs and it does mention that there is an option to
> throw away the 'file' rather than write it to disk.
> Not sure how to do that and can't actually think of a use case where I
> would want to :-)
>
> And be wary, it is H5FD_CORE.
>
>
> On Jun 05, 2013, at 08:38 AM, Anthony Scopatz <scop...@gmail.com> wrote:
>
>
> I think that you want to set parameters.DRIVER to H5DF_CORE [1]. I
> haven't ever used this personally, but it would be great to have an example
> script, if someone wants to write one ;)
>
>
>
> import numpy as np
> import tables
>
> CHUNKY = 30
> CHUNKX = 8640
>
> if __name__ == '__main__':
>
> # create dataset and add global attrs
>
> file_path = 'demofile_chunk%sx%d.h5' % (CHUNKY, CHUNKX)
>
> with tables.open_file(file_path, 'w', title='PyTables HDF5 In-memory
> example', driver='H5FD_CORE') as h5f:
>
> # dummy some data
> lats = np.empty([4320])
> lons = np.empty([8640])
>
> # create some simple arrays
> lat_node = h5f.create_array('/', 'lat', lats, title='latitude')
> lon_node = h5f.create_array('/', 'lon', lons, title='longitude')
>
> # create a 365 x 4320 x 8640 CArray of 32bit float
> shape = (365, 4320, 8640)
> atom = tables.Float32Atom(dflt=np.nan)
>
> # chunk into daily slices and then further chunk days
> sst_node = h5f.create_carray(h5f.root, 'sst', atom, shape,
> chunkshape=(1, CHUNKY, CHUNKX))
>
> # dummy up an ndarray
> sst = np.empty([4320, 8640], dtype=np.float32)
> sst.fill(30.0)
>
> # write ndarray to a 2D plane in the HDF5
> sst_node[0] = sst
>
>
>
> ------------------------------------------------------------------------------
> How ServiceNow helps IT people transform IT departments:
> 1. A cloud service to automate IT design, transition and operations
> 2. Dashboards that offer high-level views of enterprise services
> 3. A single system of record for all IT processes
> http://p.sf.net/sfu/servicenow-d2d-j
> _______________________________________________
> Pytables-users mailing list
> Pytables-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>
------------------------------------------------------------------------------
How ServiceNow helps IT people transform IT departments:
1. A cloud service to automate IT design, transition and operations
2. Dashboards that offer high-level views of enterprise services
3. A single system of record for all IT processes
http://p.sf.net/sfu/servicenow-d2d-j
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
