On 05.06.2013 03:29, Tim Burgess wrote:
> I was playing around with in-memory HDF5 prior to the 3.0 release.
> Here's an example based on what I was doing.
> I looked over the docs and it does mention that there is an option to
> throw away the 'file' rather than write it to disk.
> Not sure how to do that and can't actually think of a use case where I
> would want to :-)
>
> And be wary, it is H5FD_CORE.
>
>
> On Jun 05, 2013, at 08:38 AM, Anthony Scopatz <scop...@gmail.com> wrote:
>>
>> I think that you want to set parameters.DRIVER to H5DF_CORE [1]. I
>> haven't ever used this personally, but it would be great to have an
>> example script, if someone wants to write one ;)
>>
>
>
> import numpy as np
> import tables
>
> CHUNKY = 30
> CHUNKX = 8640
>
> if __name__ == '__main__':
>
> # create dataset and add global attrs
>
> file_path = 'demofile_chunk%sx%d.h5' % (CHUNKY, CHUNKX)
>
> with tables.open_file(file_path, 'w', title='PyTables HDF5 In-memory
> example', driver='H5FD_CORE') as h5f:
>
> # dummy some data
> lats = np.empty([4320])
> lons = np.empty([8640])
>
> # create some simple arrays
> lat_node = h5f.create_array('/', 'lat', lats, title='latitude')
> lon_node = h5f.create_array('/', 'lon', lons, title='longitude')
>
> # create a 365 x 4320 x 8640 CArray of 32bit float
> shape = (365, 4320, 8640)
> atom = tables.Float32Atom(dflt=np.nan)
>
> # chunk into daily slices and then further chunk days
> sst_node = h5f.create_carray(h5f.root, 'sst', atom, shape,
> chunkshape=(1, CHUNKY, CHUNKX))
>
> # dummy up an ndarray
> sst = np.empty([4320, 8640], dtype=np.float32)
> sst.fill(30.0)
>
> # write ndarray to a 2D plane in the HDF5
> sst_node[0] = sst
Thanks Tim,
I adapted your example for my use case (I'm using the EArray class,
because I need to continuously update my database), and it works well.
However, when I use this with my own data (but also creating the arrays
like you did), I'm running into errors like "Could not wait on barrier".
It seems like the HDF library is spawing several threads.
Any idea what's going wrong? Can I somehow avoid HDF5 multithreading at
runtime?
Cheers, Andreas.
------------------------------------------------------------------------------
How ServiceNow helps IT people transform IT departments:
1. A cloud service to automate IT design, transition and operations
2. Dashboards that offer high-level views of enterprise services
3. A single system of record for all IT processes
http://p.sf.net/sfu/servicenow-d2d-j
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
