----- Original Message ----
> From: FrancescAlted <fal...@pytables.org>
> To: Discussion list for PyTables <pytables-users@lists.sourceforge.net>
> Sent: Fri, March 25, 2011 11:17:29 AM
> Subject: Re: [Pytables-users] Problem writing strings to a CArray. Could this
>be a bug?
>
> A Friday 25 March 2011 00:05:19 Adriano Vilela Barbosa escrigué:
> > Yes, the problem is when assigning a string (any string, not only one
> > obtained from a numpy array) to the CArray. The trailing '\x00'
> > items are simply lost. In the examples you gave before with numpy
> > arrays, you could still see the trailing '\x00' elements were there
> > by doing a.data[:]; however, after assigning the string to the
> > CArray, even if I do
> >
> > fid.root.table.bin_table[0].data[:]
> >
> > I can't see anything. Is this really the way this is supposed to
> > work?
>
> Probably not, but as I said before, trying to pack binary data as
> strings is asking for problems. Please use a bytes array instead. If
> what you are after is performance, then I'd say that Blosc/VLArray is
> the way to go.
I understand. As I said before, I was using strings because that's what the
OpenCV Python bindings use to represent image data (though they've been moving
towards numpy in their latest releases). Actually, representing byte streams as
strings seems to be the standard in Python 2.x, which was kind of surprising to
me when I first started programming in Python.
>
> > In my previous, long (sorry about that) email I told you the reason
> > I'm using strings: because of OpenCV. However, I converted my OpenCV
> > images (actually, optical flow frames) to numpy arrays and I'm
> > trying to store them in a CArray. The data can be seen as a (n_rows,
> > n_cols, n_frames) array, where n_rows and n_cols are the number of
> > rows and columns in each frame, respectively, and n_frames is the
> > number of frames. The optical flow values are represented as int16.
> > Initially, I did
> >
> > array_shape = (n_rows,n_cols,n_frames)
> > array_atom = tables.Int16Atom()
> >
> > and that works fine, although this is much slower and results in
> > quite bigger files (compared to the string approach). Next, I did
> >
> > array_shape = (n_frames,)
> > array_atom = tables.Int16Atom((n_rows,n_cols))
> >
> > in the hope that this would be faster and more compression efficient.
> > However, when creating the second CArray (I need two of them, for
> > the horizontal and vertical pixel displacements) I get the following
> > error:
> [clip]
> >
> > This is the memory error I mentioned before. Any ideas why this
> > happens?
>
> Could you send a self-contained example reproducing your problem?
Please, see the code below.
Thanks a lot,
Adriano
>
> --
> FrancescAlted
>
import tables
import numpy
# ----- Writing data to file ----- #
# Open the output file for writing
fid = tables.openFile("carray_error.hdf","w")
# Create a table group
fid.createGroup("/", 'table', 'Flow table')
# The number of rows and columns in a frame, and the number of frames
n_rows = 480
n_cols = 720
n_frames = 2
# Create a numpy vector to be stored in the Carray
matrix = numpy.random.randn(n_rows,n_cols)
# The CArray shape
array_shape = (n_frames,)
# The CArray atom
array_atom = tables.Atom.from_dtype(numpy.dtype((numpy.int16, (n_rows,n_cols))))
# Create a Carray for holding horizontal flow values
fid.createCArray(fid.root.table,'flow_x',array_atom,array_shape)
# Create a Carray for holding vertical flow values. This is where we get an
error;
# working with smaller values of n_rows and n_cols works fine though.
fid.createCArray(fid.root.table,'flow_y',array_atom,array_shape)
for m in range(n_frames):
fid.root.table.flow_x[0] = matrix
fid.root.table.flow_y[0] = matrix
# Close the output file
fid.close()
------------------------------------------------------------------------------
Enable your software for Intel(R) Active Management Technology to meet the
growing manageability and security demands of your customers. Businesses
are taking advantage of Intel(R) vPro (TM) technology - will your software
be a part of the solution? Download the Intel(R) Manageability Checker
today! http://p.sf.net/sfu/intel-dev2devmar
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
