Hi Steve,
HDF5 does allow creating bitfield datatypes, but the underlying type
must currently be an integral number of bytes in size. It sounds like a
reasonable extension to allow some way to pack bitfields into one underlying
byte, but we haven't explored it seriously. If you'd like to think about it
and propose an interface that you think would work, that would kick off the
discussion nicely. :-)
Quincey
On Aug 13, 2010, at 7:04 AM, Steve Bissell wrote:
>
> I am working on an application to record data in HDF5 format, and I'm
> completely new to it.
> The data is in the form of packets, each of which has an associated
> timestamp and class.
> Therefore, it would seem appropriate to use the FL_PacketTable class (99% of
> the packets are fixed length, so this is my core use case).
> The class of the packet indicates the packet contents, and each class
> appears to map naturally to the HDF5 "Compound" data type, with a struct for
> each class of packet.
> Note also that data is retrieved from a legacy file format that uses
> individual bits to represent certain data.
>
> So far, so good. I can produce an hdf5 file with the following code
> (C++/win32/VisStudio2005); assume that the file object and the group V3 are
> defined.
>
> //structured data - "compound" in the HDF5 terminology.
> struct _my_type {
> double t;//e.g. time.
> int a;
> float b;
> };
> CompType mtype1( sizeof(_my_type) );
> mtype1.insertMember( "time", HOFFSET(_my_type, t), PredType::NATIVE_DOUBLE);
> mtype1.insertMember( "alt", HOFFSET(_my_type, a), PredType::NATIVE_INT);
> mtype1.insertMember( "math", HOFFSET(_my_type, b), PredType::NATIVE_FLOAT);
>
> FL_PacketTable pt(V3.getId(),"Packets",mtype1.getId(),500,6);
> _my_type s1;
> for (int i = 0; i< 400000; i++)
> {
> s1.t = i/10.f;//monotonic time
> s1.a = i % 10;//sawtooth integer data
> s1.b = 100.f/(i+1);//math function
> pt.AppendPacket(&s1);
> }
>
> The resulting file is maximally self describing, in that when opened with
> hdfView, I see a packet table with columns headed time, alt, math, and my
> "packets" in the records below.
>
> Now what I would like to do is achieve the same maximally self describing
> file for the amended compound type:
>
> struct _my_type {
> double t;//e.g. time.
> int a;
> float b;//so far, so easy....
> //BUT, we would also like...
> union {
> struct {
> unsigned char bit0 : 1;//ideally, should be able to map each
> bit's
> value
> unsigned char bit1 : 1;//to one of a pair of strings, e.g.
> "VALVE_OPEN"
> / "VALVE_CLOSED"
> unsigned char bit2 : 1;//by using, perhaps, something like
> the
> ENUMERATION feature of
> unsigned char bit3 : 1;//HDF5.
> unsigned char bit4 : 1;
> unsigned char bit5 : 1;
> unsigned char bit6 : 1;
> unsigned char bit7 : 1;
> };
> //..and ideally would ALSO like to be able to retrieve the entire
> field,
> as below....
> unsigned char wholebyte;
> };
> };
>
>
> If I now amend my code to do:
>
> mtype1.insertMember( "wholebyte", HOFFSET(_my_type, wholebyte),
> PredType::NATIVE_UCHAR);
> s1.wholebyte = 0;
>
> for (int i = 0; i< 400000; i++)
> {
> s1.t = i/10.f;//monotonic time
> s1.a = i % 10;//sawtooth integer data
> s1.b = 100.f/(i+1);//math function
> s1.bit1 = ( (0 == (i % 20)) ? 1 : 0);//bit1 goes true every 20th element
> s1.bit2 = ( (10 < (i % 20)) ? 1 : 0);//bit2 goes true about 1/2 the time
> s1.bit3 = ( (10 > (i % 30)) ? 1 : 0);//bit3 goes true about 1/3 the time
> pt.AppendPacket(&s1);
> }
>
>
> then I do indeed see "wholebyte" and its data as an extra column in hdfview.
> But end-users will certainly want to see individual bit values, rather than
> the entire byte.
>
> So - and this is my problem - if I do this instead (i.e. I do not insert
> wholebyte):
>
> //Create single bit transient types, then commit them to the dataset.
> //Q: are these types modifying the original types, or are they "copies" in
> the H5Tcopy sense?
> //Not yet clear without examining c++ library behaviour further.....
> IntType mySingleBit1Type(PredType::STD_B8LE);
> mySingleBit1Type.setPrecision(1);
> mySingleBit1Type.setOffset(1);
> mySingleBit1Type.commit(V3,"Bit1Type");
>
> mtype1.insertMember( "bit1", HOFFSET(_my_type,wholebyte), mySingleBit1Type);
>
> Then I do NOT see “bit1” as a field in the packet table using hdfview – that
> is, the “self describing” aspect fails.
>
> Worse, if I attempt to define and insert another bit type, as below:
>
> IntType mySingleBit2Type(PredType::STD_B8LE);
> mySingleBit2Type.setPrecision(1);
> mySingleBit2Type.setOffset(2);
> mySingleBit2Type.commit(V3,"Bit2Type");
> mtype1.insertMember( "bit2", HOFFSET(_my_type,wholebyte), mySingleBit2Type);
>
> Then I get a "member overlaps with another member" exception from
> H5Tcompound.c. This is not surprising, since the API only appears to allow
> BYTE offsets.
>
> Now some obvious, but ugly workarounds exist. I could, for example, store my
> original bit data as bytes. But this would be very inefficient, in terms of
> storage, unless the magic of compression would reduce the problem …..
>
> I can’t believe I’m the first person to encounter this issue, much more
> likely is that I’m still too stupid to understand how best to define the bit
> fields. Does anyone have any ideas? I'm aware that the above code may not be
> completely platform portable in theory due to the C specification not
> specifying exactly where bits might be put within the machine word, but this
> isn't an issue in our case (at the moment!)
> Thanks!
>
> --
> View this message in context:
> http://hdf-forum.184993.n3.nabble.com/Possible-to-pack-bit-types-into-compound-data-types-tp1131024p1131024.html
> Sent from the hdf-forum mailing list archive at Nabble.com.
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [email protected]
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
