On Thu, Jul 12, 2012 at 10:17 AM, <benjamin.bertr...@lfv.se> wrote:
> > Hello Benjamin,
> >
> > Not knowing to much about the ASTERIX format, other than what you said
> > and what is in the links, I would say that this is a good fit for HDF5
> > and PyTables. PyTables will certainly help you read in the data and
> > manipulate it.
> >
> > However, before you abandon hachoir completely, I will say it is a lot
> > easier to write hdf5 files in PyTables than to use the HDF5 C API. If
> > hachoir is too slow, have you tried profiling the code to see what is
> > taking up the most time? Maybe you could just rewrite these parts in
> > C? Have you looked into Cythonizing it? Also, you don't seem to be
> > using numpy to read in the data... (there are some tricks given ASTERIX
> > here, but not insurmountable).
> >
> > I ask the above, just so you don't have to completely rewrite
> > everything. You are correct though that pure python is probably not
> > sufficient. Feel free to ask more questions here.
>
> Hello Anthony,
>
> Thanks for your answer.
> Now that I discovered ipython and the line_profiler extension, I've done
> more profiling.
> Most of the time is spent in creating the field objects defined in hachoir.
> A field object has nice attributes (size, value, description, address...)
> but it adds a lot of overhead.
>
Understood! Good luck.
> Contrary to my intuition, not that much time is spent in reading the data
> (despite some bitfields to read).
> So it's probably worth trying to parse and write an hdf5 file directly
> with PyTables.
> I need to read more PyTables doc and examples.
> I might ask more questions when I have :-)
>
Feel free ;)
>
> Regards,
>
> Benjamin
>
>
> >
> > Be Well
> > Anthony
> >
> > On Wed, Jul 11, 2012 at 6:52 AM, <benjamin.bertr...@lfv.se> wrote:
> >
> >
> > Hi,
> >
> > I'm working with Air Traffic Management and would like to
> > perform checks / compute statistics on ASTERIX data.
> > ASTERIX is an ATM Surveillance Data Binary Messaging Format
> > (http://www.eurocontrol.int/asterix/public/standard_page/overview.html)
> >
> > The data consist of a concatenation of consecutive data
> > blocks.
> > Each data block consists of data category + length +
> > records.
> > Each record is of variable length and consists of several
> > data items (that are well defined for each category).
> > Some data items might be present or not depending on a field
> > specification (bitfield).
> >
> > I started to write a parser using hachoir
> > (https://bitbucket.org/haypo/hachoir/overview) a pure python library.
> > But the parsing was really too slow and taking a lot of
> > memory.
> > That's not really useable.
> >
> > >From what I read, PyTables could really help to manipulate
> > and analyze the data.
> > So I've been thinking about writing a tool (probably in C)
> > to convert my ASTERIX format to HDF5.
> >
> > Before I start, I'd like confirmation that this seems like a
> > suitable application for PyTables.
> > Is there another approach than writing a conversion tool to
> > HDF5?
> >
> > Thanks in advance
> >
> > Benjamin
> >
> > ------------------------------------------------------------
> > ------------------
> > Live Security Virtual Conference
> > Exclusive live event will cover all the ways today's
> > security and
> > threat landscape has changed and how IT managers can
> > respond. Discussions
> > will include endpoint security, mobile security and the
> > latest in malware
> > threats.
> > http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> > _______________________________________________
> > Pytables-users mailing list
> > Pytables-users@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/pytables-users
> >
> >
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pytables-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
