Re: [Pytables-users] [Pytables-announce] ANN: PyTables 2.4.0 beta1
Hey Antonio, this looks great. BTW, which is the status of the 3.x support? I vaguely remember you asking me for some help on this, but I don't remember well. Not that I have a lot of time to spend on it, but perhaps I can use some hours in the next days. Thanks, Francesc On 7/7/12 8:47 PM, Antonio Valentino wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 === Announcing PyTables 2.4.0b1 === We are happy to announce PyTables 2.4.0b1. This is an incremental release which includes many changes to prepare for future Python 3 support. What's new == This release includes support for the float16 data type and read-only support for variable length string attributes. The handling of HDF5 errors has been improved. The user will no longer see HDF5 error stacks dumped to the console. All HDF5 error messages are trapped and attached to a proper Python exception. Now PyTables only supports HDF5 v1.8.4+. All the code has been updated to the new HDF5 API. Supporting only HDF5 1.8 series is beneficial for future development. As always, a large amount of bugs have been addressed and squashed as well. In case you want to know more in detail what has changed in this version, please refer to: http://pytables.github.com/release_notes.html You can download a source package with generated PDF and HTML docs, as well as binaries for Windows, from: http://sourceforge.net/projects/pytables/files/pytables/2.4.0b1 For an online version of the manual, visit: http://pytables.github.com/usersguide/index.html What it is? === PyTables is a library for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data with support for full 64-bit file addressing. PyTables runs on top of the HDF5 library and NumPy package for achieving maximum throughput and convenient use. PyTables includes OPSI, a new indexing technology, allowing to perform data lookups in tables exceeding 10 gigarows (10**10 rows) in less than a tenth of a second. Resources = About PyTables: http://www.pytables.org About the HDF5 library: http://hdfgroup.org/HDF5/ About NumPy: http://numpy.scipy.org/ Acknowledgments === Thanks to many users who provided feature improvements, patches, bug reports, support and suggestions. See the ``THANKS`` file in the distribution package for a (incomplete) list of contributors. Most specially, a lot of kudos go to the HDF5 and NumPy (and numarray!) makers. Without them, PyTables simply would not exist. Share your experience = Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. - **Enjoy data!** - -- The PyTables Team -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk/4hDwACgkQ1JUs2CS3bP7TUwCfcobS3KI7L/6k3Bbbt2VBOz5B TqAAn0DhrSdtd7XTPOj0RR/mpr2FtseE =T5iQ -END PGP SIGNATURE- -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-announce mailing list pytables-annou...@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-announce -- Francesc Alted -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] [Pytables-announce] ANN: PyTables 2.4.0 beta1
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Francesc, Il 11/07/2012 11:33, Francesc Alted ha scritto: Hey Antonio, this looks great. Thanks :) BTW, which is the status of the 3.x support? I vaguely remember you asking me for some help on this, but I don't remember well. Not that I have a lot of time to spend on it, but perhaps I can use some hours in the next days. well, in PyTables 2.4 we made some job in preparation of the porting to python3, but the porting itself is still not started. One of the main issues is that numexpr still not supports python3 so we have a missing dependency. I started a porting of numexpr to python3 (see [1]) but it is still incomplete. I hope it is good enough to let us start working on the porting of PyTables. Of course if you would like to give a look to numexpr for python3 it would be of great help. After the PyTables 2.4 final I plan to publish a wiki page with my roadmap proposal. IMHO main points are: * open a new branch in the repo * remove al deprecated code (Numeric, numarray, netcdf3, etc). This breaks the API and, IMHO, we will also need to bump the format version * ensure that all the required SW work (enough) on python3 * handle str/unicode issues * full support to unicode HDF5 object names * start working an a good setup for 2to3 (needs some investigation) * ... Please let me know if you think there are other point that are important for python3 support [1] https://groups.google.com/forum/?fromgroups#!topic/numexpr/M2MVjXsBR0c cheers Thanks, Francesc On 7/7/12 8:47 PM, Antonio Valentino wrote: === Announcing PyTables 2.4.0b1 === We are happy to announce PyTables 2.4.0b1. [CUT] - -- Antonio Valentino -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk/9soUACgkQ1JUs2CS3bP4u1ACeJKnMQRFF1hATXFMG3lPH2xyU 9DwAoJNPp6L8gHf+s5hA2Jhj4JLyl3jr =AYqd -END PGP SIGNATURE- -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] advice on using PyTables
Hello Benjamin, Not knowing to much about the ASTERIX format, other than what you said and what is in the links, I would say that this is a good fit for HDF5 and PyTables. PyTables will certainly help you read in the data and manipulate it. However, before you abandon hachoir completely, I will say it is a lot easier to write hdf5 files in PyTables than to use the HDF5 C API. If hachoir is too slow, have you tried profiling the code to see what is taking up the most time? Maybe you could just rewrite these parts in C? Have you looked into Cythonizing it? Also, you don't seem to be using numpy to read in the data... (there are some tricks given ASTERIX here, but not insurmountable). I ask the above, just so you don't have to completely rewrite everything. You are correct though that pure python is probably not sufficient. Feel free to ask more questions here. Be Well Anthony On Wed, Jul 11, 2012 at 6:52 AM, benjamin.bertr...@lfv.se wrote: Hi, I'm working with Air Traffic Management and would like to perform checks / compute statistics on ASTERIX data. ASTERIX is an ATM Surveillance Data Binary Messaging Format ( http://www.eurocontrol.int/asterix/public/standard_page/overview.html) The data consist of a concatenation of consecutive data blocks. Each data block consists of data category + length + records. Each record is of variable length and consists of several data items (that are well defined for each category). Some data items might be present or not depending on a field specification (bitfield). I started to write a parser using hachoir ( https://bitbucket.org/haypo/hachoir/overview) a pure python library. But the parsing was really too slow and taking a lot of memory. That's not really useable. From what I read, PyTables could really help to manipulate and analyze the data. So I've been thinking about writing a tool (probably in C) to convert my ASTERIX format to HDF5. Before I start, I'd like confirmation that this seems like a suitable application for PyTables. Is there another approach than writing a conversion tool to HDF5? Thanks in advance Benjamin -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
