I'd like to announce the merger of a massive revamping of HDF5 & JLD. The bottom line is that now many immutables and types will be saved in a far more efficient fashion to disk, particularly for arrays of immutables. In such cases, the performance gains are quite extraordinary, often hundred-fold or larger in terms of time, and the resulting files are much smaller. For many situations, HDF5 is now comparable to the serializer (and sometimes faster) for reading from and writing to disk.
In addition to the performance enhancement, the whole system in JLD for saving and loading Julia types has been given a facelift. Rather than barfing on broken or missing types, there's a sophisticated method to reconstruct types based on data in the disk file. You might appreciate this if you save data, change your type definitions in your code, and then want the data back again. While these types won't allow you to proceed entirely as if nothing is wrong, it does give you an easier path to recovery. Finally, all this has been done with an effort to preserve backwards compatibility. All tests pass with the new format (and many new tests have been added), and the JLDArchives tests pass, suggesting that at least for the files checked into JLDArchives, there's been no hit to the ability to read old files. Before tagging a new version, we're hoping that a few hardy souls will experiment by checking out master for the HDF5 package. Testing is needed because we're talking about a long-term data storage format here. Once this becomes "official" (by tagging a new version), I'll deposit a copy of a test file in JLDArchives, so that format becomes something we have to support going forward. It would be much nicer to uncover any problems quickly so we don't have to worry about ugly kludges :-). My role in this has exclusively been that of cheerleader: the work was done by Simon Kornblith and incorporated some great preliminary work by Matt Baumann. This is a 30-commit, 2,000 change, so it's a huge effort and very beautifully done. My sincere thanks go to them! --Tim
