That sounds amazing.
On Mon, Aug 25, 2014 at 1:18 PM, Tim Holy <[email protected]> wrote: > I'd like to announce the merger of a massive revamping of HDF5 & JLD. The > bottom line is that now many immutables and types will be saved in a far > more > efficient fashion to disk, particularly for arrays of immutables. In such > cases, > the performance gains are quite extraordinary, often hundred-fold or > larger in > terms of time, and the resulting files are much smaller. For many > situations, > HDF5 is now comparable to the serializer (and sometimes faster) for reading > from and writing to disk. > > In addition to the performance enhancement, the whole system in JLD for > saving > and loading Julia types has been given a facelift. Rather than barfing on > broken or missing types, there's a sophisticated method to reconstruct > types > based on data in the disk file. You might appreciate this if you save data, > change your type definitions in your code, and then want the data back > again. > While these types won't allow you to proceed entirely as if nothing is > wrong, > it does give you an easier path to recovery. > > Finally, all this has been done with an effort to preserve backwards > compatibility. All tests pass with the new format (and many new tests have > been added), and the JLDArchives tests pass, suggesting that at least for > the > files checked into JLDArchives, there's been no hit to the ability to read > old > files. > > Before tagging a new version, we're hoping that a few hardy souls will > experiment by checking out master for the HDF5 package. Testing is needed > because we're talking about a long-term data storage format here. Once this > becomes "official" (by tagging a new version), I'll deposit a copy of a > test file > in JLDArchives, so that format becomes something we have to support going > forward. It would be much nicer to uncover any problems quickly so we don't > have to worry about ugly kludges :-). > > My role in this has exclusively been that of cheerleader: the work was > done by > Simon Kornblith and incorporated some great preliminary work by Matt > Baumann. > This is a 30-commit, 2,000 change, so it's a huge effort and very > beautifully > done. My sincere thanks go to them! > > --Tim > >
