Hi, On 10 November 2010 14:39, Francesc Alted <fal...@pytables.org> wrote: > A Wednesday 10 November 2010 12:08:34 Francesc Alted escrigué: >> A Tuesday 09 November 2010 16:38:10 Francesc Alted escrigué: >> > Hmm, this seems a problem with Blosc indeed. Could you put the >> > compressed datafile (for example, noaa18_2008_zlib3.h5) in a public >> > place so that I can see what's going on? >> >> Thanks Gerrit for your sample file. I've looked at the issue, but it >> looks like a (complex) inefficiency in the Lempel-Ziv part of Blosc >> (BloscLZ). I've no time now to look into this, so I created a >> ticket: >> >> http://blosc.pytables.org/trac/ticket/7 >> >> Hope this can be addressed in the near future. > > Hmm, I realized what's going on. The fix is in PyTables trunk now (or > std-2.2 branch, if you prefer). Can you have a try at it?
I tested for the reduced version below: -rw-r--r-- 1 gerrit students 3.2M Wednesday 10-11-2010 15:36:20 noaa18_2008_blosc1_reduced_newversion.h5 -rw-r--r-- 1 gerrit students 3.2M Wednesday 10-11-2010 15:36:20 noaa18_2008_blosc2_reduced_newversion.h5 -rw-r--r-- 1 gerrit students 2.8M Wednesday 10-11-2010 15:36:20 noaa18_2008_blosc3_reduced_newversion.h5 -rw-r--r-- 1 gerrit students 670K Wednesday 10-11-2010 15:36:20 noaa18_2008_blosc4_reduced_newversion.h5 -rw-r--r-- 1 gerrit students 670K Wednesday 10-11-2010 15:36:20 noaa18_2008_blosc5_reduced_newversion.h5 -rw-r--r-- 1 gerrit students 636K Wednesday 10-11-2010 15:36:21 noaa18_2008_blosc6_reduced_newversion.h5 -rw-r--r-- 1 gerrit students 640K Wednesday 10-11-2010 15:36:21 noaa18_2008_blosc7_reduced_newversion.h5 -rw-r--r-- 1 gerrit students 638K Wednesday 10-11-2010 15:36:21 noaa18_2008_blosc8_reduced_newversion.h5 -rw-r--r-- 1 gerrit students 640K Wednesday 10-11-2010 15:36:21 noaa18_2008_blosc9_reduced_newversion.h5 7, 8 and 9 are still larger than 6, but now by a very slight amount. Write speeds were similar, I didn't test read speeds, but usually higher compressions are slower. So then maybe number 6 (as I'm using now) is still optimal. Might this be expected? One side note: I forgot to update numexpr before recompiling pytables, and pytables only complained run-time, not compile-time. It would be nice if pytables checked such essential dependencies compile- or install-time. Gerrit. -- Gerrit Holl PhD student at Department of Space Science, Luleå University of Technology, Kiruna, Sweden http://www.sat.ltu.se/members/gerrit/ ------------------------------------------------------------------------------ The Next 800 Companies to Lead America's Growth: New Video Whitepaper David G. Thomson, author of the best-selling book "Blueprint to a Billion" shares his insights and actions to help propel your business during the next growth cycle. Listen Now! http://p.sf.net/sfu/SAP-dev2dev _______________________________________________ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
