Hi,
I'm experimenting in getting the fastest read-speed on selecting
queries from my 43 GiB table. I'm quite impressed with the results so
far, but I'm confused by the different compression levels of blosc.
The compressed size is not what I expect. I compressed with all nine
compression levels, and after compression level 6, the sizes go up
again. Why?
$ l /local/gerrit/tmp/noaa18_2008_blosc?.h5
-rw-r--r-- 1 gerrit students 41G 2010-11-08 14:11
/local/gerrit/tmp/noaa18_2008_blosc1.h5
-rw-r--r-- 1 gerrit students 41G 2010-11-08 14:33
/local/gerrit/tmp/noaa18_2008_blosc2.h5
-rw-r--r-- 1 gerrit students 28G 2010-11-08 14:44
/local/gerrit/tmp/noaa18_2008_blosc3.h5
-rw-r--r-- 1 gerrit students 7.3G 2010-11-08 14:48
/local/gerrit/tmp/noaa18_2008_blosc4.h5
-rw-r--r-- 1 gerrit students 7.3G 2010-11-08 13:34
/local/gerrit/tmp/noaa18_2008_blosc5.h5
-rw-r--r-- 1 gerrit students 7.0G 2010-11-08 14:52
/local/gerrit/tmp/noaa18_2008_blosc6.h5
-rw-r--r-- 1 gerrit students 23G 2010-11-08 13:57
/local/gerrit/tmp/noaa18_2008_blosc7.h5
-rw-r--r-- 1 gerrit students 23G 2010-11-08 15:01
/local/gerrit/tmp/noaa18_2008_blosc8.h5
-rw-r--r-- 1 gerrit students 33G 2010-11-08 13:44
/local/gerrit/tmp/noaa18_2008_blosc9.h5
To generate those, I used something like:
for i in 1 2 3 4 5 6 7 8 9
do
./ptrepack --complevel=$i --complib=blosc -v
/local/gerrit/data/collocations_hdf5/collocations_cloudsat_mhs/noaa18_2008.h5
/local/gerrit/tmp/noaa18_2008_blosc${i}.h5
done
Why are 7, 8, 9 much larger than 4, 5, 6?
Gerrit.
--
Gerrit Holl
PhD student at Department of Space Science, Luleå University of
Technology, Kiruna, Sweden
http://www.sat.ltu.se/members/gerrit/
------------------------------------------------------------------------------
The Next 800 Companies to Lead America's Growth: New Video Whitepaper
David G. Thomson, author of the best-selling book "Blueprint to a
Billion" shares his insights and actions to help propel your
business during the next growth cycle. Listen Now!
http://p.sf.net/sfu/SAP-dev2dev
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
