A Monday 08 November 2010 15:08:20 Gerrit Holl escrigué:
> Hi,
>
> I'm experimenting in getting the fastest read-speed on selecting
> queries from my 43 GiB table. I'm quite impressed with the results so
> far, but I'm confused by the different compression levels of blosc.
> The compressed size is not what I expect. I compressed with all nine
> compression levels, and after compression level 6, the sizes go up
> again. Why?
>
> $ l /local/gerrit/tmp/noaa18_2008_blosc?.h5
> -rw-r--r-- 1 gerrit students 41G 2010-11-08 14:11
> /local/gerrit/tmp/noaa18_2008_blosc1.h5
> -rw-r--r-- 1 gerrit students 41G 2010-11-08 14:33
> /local/gerrit/tmp/noaa18_2008_blosc2.h5
> -rw-r--r-- 1 gerrit students 28G 2010-11-08 14:44
> /local/gerrit/tmp/noaa18_2008_blosc3.h5
> -rw-r--r-- 1 gerrit students 7.3G 2010-11-08 14:48
> /local/gerrit/tmp/noaa18_2008_blosc4.h5
> -rw-r--r-- 1 gerrit students 7.3G 2010-11-08 13:34
> /local/gerrit/tmp/noaa18_2008_blosc5.h5
> -rw-r--r-- 1 gerrit students 7.0G 2010-11-08 14:52
> /local/gerrit/tmp/noaa18_2008_blosc6.h5
> -rw-r--r-- 1 gerrit students 23G 2010-11-08 13:57
> /local/gerrit/tmp/noaa18_2008_blosc7.h5
> -rw-r--r-- 1 gerrit students 23G 2010-11-08 15:01
> /local/gerrit/tmp/noaa18_2008_blosc8.h5
> -rw-r--r-- 1 gerrit students 33G 2010-11-08 13:44
> /local/gerrit/tmp/noaa18_2008_blosc9.h5
>
> To generate those, I used something like:
> for i in 1 2 3 4 5 6 7 8 9
> do
> ./ptrepack --complevel=$i --complib=blosc -v
> /local/gerrit/data/collocations_hdf5/collocations_cloudsat_mhs/noaa18
> _2008.h5 /local/gerrit/tmp/noaa18_2008_blosc${i}.h5
> done
>
> Why are 7, 8, 9 much larger than 4, 5, 6?
Don't know, really, but this is pretty strange :-/ Could you show us
the results for using zlib and/or lzo? Maybe this may shed some light
on this.
Cheers,
--
Francesc Alted
------------------------------------------------------------------------------
The Next 800 Companies to Lead America's Growth: New Video Whitepaper
David G. Thomson, author of the best-selling book "Blueprint to a
Billion" shares his insights and actions to help propel your
business during the next growth cycle. Listen Now!
http://p.sf.net/sfu/SAP-dev2dev
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
