Hi John, I've just stumbled upon a LZO data compression library (http://www.oberhumer.com/opensource/lzo/), which is used for example in nfdump, to cut down the space needed to store data. I wonder whether you ever considered adding some realtime compression/decompression to FastBit, since you work with a lot o data. I know that the indexes use a compression, but the data is not.
How difficult would it be to have columns optionally compressed when writing to disk (maybe indicated by some file extension) and decompressed when needed to answer a query (which might even not be necessary if we look at the indexes first)? I do not know if anyone needs such a feature, but I can see some benefits in it for our usecase. Regards, Petr _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
