On March 25, Tal Tabakman wrote:
Hi, I am writing an application that performs a lot of DB writes. I am using a lot of recommended optimizations (like using transactions and more...) I want to improve my recording time by reducing the amount of I/O. one way to do so is by compressing the data before dumping it to DISK. I am evaluating a sqlite extension called zipvfs. this VFS extension compresses pages before writing them to disk I am using zlib compress/uncompress as my compression callback functions for this VFS. I assumed that database writing will be faster with this VFS since compression [means less I/O], in reality I see no difference (but the data is indeed compressed)... any idea why I don't see any recording time improvement ? is there an overhead with zipvfs ? any other recommended compression callback functions ?
I suspect you are seeing the effect of SQLite's choice of block size on disk. If, for some reason, you application would work better with much larger block sizes than SQLite uses by default, you might see an improvement with on-the-fly compression/decompression, (provided CPU time does not take back the I/O time saving). The compression schemes generally have some overhead, creating and writing out a dictionary in ZLib's case. You are likely not reducing the number of I/O operations, but just reducing their size a little. With disk caching and read-ahead working pretty well for disk sectors that are contiguous, the effect is bound to be small with block size already resembling the cluster size.
You are tackling a problem that has already been subject to a lot of optimization effort. Further improvement is likely to be difficult unless there is something unusual about the access pattern of your application.
cheers Tal
Good luck. -- Larry Brasfield _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
