Hi Teg, Presumably you have a transaction in place around the whole of your inserts and that you have the PRAGMA synchronous = OFF; set.
Have you looked at perhaps not creating the database on the server, but merely creating the INSERT statements in one big file that you compress and send down to the client, who then decompresses and runs the inserts? You could even abbreviate the insert statements but I've always found (possibly because the indices don't compress well) that compressing the source of a database gets you a much smaller payload than compressing the finished database. Steve -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] org] On Behalf Of Teg Sent: 07 February 2006 15:40 To: Andrew Piskorski Subject: [RBL] Re[2]: [sqlite] R: [sqlite] Snapshot database creation performance Hello Andrew, My purpose is primarily disk storage savings, the data's mainly text so it's highly compressible. 500K on disk chunks of data decompress out to about 8 megabytes of text. What compression scheme do they use? I might consider trading some disk space for faster compression/decompression. C Tuesday, February 7, 2006, 10:26:02 AM, you wrote: AP> On Tue, Feb 07, 2006 at 08:51:43AM -0500, Teg wrote: >> My application uses compressed data (gzip) but, the tradeoff to small >> data files is exceptionally heavy CPU usage when the data is >> decompressed/compressed. AP> Incidentally, the MonetDB folks have done research on that sort of AP> thing. In their most recent project, "X100", they keep the data AP> compressed both on disk AND in main memory, and decompress it only in AP> the CPU cache when actually manipulating values. AP> They do that not primarily to save disk space, but to reduce the AP> amount of memory bandwith needed. Apparently in some cases it's a big AP> speed-up, and shifts the query from being memory I/O bound to CPU AP> bound. Of course, in order for that to work they have to use very AP> lightweight compression/decompression algorithms. Gzip gives much AP> better compression, but in comparison it's extremely slow. AP> Probably not immediately useful, but it seems like interesting stuff: AP> http://monetdb.cwi.nl/ AP> http://homepages.cwi.nl/~mk/MonetDB/ AP> http://sourceforge.net/projects/monetdb/ AP> http://homepages.cwi.nl/~boncz/ AP> "MonetDB/X100 - A DBMS In The CPU Cache" AP> by Marcin Zukowski, Peter Boncz, Niels Nes, Sandor Himan AP> ftp://ftp.research.microsoft.com/pub/debull/A05june/issue1.htm AP> Btw, apparently the current stable version of MonetDB is open source AP> but they haven't decided whether the X100 work will be or not. AP> Googling just now, there seems to have been a fair amount of research AP> and commercialization of this sort of stuff lately, e.g.: AP> http://db.csail.mit.edu/projects/cstore/ -- Best regards, Teg mailto:[EMAIL PROTECTED]
