Re: [PATCHES] [PATCH] Compression and on-disk sorting

2006-05-19 Thread Martijn van Oosterhout
On Fri, May 19, 2006 at 12:43:05PM +0100, Simon Riggs wrote: > We need to test "SELECT aid from accounts" also, or some other scenarios > where the data is as uncompressible as possible. We should also try this > on a table where the rows have been inserted by different transactions, > so that the

Re: [PATCHES] [PATCH] Compression and on-disk sorting

2006-05-19 Thread Simon Riggs
On Thu, 2006-05-18 at 17:10 -0500, Jim C. Nasby wrote: > On Thu, May 18, 2006 at 10:31:03AM +0200, Martijn van Oosterhout wrote: > > Besides, we still havn't got any reports yet that this actually > > provides a benefit on any machine less than five years ago. Anyone out > > there doing tests? > >

Re: [PATCHES] [PATCH] Compression and on-disk sorting

2006-05-18 Thread Jim C. Nasby
On Thu, May 18, 2006 at 10:31:03AM +0200, Martijn van Oosterhout wrote: > Besides, we still havn't got any reports yet that this actually > provides a benefit on any machine less than five years ago. Anyone out > there doing tests? Yes. I'm compiling the patched binaries right now, but the baselin

Re: [PATCHES] [PATCH] Compression and on-disk sorting

2006-05-18 Thread Martijn van Oosterhout
On Thu, May 18, 2006 at 11:34:36AM +0100, Simon Riggs wrote: > Just do a Z_FULL_FLUSH when you hit end of block. That way all blocks > will be independent of each other and you can rewind as much as you > like. We can choose the block size to be 32KB or even 64KB, there's no > dependency there, jus

Re: [PATCHES] [PATCH] Compression and on-disk sorting

2006-05-18 Thread Simon Riggs
On Thu, 2006-05-18 at 10:31 +0200, Martijn van Oosterhout wrote: > On Wed, May 17, 2006 at 06:38:47PM +0100, Simon Riggs wrote: > > > - Each tape is compressed as one long compressed stream. Currently no > > > seeking is allowed, so only sorts, no joins! (As tom said, quick and > > > dirty numbers)

Re: [PATCHES] [PATCH] Compression and on-disk sorting

2006-05-18 Thread Martijn van Oosterhout
On Wed, May 17, 2006 at 06:38:47PM +0100, Simon Riggs wrote: > > - Each tape is compressed as one long compressed stream. Currently no > > seeking is allowed, so only sorts, no joins! (As tom said, quick and > > dirty numbers). This should show this possibility in its best light > > but if we want

Re: [PATCHES] [PATCH] Compression and on-disk sorting

2006-05-17 Thread Simon Riggs
On Wed, 2006-05-17 at 18:17 +0200, Martijn van Oosterhout wrote: > Persuant to the discussions currently on -hackers, here's a patch that > uses zlib to compress the tapes as they go to disk. I default to the > compression level 3 (think gzip -3). > > Please speed test all you like, I *think* it's

[PATCHES] [PATCH] Compression and on-disk sorting

2006-05-17 Thread Martijn van Oosterhout
Persuant to the discussions currently on -hackers, here's a patch that uses zlib to compress the tapes as they go to disk. I default to the compression level 3 (think gzip -3). Please speed test all you like, I *think* it's bug free, but you never know. Outstanding questions: - I use zlib becaus