On Wed, May 17, 2006 at 06:38:47PM +0100, Simon Riggs wrote: > > - Each tape is compressed as one long compressed stream. Currently no > > seeking is allowed, so only sorts, no joins! (As tom said, quick and > > dirty numbers). This should show this possibility in its best light > > but if we want to support seeking we're going to need to change that. > > Maybe no compression on the last pass? > > We should be able to do this without significant loss of compression by > redefining the lts block size to be 32k. That's the size of the > look-back window anyhow, so compressing the whole stream doesn't get us > much more.
The major problem is looking back costs significantly more with compression. If you need to look back into the previous compressed block, you need to decompress the whole previous block. The simple solution would be to keep a buffer of the last 32KB. Another posibility would be to have a limit of 32KB of uncompressed data per block and just remember the whole previous block. Seek/Tell is not the hard part, it's the backspace. It would probably be smart to make backspace call Seek, rather than trying to be smart about it. Another issue is that currently the compression code is completely within logtape.c. To be able to seek backwards efficiently you might have to change the abstraction so that it knows about the records from tuplesort.c. That's much more work, which needs a lot more thinking. Besides, we still havn't got any reports yet that this actually provides a benefit on any machine less than five years ago. Anyone out there doing tests? Have a nice day, -- Martijn van Oosterhout <firstname.lastname@example.org> http://svana.org/kleptog/ > From each according to his ability. To each according to his ability to > litigate.
Description: Digital signature