On Sat, May 7, 2011 at 3:01 PM, Christian Vetter <veaac.fdi...@gmail.com> wrote: > On Sat, May 7, 2011 at 9:23 PM, Scott Crosby <sc...@sacrosby.com> wrote: >> On Sat, May 7, 2011 at 1:25 PM, Christian Vetter <veaac.fdi...@gmail.com> >> wrote: >>> Hi, >>> >>> On Sat, May 7, 2011 at 5:46 AM, Scott Crosby <sc...@sacrosby.com> wrote: >>>> On Fri, May 6, 2011 at 9:24 PM, Christian Vetter <veaac.fdi...@gmail.com> >>>> wrote: >>>>> With regard to LZMA: I have some C++ code lying around to compress / >>>>> decompress LZMA... I can test how much it would affect file size / >>>>> decoding speed. >>>> >>>> Cool. You don't need a full-fledged PBF reader&writer to test it. Just >>>> enough to parse out blobs and write blobs. >>> >>> I quickly hacked it into MoNav's importer and tested it on the extract >>> of Germany. I used maximum compression ( dictionary size == blob size >>> ): >>> >>> size of zlib blobs: 849MB >>> size of lzma blobs: 762MB >>> time spent decoding zlib blobs: 6.986 s >>> time spent decoding lzma blobs: 49.078 s >>> >>> We can reduce the size a bit by using lzma ( ~10% ) and adding it >>> isn't much work ( about 10 lines of code for encoding / decoding ). >>> However, it doesn't seem worth it, considering that it makes parsing >>> slower. Increasing the block size would most likely not increase the >>> compression: I tested compressing all uncompressed blobs at once using >>> a 64MB dictionary and the size only decreased to 728MB >> >> Interesting. This doesn't rule out using LZMA. With osmosis, >> decompression time is small compared to the other overheads. >> (uncompressed files are no faster to read than compressed files), so >> the decompression hit isn't too bad. How bad is the compression >> performance hit? Deflate halves the compression performance in osmosis >> compared to uncompressed. > > About 312s to compress all blobs for Germany. Changing the dictionary > size does not change much. I lowered it all the way down to 64kb and > the values stayed the same essentially. >
And deflate? Scott _______________________________________________ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev