On Sat, May 7, 2011 at 1:25 PM, Christian Vetter <veaac.fdi...@gmail.com> wrote: > Hi, > > On Sat, May 7, 2011 at 5:46 AM, Scott Crosby <sc...@sacrosby.com> wrote: >> On Fri, May 6, 2011 at 9:24 PM, Christian Vetter <veaac.fdi...@gmail.com> >> wrote: >>> With regard to LZMA: I have some C++ code lying around to compress / >>> decompress LZMA... I can test how much it would affect file size / >>> decoding speed. >> >> Cool. You don't need a full-fledged PBF reader&writer to test it. Just >> enough to parse out blobs and write blobs. > > I quickly hacked it into MoNav's importer and tested it on the extract > of Germany. I used maximum compression ( dictionary size == blob size > ): > > size of zlib blobs: 849MB > size of lzma blobs: 762MB > time spent decoding zlib blobs: 6.986 s > time spent decoding lzma blobs: 49.078 s > > We can reduce the size a bit by using lzma ( ~10% ) and adding it > isn't much work ( about 10 lines of code for encoding / decoding ). > However, it doesn't seem worth it, considering that it makes parsing > slower. Increasing the block size would most likely not increase the > compression: I tested compressing all uncompressed blobs at once using > a 64MB dictionary and the size only decreased to 728MB
Interesting. This doesn't rule out using LZMA. With osmosis, decompression time is small compared to the other overheads. (uncompressed files are no faster to read than compressed files), so the decompression hit isn't too bad. How bad is the compression performance hit? Deflate halves the compression performance in osmosis compared to uncompressed. Ultimately, I think the decision will come down to whether a 10% space savings is worth adding a dependency on LZMA. Opinions? Scott _______________________________________________ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
