On Sat, May 7, 2011 at 9:23 PM, Scott Crosby <[email protected]> wrote: > On Sat, May 7, 2011 at 1:25 PM, Christian Vetter <[email protected]> > wrote: >> Hi, >> >> On Sat, May 7, 2011 at 5:46 AM, Scott Crosby <[email protected]> wrote: >>> On Fri, May 6, 2011 at 9:24 PM, Christian Vetter <[email protected]> >>> wrote: >>>> With regard to LZMA: I have some C++ code lying around to compress / >>>> decompress LZMA... I can test how much it would affect file size / >>>> decoding speed. >>> >>> Cool. You don't need a full-fledged PBF reader&writer to test it. Just >>> enough to parse out blobs and write blobs. >> >> I quickly hacked it into MoNav's importer and tested it on the extract >> of Germany. I used maximum compression ( dictionary size == blob size >> ): >> >> size of zlib blobs: 849MB >> size of lzma blobs: 762MB >> time spent decoding zlib blobs: 6.986 s >> time spent decoding lzma blobs: 49.078 s >> >> We can reduce the size a bit by using lzma ( ~10% ) and adding it >> isn't much work ( about 10 lines of code for encoding / decoding ). >> However, it doesn't seem worth it, considering that it makes parsing >> slower. Increasing the block size would most likely not increase the >> compression: I tested compressing all uncompressed blobs at once using >> a 64MB dictionary and the size only decreased to 728MB > > Interesting. This doesn't rule out using LZMA. With osmosis, > decompression time is small compared to the other overheads. > (uncompressed files are no faster to read than compressed files), so > the decompression hit isn't too bad. How bad is the compression > performance hit? Deflate halves the compression performance in osmosis > compared to uncompressed.
About 312s to compress all blobs for Germany. Changing the dictionary size does not change much. I lowered it all the way down to 64kb and the values stayed the same essentially. Regards, Christian Vetter _______________________________________________ dev mailing list [email protected] http://lists.openstreetmap.org/listinfo/dev
