On Wed, Apr 29, 2015, at 11:45, François Battail wrote: > with liprotobuf-c.so, default allocator, without assembly support: > 799s (bandwidth: 33.9 MiB/s) with liprotobuf-c.so, sw_pool_t > allocator, little assembly support: 629s (bandwidth: 43.1 MiB/s)"
Point taken -- in future comparisons I will focus on throughput as much or more than size itself. I did notice that protobuf-c allows for customizing ProtobufCAllocator, and it's interesting to see the results of using a pool allocator. However my first reaction is that we really shouldn't need to do much dynamic allocation at all to handle OSM data. The use of dynamic allocation is a result of the general-purpose nature of Protobuf. My sense is that Protobuf is not even a particularly good fit for bulk OSM data transfer, considering the hoops that we must jump through in dense-nodes to bypass the natural mapping of one Protobuf message to one OSM entity. > Most of the time is spent in zlib, libprotobuf-c and memory > allocations. I've addressed the last point using x86_64 assembly > language and a pool allocator. I think a rewrite of an optimized > libprotobuf library would help to gain some speed but the cost is very > high (at least for my application). The alternative being to use a format that is not based on Protobuf at all, but only uses its variable-byte encoding scheme. Thanks for your comments! Andrew
_______________________________________________ dev mailing list [email protected] https://lists.openstreetmap.org/listinfo/dev
