[OSM-dev] Memory-Leak in osmium
Hi I found a memory-leak in osmium-master. I compiled the osmium_progress example and fed germany.osh.pbf http://ftp5.gwdg.de/pub/misc/openstreetmap/osm-full-history-extracts/110919/pbf/europe/germany.osh.pbf into it. During the nodes processing the program takes 70 MB of RAM. When it starts to process Ways, the required memory raises constantly up to 300 MB. Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Memory-Leak in osmium
Hi Am 21.11.2011 12:07, schrieb Peter Körner: I found a memory-leak in osmium-master. Digging into it, it seems that the pbf reader is leaking memory. I attached valgrind-massif output for running osmium_progess against mainz.osh.pbf and mainz.osh. The pbf-run took nearly 44x the memory of the xml-run: http://daten.marvinstudio.de/osmium_progress-on-pbf.txt http://daten.marvinstudio.de/osmium_progress-on-xml.txt Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Memory-Leak in osmium
On Mon, Nov 21, 2011 at 12:07:42PM +0100, Peter Körner wrote: I found a memory-leak in osmium-master. I compiled the osmium_progress example and fed germany.osh.pbf http://ftp5.gwdg.de/pub/misc/openstreetmap/osm-full-history-extracts/110919/pbf/europe/germany.osh.pbf into it. During the nodes processing the program takes 70 MB of RAM. When it starts to process Ways, the required memory raises constantly up to 300 MB. I don't think that there is a memory leak. Valgrind tells me that all is ok. The behaviour you see comes from the PBF reader and especially the Google Protobuf stuff. Protobuf does not free memory but keeps it around until the next time it is needed. So every time you hit a PBF block with more data in it than all the other blocks before (for instance more nodes per way), PBF will allocated more memory and not de-allocate it. So for a while you'll see the process slowly using more and more memory until it hits the peak. Only google::protobuf::ShutdownProtobufLibrary(); will release all the memory. Thats done at the end of the program. (Could be done at the end of the read, but for most programs thats the same as the end of the program anyway.) So there is no leak, all memory is accounted for. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Memory-Leak in osmium
Am 21.11.2011 15:36, schrieb Jochen Topf: I don't think that there is a memory leak. Valgrind tells me that all is ok. A program can leak memory without valgrind showing it. I already guessed that this data is not completely useless. The behaviour you see comes from the PBF reader and especially the Google Protobuf stuff. Protobuf does not free memory but keeps it around until the next time it is needed. So every time you hit a PBF block with more data in it than all the other blocks before (for instance more nodes per way), PBF will allocated more memory and not de-allocate it. So for a while you'll see the process slowly using more and more memory until it hits the peak. 260 MB for a city-sized file (mainz) and 500 MB for a country (germany) is a lot. That explains, why my memory calculations always were wrong. Thanks for that explanation. Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
