Re: [OSM-dev] osmium trouble
On Sun, Nov 20, 2011 at 03:34:27PM -0700, Martijn van Exel wrote: I also ran into trouble with files of around 40MB or larger. I still get the same errors though, either using array or sparsetable for node storage. I'm on Ubuntu 10.04 64 bit with 16GB RAM. Can you post the script you are using? Maybe we can find out if its doing something that triggers a bug. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osmium trouble
On 21/11/2011 08:13, Jochen Topf wrote:
On Sun, Nov 20, 2011 at 03:34:27PM -0700, Martijn van Exel wrote:
I also ran into trouble with files of around 40MB or larger. I still
get the same errors though, either using array or sparsetable for node
storage.
I'm on Ubuntu 10.04 64 bit with 16GB RAM.
Can you post the script you are using? Maybe we can find out if its doing
something that triggers a bug.
Just to confirm that osmjs works again for me after applying your fix
from yesterday:
commit 10ec36fbf104c5e1ae8ae1e0c5199cf072acb143
Author: Jochen Topf joc...@topf.org
Date: Sun Nov 20 15:06:09 2011 +0100
fix memory leak in Javascript glue code
I just can't confirm your comment from an earlier commit (22 Oct): This
change does not lead to much extra overhead... [1]. Some more figures
to illustrate the performance difference:
austria.osm.pbf (164M)
Before: After:
real0m58.012s real1m19.850s
user0m52.899s user1m16.013s
sys 0m2.980ssys 0m1.784s
france.osm.pbf (1.7G)
Before: After:
real5m43.047s real11m4.719s
user5m25.260s user10m49.733s
sys 0m13.881s sys 0m14.349s
I was using again the same simple Javascript code [2].
Hermann
[1]
https://github.com/joto/osmium/commit/17604decdd2f26d9fc831cc3c2412a8ca97dc2b0
[2]
Osmium.Callbacks.way = function() {
if ( this.tags[highway] ) {
print(this.tags[highway] + '\t' + this.geom.toWKT());
}
}
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Memory-Leak in osmium
Hi I found a memory-leak in osmium-master. I compiled the osmium_progress example and fed germany.osh.pbf http://ftp5.gwdg.de/pub/misc/openstreetmap/osm-full-history-extracts/110919/pbf/europe/germany.osh.pbf into it. During the nodes processing the program takes 70 MB of RAM. When it starts to process Ways, the required memory raises constantly up to 300 MB. Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Memory-Leak in osmium
Hi Am 21.11.2011 12:07, schrieb Peter Körner: I found a memory-leak in osmium-master. Digging into it, it seems that the pbf reader is leaking memory. I attached valgrind-massif output for running osmium_progess against mainz.osh.pbf and mainz.osh. The pbf-run took nearly 44x the memory of the xml-run: http://daten.marvinstudio.de/osmium_progress-on-pbf.txt http://daten.marvinstudio.de/osmium_progress-on-xml.txt Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Memory-Leak in osmium
On Mon, Nov 21, 2011 at 12:07:42PM +0100, Peter Körner wrote: I found a memory-leak in osmium-master. I compiled the osmium_progress example and fed germany.osh.pbf http://ftp5.gwdg.de/pub/misc/openstreetmap/osm-full-history-extracts/110919/pbf/europe/germany.osh.pbf into it. During the nodes processing the program takes 70 MB of RAM. When it starts to process Ways, the required memory raises constantly up to 300 MB. I don't think that there is a memory leak. Valgrind tells me that all is ok. The behaviour you see comes from the PBF reader and especially the Google Protobuf stuff. Protobuf does not free memory but keeps it around until the next time it is needed. So every time you hit a PBF block with more data in it than all the other blocks before (for instance more nodes per way), PBF will allocated more memory and not de-allocate it. So for a while you'll see the process slowly using more and more memory until it hits the peak. Only google::protobuf::ShutdownProtobufLibrary(); will release all the memory. Thats done at the end of the program. (Could be done at the end of the read, but for most programs thats the same as the end of the program anyway.) So there is no leak, all memory is accounted for. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osmium trouble
On Mon, Nov 21, 2011 at 10:33:40AM +0100, Hermann Peifer wrote: On 21/11/2011 08:13, Jochen Topf wrote: On Sun, Nov 20, 2011 at 03:34:27PM -0700, Martijn van Exel wrote: I also ran into trouble with files of around 40MB or larger. I still get the same errors though, either using array or sparsetable for node storage. I'm on Ubuntu 10.04 64 bit with 16GB RAM. Can you post the script you are using? Maybe we can find out if its doing something that triggers a bug. Just to confirm that osmjs works again for me after applying your fix from yesterday: commit 10ec36fbf104c5e1ae8ae1e0c5199cf072acb143 Author: Jochen Topf joc...@topf.org Date: Sun Nov 20 15:06:09 2011 +0100 fix memory leak in Javascript glue code I just can't confirm your comment from an earlier commit (22 Oct): This change does not lead to much extra overhead... [1]. Some more I had carefully benchmarked several things on the C++ side to make sure the changes didn't slow things down too much. But unfortunately I didn't check the Javascript side. I didn't expect it to be so bad. I'll have to dig into that to see what can be done. Actually it will probably get even slower at some point. Thats because I am taking some shortcuts (in the old and in the new code) which will lead to crashes if you use the OSM object outside the handler function... Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Memory-Leak in osmium
Am 21.11.2011 15:36, schrieb Jochen Topf: I don't think that there is a memory leak. Valgrind tells me that all is ok. A program can leak memory without valgrind showing it. I already guessed that this data is not completely useless. The behaviour you see comes from the PBF reader and especially the Google Protobuf stuff. Protobuf does not free memory but keeps it around until the next time it is needed. So every time you hit a PBF block with more data in it than all the other blocks before (for instance more nodes per way), PBF will allocated more memory and not de-allocate it. So for a while you'll see the process slowly using more and more memory until it hits the peak. 260 MB for a city-sized file (mainz) and 500 MB for a country (germany) is a lot. That explains, why my memory calculations always were wrong. Thanks for that explanation. Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osmium trouble
Sure, it's on github -- https://github.com/mvexel/OSMQualityMetrics FYI I did compile against the latest v8 from svn. Martijn On Mon, Nov 21, 2011 at 12:13 AM, Jochen Topf joc...@remote.org wrote: On Sun, Nov 20, 2011 at 03:34:27PM -0700, Martijn van Exel wrote: I also ran into trouble with files of around 40MB or larger. I still get the same errors though, either using array or sparsetable for node storage. I'm on Ubuntu 10.04 64 bit with 16GB RAM. Can you post the script you are using? Maybe we can find out if its doing something that triggers a bug. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 -- martijn van exel schaaltreinen.nl ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osmium trouble
Hi! You are keeping a lot of data around (users and ages array, nodes hash). Are you sure thats not the problem? There are about 50 user IDs now, so if a user with id 50 comes along the users array is expanded to that size. I have no idea how much memory Javascript will need for an empty array that size, but I could imagine that its quite a bit. Jochen On Mon, Nov 21, 2011 at 10:16:25AM -0700, Martijn van Exel wrote: Date: Mon, 21 Nov 2011 10:16:25 -0700 From: Martijn van Exel mve...@gmail.com To: Jochen Topf joc...@remote.org Cc: Hermann Peifer pei...@gmx.eu, ThomasB toba0...@yahoo.de, dev@openstreetmap.org Subject: Re: [OSM-dev] osmium trouble Sure, it's on github -- https://github.com/mvexel/OSMQualityMetrics FYI I did compile against the latest v8 from svn. Martijn On Mon, Nov 21, 2011 at 12:13 AM, Jochen Topf joc...@remote.org wrote: On Sun, Nov 20, 2011 at 03:34:27PM -0700, Martijn van Exel wrote: I also ran into trouble with files of around 40MB or larger. I still get the same errors though, either using array or sparsetable for node storage. I'm on Ubuntu 10.04 64 bit with 16GB RAM. Can you post the script you are using? Maybe we can find out if its doing something that triggers a bug. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 -- martijn van exel schaaltreinen.nl -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Overpass API: new big server, IPv6
Hello everybody, there are two new amazing news around Overpass API: I'm happy to announce a much more powerful (8-core, 64 GB RAM) machine, hosted voluntarily by Rambler. It offers the full interface of currently Overpass API 0.6.94, including meta data. Please have a try at http://overpass.osm.rambler.ru/query_form.html More information is on the wiki http://wiki.openstreetmap.org/wiki/Overpass_API http://wiki.openstreetmap.org/wiki/OSM_Servers_in_Rambler Second, the server at overpass-api.de now supports IPv6. Because I'm a complete newbie on IPv6 and I don't have IPv6 on my personal internet access, I've set up a second address http://ipv6.overpass-api.de/query_form.html pointing to the same server overpass-api.de/query_form.html . If it doesn't work, please contact me at my personal mail address. Happy access, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[Potlatch-dev] [OpenStreetMap] #4098: Wireframe style to show licence status
#4098: Wireframe style to show licence status -+-- Reporter: Socks| Owner: potlatch-dev@… Type: enhancement | Status: new Priority: minor| Milestone: Component: potlatch2| Version: Keywords: | -+-- Could the wireframe style show the licence-status colours if that option is selected? -- Ticket URL: https://trac.openstreetmap.org/ticket/4098 OpenStreetMap http://www.openstreetmap.org/ OpenStreetMap is a free editable map of the whole world ___ Potlatch-dev mailing list Potlatch-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/potlatch-dev
