Re: [osmosis-dev] Osmosis Java heap space
Hi Brett, that is what I typed and get back: C:\Users\u.stoertz\Downloads\s\osmosis-0.41\binosmosis --rx 1.osm.bz2 --tf acce pt-nodes addr:housenumber=* --tf reject-relations --un --wx 1_nodes.osm.bz2 25.10.2012 09:37:45 org.openstreetmap.osmosis.core.Osmosis run INFO: Osmosis Version 0.41 25.10.2012 09:37:46 org.java.plugin.registry.xml.ManifestParser init INFO: got SAX parser factory - org.apache.xerces.jaxp.SAXParserFactoryImpl@1be2d 65 25.10.2012 09:37:46 org.java.plugin.registry.xml.PluginRegistryImpl configure INFO: configured, stopOnError=false, isValidating=true 25.10.2012 09:37:47 org.java.plugin.registry.xml.PluginRegistryImpl register INFO: plug-in and fragment descriptors registered - 1 25.10.2012 09:37:47 org.java.plugin.standard.StandardPluginManager activatePlugi n INFO: plug-in started - org.openstreetmap.osmosis.core.plugin.Core@0.41.0 25.10.2012 09:37:47 org.openstreetmap.osmosis.core.Osmosis run INFO: Preparing pipeline. 25.10.2012 09:37:47 org.openstreetmap.osmosis.core.Osmosis run INFO: Launching pipeline execution. 25.10.2012 09:37:47 org.openstreetmap.osmosis.core.Osmosis run INFO: Pipeline executing, waiting for completion. 25.10.2012 09:59:46 org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskMan ager waitForCompletion SCHWERWIEGEND: Thread for task 1-rx failed java.lang.OutOfMemoryError: Java heap space at org.openstreetmap.osmosis.core.filter.common.ListIdTracker.extendIdLi st(ListIdTracker.java:71) at org.openstreetmap.osmosis.core.filter.common.ListIdTracker.set(ListId Tracker.java:122) at org.openstreetmap.osmosis.tagfilter.v0_6.UsedNodeFilter.process(UsedN odeFilter.java:106) at org.openstreetmap.osmosis.core.container.v0_6.WayContainer.process(Wa yContainer.java:58) at org.openstreetmap.osmosis.tagfilter.v0_6.UsedNodeFilter.process(UsedN odeFilter.java:75) at org.openstreetmap.osmosis.tagfilter.v0_6.TagFilter.process(TagFilter. java:141) at org.openstreetmap.osmosis.tagfilter.v0_6.TagFilter.process(TagFilter. java:141) at org.openstreetmap.osmosis.xml.v0_6.impl.WayElementProcessor.end(WayEl ementProcessor.java:117) at org.openstreetmap.osmosis.xml.v0_6.impl.OsmHandler.endElement(OsmHand ler.java:107) at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source ) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanEndElement( Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContent Dispatcher.dispatch(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Un known Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Sour ce) at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source) at javax.xml.parsers.SAXParser.parse(Unknown Source) at org.openstreetmap.osmosis.xml.v0_6.XmlReader.run(XmlReader.java:111) at java.lang.Thread.run(Unknown Source) 25.10.2012 09:59:46 org.openstreetmap.osmosis.core.Osmosis main SCHWERWIEGEND: Execution aborted. org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks failed . at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForComple tion(Pipeline.java:146) at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:92) at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Laun cher.java:329) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.jav a:239) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(La uncher.java:409) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java: 352) at org.codehaus.classworlds.Launcher.main(Launcher.java:47) C:\Users\u.stoertz\Downloads\s\osmosis-0.41\bin 2012/10/25 Brett Henderson br...@bretth.com Hi Uli, On 25 October 2012 01:56, Uli Strötz ustro...@gmail.com wrote: Hello, I keep getting the error Java heap space. I set the Java space in the osmosis.bat file to 2200MB with the following line: SET JAVACMD_OPTIONS=-Xmx2200m During the process the RAM does not exceedhttp://www.dict.cc/englisch-deutsch/exceed.html these 2200MB. Actually the RAM hard changes at all. The file itself is only of the size 200MB. The process I want to do is the
Re: [OSM-dev] ODbL full history planet
On Wed, Oct 24, 2012 at 05:51:49PM +0200, Peter Körner wrote: Am 23.10.2012 23:26, schrieb Pedro Larroy: Do you have a pointer to the pbf definition files? It's in the usual osm.pbf-File-Format: https://github.com/scrosby/OSM-binary The only difference is, that the visible-field is filled and there are more then one entity per id (one for each version). See: https://github.com/scrosby/OSM-binary/blob/master/src/osmformat.proto#L130 What about redacted versions of nodes without coordinates? Does that work properly? Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
On 23 October 2012 09:15, kimaidou kimai...@gmail.com wrote: Hi Paul, Thanks for pointing this out. I would say this file is the one you pointed : https://github.com/gravitystorm/openstreetmap-license-change/blob/master/check_history.rb It's actually a different file for loading history extracts into an apidb database: https://github.com/gravitystorm/openstreetmap-license-change/blob/master/extract_loader.rb There's a lot of stuff in there about license acceptance, which of course was more relevant to the original purpose of the script. Don't worry though, all the licensing stuff is only for calculating the agreed_time column in the users table - it doesn't affect the main output of nodes/ways/relations. As far as I'm aware, it's still the only code that can load .osh files into an apidb database. Cheers, Andy ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Hi, Jochen Topf schrieb: What about redacted versions of nodes without coordinates? can you please give an example Node ID for that? I mean, did the bot really only removed lat/lon of a node and not the entire version no. of a node? All the best, Pascal ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] OWL resurrection branch demo instance
Hi all, As I mentioned yesterday, I've been preparing a tool that would allow to analyze changes in OSM data. This work is based on OWL although it includes major alterations to the original code so it is not certain that it should be called OWL anymore (Matt's opinion pending)... Anyway, after I resurrected the database part I went to work on the owl_viewer component and this is what I have to show you today: http://suncobalt.dyndns.org:8082/map It is based on the original owl_viewer with couple of major changes: * There is no tiling of changes - it works based on the current viewport - what you see is what you get in the changeset list and RSS feed. * Full changeset geometry is shown for each changeset (marked in yellow). * You can click on a specific changeset row to zoom to its extent. * Right now it is limited to min zoom level 12 and shows max 100 last changesets at once because of server overload issues. * Changesets that contain only changes on relations or only removals currently don't have any geometry in the database thus will not be display on the map - it is on the TODO list. * The database contains changesets starting from 2012-10-11 11:59:01. * Data replication on the server is currently at 2012-10-12 16:58:57 and going (very) slowly so new changesets are appearing. This is mainly an initial experiment to see if current database schema scales to large number of changesets. Paweł ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
On Thu, Oct 25, 2012 at 10:30:50AM +0200, Pascal Neis wrote: Jochen Topf schrieb: What about redacted versions of nodes without coordinates? can you please give an example Node ID for that? I mean, did the bot really only removed lat/lon of a node and not the entire version no. of a node? I don't have an example. I just heard months back that this would happen when the redaction bot went through. I have never looked at the data to see how things were done. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Pascal Neis writes: Jochen Topf schrieb: What about redacted versions of nodes without coordinates? can you please give an example Node ID for that? I mean, did the bot really only removed lat/lon of a node and not the entire version no. of a node? for example here: http://www.openstreetmap.org/api/0.6/node/573785524/history entry does not contain lat/lon, also versions are missing. Was also discussed in this thread: http://lists.openstreetmap.org/pipermail/rebuild/2012-July/000299.html Can the software/osmium cope with that? what will end up in the pbf file? Stephan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OWL resurrection branch demo instance
Hi Paweł, * Full changeset geometry is shown for each changeset (marked in yellow). I'm working on a change viewer based on OpenLayers. It's currently aimed at visualizing Overpass API Augmented Diffs. But maybe parts of it could be reused for your project as well. I'll see if I can publish the current state by tomorrow. Norbert ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Hi, Looking at this file https://github.com/gravitystorm/openstreetmap-license-change/blob/master/extract_loader.rb It seems if we do not use the --file, --users-agreed nor --changesets-agreed options, this will import the whole history file into an apidb. This looks promising ! I am not a ruby guy, so I will surely need some time to understand how to install and run this file, but I will give it a try. What about performance ? Does it handle a big history file ? Michael 2012/10/25 Andy Allan gravityst...@gmail.com On 23 October 2012 09:15, kimaidou kimai...@gmail.com wrote: Hi Paul, Thanks for pointing this out. I would say this file is the one you pointed : https://github.com/gravitystorm/openstreetmap-license-change/blob/master/check_history.rb It's actually a different file for loading history extracts into an apidb database: https://github.com/gravitystorm/openstreetmap-license-change/blob/master/extract_loader.rb There's a lot of stuff in there about license acceptance, which of course was more relevant to the original purpose of the script. Don't worry though, all the licensing stuff is only for calculating the agreed_time column in the users table - it doesn't affect the main output of nodes/ways/relations. As far as I'm aware, it's still the only code that can load .osh files into an apidb database. Cheers, Andy ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[Potlatch-dev] [OpenStreetMap] #4650: Keystrokes in the 'Load vector file' dialog are not trapped
#4650: Keystrokes in the 'Load vector file' dialog are not trapped ---+ Reporter: smsm1 | Owner: potlatch-dev@… Type: defect | Status: new Priority: minor | Milestone: Component: potlatch2 |Version: Keywords: | ---+ * Open Potlatch 2 * In the background menu in the top right choose Vector file... at the bottom right. * Press s or h and it will run the actions that you wouldn't expect, e.g. saving or history. Other dialogs don't do this. I can reproduce on Safari 6.0.1 (8536.26.14), and on Chrome 22.0.1229.94 both with flash version: 11,4,402,287 -- Ticket URL: https://trac.openstreetmap.org/ticket/4650 OpenStreetMap http://www.openstreetmap.org/ OpenStreetMap is a free editable map of the whole world ___ Potlatch-dev mailing list potlatch-...@openstreetmap.org http://lists.openstreetmap.org/listinfo/potlatch-dev
Re: [OSM-dev] ODbL full history planet
On 25 October 2012 14:43, kimaidou kimai...@gmail.com wrote: I am not a ruby guy, so I will surely need some time to understand how to install and run this file, but I will give it a try. There is a readme, but in short use ruby 1.9 and run bundle install to install the dependencies. What about performance ? Does it handle a big history file ? It was used to load a whole-country extract (Ireland) before, but it's no where near as fast as other tools. I'd suggest creating a very small history extract first, and see how you get on. Cheers, Andy ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Am 25.10.2012 15:43, schrieb kimaidou: Hi, Looking at this file https://github.com/gravitystorm/openstreetmap-license-change/blob/master/extract_loader.rb It seems if we do not use the --file, --users-agreed nor --changesets-agreed options, this will import the whole history file into an apidb. This looks promising ! I am not a ruby guy, so I will surely need some time to understand how to install and run this file, but I will give it a try. What about performance ? Does it handle a big history file ? AFAIK it took multiple days to import Ireland (which indicates a year or so for the whole planet). Simon ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Thanks for your answers. I will try with a smal extract first. 2012/10/25 Simon Poole si...@poole.ch Am 25.10.2012 15:43, schrieb kimaidou: Hi, Looking at this file https://github.com/gravitystorm/openstreetmap-license-change/blob/master/extract_loader.rb It seems if we do not use the --file, --users-agreed nor --changesets-agreed options, this will import the whole history file into an apidb. This looks promising ! I am not a ruby guy, so I will surely need some time to understand how to install and run this file, but I will give it a try. What about performance ? Does it handle a big history file ? AFAIK it took multiple days to import Ireland (which indicates a year or so for the whole planet). Simon ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Me again : I managed to install ruby on ubuntu 12.04 with these commands : sudo apt-get install ruby1.9.1 libxslt-dev ruby1.9.1-dev sudo gem1.9.1 install text libxml-ruby mechanize algorithms # Then I downloaded the sources : cd /tmp/ wget https://github.com/gravitystorm/openstreetmap-license-change/zipball/master unzip gravitystorm-openstreetmap-license-change-2d01fab.zip cd /tmp/gravitystorm-openstreetmap-license-change-2d01fab cp example.auth.yaml auth.yaml Now I need a small history extract to perform tests. I have seen these places : * full planet : BIG file (too big for my small machine) : http://planet.openstreetmap.org/planet/full-history/ * Simon's extracts : available only unti redaction period, but there are some small files to try : http://odbl.poole.ch/extracts/ Am I missing some other URLs ? I will try asap with Andora : http://odbl.poole.ch/extracts/andorra.osh.bz2 ++ 2012/10/25 kimaidou kimai...@gmail.com Thanks for your answers. I will try with a smal extract first. 2012/10/25 Simon Poole si...@poole.ch Am 25.10.2012 15:43, schrieb kimaidou: Hi, Looking at this file https://github.com/gravitystorm/openstreetmap-license-change/blob/master/extract_loader.rb It seems if we do not use the --file, --users-agreed nor --changesets-agreed options, this will import the whole history file into an apidb. This looks promising ! I am not a ruby guy, so I will surely need some time to understand how to install and run this file, but I will give it a try. What about performance ? Does it handle a big history file ? AFAIK it took multiple days to import Ireland (which indicates a year or so for the whole planet). Simon ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [osmosis-dev] Geometries for the relations table of pgsnapshot
Routes definitely have a relation to geometry. It is meaningful to ask what the length of a route is. Statistically speaking, relations are multipolygons/boundaries, routes, restrictions, site and associatedStreet. For collection relations, there’s hardly any of them, generally they make sense as a GEOMETRYCOLLECTION, and many of them shouldn’t really be there as relations are not categories (http://wiki.openstreetmap.org/wiki/Relations/Relations_are_not_Categories) From: Кирилл Бестужев [mailto:bestou...@gmail.com] Sent: Thursday, October 25, 2012 1:45 AM To: Paweł Paprota Cc: osmosis-...@openstreetmap.org Subject: Re: [osmosis-dev] Geometries for the relations table of pgsnapshot What's the use of converting ALL relations into geometries? Much of them have no relation to geometry - they are collections, routes and so on!!! Kirill 2012/10/25 Paweł Paprota ppa...@fastmail.fm Hi Paul, I've been doing some similar work on aggregating way/node geometries and there are some questions that need to be answered when considering something like this: 1. What spatial type to store? OSM relations obviously can contain both ways and nodes so when you do a simple ST_Union/ST_Collect you will get a GEOMETRYCOLLECTION of LINESTRINGs and POINTs; GEOMETRYCOLLECTION is a bit problematic as it is not supported by many PostGIS ST_* functions. What I've done myself is I convert node points into closed linestrings with two points (X, X) thus when I aggregate node and way geometries I get a MULTILINESTRING which is much more useful than GEOMETRYCOLLECTION. 2. What to do about hierarchical relations (that is, relations that have other relations as members)? Should they be resolved down to nodes/ways? Paweł On 10/25/2012 02:37 AM, Paul Norman wrote: I keep a pgsnapshot database up to date on my home server for a local jxapi server and also for analysis. I have a linestring column built for the ways table, but there's no geometry for relations. I am wondering what would be involved with getting a geometry column built for the relations table. I expect this would slow down updates, but it would be invaluable to for analysis. I believe Ian Dees mentioned to me something about some work that had been done on this. Is there anything that I could do to help? I have minimal java knowledge and some postgis knowledge. ___ osmosis-dev mailing list osmosis-...@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-...@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-...@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [OSM-dev] ODbL full history planet
Hi, kimaidou schrieb: Now I need a small history extract to perform tests. I have seen these places : * full planet : BIG file (too big for my small machine) : http://planet.openstreetmap.org/planet/full-history/ * Simon's extracts : available only unti redaction period, but there are some small files to try : http://odbl.poole.ch/extracts/ Am I missing some other URLs ? You will find Peter's files here: http://osm.personalwerk.de/full-history-extracts/history_2012-10-13_13:35/europe/ and some small files here: http://osm.personalwerk.de/full-history-extracts/history_2012-10-13_13:35/europe/germany/ http://osm.personalwerk.de/full-history-extracts/history_2012-10-13_13:35/europe/germany/rheinland-pfalz/ All the best, Pascal ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Thanks a lot for the updated url. One small addition to the installation for ubuntu : I needed to add the pg gem package : sudo gem1.9.1 install text libxml-ruby mechanize algorithms pg Then download the andora.osh, untar it and run the importer via the command: ruby1.9.1 extract_loader.rb -f andorra.osh It took ~ 20 minutes for a 82Mo uncompressed file Now I will read some lines in the database to see the result, and report back here 2012/10/25 Pascal Neis pascal.n...@gmail.com Hi, You will find Peter's files here: http://osm.personalwerk.de/**full-history-extracts/history_** 2012-10-13_13:35/europe/http://osm.personalwerk.de/full-history-extracts/history_2012-10-13_13:35/europe/ and some small files here: http://osm.personalwerk.de/**full-history-extracts/history_** 2012-10-13_13:35/europe/**germany/http://osm.personalwerk.de/full-history-extracts/history_2012-10-13_13:35/europe/germany/ http://osm.personalwerk.de/**full-history-extracts/history_** 2012-10-13_13:35/europe/**germany/rheinland-pfalz/http://osm.personalwerk.de/full-history-extracts/history_2012-10-13_13:35/europe/germany/rheinland-pfalz/ All the best, Pascal ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Hi Peter, I would like to know if you could build a small extract for me (or point me to a tutorial to do so myself) . I need to run a test for the europe/france/provence-alpes-cote-d-azur.poly contained in the geofabrik zip here : http://download.geofabrik.de/clipbounds/clipbounds.tgz Or the bbox if easier : europe/germany/rheinland-pfalz.osh.pbf BBOX 4.303,43.847,5.375,44.493 Thanks in advance Michael 2012/10/22 Peter Körner osm-li...@mazdermind.de As always, if you need another extract (and don't want/can't to generate it yourself), drop me a line. Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Hi again peter. sorry for the noise, I just read your very complete tutorial here : https://github.com/MaZderMind/osm-history-splitter I think I would give it a try to split the France extract into a smaller extract. Michael 2012/10/25 kimaidou kimai...@gmail.com Hi Peter, I would like to know if you could build a small extract for me (or point me to a tutorial to do so myself) . I need to run a test for the europe/france/provence-alpes-cote-d-azur.poly contained in the geofabrik zip here : http://download.geofabrik.de/clipbounds/clipbounds.tgz Or the bbox if easier : europe/germany/rheinland-pfalz.osh.pbf BBOX4.303,43.847,5.375,44.493 Thanks in advance Michael 2012/10/22 Peter Körner osm-li...@mazdermind.de As always, if you need another extract (and don't want/can't to generate it yourself), drop me a line. Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[josm-dev] Exporting .OSM from JOSM without changeset information
Hi, I'm trying to create a test dataset from scratch using JOSM which I ultimately need to run through osmosis. However, the output of the JOSM export is a change set, not a proper OSM file. This means that the file is very large as it contains deletions and modifications etc. At the moment, I've having to import this data into Merkaartor and then outputting it again as Merkaartor produces a better output. This is a bit of a pain, though. Is there currently a way of exporting a .OSM file from JOSM without this change set information (i.e. just a list of nodes, ways, relations...)? Also, I apologise if this is the wrong place to ask this question! Thanks, Jon ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
