[OSM-dev] First version of long way splitter for gdalcontour output
Hi all, As you my know there is a great tool call gdalcontour that will help trace sat images. The problem with it, it produces a huge amount of data. I have been working on how to process this data. First I used shp2osm.pl to produce osm files from the output, but they are still huge. Then I used my modified version that splits into 100 way packages. http://xhema.flossk.org:8080/mapdata/02/mdupont/dlr/contours/shp2osm_split.pl Now today, I have rewarmed my c++ processing tool for high speed, single-pass osm processing, I have a program FindClosedWays (compiled version) http://xhema.flossk.org:8080/mapdata/03/EPANatReg/FindClosedWaysthat will emit only closed ways and only the ones above a certain length. That way you can remove alot of the junk. This can be used for lakefinding and also forest finding. The image data can be preprocessed with different tools to extract colors, or apply filtering. here is my blog post for it, http://osmopenlayers.blogspot.com/2010/03/first-version-of-way-cutter.html Code and compiled version is here : http://xhema.flossk.org:8080/mapdata/03/EPANatReg/?C=M;O=A The code is also checked into the bzr repo on launchpad. I hope to make more filter features, for example to split data on a keyfield into new files, or to be able to process I will also look into how my tools can fit into the ogr/gdal toolkit, I wonder why they dont deal with OSM data, or do they? there is the ogr2osm tool as well. Also we have this http://www.gnu.org/software/libredwg/ as a possible future datasource as well as my work on the dime/twonickets dxf2osm. And I have been working on the mapnik c++ code that is stalling atm. All of those programs work on structured geometry files. We should have a simple template callback system that can be adapted to all these tools to be able to process and emit structured data from them. Of course we can just use OSM as our file format, and that is what I am doing here. I am also working with skippern on brasilian cad files. We are in the negotiations on getting a huge load of cad files from albania at the moment. When that happens, we will need tools to splitup and extract features from them. Now my code works in a single pass, at least over the entire data. It builds up node indexes to remove duplicate, and processes the waypoints of each roads 3 times, one to read them and look for duplicates and one to emit the nodes. This can be optimized further. I have exploited the following optimisations : 1. the nodes are all declared before they are referenced. 2. the xml is not nested, its basically flat. I admit the code is not as clean as I would like it, but the speed and flexibility is promising, one area of the code that is a problem is the end node handling, there I introduced an enum to track in what node type I am in. this code needs to be refactored into the base class. The program is invoked like this : ./FindClosedWays INPUT LENGTH OUTPUT ./FindClosedWays inputways_2054100.osm 50 test_out_contour_ways_2054100.osm As I have talked about previously, there is a great need for some high speed c++ processing in osm. I hope to make these tools to be usable in a plug in processing schema. So you can activate certain features via including the templates for the algorithms and have the data shared. Anyway, I hope some of my ideas will be adopted in OSM some day. mike ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] First version of long way splitter for gdalcontour output
On Wed, Mar 10, 2010 at 8:54 PM, jamesmikedup...@googlemail.com jamesmikedup...@googlemail.com wrote: ... Now my code works in a single pass, at least over the entire data. It builds ... This can be optimized further. I have exploited the following optimisations ... I process the entire planet every week (+- 5 passes, +-12 counting compression and decompression). I guess I'll be able to cope if the planet grows by 40% p.a, but you make it sound if that is too slow for you. On a less predicting the future topic: Can you please point me to some of these monster forests or lakes you've uploaded, so that I can check that my software can handle them ? What about Potlatch ? ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] First version of long way splitter for gdalcontour output
I have not done any monster forests, It is just in the beginning. Feel free to try the program on a big file, I would be interested to know how it works. any bug reports will be processed asap. I have updated the blog post, at the bottom you will see two features i extracted and the source. I hope to have some more results soon, mike On Wed, Mar 10, 2010 at 9:21 PM, Nic Roets nro...@gmail.com wrote: On Wed, Mar 10, 2010 at 8:54 PM, jamesmikedup...@googlemail.com jamesmikedup...@googlemail.com wrote: ... Now my code works in a single pass, at least over the entire data. It builds ... This can be optimized further. I have exploited the following optimisations ... I process the entire planet every week (+- 5 passes, +-12 counting compression and decompression). I guess I'll be able to cope if the planet grows by 40% p.a, but you make it sound if that is too slow for you. On a less predicting the future topic: Can you please point me to some of these monster forests or lakes you've uploaded, so that I can check that my software can handle them ? What about Potlatch ? ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] First version of long way splitter for gdalcontour output
James, This sounds interesting - to make sure I understand are you saying that you can use gdalcontour to process image files and extract boundaries from itlike forests from satellite images? I had never thought of doing that - what a good idea! I thought it would only have worked with monochrome images. I would really like this because I am keen to add forests to the map to help with outdoor navigation, but my eyes are not up to tracing them from satellite images. Is this something that could be automated into an editor to highlight areas you might want to trace around? Regards Graham. On 10 March 2010 20:25, jamesmikedup...@googlemail.com jamesmikedup...@googlemail.com wrote: I have not done any monster forests, It is just in the beginning. Feel free to try the program on a big file, I would be interested to know how it works. any bug reports will be processed asap. I have updated the blog post, at the bottom you will see two features i extracted and the source. I hope to have some more results soon, mike On Wed, Mar 10, 2010 at 9:21 PM, Nic Roets nro...@gmail.com wrote: On Wed, Mar 10, 2010 at 8:54 PM, jamesmikedup...@googlemail.com jamesmikedup...@googlemail.com wrote: ... Now my code works in a single pass, at least over the entire data. It builds ... This can be optimized further. I have exploited the following optimisations ... I process the entire planet every week (+- 5 passes, +-12 counting compression and decompression). I guess I'll be able to cope if the planet grows by 40% p.a, but you make it sound if that is too slow for you. On a less predicting the future topic: Can you please point me to some of these monster forests or lakes you've uploaded, so that I can check that my software can handle them ? What about Potlatch ? ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev -- Dr. Graham Jones Hartlepool, UK email: grahamjones...@gmail.com ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] First version of long way splitter for gdalcontour output
yes, you can also process the images beforehand to increase the contrast. I have been experimenting with that, using gimp , imagemagick and some of the gdal tools. Will be posting updates when I have more info. Yes I think this could be put into an editor. It could even be made an online service. This method extracts areas of the same color, so you can see from the radarsat that it can extract the purple areas. the only problem is the CPU, it uses alot, gdalcontour and also produces huge files... but we will get there some day. mike On Wed, Mar 10, 2010 at 10:19 PM, Graham Jones grahamjones...@googlemail.com wrote: James, This sounds interesting - to make sure I understand are you saying that you can use gdalcontour to process image files and extract boundaries from itlike forests from satellite images? I had never thought of doing that - what a good idea! I thought it would only have worked with monochrome images. I would really like this because I am keen to add forests to the map to help with outdoor navigation, but my eyes are not up to tracing them from satellite images. Is this something that could be automated into an editor to highlight areas you might want to trace around? Regards Graham. On 10 March 2010 20:25, jamesmikedup...@googlemail.com jamesmikedup...@googlemail.com wrote: I have not done any monster forests, It is just in the beginning. Feel free to try the program on a big file, I would be interested to know how it works. any bug reports will be processed asap. I have updated the blog post, at the bottom you will see two features i extracted and the source. I hope to have some more results soon, mike On Wed, Mar 10, 2010 at 9:21 PM, Nic Roets nro...@gmail.com wrote: On Wed, Mar 10, 2010 at 8:54 PM, jamesmikedup...@googlemail.com jamesmikedup...@googlemail.com wrote: ... Now my code works in a single pass, at least over the entire data. It builds ... This can be optimized further. I have exploited the following optimisations ... I process the entire planet every week (+- 5 passes, +-12 counting compression and decompression). I guess I'll be able to cope if the planet grows by 40% p.a, but you make it sound if that is too slow for you. On a less predicting the future topic: Can you please point me to some of these monster forests or lakes you've uploaded, so that I can check that my software can handle them ? What about Potlatch ? ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev -- Dr. Graham Jones Hartlepool, UK email: grahamjones...@gmail.com ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] First version of long way splitter for gdalcontour output
Ok Mike, I've look at it. It's in an urban area so chances are that it will be useful to someone. But I'm still a bit worried that there is too much detail there, considering what is being imported. One of those segments is shorter than 4m. It's not like a building, where you know someone will be happy if they see their house, or a road where it can influence routing. And you haven't documented your tag (at least not in the obvious place): http://wiki.openstreetmap.org/wiki/Tag:natural%3Dpurple_floodzone http://wiki.openstreetmap.org/wiki/Tag:natural=floodzone http://wiki.openstreetmap.org/wiki/Key:natural Regards, Nic On Wed, Mar 10, 2010 at 10:25 PM, jamesmikedup...@googlemail.com jamesmikedup...@googlemail.com wrote: I have not done any monster forests, It is just in the beginning. Feel free to try the program on a big file, I would be interested to know how it works. any bug reports will be processed asap. I have updated the blog post, at the bottom you will see two features i extracted and the source. I hope to have some more results soon, mike On Wed, Mar 10, 2010 at 9:21 PM, Nic Roets nro...@gmail.com wrote: On Wed, Mar 10, 2010 at 8:54 PM, jamesmikedup...@googlemail.com jamesmikedup...@googlemail.com wrote: ... Now my code works in a single pass, at least over the entire data. It builds ... This can be optimized further. I have exploited the following optimisations ... I process the entire planet every week (+- 5 passes, +-12 counting compression and decompression). I guess I'll be able to cope if the planet grows by 40% p.a, but you make it sound if that is too slow for you. On a less predicting the future topic: Can you please point me to some of these monster forests or lakes you've uploaded, so that I can check that my software can handle them ? What about Potlatch ? ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] First version of long way splitter for gdalcontour output
yes, we are just in the process of creating tagging guidelines. setting up the server. There is alot of tagcleanup to be done. We are setting up a mapfish server , and later will have a mapnik and a railsport. I think we can pull all the floodzone data into a shapefile when we are finished, it wont be needed. But a lot of the flooded houses were built illegally, and in fact the fact that the water floods that area easily could be a feature of the map. For the urban planning project I am trying to support in dragash, the southern tip of kosovo, next to albania, they are also interested in doing civil planning using some osm data and other tools. I dont know enough about it, but maybe it would be interesting to model the ground contours enough to model where the water can raise to. We also have the DEM and STRM data for that area from bekim from immap, and still need to process that. We have also more shapefiles with the rivers and streams. http://wiki.openstreetmap.org/wiki/AlbanianFloodingCrisisCamp/Layers see the rivers section. We need to create tagging guidelines for all those areas in kosovo and albania we have been working on. I am painfully aware of the shortcomings, and hope that you will have patience. mike On Wed, Mar 10, 2010 at 10:35 PM, Nic Roets nro...@gmail.com wrote: Ok Mike, I've look at it. It's in an urban area so chances are that it will be useful to someone. But I'm still a bit worried that there is too much detail there, considering what is being imported. One of those segments is shorter than 4m. It's not like a building, where you know someone will be happy if they see their house, or a road where it can influence routing. And you haven't documented your tag (at least not in the obvious place): http://wiki.openstreetmap.org/wiki/Tag:natural%3Dpurple_floodzone http://wiki.openstreetmap.org/wiki/Tag:natural=floodzone http://wiki.openstreetmap.org/wiki/Key:natural Regards, Nic On Wed, Mar 10, 2010 at 10:25 PM, jamesmikedup...@googlemail.com jamesmikedup...@googlemail.com wrote: I have not done any monster forests, It is just in the beginning. Feel free to try the program on a big file, I would be interested to know how it works. any bug reports will be processed asap. I have updated the blog post, at the bottom you will see two features i extracted and the source. I hope to have some more results soon, mike On Wed, Mar 10, 2010 at 9:21 PM, Nic Roets nro...@gmail.com wrote: On Wed, Mar 10, 2010 at 8:54 PM, jamesmikedup...@googlemail.com jamesmikedup...@googlemail.com wrote: ... Now my code works in a single pass, at least over the entire data. It builds ... This can be optimized further. I have exploited the following optimisations ... I process the entire planet every week (+- 5 passes, +-12 counting compression and decompression). I guess I'll be able to cope if the planet grows by 40% p.a, but you make it sound if that is too slow for you. On a less predicting the future topic: Can you please point me to some of these monster forests or lakes you've uploaded, so that I can check that my software can handle them ? What about Potlatch ? ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
