[Talk-ca] Correcting Geobase_import_2009
Looking at my local map in Orleans Ontario noticed that Merkley Drive does not connect up to Charlemagne. Merkley Drive is tagged Geobase_import_2009 and all sorts of interesting things. Charlemagne is just tagged potlach and residential road. Is there an easy way to extend Merkley Drive so it does connect? ie add another node but copy the attributes of the existing road? Should I simply hold off until I know that the Geobase_import_2009 has been completed? Is there a way to submit a GPS trace into Geobase to get their base map corrected? Thanks John ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca
Re: [Talk-ca] Correcting Geobase_import_2009
Hi, just use potlatch connect the roads. No harm done :-) The OSM changeset history keeps track of attribution and such. If you want a copy of where the geobase version of that road is, i can provide that. The system that imported the roads didnt want to interfeer with what work previous people did. Cheers, Sam On 10/25/09, john whelan jwhelan0...@gmail.com wrote: Looking at my local map in Orleans Ontario noticed that Merkley Drive does not connect up to Charlemagne. Merkley Drive is tagged Geobase_import_2009 and all sorts of interesting things. Charlemagne is just tagged potlach and residential road. Is there an easy way to extend Merkley Drive so it does connect? ie add another node but copy the attributes of the existing road? Should I simply hold off until I know that the Geobase_import_2009 has been completed? Is there a way to submit a GPS trace into Geobase to get their base map corrected? Thanks John ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca -- Twitter: @Acrosscanada Blog: http://Acrosscanadatrails.blogspot.com Facebook: http://www.facebook.com/sam.vekemans OpenStreetMap IRC: http://irc.openstreetmap.org @Acrosscanadatrails ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca
Re: [Talk-ca] Correcting Geobase_import_2009
Yes I'm new and my background is red tape. I understand there is a lot going on and I appreciate the work that has been done. My background is in databases etc. If you ask clients what they want they always rate reliability above anything else. To me the data from geobase is good high quality data with input from multiple sources including municipal governments. Talk to them nicely and we can get all sorts of high quality things like bus stops, speed limits etc direct from the municipalities. What appears to have happened is where the data has been merged roads that are in the geobase database no longer connect to roads that have been put in via potlatch. I think the end point is dropped. The older potlatch roads do not have the same depth of tags as the geobase data. Not only that but roads that run close to the old roads appear to be deleted for the section close to the potlatch inputted roads. Oh and my WAAS enabled GPS thinks at least one Potlatch inputted road is in the wrong place. My personal view is that we should go with the geobase database and see if we can layer on tags etc. That way we can update the database from time to time by replacing the entire geobase data layer. I think OSM can add value in Canada basically by tagging and enhancing data already there, if a client can get geobase data elsewhere that actually has connecting roads and complete roads they may prefer that to OSM where the data quality isn't so consistent. Oh and I prefer JOSM to potlatch for some reason. Thoughts? Cheerio John 2009/10/25 Sam Vekemans acrosscanadatra...@gmail.com: Hi, just use potlatch connect the roads. No harm done :-) The OSM changeset history keeps track of attribution and such. If you want a copy of where the geobase version of that road is, i can provide that. The system that imported the roads didnt want to interfeer with what work previous people did. Cheers, Sam On 10/25/09, john whelan jwhelan0...@gmail.com wrote: Looking at my local map in Orleans Ontario noticed that Merkley Drive does not connect up to Charlemagne. Merkley Drive is tagged Geobase_import_2009 and all sorts of interesting things. Charlemagne is just tagged potlach and residential road. Is there an easy way to extend Merkley Drive so it does connect? ie add another node but copy the attributes of the existing road? Should I simply hold off until I know that the Geobase_import_2009 has been completed? Is there a way to submit a GPS trace into Geobase to get their base map corrected? Thanks John ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca -- Twitter: @Acrosscanada Blog: http://Acrosscanadatrails.blogspot.com Facebook: http://www.facebook.com/sam.vekemans OpenStreetMap IRC: http://irc.openstreetmap.org @Acrosscanadatrails ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca
Re: [Talk-ca] Correcting Geobase_import_2009
On Sun, 25 Oct 2009, john whelan wrote: What appears to have happened is where the data has been merged roads that are in the geobase database no longer connect to roads that have been put in via potlatch. I think the end point is dropped. The older potlatch roads do not have the same depth of tags as the geobase data. Not only that but roads that run close to the old roads appear to be deleted for the section close to the potlatch inputted roads. Oh and my WAAS enabled GPS thinks at least one Potlatch inputted road is in the wrong place. Yes, in places where a road already existed in OSM the software we used to avoid duplicating the road with the geobase version sometimes results in missing the connecting segment or other segments of the geobase road close to an existing OSM road. If you see places like this it is best to just fix things up. Steve ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca
Re: [Talk-ca] Correcting Geobase_import_2009
On Sun, Oct 25, 2009 at 10:43 AM, john whelan jwhelan0...@gmail.com wrote: What appears to have happened is where the data has been merged roads that are in the geobase database no longer connect to roads that have been put in via potlatch. I think the end point is dropped. There is a discontinuity between the GeoBase data, and existing OSM data. The script that was used to determine which Geobase data to import was designed to stop short of the OSM data. Since the OSM data does not match exactly to the GeoBase data, it was impossible to have the two seamlessly merge. It is up to the OSM users to stitch the two together. The GeoBase data allows the OSM community to leverage the massive amount of data available in the GeoBase database. The older potlatch roads do not have the same depth of tags as the geobase data. This sounds like you believe that the quantity of tags associated with a way some how infers higher quality data. One can not automatically assume that the GeoBase data is of higher quality than the OSM data simply because there are more tags associated. In some cases, the positional accuracy of the GeoBase data is better than OSM data, but in other instances, the OSM data positional accuracy is better than GeoBase. It all depends upon the amount of effort expended during the input process. Not only that but roads that run close to the old roads appear to be deleted for the section close to the potlatch inputted roads. That's the GeoBase input script trying to match GeoBase roads to existing OSM roads. When they are close enough in geometry, the GeoBase roads get dropped. Preference is given to OSM data over GeoBase because the OSM data has been provided by the OSM community. Real people have invested many hours of their time to input their data. GeoBase data is being automatically input by a machine. We need to respect the OSM community investment over the automated imports. Oh and my WAAS enabled GPS thinks at least one Potlatch inputted road is in the wrong place. Yes, that can be very true. Some roads have been input by simply tracing low resolution imagery, and the resultant roads can be fairly crude. As an OSM participant, you can help improve the database by converting your GPS traces to roads. You can also use the real intelligence built into the human brain to make informed decisions about how to merge the OSM and GeoBase data into the best map database possible. My personal view is that we should go with the geobase database and see if we can layer on tags etc. That way we can update the database from time to time by replacing the entire geobase data layer. Of course you are entitled to your opinion, but options were discussed in this forum, and decisions made on how best to go about merging the OSM and GeoBase data. From those decisions, the scripts were created, and import activities started. I think you will find very little support for keeping the GeoBase data as a pristine import layer that can be changed out en masse. This would limit the amount of mapping that the OSM community could do in Canada. We would be limited to only adding data that is not maintained in the GeoBase database. There would be no ability to correct or modify the GeoBase road database as any changes made would be wiped out by the next en masse GeoBase layer import. The reason for the GeoBase import is to simply add a large amount of fairly accurate road data to an area of low population density. in Canada, we have a huge road network, and very few people to import that data. By importing GeoBase data provided by the government, we get a large amount of data, but we still have the capability of modifying that data. I think OSM can add value in Canada basically by tagging and enhancing data already there, if a client can get geobase data elsewhere that actually has connecting roads and complete roads they may prefer that to OSM where the data quality isn't so consistent. Here you are absolutely correct. OSM is all about providing the best FREE map data for the world. Anyone is free to use the OSM data, or they can go elsewhere to find the data they require. However, it is not possible to upload changes, corrections and additions to the GeoBase data as an end user. It is possible to do these things as an OSM end user. The GeoBase import was simply a shortcut implemented to get a large quantity of data into the database, and now it is up to the OSM community to manipulate that data as they see fit. James VE6SRV ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca
Re: [Talk-ca] Correcting Geobase_import_2009
OK accepting what you say is there a way to identify where an old OSM road was so that some one can go back and clean up the new geobase added data? Connect the roads and insert road sections that have been deleted? I think Toronto organised something that recognised the quality of the data by marking it verified etc. On the clean up side is there an easy way to copy the tags on one section of road onto another? For example when I extend a geobase road up to the old OSM road I'd like to have the same tags as the other sections of the road. At a quick glance I can see at least seven sections that need to be added back in within 600 meters. My personal view is the amount of effort to clean up the data required is probably often going to be greater than the amount of effort put into creating the OSM map in the first place. On the tag issue, its not simply a matter of the quantity of tags means higher quality data but if the tag doesn't have a value then the information doesn't exist. For example Merkley Drive geobase import has a tag saying 2 lanes. Charlemange has four lanes but no tag giving that value in the potlatch input. I've learnt over the years with databases that some idiot somewhere will invariably want to use a tag like this and not realise that some roads don't have a value. It is unfortunate that the geobase tag data can't be dragged over and associated with the OSM roads. Is there a geobase identifier on a road that could be added manually as a tag so that a script could drag in the rest of the tag values? This would probably mean having a pure geobase OSM map somewhere that could be used to pick out these values. Thanks Cheerio John James Ewen wrote: On Sun, Oct 25, 2009 at 10:43 AM, john whelan jwhelan0...@gmail.com wrote: What appears to have happened is where the data has been merged roads that are in the geobase database no longer connect to roads that have been put in via potlatch. I think the end point is dropped. There is a discontinuity between the GeoBase data, and existing OSM data. The script that was used to determine which Geobase data to import was designed to stop short of the OSM data. Since the OSM data does not match exactly to the GeoBase data, it was impossible to have the two seamlessly merge. It is up to the OSM users to stitch the two together. The GeoBase data allows the OSM community to leverage the massive amount of data available in the GeoBase database. The older potlatch roads do not have the same depth of tags as the geobase data. This sounds like you believe that the quantity of tags associated with a way some how infers higher quality data. One can not automatically assume that the GeoBase data is of higher quality than the OSM data simply because there are more tags associated. In some cases, the positional accuracy of the GeoBase data is better than OSM data, but in other instances, the OSM data positional accuracy is better than GeoBase. It all depends upon the amount of effort expended during the input process. Not only that but roads that run close to the old roads appear to be deleted for the section close to the potlatch inputted roads. That's the GeoBase input script trying to match GeoBase roads to existing OSM roads. When they are close enough in geometry, the GeoBase roads get dropped. Preference is given to OSM data over GeoBase because the OSM data has been provided by the OSM community. Real people have invested many hours of their time to input their data. GeoBase data is being automatically input by a machine. We need to respect the OSM community investment over the automated imports. Oh and my WAAS enabled GPS thinks at least one Potlatch inputted road is in the wrong place. Yes, that can be very true. Some roads have been input by simply tracing low resolution imagery, and the resultant roads can be fairly crude. As an OSM participant, you can help improve the database by converting your GPS traces to roads. You can also use the "real intelligence" built into the human brain to make informed decisions about how to merge the OSM and GeoBase data into the best map database possible. My personal view is that we should go with the geobase database and see if we can layer on tags etc. That way we can update the database from time to time by replacing the entire geobase data layer. Of course you are entitled to your opinion, but options were discussed in this forum, and decisions made on how best to go about merging the OSM and GeoBase data. From those decisions, the scripts were created, and import activities started. I think you will find very little support for keeping the GeoBase data as a pristine import layer that can be changed out en masse. This would limit the amount of mapping that the OSM community could do in Canada. We would be limited to only adding data that is not maintained in the GeoBase database. There would
Re: [Talk-ca] canvec / shp-to-osm
Hi Sam, I've just downloaded some CanVec data, and had a look at sheets 031I07 and -08. I wonder what you mean by uploading all sub-residential files. I understand that the data is separated over multiple files, because of certain limitations. In the residential OSM files I also see no polygons with a multipolygon relationship of outer. So,this means that the outlines of places like Trois-Rivieres and others are missing. The same issue is going on with wooded areas. The data is converted with Canvec2OSM version 0.9.4. I had a closer look at the raster file (from Toporama) of sheet 031I08, because there is much less data, and I looked at the village of Gentilly (see [1]). This is in the center of the sheet. The raster file suggests that a multipolygon relationship should be in place, but the vector file (BS_1370009_2_Residential_area0.osm) shows only the two inner polygons. Are the outer polygons stored in a different file, or are they not converted at all? The shape of the outer polygon doesn't look to be complex, so I don't think the max_nodes threshold would be exceeded. Looking at the OSM file: there is only one multipolygon relationship in it, but it only refers to the two inner polygons, and not to any outer polygon at all. One note regarding multipolygons: the inner polygons shouldn't have any tags at all. See [2]. Anyways, some clarifications about what is going on, and how the data should be interpreted would be welcome. I'm reluctant to import data which looks not correct. For the rest, keep up your good work :) Regards, Frank [1] http://osm.org/go/cKHX9ApT- [2] http://wiki.openstreetmap.org/wiki/Multipolygon Sam Vekemans wrote: Hi Richard, i think your refering to the large multi-polygons such as 'residential_area', and it 'appears' to be inverted. Here's the majic; when all the sub- residential.osm files are uploaded to OSM, it renders correctly. In JOSM, you need to zoom out and load the area, to see it. I think i'll load a region of NFLD in the next cuple days to test my hypothises. Sam ps. I cc'd talk-ca as this was mentioned b4. On 9/22/09, Richard Weait rich...@weait.com wrote: Dear gentlemen, I've had a look at some of Sam's test areas. In 1435 files there are zero occurrences of Relation=outer. So at some point we started calling relation=outer, relation=inner or completely dropping outer relations by mistake. I do still see rare nested ways, but both are marked as inner, and are on separate layers after --maxnodes I've run 0.6.1 again with an old rules file and see the same problem so I believe that this is an issue in shp-to-osm. Ian can you check a 0.5.0 - generated file and see if it contains any outer? Best regards, Richard ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca
Re: [Talk-ca] Correcting Geobase_import_2009
On Sun, Oct 25, 2009 at 1:39 PM, John Whelan jwhelan0...@gmail.com wrote: OK accepting what you say is there a way to identify where an old OSM road was so that some one can go back and clean up the new geobase added data? Actually it's more like the opposite. The old OSM road gets priority, an OSM road will not get deleted in favour of a GeoBase way. The scripts are designed to not import any GeoBase data that may interfere with existing OSM data. On the clean up side is there an easy way to copy the tags on one section of road onto another? For example when I extend a geobase road up to the old OSM road I'd like to have the same tags as the other sections of the road. In Potlatch, you select an existing way, then select the new way, and press 'R' this repeats the tags from the first way to the second way. However, you don't want to copy all the tags from a GeoBase way, as the new way would not be attributed to GeoBase, since you are creating it. You also should not copy the GeoBase UUID to the new way as it is a different segment that would have a different GeoBase UUID. At a quick glance I can see at least seven sections that need to be added back in within 600 meters. In an area where a single road has been mapped through a neighborhood, every intersecting road will need to be attached. My personal view is the amount of effort to clean up the data required is probably often going to be greater than the amount of effort put into creating the OSM map in the first place. If you are looking at a very small area, it might be easy to make that assumption. Let's think about it though. Imagine a main roadway running past a horseshoe shaped side road. The main road is in the OSM database, while the side road is imported from GeoBase. Now all that has to be done, is to connect the GeoBase way to the OSM way. If you don't use the GeoBase data, you'll need to physically collect the path of the side road with a GPS, import that into the OSM database, and convert it to a way. You'll also need to collect the rest of the physical attributes for the roadway. Yes, some areas have high resolution imagery, but most of Canada is only covered with very low resolution imagery. Now think about having to go out and collect that GPS data for millions of miles of roadway across Canada. I spent over $300 dollars on fuel, and 3 days driving just a portion of a few highways into Northern BC, capturing that data on the GPS. I added this data to the OSM database. There's no one else working on roads in the area, so it's up to just a handful of people to try and map the millions of miles of roads in this country. I tracked thousands of kilometres of highway in the year before GeoBase became available, adding it to the database, or replacing low resolution imagery tracings. Those highways still exist in the OSM database, describing the route of the highways in higher resolution than the GeoBase data does. I am grateful that the time and effort invested in tracking and importing that data was not thrown out in favour of an automated GeoBase import. The GeoBase data gives us just about every road in Canada at a very low cost. In areas where there are OSM users, the data can easily be modified, removed or replaced, just as it can for any other type of OSM data. On the tag issue, its not simply a matter of the quantity of tags means higher quality data but if the tag doesn't have a value then the information doesn't exist. For example Merkley Drive geobase import has a tag saying 2 lanes. Charlemange has four lanes but no tag giving that value in the potlatch input. I've learnt over the years with databases that some idiot somewhere will invariably want to use a tag like this and not realise that some roads don't have a value. It is unfortunate that the geobase tag data can't be dragged over and associated with the OSM roads. It is fortunate that ANY user can add a tag to any OSM road. The O in OSM stands for Open, meaning that anyone can add tags to the ways in the database. If you want Charlemange to have a tag that says lanes=4, get in there and add it. Is there a geobase identifier on a road that could be added manually as a tag so that a script could drag in the rest of the tag values? This would probably mean having a pure geobase OSM map somewhere that could be used to pick out these values. You would like to have some type of tag that says Import please? When the import scripts are run, they convert the GeoBase data into OSM compatible data. They can be used to create 3 different types of data. The usual is a database of GeoBase roads that are NOT included in the OSM data (as determined by the roadmatcher script). They can also create the inverse, a database of roads that are duplicates of roads in the OSM data. They can also simply create a full GeoBase database compatible with OSM. You are probably talking about having a layer that is the full GeoBase database, so that you can see
Re: [Talk-ca] Correcting Geobase_import_2009
In JOSM, another way to quickly add on to a way like that is to use the select tool to select the way you will be adding on to, holding the shift (or control) key, selecting the last node in the way, then use the draw (add) tool to continue drawing the way. By default, in JOSM, if you have a node selected that only belongs to one way, and you start using the draw tool, that way will be extended, including tags. However, if a node belongs to multiple ways (an intersection), then a brand new way will be created (since it doesn't know which of the ways to extend), and you have to do a combine operation. So selecting a node and a way together tells JOSM which of the ways you want to extend. To copy tags from one way to another, you can use the paste tags operation. Select the way to take the tags from, and use the copy operation (ctrl-c). At this point you have the choice of pasting the entire way (ctrl-v), or just pasting the tags (ctrl-shift-v) into an existing way. When you paste tags it will overwrite tags with new values if they already exist, but will leave tags intact if there is no new tag value coming in (there is no dialog to merge, at least on version 2300). One unfortunate aspect of doing this with GeoBase is that GeoBase assigns a new uuid to a way separated by an intersection (ie. a road will be split into separate ways ever time there is an intersection, and each of those ways gets a unique uuid) I've learnt over the years with databases that some idiot somewhere will invariably want to use a tag like this and not realise that some roads don't have a value. If some idiot is using the OSM data in a manner that assumes some tag exists, then they won't get very far at all. OSM allows anyone to use whatever tags they want, by allowing any combination of keys and values. All tags that are used in OSM are merely *suggestions* and are not mandatory. It would be foolish to assume even the simple such as highway=tertiary will exist on tertiary roads. Having said that, it would be nice to have as much information as possible taken in from government sources. That is why Sam V is advocating making .osm files available for people to obtain, so that people without postgis/sql/roadmatcher experience can still use josm to copy over the stuff that they want. There has been much discussion on how to make this as automatic as possible, but uuid can only do so much Anybody can get their hands on the GeoBase data and start copying tags over if they want. The Roadmatcher Excluded files listed in the import spreadsheet represent the roads that were already existing in OSM. It looks like James beat me to the reply, but hopefully this also helps. Adam On Sun, Oct 25, 2009 at 12:51 PM, john whelan jwhelan0...@gmail.com wrote: Looks like I can do an extend and combine in JOSM. Thanks John On the clean up side is there an easy way to copy the tags on one section of road onto another? For example when I extend a geobase road up to the old OSM road I'd like to have the same tags as the other sections of the road. At a quick glance I can see at least seven sections that need to be added back in within 600 meters. ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca
Re: [Talk-ca] canvec / shp-to-osm
Hi, I think to know what is going on. I've tried to convert the residential areas of 031I08 myself, and I got an OSM file with an outer polygon. However, the outer polygon has no tags. Also, it looks that Sam's batch files run shp-to-osm with the -t parameter, which suppresses the output of features without any tags. Solution: * shp-to-osm needs to be adjusted, so that the outer polygon will get the tags, but the inner polygons will not. * shp-to-osm should be called without the -t parameter. Is this possible? Frank Frank Steggink wrote: Hi Sam, I've just downloaded some CanVec data, and had a look at sheets 031I07 and -08. I wonder what you mean by uploading all sub-residential files. I understand that the data is separated over multiple files, because of certain limitations. In the residential OSM files I also see no polygons with a multipolygon relationship of outer. So,this means that the outlines of places like Trois-Rivieres and others are missing. The same issue is going on with wooded areas. The data is converted with Canvec2OSM version 0.9.4. I had a closer look at the raster file (from Toporama) of sheet 031I08, because there is much less data, and I looked at the village of Gentilly (see [1]). This is in the center of the sheet. The raster file suggests that a multipolygon relationship should be in place, but the vector file (BS_1370009_2_Residential_area0.osm) shows only the two inner polygons. Are the outer polygons stored in a different file, or are they not converted at all? The shape of the outer polygon doesn't look to be complex, so I don't think the max_nodes threshold would be exceeded. Looking at the OSM file: there is only one multipolygon relationship in it, but it only refers to the two inner polygons, and not to any outer polygon at all. One note regarding multipolygons: the inner polygons shouldn't have any tags at all. See [2]. Anyways, some clarifications about what is going on, and how the data should be interpreted would be welcome. I'm reluctant to import data which looks not correct. For the rest, keep up your good work :) Regards, Frank [1] http://osm.org/go/cKHX9ApT- [2] http://wiki.openstreetmap.org/wiki/Multipolygon Sam Vekemans wrote: Hi Richard, i think your refering to the large multi-polygons such as 'residential_area', and it 'appears' to be inverted. Here's the majic; when all the sub- residential.osm files are uploaded to OSM, it renders correctly. In JOSM, you need to zoom out and load the area, to see it. I think i'll load a region of NFLD in the next cuple days to test my hypothises. Sam ps. I cc'd talk-ca as this was mentioned b4. On 9/22/09, Richard Weait rich...@weait.com wrote: Dear gentlemen, I've had a look at some of Sam's test areas. In 1435 files there are zero occurrences of Relation=outer. So at some point we started calling relation=outer, relation=inner or completely dropping outer relations by mistake. I do still see rare nested ways, but both are marked as inner, and are on separate layers after --maxnodes I've run 0.6.1 again with an old rules file and see the same problem so I believe that this is an issue in shp-to-osm. Ian can you check a 0.5.0 - generated file and see if it contains any outer? Best regards, Richard ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca
Re: [Talk-ca] Correcting Geobase_import_2009
I found both James's and Sam's comments very useful. It gives me a much clearer idea of what you are trying to do and the limitations involved. It would appear that we have the ability to identify roads omitted from the geobase import which means at some point in time given enough resources a clean up could be done. Until then as you say areas where one road is already in OSM that have a number of residential roads connected to it can be cleaned up on an if spotted basis. So be it. Thanks Cheerio John Adam Dunn wrote: In JOSM, another way to quickly add on to a way like that is to use the select tool to select the way you will be adding on to, holding the shift (or control) key, selecting the last node in the way, then use the draw (add) tool to continue drawing the way. By default, in JOSM, if you have a node selected that only belongs to one way, and you start using the draw tool, that way will be extended, including tags. However, if a node belongs to multiple ways (an intersection), then a brand new way will be created (since it doesn't know which of the ways to extend), and you have to do a combine operation. So selecting a node and a way together tells JOSM which of the ways you want to extend. To copy tags from one way to another, you can use the "paste tags" operation. Select the way to take the tags from, and use the copy operation (ctrl-c). At this point you have the choice of pasting the entire way (ctrl-v), or just pasting the tags (ctrl-shift-v) into an existing way. When you paste tags it will overwrite tags with new values if they already exist, but will leave tags intact if there is no new tag value coming in (there is no dialog to merge, at least on version 2300). One unfortunate aspect of doing this with GeoBase is that GeoBase assigns a new uuid to a way separated by an intersection (ie. a road will be split into separate ways ever time there is an intersection, and each of those ways gets a unique uuid) I've learnt over the years with databases that some idiot somewhere will invariably want to use a tag like this and not realise that some roads don't have a value. If "some idiot" is using the OSM data in a manner that assumes some tag exists, then they won't get very far at all. OSM allows anyone to use whatever tags they want, by allowing any combination of keys and values. All tags that are used in OSM are merely *suggestions* and are not mandatory. It would be foolish to assume even the simple such as highway=tertiary will exist on tertiary roads. Having said that, it would be nice to have as much information as possible taken in from government sources. That is why Sam V is advocating making .osm files available for people to obtain, so that people without postgis/sql/roadmatcher experience can still use josm to copy over the stuff that they want. There has been much discussion on how to make this as automatic as possible, but uuid can only do so much Anybody can get their hands on the GeoBase data and start copying tags over if they want. The Roadmatcher Excluded files listed in the import spreadsheet represent the roads that were already existing in OSM. It looks like James beat me to the reply, but hopefully this also helps. Adam On Sun, Oct 25, 2009 at 12:51 PM, john whelan jwhelan0...@gmail.com wrote: Looks like I can do an extend and combine in JOSM. Thanks John On the clean up side is there an easy way to copy the tags on one section of road onto another? For example when I extend a geobase road up to the old OSM road I'd like to have the same tags as the other sections of the road. At a quick glance I can see at least seven sections that need to be added back in within 600 meters. ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca
Re: [Talk-ca] Correcting Geobase_import_2009
Hi John, I personally think it would be better to do the cleanup immediately after the import, or possible during the import. Of course it is very tedious to do so, and it will slow down the import, but the person who is doing the import, knows best which roads were omitted. The goal is not to import as fast as possible (it's not a match), but to get high quality data. Here is the process I have done so far. When using the process with RoadMatcher, I made sure that only missing roads are imported. Since RoadMatcher isn't working perfectly, I added or removed some features which should be imported. Then I uploaded the data, and after the upload, I downloaded the area again, and made the connections. Yesterday and today I happened to have imported a few sheets, where I haven't used RoadMatcher, because it isn't working terribly well. What I have done is basically Sam's more recent suggestion, to convert the Geobase data to separate OSM files, and copy the features over which should be imported. When doing that I made the connections immediately, using the validator tools in JOSM to ensure that I haven't forgotten anything. It is slow, but after a while you get used to it, and you're able to make more progress. So far, in nearly all of the cases I just extended / shortened the Geobase segments, because the deviations weren't that big. It didn't warrant the introduction of new segments, and it is also inevitable that Geobase data would never be edited by a different person. By the way, a tool that is useful, and which now has worldwide coverage is KeepRight: [1]. It also detects missing intersections / overlapping roads, and even stuff like one-way dead ends. There are many different checks, which will all help us improving the data in OpenStreetMap. Frank [1] http://keepright.ipax.at/ John Whelan wrote: I found both James's and Sam's comments very useful. It gives me a much clearer idea of what you are trying to do and the limitations involved. It would appear that we have the ability to identify roads omitted from the geobase import which means at some point in time given enough resources a clean up could be done. Until then as you say areas where one road is already in OSM that have a number of residential roads connected to it can be cleaned up on an if spotted basis. So be it. Thanks Cheerio John Adam Dunn wrote: In JOSM, another way to quickly add on to a way like that is to use the select tool to select the way you will be adding on to, holding the shift (or control) key, selecting the last node in the way, then use the draw (add) tool to continue drawing the way. By default, in JOSM, if you have a node selected that only belongs to one way, and you start using the draw tool, that way will be extended, including tags. However, if a node belongs to multiple ways (an intersection), then a brand new way will be created (since it doesn't know which of the ways to extend), and you have to do a combine operation. So selecting a node and a way together tells JOSM which of the ways you want to extend. To copy tags from one way to another, you can use the paste tags operation. Select the way to take the tags from, and use the copy operation (ctrl-c). At this point you have the choice of pasting the entire way (ctrl-v), or just pasting the tags (ctrl-shift-v) into an existing way. When you paste tags it will overwrite tags with new values if they already exist, but will leave tags intact if there is no new tag value coming in (there is no dialog to merge, at least on version 2300). One unfortunate aspect of doing this with GeoBase is that GeoBase assigns a new uuid to a way separated by an intersection (ie. a road will be split into separate ways ever time there is an intersection, and each of those ways gets a unique uuid) I've learnt over the years with databases that some idiot somewhere will invariably want to use a tag like this and not realise that some roads don't have a value. If some idiot is using the OSM data in a manner that assumes some tag exists, then they won't get very far at all. OSM allows anyone to use whatever tags they want, by allowing any combination of keys and values. All tags that are used in OSM are merely *suggestions* and are not mandatory. It would be foolish to assume even the simple such as highway=tertiary will exist on tertiary roads. Having said that, it would be nice to have as much information as possible taken in from government sources. That is why Sam V is advocating making .osm files available for people to obtain, so that people without postgis/sql/roadmatcher experience can still use josm to copy over the stuff that they want. There has been much discussion on how to make this as automatic as possible, but uuid can only do so much Anybody can get their hands on the GeoBase data and start copying tags over if they want. The
Re: [Talk-ca] canvec / shp-to-osm
On Sun, Oct 25, 2009 at 3:15 PM, Frank Steggink stegg...@steggink.orgwrote: Hi, I think to know what is going on. I've tried to convert the residential areas of 031I08 myself, and I got an OSM file with an outer polygon. However, the outer polygon has no tags. Also, it looks that Sam's batch files run shp-to-osm with the -t parameter, which suppresses the output of features without any tags. Solution: * shp-to-osm needs to be adjusted, so that the outer polygon will get the tags, but the inner polygons will not. Im running the script how with that change.. to see how it works... * shp-to-osm should be called without the -t parameter. Is this possible? However, that means for tiles that have no residential areas a file with the size of 0 bytes will be created. (not a problem, as i could do that for all the features... but you'd end up with 80 0meg files. that would cause a headache when someone looks at it for the 1st time) ... or we could just do that for residential areas. What i DID do was create an 'extra' file, in the 'extra' folder, that extended the max nodes to 2 million. (or i could just remove that toggle), and a full .osm file will be created. ... but remember that the API can only handle 2000 nodes. and what i also did was create a 3rd line on the bat file that omits the '-t' and also in the 'extra' folder, as that should do the trick. Frank Frank Steggink wrote: Hi Sam, I've just downloaded some CanVec data, and had a look at sheets 031I07 and -08. I wonder what you mean by uploading all sub-residential files. I understand that the data is separated over multiple files, because of certain limitations. In the residential OSM files I also see no polygons with a multipolygon relationship of outer. So,this means that the outlines of places like Trois-Rivieres and others are missing. The same issue is going on with wooded areas. The data is converted with Canvec2OSM version 0.9.4. I had a closer look at the raster file (from Toporama) of sheet 031I08, because there is much less data, and I looked at the village of Gentilly (see [1]). This is in the center of the sheet. The raster file suggests that a multipolygon relationship should be in place, but the vector file (BS_1370009_2_Residential_area0.osm) shows only the two inner polygons. Are the outer polygons stored in a different file, or are they not converted at all? The shape of the outer polygon doesn't look to be complex, so I don't think the max_nodes threshold would be exceeded. Looking at the OSM file: there is only one multipolygon relationship in it, but it only refers to the two inner polygons, and not to any outer polygon at all. One note regarding multipolygons: the inner polygons shouldn't have any tags at all. See [2]. Ya, i noticed that with the water features i was playing with the other day. So that needs to have a closer look into. Anyways, some clarifications about what is going on, and how the data should be interpreted would be welcome. Thats where the readme.txt file comes in to play. As it gives some instructions. But it might need a little fixing up. I'm reluctant to import data which looks not correct. For the rest, keep up your good work :) Thanks :) Regards, Frank [1] http://osm.org/go/cKHX9ApT- [2] http://wiki.openstreetmap.org/wiki/Multipolygon Sam Vekemans wrote: Hi Richard, i think your refering to the large multi-polygons such as 'residential_area', and it 'appears' to be inverted. Here's the majic; when all the sub- residential.osm files are uploaded to OSM, it renders correctly. In JOSM, you need to zoom out and load the area, to see it. I think i'll load a region of NFLD in the next cuple days to test my hypothises. Sam ps. I cc'd talk-ca as this was mentioned b4. On 9/22/09, Richard Weait rich...@weait.com wrote: Dear gentlemen, I've had a look at some of Sam's test areas. In 1435 files there are zero occurrences of Relation=outer. So at some point we started calling relation=outer, relation=inner or completely dropping outer relations by mistake. I do still see rare nested ways, but both are marked as inner, and are on separate layers after --maxnodes I've run 0.6.1 again with an old rules file and see the same problem so I believe that this is an issue in shp-to-osm. Ian can you check a 0.5.0 - generated file and see if it contains any outer? Best regards, Richard ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca
Re: [Talk-ca] Correcting Geobase_import_2009
The difficulty with this approach is people keep adding data. Which is fine if its high quality but where people are doing tracing on lower quality data you end up with a mess. I think you have to accept that with the OSM approach the data will be of variable quality. Even users with WAAS GPS devices may not realise that WAAS enabled is not the default. As the numbers grow so it becomes more difficult to control the data quality, which is why the idea of taking the geobase data as a base is so attractive. We have the old OSM data available so my preferred approach would be to pull in the geobase data then add in any gps traces and other tags that exist in the OSM datasets over time. The original OSM Charlemagne was so far off that in parts the geobase Charlemagne was imported. One end was at least a hundred meters off. I've done a clean up based on local knowledge but really it could do with a couple of GPS tracks down all the roads to verify their positions but that again can be done over time and at least its better quality now than it was. My thoughts would be import as quickly as possible then clean up afterwards. That way you avoid helpful users adding more lower quality data in. Perhaps some sort of tag could be added to a road saying verified so that a search could be done for unverified roads close to a geobase import. I don't think there is a perfect answer accepting there is a desire to use existing data that people have spent time inputting. It is difficult telling people we've decided to scrap the bit you spent time inputting. Cheerio John 2009/10/25 Frank Steggink stegg...@steggink.org: Hi John, I personally think it would be better to do the cleanup immediately after the import, or possible during the import. Of course it is very tedious to do so, and it will slow down the import, but the person who is doing the import, knows best which roads were omitted. The goal is not to import as fast as possible (it's not a match), but to get high quality data. Here is the process I have done so far. When using the process with RoadMatcher, I made sure that only missing roads are imported. Since RoadMatcher isn't working perfectly, I added or removed some features which should be imported. Then I uploaded the data, and after the upload, I downloaded the area again, and made the connections. Yesterday and today I happened to have imported a few sheets, where I haven't used RoadMatcher, because it isn't working terribly well. What I have done is basically Sam's more recent suggestion, to convert the Geobase data to separate OSM files, and copy the features over which should be imported. When doing that I made the connections immediately, using the validator tools in JOSM to ensure that I haven't forgotten anything. It is slow, but after a while you get used to it, and you're able to make more progress. So far, in nearly all of the cases I just extended / shortened the Geobase segments, because the deviations weren't that big. It didn't warrant the introduction of new segments, and it is also inevitable that Geobase data would never be edited by a different person. By the way, a tool that is useful, and which now has worldwide coverage is KeepRight: [1]. It also detects missing intersections / overlapping roads, and even stuff like one-way dead ends. There are many different checks, which will all help us improving the data in OpenStreetMap. Frank [1] http://keepright.ipax.at/ John Whelan wrote: I found both James's and Sam's comments very useful. It gives me a much clearer idea of what you are trying to do and the limitations involved. It would appear that we have the ability to identify roads omitted from the geobase import which means at some point in time given enough resources a clean up could be done. Until then as you say areas where one road is already in OSM that have a number of residential roads connected to it can be cleaned up on an if spotted basis. So be it. Thanks Cheerio John Adam Dunn wrote: In JOSM, another way to quickly add on to a way like that is to use the select tool to select the way you will be adding on to, holding the shift (or control) key, selecting the last node in the way, then use the draw (add) tool to continue drawing the way. By default, in JOSM, if you have a node selected that only belongs to one way, and you start using the draw tool, that way will be extended, including tags. However, if a node belongs to multiple ways (an intersection), then a brand new way will be created (since it doesn't know which of the ways to extend), and you have to do a combine operation. So selecting a node and a way together tells JOSM which of the ways you want to extend. To copy tags from one way to another, you can use the paste tags operation. Select the way to take the tags from, and use the copy operation (ctrl-c). At this point you have the choice of pasting the entire way (ctrl-v), or just
Re: [Talk-ca] canvec / shp-to-osm
Hi Sam, It's either that you'll end up with 0 byte files, or with files without any outer polygons. The former is just an inconvenience, while the latter is a problem. ;) Anyways, before you generate new data, I think someone should have a look at shp-to-osm, to check if my assumption that the tags are written to the wrong polygons (inner polygon, instead of outer polygon) is right. I could give it a try, but I'm not very familiar with Java, so I hope that Ian or someone who is more knowledgeable is willing to check this out. And maybe the working of the -t switch should be revisited, so that tagless elements which are part of a multipolygon relationship are still exported. I'll have a look at the extra file, to see if it contains the data with outer polygons, although I actually want to upload one other sheet tonight. Regarding the nodes: I'm using JOSM to upload data, and although this might not be the most ideal solution, it is working fine. I've uploaded more than 1 elements at once, and I just uploaded more than 5000 elements (see [1], sheet 031I01), so this is still possible with JOSM. Anyways, what good is a bulk upload tool, if it doesn't really support bulk ;) By the way, I also uploaded the residential areas of sheet 031I08. I've copied the tags of the shape of Gentilly to the outer polygon, removed them from the inner polygons, and uploaded the data. It looks OK in OSM. (Actually, I don't think that the CanVec residential areas are that good, but at least they correspond to the areas in the raster file.) Frank [1] http://www.openstreetmap.org/browse/changeset/2952729 Sam Vekemans wrote: On Sun, Oct 25, 2009 at 3:15 PM, Frank Steggink stegg...@steggink.org mailto:stegg...@steggink.org wrote: Hi, I think to know what is going on. I've tried to convert the residential areas of 031I08 myself, and I got an OSM file with an outer polygon. However, the outer polygon has no tags. Also, it looks that Sam's batch files run shp-to-osm with the -t parameter, which suppresses the output of features without any tags. Solution: * shp-to-osm needs to be adjusted, so that the outer polygon will get the tags, but the inner polygons will not. Im running the script how with that change.. to see how it works... * shp-to-osm should be called without the -t parameter. Is this possible? However, that means for tiles that have no residential areas a file with the size of 0 bytes will be created. (not a problem, as i could do that for all the features... but you'd end up with 80 0meg files. that would cause a headache when someone looks at it for the 1st time) ... or we could just do that for residential areas. What i DID do was create an 'extra' file, in the 'extra' folder, that extended the max nodes to 2 million. (or i could just remove that toggle), and a full .osm file will be created. ... but remember that the API can only handle 2000 nodes. and what i also did was create a 3rd line on the bat file that omits the '-t' and also in the 'extra' folder, as that should do the trick. Frank Frank Steggink wrote: Hi Sam, I've just downloaded some CanVec data, and had a look at sheets 031I07 and -08. I wonder what you mean by uploading all sub-residential files. I understand that the data is separated over multiple files, because of certain limitations. In the residential OSM files I also see no polygons with a multipolygon relationship of outer. So,this means that the outlines of places like Trois-Rivieres and others are missing. The same issue is going on with wooded areas. The data is converted with Canvec2OSM version 0.9.4. I had a closer look at the raster file (from Toporama) of sheet 031I08, because there is much less data, and I looked at the village of Gentilly (see [1]). This is in the center of the sheet. The raster file suggests that a multipolygon relationship should be in place, but the vector file (BS_1370009_2_Residential_area0.osm) shows only the two inner polygons. Are the outer polygons stored in a different file, or are they not converted at all? The shape of the outer polygon doesn't look to be complex, so I don't think the max_nodes threshold would be exceeded. Looking at the OSM file: there is only one multipolygon relationship in it, but it only refers to the two inner polygons, and not to any outer polygon at all. One note regarding multipolygons: the inner polygons shouldn't have any tags at all. See [2]. Ya, i noticed that with the water features i was playing with the other day. So that needs to have a closer look into. Anyways, some clarifications about what is going on, and how the data should be
Re: [Talk-ca] canvec / shp-to-osm -multi-polygons -residential area
Ok, for the residential area, i have the tags in the relation, and not on the outer, nor on the inner. Is that OK? It still renders (i would think) Sam ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca
