[Talk-ca] Sam's summery essay (was Re: Correcting Geobase_import_2009)
Hi John, Sorry for the delay... and thanks everyone else for chiming in. Others can (as you saw) explain it better. :-) This message is directed to the talk-ca list, as it serves as a summery for the latest and greatest. In a month or so, I'll be able to summarize in 1 page. But for now, I've put a lot of thought into the below message, so, although long and rambley... it's the best answer i got :) First off, let me introduce the team... lets call it 'the GeoBase Road Crew'. These folks all know how to use python scripting and know how to use a 'postGIG' database. And are brilliant! And going a super awesome job! From west to east, we have; Steve Singer - who loaded 092c area for me - And kicked of the whole province of Alberta Most of Ontario... doing a super job .. and we have to thank for all the hard work done. .. paving the way for the others to follow. Jason Reid - who started the geobase2osm script (i think getting the initial geobase tag matching done) revising the script from shp2osm user: Austin Henry - Who has converted All of Vancouver Island. And for the 092B (Victoria area) he made the complete.osm area file available. As it was easier to manually add in roads.. than to pick-and-choose from RoadMatcher. (Since there's a lot of mappers around, this makes sense). And for the rest of the island, he manually checked out roadmatchers results and tried to attach as many geobase roads to OSM roads as he could. user: Michael Barbanov - Who is working on the 092G area (Vancouver area) where 092G07 (port Coquitlam) is awesome. User: Neskie - who is in the 082L area and did some roads first, then went on to be in charge of the Aboriginal Lands Database.. user: Adam Dunn - who is now going full at it, and getting (what looks like) the rest of BC done. Starting with 092H Wow, awesome! user: Steve Singer - again did all of Alberta, and most of Ontario user: John Peterson - who did some of southwestern Ontario (and used the Stats Can roadnames) before roadnames became available in Ontario, using that cool script to match it up. Awesome! user: Frank Steggink - who is doing a super job in Quebec - getting it all done. user: Yan Morin - who did the Mont Laurier area . and also started the geobase NHN data, and make a cool python version of the script. and of course, the entire talk-ca list, as anyone who has ideas, chimes in when they can :) ... and me user:Acrosscanadatrails, who doesn't know a think about python except that it works great, and we all like the results. :) I'm the one who's keeping track of the progress on a national level using the 'Canada Data Import Chart' [1]. ... I'm also the one who is managing the conversion process for the 'CanVec data (all the other map features). For this conversion I'm using Ian Dees's script shp-to-osm (which is simple java script and command line function with a rules.txt file). Who's helping along. http://wiki.openstreetmap.org/wiki/Canvec2osm .. as well, I'm working he geobaseNHN data (that user: Yan Morn started) and found errors). Frank is helping right now get the details on that.(considering that the data can also be found on canvec. So I'm sorting all that out. We now have a imports@ discussion list[2], where the folks around the world who have conducted (or will be) or interested in. dealing with large amounts of data, and how it effects the community. etc. Brilliant minds, where I'm amazed everyday. How ALL questions can be answered. And so, (recently) I've been using the IRC #osm channel[3], where there are on average 150 awesome mappers/programmers/GIS professionals/architects... and more who basically can answer any question asked. ... and debate about that, with fun and sometimes heated discussions. ... always ending up in some kind of change for the better :-) Also, the talk-au the talk-us list, where in the USA, there dealing with the post-Tiger trauma... and in Australia where they now have access to all this data and working it all out.And Also, new Zealand, where they are working on converting .mp files converting to OSM (LINZ data).[4] On Sun, Oct 25, 2009 at 9:43 AM, john whelan jwhelan0...@gmail.com wrote: Yes I'm new and my background is red tape. I understand there is a lot going on and I appreciate the work that has been done. My background is in databases etc. If you ask clients what they want they always rate reliability above anything else. To me the data from geobase is good high quality data with input from multiple sources including municipal governments. Talk to them nicely and we can get all sorts of high quality things like bus stops, speed limits etc direct from the municipalities. So ya, right now i have a stack of data available,, and ready.. I DO think that the community would appreciate me just doing one thing at a time. I volunteer to get it done. Others can help if they like. But the community would agree that getting all the roads in is 1st
[Talk-ca] Formal proposal to replace the existing Ottawa information with Geobase information.
This is purely local and it does not impact any other parts of Canada. If you have been involved in creating the existing map and have input please input it now. I'm under the impression that locally so far there is not a major community effort going on with map parties etc which would be disrupted by this action. The reasoning behind this proposal is the road data sampled so far seems to be of dubious quality. I've been talking to some residents associations and people interested in heritage and they are interested in what OSM can offer them, especially when involved in the planning process. However to be of most use the roads have to be in the correct place. They also have access to volunteers. I've also been in communication with a group of people who are interested in buses. Some of which have created applications to work with PDAs and other web enabled devices to see when the next bus is due and which route to take. Bus stop locations are important to them and I think I can obtain their locations in the same way that they have been imported recently in the UK. I accept that just because Geobase data is from the government it does not mean that is is 100% accurate but it is consistent. Roads do connect which is important when putting together routing information. I suggest that if there is no local objection within 30 days that implement this. Cheerio John ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca
Re: [Talk-ca] Users in Ottawa and Geobase
Ladies and Gentlemen, James Treacy mentioned On the other hand, there are many areas that have had the occasional, inexperienced mapper enter a few roads. These are generally of low quality and make a mess of the geobase import. These people are not on talk-ca so never had a chance to say, 'replace my work as it is of dubious quality.' Fixing the roads can be a lot of work in this case. Much more than simply replacing the work and then fixing a few modified intersections and moving some roads using data from gps traces. In spite of the second case, anyone asking about poor quality roads is given the party line that the existing work should stay. The data contributed by all contributors should stay regardless of the apparent quality. The OpenStreetMap wiki about imports ( http://wiki.openstreetmap.org/wiki/Bulk_Importing ) clearly states: Bulk imports are undertaken as a supplement to data collected by individual volunteers, and their data and ability to work in an area *always * has priority. which should be well regarded as a guideline for all of the imports in Canada and abroad. They do not differentiate between good quality and poor quality work nor about the apparent quality of the imported data. And the imported data varies in quality as well. Some of it is very high quality Differential GPS but other areas are of a lower quality. James Ewen has stated, recently (Oct 25, 2009), ... I tracked thousands of kilometres of highway in the year before GeoBase became available, adding it to the database, or replacing low resolution imagery tracings. Those highways still exist in the OSM database, describing the route of the highways in higher resolution than the GeoBase data does. and, again from James Ewen (Oct 25, 2009), ...One can not automatically assume that the GeoBase data is of higher quality than the OSM data simply because there are more tags associated. In some cases, the positional accuracy of the GeoBase data is better than OSM data, but in other instances, the OSM data positional accuracy is better than GeoBase. It all depends upon the amount of effort expended during the input process. Among the tags for the GeoBase import there is usually a tag describing the source of the original data from the municipality and it varies with age and location. Now do you want to replace higher quality data submitted by a user, even if that user has contributed little and nothing for a while, with something from GeoBase that is not really accurate? I have updated data that was imported as part of the GeoBase import that was certainly not as accurate as it should, and could, have been so I personally know that the GeoBase import is not always the best source of data. On the other hand I am grateful for the data being available sine it is sometimes the only way we are going to get data for some areas within a reasonable time frame, or even our lifetimes. If you are in the area that you are looking at the best, and really only, option is to check out the area yourself with a GPS and to correct any discrepancies regardless of if they were made by a individual mapper or as part of the GeoBase import. It is possible to modify the data regardless of the source and doing so improves the map. Replacing all of the user contributed data automatically with GeoBase data is not the answer and can even cause errors because the GeoBase data is not always the most current either. So update, and correct, the data regardless of the source by physically checking it, preferably with a GPS. ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca
Re: [Talk-ca] Formal proposal to replace the existing Ottawa information with Geobase information.
If Ottawa had a crowd who had done a considerable amount of GPS based mapping then I would totally agree with you. My sampling has been done with a hand held WAAS GPS unit used whilst walking, I am not comparing the original OSM roads to the Geobase data but to a hand held GPS unit. The Geobase tagged data seems to have a much closer agreement with the GPS unit. I used to work for Statistics Canada and I did some work in quality control in industry so I am aware that my samples to verify what is there so far doesn't prove everything is bad unfortunately I am also aware that the probability of the rest of the data being dubious increases with every new error found. One concern comes from talking to people who are interested in trees in Ottawa. They like the idea of taking a photograph with a time stamp on it so its GPS position can be determined then have a more knowledgeable person identify the tree more exactly. The city centre looks like its about 30 meters out. Currently I have identified a fire station being roughly 100 meters off. One road was more than 100 meters off at one end, causing parts of other roads to be omitted on the import. Are we saying the the tree people take your photo and the GPS coordinates put it on our map but accept that it might appear 100 meters away from its location? I honestly don't think this data is based on a GPS trace. If you are in the area that you are looking at the best, and really only, option is to check out the area yourself with a GPS and to correct any discrepancies regardless of if they were made by a individual mapper or as part of the GeoBase import. It is possible to modify the data regardless of the source and doing so improves the map. Replacing all of the user contributed data automatically with GeoBase data is not the answer and can even cause errors because the GeoBase data is not always the most current either. So update, and correct, the data regardless of the source by physically checking it, preferably with a GPS. I agree with this statement as well however given the current known resources in Ottawa then I think it is better to go clean import from Geobase so we have a known quality of data then use groups such as the residents associations, the bus fanatics, the tree huggers whose primary interest may not be mapping to improve the result. That way we end up with something usable quickly. We are leveraging on the resources available. However to leverage them I have to be able to say to them this is something useful for you today, not when we have done a GPS survey and corrected the map it will be useful. Current known resources are one person who lives in the suburbs walking with a GPS unit and there are a lot of roads to cover. I haven't come across a community of OSM mappers who party every week in Ottawa. To me OSM is not a printed map in the traditional sense but a GIS system which means the ability to add tags easily is very important. The data contributed by all contributors should stay regardless of the apparent quality. The OpenStreetMap wiki about imports (http://wiki.openstreetmap.org/wiki/Bulk_Importing ) clearly states: "Bulk imports are undertaken as a supplement to data collected by individual volunteers, and their data and ability to work in an area always has priority." which should be well regarded as a guideline for all of the imports in Canada and abroad. They do not differentiate between good quality and poor quality work nor about the apparent quality of the imported data. And the imported data varies in quality as well. Some of it is very high quality Differential GPS but other areas are of a lower quality. I'm permitted to correct any data in an existing map, so "data contributed by all contributors should stay regardless of apparent quality" only means what I want it to mean. These are general guidelines drawn up in Europe where the ratio of GPS equipped mappers to road distance is quite reasonable. Ottawa does not have the same resources and I think in this particular case we should look for a more appropriate solution. I think the issue is a local one, I'm not asking for this for everywhere in Canada and I think the decision should be a local one. So far I have not seen any input from some one doing OSM mapping in Ottawa. Cheerio John Gerald A wrote: Hi, On Wed, Oct 28, 2009 at 9:52 AM, john whelan jwhelan0...@gmail.com wrote: I accept that just because Geobase data is from the government it does not mean that is is 100% accurate but it is consistent. Roads do connect which is important when putting together routing information. I'm not local to Ottawa, so you can take this with a grain of salt. First, I have mapped some stuff in the Toronto area which was "adjusted" better with the Geobase data. It really discouraged me from participating, because why actually map when someone else will do it different "better" eventually? I think this goes against the crowdsource we draw
Re: [Talk-ca] Users in Ottawa and Geobase
Richard, A mapper notices a subdivision in openstreetmap does not jibe with his experience so he goes and traces it using his gps. He loads it into JOSM along with the current openstreetmap data and the geobase streets. He then notices that the existing data is worse than he thought: streets aren't joined, streets are substantially displaced from what the other data show and many streets are missing because the displaced roads caused RoadMatcher to drop them. Further, the mapper's trace and geobase are in good agreement. This is not an exceptional example. I have encountered similar situations many times. Are you really suggesting that the mapper should spend hours massaging the existing map into place and then adding the missing roads? The alternative is to replace the area using geobase and then do some road alignment from the gps traces, probably taking 1/4 the time. In Southwester Ontario, I have found that the geobase data is excellent for getting the topology right and quite good for location. On the other hand, the concession roads in SW Ontario mapped before geobase came along were 95% crap. That doesn't mean they should all be junked wholesale, but if data shows a road to be awful and causes a lot of roads to be dropped, I think it is fair to let a mapper fix things in the easiest way. -- James Treacy tre...@debian.org ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca
Re: [Talk-ca] Users in Ottawa and Geobase
Hi James, i know you sent it to Richard, but i'll chime in :-) On Wed, Oct 28, 2009 at 12:13 PM, James A. Treacy tre...@debian.org wrote: Richard, A mapper notices a subdivision in openstreetmap does not jibe with his experience so he goes and traces it using his gps. He loads it into JOSM along with the current openstreetmap data and the geobase streets. He then notices that the existing data is worse than he thought: streets aren't joined, streets are substantially displaced from what the other data show and many streets are missing because the displaced roads caused RoadMatcher to drop them. Further, the mapper's trace and geobase are in good agreement. This is not an exceptional example. I have encountered similar situations many times. Are you really suggesting that the mapper should spend hours massaging the existing map into place and then adding the missing roads? The alternative is to replace the area using geobase and then do some road alignment from the gps traces, probably taking 1/4 the time. In Southwester Ontario, I have found that the geobase data is excellent for getting the topology right and quite good for location. On the other hand, the concession roads in SW Ontario mapped before geobase came along were 95% crap. That doesn't mean they should all be junked wholesale, but if data shows a road to be awful and causes a lot of roads to be dropped, I think it is fair to let a mapper fix things in the easiest way. Well, just looking at the Canada Data Import Chart, all of the geobase roads are available as stand-alone .osm files. ... These can be used how you like. Provided that you have direct permission from the person who origionally traced those roads to change them over. Then it's fine to swap those OSM roads, to add in geobase roads. (But the whole point of our discussions is that we need to act as a community, and work with those local mappers who spent all that time, to locally decide if they want to manually adjust the roads to be a bit better, they want to replace it (that decision is to be made by the person who originally drew in the road. (if you cant contact that person, ask on the talk list, we'll try :). And if you still cant reach that person, ask for help with manually adjusting. I'll begin (or continue) converting the NTS tile areas for each of the tiles where mappers are.So that should be of assistance to everyone. And so, if you find that there are missing .osm standalone geobaseRoads files, from that chart, please post it to the talk-ca list, and someone with the skills to use geobase2osm will be able to make those files available for you. (were all volunteers, so it will take some time) :-) So anyway, (if others dont mind). What i'll do is make the Ottawa area canvec data available, along with the canvec roads (to be used as a reference). I'll go ahead and do a little adjusting. Just like how downtown Vancouver, and Victoria were off by 10 meters or so. ... what i did (and encourage others todo) is to ONLY move the road over when i'm doing things like adding in a point of interest and i know its on 1 side of the road, adding in points of interest on the other side of the road. (this adds VALUE to the map), then it's easy to fix up the map. Even with WAAS enabled GPS's, having buildings around, it does mess it up a little. BTW, when we have 'mapping parties' these are the things that we discuss, so everyone understand that there doing. (If anyone want to host a mapping party in Ottawa, i'm happy to be attending remotely via. cell phone on speaker phone :-). Cheers, Sam -- James Treacy tre...@debian.org ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca
Re: [Talk-ca] Users in Ottawa and Geobase
On Wed, Oct 28, 2009 at 7:19 PM, john whelan jwhelan0...@gmail.com wrote: My objective at the moment is to get something sane that can be used by various groups in the city with GPS devices to tag trees, heritage buildings etc. Leverage those people with geographic interests... If they go out to tag a tree, and see that the road is out of alignment by a couple metres, then show them how to enhance the map. That's what OSM is all about, getting the community involved in making the map. As opposed to Google, Mapquest, TeleAtlas, or whoever else, here at OSM, you as a user can make the map, not just look at it. Those who are interested in heritage buildings can put a POI in place, or even better, draw in the building outline. These aren't quite heritage buildings, but rather commercial buildings in a shopping complex, but the concept is the same. http://www.openstreetmap.org/?lat=53.4480357170105lon=-113.482160568237zoom=15 James VE6SRV ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca
Re: [Talk-ca] Users in Ottawa and Geobase
On Wed, Oct 28, 2009 at 9:19 PM, john whelan jwhelan0...@gmail.com wrote: My objective at the moment is to get something sane that can be used by various groups in the city with GPS devices to tag trees, heritage buildings etc. If you are saying that provided I can do a couple of GPS traces down some of the city center roads then based on that we can adjust the road system wholesale to more nearly agree with the GPS as was done in Vancouver etc. then I'm more than happy. The Yahoo! satellite imagery is pretty good for some places and I believe that Ottawa is one of those places, but I am finding that in some places it is slightly out of alignment. I do not know if it is intentional or due to some kinds of errors. The same issue was raised in Australia recently. If you can find the shift, and it may vary from one area to another, and can easily correct it then, from my perspective at least, go ahead and try. The more traces you use to set your baseline the better, and I would certainly do it in a grid pattern to ensure you are getting the proper shift. Of course if you are doing GPS traces to find the shift in the map why not use the data to correct the traces completely. And if you can encourage those that are wanting to mark specific points to also create traces and to submit them as well then you have an even better source to correct the local map. It's been a long time since I took a look at GIS systems professionally and I'm not familiar with what is the best approach. I would prefer not to put too much load on others knowing they are all volunteers. I have a background in programming but its been 30 years since I was involved in teaching computer science at university level. On mapping parties realistically this is better done in the spring / summer / fall time frame when bicycles can be used, Ottawa gets a bit chilly at this time of the year for organising something in the next four months. True and I have to agree with you there. But at the same time you can start to plan for one, even of you know that it will be at least four months away, and starting to tell people about it so that they can plan for it. Also having some chances to talk about exactly what you want to accomplish and how to go about doing it, such as using the editors, may help as well. Even when it is really cold people can still be making GPS tracks while they are driving, which is a pretty quick way to generate a great deal of tracks quickly, and so improve the data to correct the map, and even to make corrections all year long. Cheerio John Richard ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca
Re: [Talk-ca] Sam's summery essay (was Re: Correcting Geobase_import_2009)
Hi all, James did a good summery explaining the concept of 'social impact of Bulk Importing', perhaps better than i did. Just substitute the word 'geobase' for 'the data source', and it really can apply internationally. And suffixing it with the assumption that the geobase-source-file.osm is available to use (and download from somewhere). Cheers, Sam On Wed, Oct 28, 2009 at 5:10 PM, James Ewen ve6...@gmail.com wrote: On Wed, Oct 28, 2009 at 1:19 AM, Sam Vekemans acrosscanadatra...@gmail.com wrote: This message is directed to the talk-ca list, as it serves as a summery for the latest and greatest. In a month or so, I'll be able to summarize in 1 page. But for now, I've put a lot of thought into the below message, so, although long and rambley... it's the best answer i got :) Wow, Sam... I made it through the whole spiel, and even stayed with your thought process through the whole thing... that's a first! 8) Here's the low-down. (social impact) We respect the integrity of the local area mapper who spent a considerable amount of time either tracing from imagery, or tracing from there own GPS tracks and place this on a HIGHER priority than that of geobase/canvec data. So, again... this is Openstreetmap, where its a collaborative community who builds the map. ... we respect the integrity of the local area mapper who spent a considerable amount of time either tracing from imagery, or tracing from there own GPS tracks and place this on a HIGHER priority than that of geobase/canvec data. I think this type of statement is what is causing problems. We should not use a blanket statement that OSM data of any quality is sacred... OSM data is a living database that everyone can work on. Data that you, I, or anyone else enters into the database is not locked into the database never to be modified. If another user comes along and wants to add tags, modify the way to (hopefully) increase the accuracy of the data, or even remove the data should the real world object the data is representing should be removed or destroyed. The issue is that data being imported by a bulk import script should not be blindly imported damaging or destroying work that has been done by a real live OSM user. The key concept in that statement is BULK IMPORT SCRIPT. If a user has the complete GeoBase file for the area and is putting the time and effort into verifying and checking the GeoBase data versus the OSM data, and comes up with the conclusion that the GeoBase data does a better job of describing the way, then they should feel free to modify/remove the lower quality OSM data, and copy the better quality GeoBase data into the OSM database. Another concept to remember is that there does not have to be an exclusion clause. One does not have to choose to go with only OSM data or GeoBase data. One could use a high resolution OSM GPS trace based way, and copy the GeoBase tags onto the OSM way. There's also the possibility that some of the tags on a low quality OSM way might be useful if copied onto the higher quality GeoBase way. What we need to do, is to take the best data that we can find from whatever source is available (that meets OSM guidelines), and merge that into the database. The bulk import scripts are written to do that, but only where an easy decision can be made, which is based only on the easily determined logical choice... Is there any existing OSM data at this location? If the answer is no, then import the GeoBase data. We need to have real people make the harder decisions where the GeoBase data and OSM data overlap. That's where we are at in areas that have been imported, and people are seeing holes between the GeoBase data, and OSM data. Feel free to get your fingers dirty... get in there and make an informed decision about what data to include in the OSM database. Just don't blindly wipe out existing OSM data to import a bunch of bulk data. It's not a GeoBase versus OSM issue, but rather a data quality issue, and it is up to the OSM community to get in there and determine which data has the best quality, and if required merge both sources to come up with an even better final product. I did the same type of thing when I was tracing hundreds of kilometres worth of highways with my GPS. I would upload the GPS trace to OSM, and then manually work my way along the highway checking my trace versus the OSM way. I would copy the tags from the OSM way to the GPS based way, I'd chop the GPS trace into pieces where I turned off one highway, and onto the next. Using aerial imagery, I would insert bridges, or other things that wouldn't be contained in a GPS trace. I didn't just wholesale delete every road in the area so I could upload my data. I used the GPS trace as another source, and using my knowledge, made the best decisions to improve the OSM database. Here are some examples... There was a very rough trace
