Re: [HOT] Broken (multi)polygon cleanup
On Thu, Apr 27, 2017 at 05:54:39PM +0200, Jochen Topf wrote: > Date: Thu, 27 Apr 2017 17:54:39 +0200 > From: Jochen Topf <joc...@remote.org> > To: hot@openstreetmap.org > Subject: Re: [HOT] Broken (multi)polygon cleanup > > On Thu, Apr 27, 2017 at 03:36:11PM +0200, Jochen Topf wrote: > > Because there is a large number of those in HOT areas, I have split > > those off into their own challenge > > (http://maproulette.org/ui/admin/list/379/Challenge/tasks/2499). > > Sorry, that was the wrong URL. The correct one is: > http://maproulette.org/map/2499 Sorry again. I pulled the challenge and recreated a new one without the cases where the "spike" had zero-length. That was understandably confusing. New link is at http://maproulette.org/map/2500. Info is still at http://area.jochentopf.com/fixing.html#spikes-buildings . Jochen -- Jochen Topf joc...@remote.org https://www.jochentopf.com/ +49-351-31778688 ___ HOT mailing list HOT@openstreetmap.org https://lists.openstreetmap.org/listinfo/hot
Re: [HOT] Broken (multi)polygon cleanup
On Thu, Apr 27, 2017 at 03:36:11PM +0200, Jochen Topf wrote: > Because there is a large number of those in HOT areas, I have split > those off into their own challenge > (http://maproulette.org/ui/admin/list/379/Challenge/tasks/2499). Sorry, that was the wrong URL. The correct one is: http://maproulette.org/map/2499 Jochen -- Jochen Topf joc...@remote.org https://www.jochentopf.com/ +49-351-31778688 ___ HOT mailing list HOT@openstreetmap.org https://lists.openstreetmap.org/listinfo/hot
Re: [HOT] Broken (multi)polygon cleanup
Hi! I just posted a new Maproulette challenge for fixing one particular case of broken polygons: Building ways with "spikes". Details are at http://area.jochentopf.com/fixing.html#spikes-buildings . Because there is a large number of those in HOT areas, I have split those off into their own challenge (http://maproulette.org/ui/admin/list/379/Challenge/tasks/2499). Would be great to get some help fixing those from the HOT community. Jochen On Wed, Mar 15, 2017 at 04:06:05PM +0100, Jochen Topf wrote: > Date: Wed, 15 Mar 2017 16:06:05 +0100 > From: Jochen Topf <joc...@remote.org> > To: hot@openstreetmap.org > Subject: [HOT] Broken (multi)polygon cleanup > > Hi! > > There are many broken (multi)polygon geometries in OSM data leading to > all sorts of problems. I have started an effort to get this cleaned up. > You can read more about this at http://area.jochentopf.com/ . > > This effort has been running for about a month now and we can already see > some good results. See > https://github.com/osmlab/fixing-polygons-in-osm/issues/15 > for some news about what's been happening. > > I am writing to the HOT list today, because a not-so-small part of the > problems come from HOT mappers. For instance, looking at one particular > problem here (spikes in closed ways), I am seeing about 5800 errors, of > which about 1600 are in areas covered by the HOT tasking manager. (Of > couse not all of these have been done in the course of HOT mapping, but > probably most of them.) > > I am reaching out to some of the regional OSM communities and also to > HOT, because I want to a) ask for the help of all the communities in OSM > to fix those errors and b) generally raise awareness, so those problems > can be minimized in the future. > > Please go to the web sites linked above and read up on what the problems > are and how we are fixing them. I also want to offer my help in > preparing special data extracts or Maproulette challenges etc. to help > the HOT community participate in the cleaning up effort in a way that > makes sense to you. I could, for instance, put those 1600 errors > mentioned above in a special Maproulette challenge, if that would be a > good way for HOT mappers to work on them. > > Tell me what you think. I appreciate any feedback. > > Jochen > -- > Jochen Topf joc...@remote.org https://www.jochentopf.com/ +49-351-31778688 > > ___ > HOT mailing list > HOT@openstreetmap.org > https://lists.openstreetmap.org/listinfo/hot -- Jochen Topf joc...@remote.org https://www.jochentopf.com/ +49-351-31778688 ___ HOT mailing list HOT@openstreetmap.org https://lists.openstreetmap.org/listinfo/hot
Re: [HOT] Broken (multi)polygon cleanup
On Fri, Mar 17, 2017 at 10:48:00AM -0400, john whelan wrote: > I've read it through but remember like many mappers I do not have a Ph.D. > in GIS, just a degree in Chemistry so I'm usually not too bad on logic. > > The first thing that stands out is the message is too complex for the > intended audience. Yes, and no. The documentation we have now is trying to be complete and thorough. The intended audience are, first, power mappers. I am totally aware that it is too much for the average mapper, but that's why I need everybodies help not only the fix things, but also to translate the problem into words that the different audiences can understand. I know that HOT has capable mappers that understand OSM well and have written documentation for new HOT mappers to use. I hope I can reach those people here so they can help. > The second "The *new style* tagging option is the recommended tagging > option these days, but some mappers still disagree." So we don't have > complete agreement, fine. Well, that was me being overcautious. I haven't heard any disagreements in the last months. So I don't think there is any major disagreements left. I have removed that sentence from the documentation. > The third is the main problem areas seem to be some rendering systems > prefer one method over the other. "Are we mapping for a specific rendering > system now?" No, we are not. I am not sure why you got that impression from the documentation. Can you point me to a specific sentence? I have highlighted Osm2pgsql, because it is by far the most often used basis for rendering maps and it is what powers the main OSM map. So changes there will affect more people. And it also means that basically every other system is trying to mimic what Osm2pgsql is doing (right or wrong), because they have to keep "in sync" with the main map that OSM mappers use to verify their work. Currently all rendering systems have to handle old- and new-style multipolygons and in the future they wont have to do that any more. This makes all systems simpler. > I suspect if we concentrate on unclosed ways and duplicate segments then > this is something that can be done without a Ph.D. in GIS concepts and > there is no disagreement. Leave the more complex problem solving to others. I totally agree. That's why I have been splitting up the work into small Maproulette tasks, each with a very easy to understand and concrete description of what the fixing task is. I don't expect most mappers to be able to solve complex multipolygon cases. But there are plenty of simple cases around. > I would recommend if you wish to use the resources of HOT that you talk > nicely to the HOT training group and see if they can sort out the message > and what training needs to be given to support these efforts including what > you would like mappers to avoid when mapping. I am hoping these people are listening here. (Are you?) Or do I have to go somewhere else to talk to them? Jochen -- Jochen Topf joc...@remote.org https://www.jochentopf.com/ +49-351-31778688 ___ HOT mailing list HOT@openstreetmap.org https://lists.openstreetmap.org/listinfo/hot
Re: [HOT] Broken (multi)polygon cleanup
Hi! did you read those documents linked from http://area.jochentopf.com/ ? https://github.com/osmlab/fixing-polygons-in-osm/blob/master/doc/background.md https://github.com/osmlab/fixing-polygons-in-osm/blob/master/doc/problems.md https://github.com/osmlab/fixing-polygons-in-osm/blob/master/doc/faq.md If there is something unclear after reading them, please ask, and I am happy to explain. Jochen On Fri, Mar 17, 2017 at 08:30:27AM -0400, john whelan wrote: > Date: Fri, 17 Mar 2017 08:30:27 -0400 > From: john whelan <jwhelan0...@gmail.com> > To: Jochen Topf <joc...@remote.org> > Cc: "hot@openstreetmap.org" <hot@openstreetmap.org>, Blake Girardot HOT/OSM > <blake.girar...@hotosm.org> > Subject: Re: [HOT] Broken (multi)polygon cleanup > > What isn't clear to me is just what you are trying to fix. The term means > nothing to me. I am correcting duplicate segments and unclosed ways which > appear to be part of your problem as I work. Perhaps someone can explain > what you are after in simpler terms because if we don't understand the > problem then trying to come up with ways to avoid it in the future isn't > going to happen. > > Thanks John > > On 17 March 2017 at 08:16, Jochen Topf <joc...@remote.org> wrote: > > > Hi! > > > > I am sure there are many other problems, but can we please keep to the > > issue at hand in this thread? I am not trying to solve all problems, I > > am trying to solve one particular set of problem related to > > (multi)polygons. > > > > And I didn't pick that particular problem at random. This is something > > that affects everybody. The short of it is: Due to upcoming software > > changes, some (multi)polygons that are rendered now will disappear from > > the map or look different in the future. Again, you can find all the > > details about the problems and of what we are trying to achieve at > > http://area.jochentopf.com/ . > > > > We are having great success working on these problems. You can see the > > stats at http://area.jochentopf.com/stats/ . This doesn't mean that the > > approach will be the best one for other problems, and it doesn't mean we > > can't do even better than we are doing until now with this problem. > > > > Btw: I am deliberately talking about "(multi)polygons". This is not > > something that only affects multipolygon relations, but all polygons, > > even the ones created from closed ways. They are simpler than mp > > relations, but that doesn't mean they can't have problems. > > > > Jochen > > > > On Thu, Mar 16, 2017 at 05:23:47PM -0400, john whelan wrote: > > > Date: Thu, 16 Mar 2017 17:23:47 -0400 > > > From: john whelan <jwhelan0...@gmail.com> > > > To: Jochen Topf <joc...@remote.org> > > > Cc: hot@openstreetmap.org, Blake Girardot HOT/OSM > > > <blake.girar...@hotosm.org> > > > Subject: Re: [HOT] Broken (multi)polygon cleanup > > > > > > This is similar to my pulling in Africa a country at a time and using > > JOSM > > > to check for errors. > > > > > > What I have noticed is one mapper will misstag then others will follow. > > So > > > sanitising the area or checking it for errors before the new HOT mappers > > > start would probably improve the overall data quality as they tend to > > > follow what is already mapped. > > > > > > Untagged ways by one mapper tend to cluster. I've seen a 100 buildings > > in > > > one small area by the same mapper left as area=yes. That's quick and > > easy > > > to fix. > > > > > > Recently I came across 42 highways left untagged by the same mapper in > > the > > > same area. The mapper had more than 2,000 edits so wasn't inexperienced. > > > > > > Crossing highways often I'll see a town where the highway residentials > > are > > > laid out in a grid but the junctions aren't nodes. > > > > > > Another nice recent one was a batch of settlements that had been tagged > > > highway=unclassified rather than landuse=residential. Searching for > > > highways above unclassified and crossing ways picked those out quite > > > quickly. > > > > > > These types of errors are handled more quickly directly in JOSM than > > > involving maproulette. > > > > > > Even the some other errors can be loaded up in the JOSM to-do plugin. > > > > > > I've been cleaning up in Nigeria for a while now. Making repeated passes > > > over the same areas over time. I'm seeing fewer errors s
Re: [HOT] Broken (multi)polygon cleanup
Hi! I am sure there are many other problems, but can we please keep to the issue at hand in this thread? I am not trying to solve all problems, I am trying to solve one particular set of problem related to (multi)polygons. And I didn't pick that particular problem at random. This is something that affects everybody. The short of it is: Due to upcoming software changes, some (multi)polygons that are rendered now will disappear from the map or look different in the future. Again, you can find all the details about the problems and of what we are trying to achieve at http://area.jochentopf.com/ . We are having great success working on these problems. You can see the stats at http://area.jochentopf.com/stats/ . This doesn't mean that the approach will be the best one for other problems, and it doesn't mean we can't do even better than we are doing until now with this problem. Btw: I am deliberately talking about "(multi)polygons". This is not something that only affects multipolygon relations, but all polygons, even the ones created from closed ways. They are simpler than mp relations, but that doesn't mean they can't have problems. Jochen On Thu, Mar 16, 2017 at 05:23:47PM -0400, john whelan wrote: > Date: Thu, 16 Mar 2017 17:23:47 -0400 > From: john whelan <jwhelan0...@gmail.com> > To: Jochen Topf <joc...@remote.org> > Cc: hot@openstreetmap.org, Blake Girardot HOT/OSM > <blake.girar...@hotosm.org> > Subject: Re: [HOT] Broken (multi)polygon cleanup > > This is similar to my pulling in Africa a country at a time and using JOSM > to check for errors. > > What I have noticed is one mapper will misstag then others will follow. So > sanitising the area or checking it for errors before the new HOT mappers > start would probably improve the overall data quality as they tend to > follow what is already mapped. > > Untagged ways by one mapper tend to cluster. I've seen a 100 buildings in > one small area by the same mapper left as area=yes. That's quick and easy > to fix. > > Recently I came across 42 highways left untagged by the same mapper in the > same area. The mapper had more than 2,000 edits so wasn't inexperienced. > > Crossing highways often I'll see a town where the highway residentials are > laid out in a grid but the junctions aren't nodes. > > Another nice recent one was a batch of settlements that had been tagged > highway=unclassified rather than landuse=residential. Searching for > highways above unclassified and crossing ways picked those out quite > quickly. > > These types of errors are handled more quickly directly in JOSM than > involving maproulette. > > Even the some other errors can be loaded up in the JOSM to-do plugin. > > I've been cleaning up in Nigeria for a while now. Making repeated passes > over the same areas over time. I'm seeing fewer errors so my suspicion is > the improved learnOSM, tutorials and simplifying what we ask mappers to map > is paying off. Validating new mappers work within 48 hours does pay off in > data quality terms. Giving feedback more than two weeks later is usually a > waste of time based on my own experience and Martin's report but I don't > think we have the validators to do the validation within the magic 24/48 > hours. > > Broken multipolyons do take a level of expertise to resolve. I usually > avoid them. However new mappers will fearlessly tread ahead and it is to > be expected they will break some. > > I don't think learnOSM goes into detail about them or about relationships. > Should they? That's another question. > > For some errors we need boots on the ground. Especially when it comes to > urban areas with limited access highways such as dual carriageways​ and a > highway approaching them can you turn in either direction? > > Often the most motivated people to fix the errors are the locals who will > be using the map. Trouble is we first have to get something in place that > is useful to get them motivated. > > So yes HOT maperthons do have a reputation of the blind leading the blind, > ie some are led by inexperienced mappers but we are putting effort into > reducing the number of errors. The training group has put together a very > good detailed set of instructions on how best to run a maperthon and if you > can identify a problem area HOT will listen and try to address the issues. > > Cheerio John > > On 16 Mar 2017 3:49 pm, "Jochen Topf" <joc...@remote.org> wrote: > > > Hi Blake, > > > > On Wed, Mar 15, 2017 at 05:29:04PM +0100, Blake Girardot HOT/OSM wrote: > > > Can you send me something that would help me see those 1600 examples > > > you found on OSM so I can see what might be going on and how to > > > improv
Re: [HOT] Broken (multi)polygon cleanup
Hi Blake, On Wed, Mar 15, 2017 at 05:29:04PM +0100, Blake Girardot HOT/OSM wrote: > Can you send me something that would help me see those 1600 examples > you found on OSM so I can see what might be going on and how to > improve things in HOT/MM workflow to reduce them? Those 1600 are some very specific errors I am currently looking at, but only have those on my local machine currently. But for the wider picture you can use the OSM Inspectors "area" view to see the scope of the problem. Just go to any place where HOT/MM was active. Here is an example: http://tools.geofabrik.de/osmi/?view=areas=33.65927=-2.07600=8 My current workflow is as follows: I choose one particular well-defined problem and extract the data from OSM that shows this problem. Then I put the data into Maproulette and document the problem and how to fix it. Some problems are easier, some harder to fix. Mappers can choose what problems to work on based on their skill level. If somebody doesn't feel comfortable with one of the harder problems, there are simpler ones they can work on. I also have some more general descriptions for the experts who want to go exploring/fixing on their own. Once the challenges are done, I choose a new problem and start the process from the beginning. Here are the current challenges: http://area.jochentopf.com/fixing.html Usually I am splitting up larger challenges into regions, for instance by continent or country. I could also do this by HOT activation area or so if that makes sense for HOT mappers. Splitting up, often makes the problems easier to fix, because problems inside an area are often very similar, but different to problems in other areas due to mapping priorities and methods. For instance, some regions have lots of problems with buildings, others with landuse areas, etc. Concentrating on one very well-defined problem at a time makes all this work simpler. Jochen -- Jochen Topf joc...@remote.org https://www.jochentopf.com/ +49-351-31778688 ___ HOT mailing list HOT@openstreetmap.org https://lists.openstreetmap.org/listinfo/hot
[HOT] Broken (multi)polygon cleanup
Hi! There are many broken (multi)polygon geometries in OSM data leading to all sorts of problems. I have started an effort to get this cleaned up. You can read more about this at http://area.jochentopf.com/ . This effort has been running for about a month now and we can already see some good results. See https://github.com/osmlab/fixing-polygons-in-osm/issues/15 for some news about what's been happening. I am writing to the HOT list today, because a not-so-small part of the problems come from HOT mappers. For instance, looking at one particular problem here (spikes in closed ways), I am seeing about 5800 errors, of which about 1600 are in areas covered by the HOT tasking manager. (Of couse not all of these have been done in the course of HOT mapping, but probably most of them.) I am reaching out to some of the regional OSM communities and also to HOT, because I want to a) ask for the help of all the communities in OSM to fix those errors and b) generally raise awareness, so those problems can be minimized in the future. Please go to the web sites linked above and read up on what the problems are and how we are fixing them. I also want to offer my help in preparing special data extracts or Maproulette challenges etc. to help the HOT community participate in the cleaning up effort in a way that makes sense to you. I could, for instance, put those 1600 errors mentioned above in a special Maproulette challenge, if that would be a good way for HOT mappers to work on them. Tell me what you think. I appreciate any feedback. Jochen -- Jochen Topf joc...@remote.org https://www.jochentopf.com/ +49-351-31778688 ___ HOT mailing list HOT@openstreetmap.org https://lists.openstreetmap.org/listinfo/hot
