I'm on board with removing redundant nodes, as long as it doesn't affect actual geometries much. Also there are quite a few duplicated nodes in buildings. They can be removed in JOSM in one click before upload but better to do it at the source.
On Fri, Jan 18, 2019 at 4:25 PM James <[email protected]> wrote: > I can run all the shapefiles through qgis simplify tool if this resolves > the issue... > > On Fri., Jan. 18, 2019, 4:08 p.m. Nate Wessel <[email protected] wrote: > >> With default settings in JOSM, sure. In the import I was working on, we >> used a Douglas-Peucker algorithm with a 20cm threshold (before the import >> started) and it worked beautifully. We had many points that seemed to have >> been introduced in the shapefiles as some kind of data artifact - they >> didn't add any detail to the shape at all. This procedure removed almost >> all of them with no discernible reduction in quality. >> Nate Wessel >> Jack of all trades, Master of Geography, PhD candidate in Urban Planning >> NateWessel.com <http://natewessel.com> >> >> On 1/18/19 4:03 PM, James wrote: >> >> dare you to run simplify tool on anything remotely round, it will make it >> look like garbage >> >> On Fri., Jan. 18, 2019, 3:49 p.m. John Whelan <[email protected] >> wrote: >> >>> The import mailing list was pointed to the correct page of the wiki. >>> The initial post was to say this is what we were thinking of and there was >>> a comment saying we needed to change the comment line. >>> >>> >There is no mention of this proposed import on the import catalogue >>> >>> >>> The import process was reviewed by the person who set up the Ottawa >>> import did we miss that step on the Ottawa import as well? Neither was it >>> raised as a concern on the import mailing list. I think this is very minor >>> and can be corrected. >>> >>> We learnt a fair bit on the Ottawa import and my expectation is since we >>> are using experienced mappers to do the import conflation would be either >>> handled by them or the building not imported. We aren't using new mappers >>> in a mapathon here and with experienced mappers then I think you have to >>> trust them. The world isn't perfect. Think in terms of service level. >>> >>> >There are 2X more nodes than needed to represent the building >>> accurately. >>> >>> The problem with correcting this is you are introducing approximations. >>> This will vary according to the source and this can be simplified or >>> corrected once its in OSM. I think this is a different issue of a >>> mechanical edit that needs to be considered separately. >>> >>> If we are concerned with database size then I suggest we change the >>> instructions to say put the source comment on the change set rather than on >>> the building outline. >>> >>> Cheerio John >>> >>> >>> Nate Wessel wrote on 2019-01-18 3:06 PM: >>> >>> John, >>> >>> You seem to be playing the long game with this data - it sounds like >>> you've been working with this a lot longer than I have, and you've put in >>> the time and effort to help make this actually-quite-incredible dataset >>> available to us. I don't want to stop the import from happening - quite the >>> opposite. I just want to make sure that the time is taken to do this right. >>> OSM deserves that. Your (our) long awaited victory will be the sweeter for >>> our patience now. >>> >>> There are several specific issues I see where the I's are not crossed, >>> nor the t's dotted. I've mentioned several already, so I'll try to be brief >>> (I really need to get back to working on my dissertation). >>> >>> 1) There was extremely limited discussion on the imports mailing list. >>> The initial email did not make clear the scope of the project. I read the >>> email and did not think twice at it, thinking it was entirely about Ottawa. >>> The link in that email was actually to the Ottawa import, and not this one, >>> which seems to have been only in draft at the time. >>> https://lists.openstreetmap.org/pipermail/imports/2018-November/005812.html >>> As such, this project has NOT been reviewed by the imports list, which >>> is a requirement for proceeding with the import. >>> >>> 2) There is no mention of this proposed import on the import catalogue ( >>> https://wiki.openstreetmap.org/wiki/Import/Catalogue) >>> which is required in the imports guidelines. I suspect many other >>> guidelines have not been followed. >>> >>> 3) The wiki page describing the import is not adequate to assess the >>> quality of the data or of the proposed import. See for example: >>> https://wiki.openstreetmap.org/wiki/WikiProject_Canada/Canada_Stats_Canada_Building_Outlines_Import/Plan#Risks >>> The import guidelines call for a description of how conflation will be >>> handled. The fact that two of the major importers seem to have a >>> substantial disagreement about how to handle existing data indicates this >>> was not well discussed and I can see that it isn't well documented. >>> >>> 4) The buildings need to be simplified, quite a bit actually. Most >>> buildings have multiple nodes representing straight lines. This bloats the >>> database and makes things harder to edit by hand later. There are probably >>> 2x more nodes than are needed to represent the data accurately, making it >>> harder for editors and data consumers to work with down the road.This is a >>> simple fix that will save countless hours later. >>> >>> ... I could go on, but I think this is plenty sufficient to justify >>> pressing pause on all this. >>> >>> Again, I don't in any way want to disrespect the work that has gone into >>> this effort already. We're all volunteers here and I know how much time >>> this all takes. However. importing all/most of the buildings in Canada is a >>> monstrously large task, which will have to dance around a lot of people's >>> toes. We should expect this to take a really damn long time if we're going >>> to do it right. We need to have the patience to learn from experience, from >>> critique, and from the wisdom of the people who've learned from flawed >>> imports in the past and have devised guidelines and processes so that we >>> can have better experiences with this in the future. >>> Nate Wessel >>> Jack of all trades, Master of Geography, PhD candidate in Urban Planning >>> NateWessel.com <http://natewessel.com> >>> >>> On 1/18/19 2:24 PM, john whelan wrote: >>> >>> My background is I'm a retired civil servant who has written and >>> overseen procurement documents and fairly large procurements. Dotting the >>> is and crossing the Ts are my speciality. >>> >>> There are two parts to an import. The first part is the part played by >>> the import mailing group. They confine themselves to is the license >>> correct and do you have a reasonable plan. In this case the license is one >>> of the few that has been confirmed by the Legal Working Group of >>> OpenStreetMap and as such no questions were raised about it on the import >>> mailing list. We have methodology that has been used before successfully >>> with the Ottawa building outline import. There were major discussions both >>> on talk-ca and the import mailing group before that import took place and >>> we took note of the issues raised and addressed them. The licensing issue >>> goes back about eight years to when I was talking to Federal Government >>> Treasury Board and explaining their Open Data license did not align with >>> OSM. That is why their license is now known as 2.0. >>> >>> The second part is the local group makes the decision to import they are >>> the authority no one else. >>> >>> Apparently you were not part of the talk-ca when the discussions took >>> place which would have been the time and place to raise concerns. >>> >>> When the Ottawa import was done there were one or two places where the >>> existing buildings and the import overlapped. In the instructions on the >>> import there are instructions to cover this. Specifically there is a >>> validation step. I seem to recall the error rate was of the order of 1% >>> and I expect this latest batch to be roughly the same. >>> >>> If you can identify which municipalities data is of poor quality then >>> I'm sure we can remove these. For the most part these are from the >>> foundation plans recorded by the municipality using professional surveying >>> techniques. >>> >>> Would you like to clarify exactly where I failed to dot the Is and cross >>> the Ts please. >>> >>> Many Thanks >>> >>> John >>> >>> >>> >>> On Fri, 18 Jan 2019 at 13:37, Nate Wessel <[email protected]> wrote: >>> >>>> Hi John, >>>> >>>> As Steve has said, you seem to be the only one suggesting that >>>> thousands of import committees might need to be formed. Certainly I'm not >>>> suggesting that. >>>> >>>> My understanding of OSM import procedure (and wiki-style projects more >>>> generally) is that imports should operate in an essentially consensual way >>>> where possible. The goal is to build consent and bring people on board with >>>> a project or a change by addressing their concerns in a meaningful and >>>> respectful way. >>>> >>>> I think that I have made some substantive and troubling claims about >>>> the quality of the data being imported. I've pointed out that this project >>>> has not followed the import procedures that were produced by a community of >>>> mappers larger than just those in Canada. >>>> >>>> So to respond to your implication, I am in some sense the one reviewing >>>> the project, just as I would welcome you to find ways that my own >>>> contributions could be better. If you want my credentials for reviewing >>>> your work, here they are: >>>> >>>> 1) I am an active contributor to OSM in Toronto, where I live (and >>>> elsewhere) >>>> >>>> 2) I am currently helping to lead a building import in Hamilton County >>>> Ohio that has better addressed some of the issues I see this import >>>> struggling with. I can help you do the same. >>>> >>>> 3) I've been doing research in GIS for a long time now, though I don't >>>> need that to tell you that the issues I've described are hardly >>>> insurmountable technically or even all that difficult to fix. It would take >>>> maybe one day's hard work to get the technical side of this right. >>>> >>>> I think Canadian OSMers will agree that we can take a pause to get >>>> things right on such a massive import. If they don't - if I'm shouted down >>>> or better, if my critiques are adequately addressed, then I will leave you >>>> to finish the project in peace. I might even lend a hand if all goes well, >>>> as I sincerely hope it does :-) >>>> >>>> Best, >>>> Nate Wessel >>>> Jack of all trades, Master of Geography, PhD candidate in Urban Planning >>>> NateWessel.com <http://natewessel.com> >>>> >>>> On 1/18/19 1:11 PM, john whelan wrote: >>>> >>>> I know of no other way to contact him but he made an interesting >>>> comment that the project is on hold in the wiki pending review. >>>> >>>> Would he care to comment on who is supposed to be reviewing the project? >>>> >>>> My understanding is that the import was raised in talk-ca before it >>>> commenced for comment and these were generally favourable. I took that as >>>> the local mappers to Canada had been consulted and they are the "local >>>> mappers" authority in this case. >>>> >>>> I understand he has concerns about local mappers making decisions but >>>> in Canada we have been importing similar data through CANVEC for some >>>> time. CANVEC data comes from a number of sources including municipal data. >>>> >>>> Is he suggesting that each of the 3,700 municipalities in Canada should >>>> form a group of local mappers who can make individual decisions on whether >>>> their municipal data should be imported and we should end up with 3,700 >>>> import plans? >>>> >>>> Thanks John >>>> >>>> >>>> >>>> _______________________________________________ >>>> Talk-ca mailing >>>> [email protected]https://lists.openstreetmap.org/listinfo/talk-ca >>>> >>>> _______________________________________________ >>>> Talk-ca mailing list >>>> [email protected] >>>> https://lists.openstreetmap.org/listinfo/talk-ca >>>> >>> >>> -- >>> Sent from Postbox >>> <https://www.postbox-inc.com/?utm_source=email&utm_medium=siglink&utm_campaign=reach> >>> _______________________________________________ >>> Talk-ca mailing list >>> [email protected] >>> https://lists.openstreetmap.org/listinfo/talk-ca >>> >> _______________________________________________ > Talk-ca mailing list > [email protected] > https://lists.openstreetmap.org/listinfo/talk-ca > -- Best Regards, Yaro Shkvorets
_______________________________________________ Talk-ca mailing list [email protected] https://lists.openstreetmap.org/listinfo/talk-ca
