Re: [Talk-ca] NRCan lakes

Fri, 10 Jul 2020 14:29:27 -0700

Dear Pierre
 
Thanks for your reply, that makes a lot of sense. I wonder whether it may make sense to have an automated bot/script or whether this really has to be done by hand? It seems like a lot of work and its quite repetitive and the potential for error is low (merging ways that are identical).
 
Best
 
Hannes
 
 
Gesendet: Mittwoch, 08. Juli 2020 um 19:09 Uhr
Von: "Pierre Béland" <pierz...@yahoo.fr>
An: "Talk-CA OpenStreetMap" <talk-ca@openstreetmap.org>
Cc: "Hannes Röst" <hannesro...@gmx.ch>, "Daniel @jfd553" <jfd...@hotmail.com>
Betreff: Re: Aw: Re: [Talk-ca] NRCan lakes
Hannes,
 
Ton exemple de doublons, on le retrouve beaucoup.  Et effectivement, le validateur JOSM permet de repérer ces doublons.  Et comme Daniel l'a clairement expliqué, il faut bien connaitre les données et effectuer les correction nécessaires.
 
Voici un script Overpass pour identifier les doublons  de polygones [natural=water] vs [roles inner - natural=wood]. Cette requête Overpass peut être lancée à partir de JOSM.
 
Il faut éviter de sélectionner une zone trop grande. Le script va extraire la relation natural=wood pour la zone et les chemins en doublon./ relations où doublons avec les éléments de cette relation natural=water.
 
Le panneau Validation permet ensuite de repérer les doublons. Pour chacun il faut ensuite corriger tout en assurant l'intégrité de la relation natural=wood.
 
Pierre
 
 
Le mercredi 8 juillet 2020 17 h 23 min 04 s UTC−4, Hannes Röst <hannesro...@gmx.ch> a écrit :
 
 
Dear Daniel
 
Thanks for your answers, I have tried to piece together this (apparently 10 year old) history of the import from the mailing list threads and the wiki and it has been somewhat difficult, especially as discussions seem to have been at multiple places. So, so many discussion about forests!...Overall there seem to be some questions about the quality and desirability of parts of the import of CanVec with the (Canadian) consus being that it is desireable to do the imports.
 
The wiki still indicates to use the canvec.osm product even though the timestamps on the files are from 2013 [1] and it is not clear whether there is a newer / updated version of the data. When I compare the OSM files of tiles from the FTP site to the Toporama product doing some spot-checks I find them to be identical for hydrological data (wetlands, rivers etc) and almost identical for forests (with Toporama having some additional "inner" ways where no forest is, but not always more accurate). If my understanding is correct that the WMS endpoints of CanVec and Toporama are up-to-date, then this allows us to compare changes in the products since 2013 when the OSM FTP dump was made. On the other hand in the release notes from 2019 [2] they point to an FTP site but that one does not contain OSM files and the release notes seem to indicate 2016-01-14 as "original release" of the current CanVec data [3]. So it seems our version from 2013 is a bit behind but probably the best we have unless somebody is willing to create another export. However, it may make sense to load the *current* Toporama WSM layer into JOSM during an import and check for any updates since the 2013 dump. On the other hand, the data is not very up to date in cities, I found a large industrial complex in the Toporama map in downtown Toronto where "Marian Engel Park" is since at least 12 years [4], so we have to keep that in mind.
 
The wiki also suggest to use a Google Sheet to track imports, but it does not seem to be used a lot - I assume from the wikipage that you have written most of it and initiated the import, correct?
 
Best regards
 
Hannes
 
 
Gesendet: Mittwoch, 08. Juli 2020 um 13:09 Uhr
Von: "Daniel @jfd553" <jfd...@hotmail.com>
An: "pierz...@yahoo.fr" <pierz...@yahoo.fr>, "Hannes Röst" <hannesro...@gmx.ch>
Cc: "Talk-CA OpenStreetMap" <talk-ca@openstreetmap.org>
Betreff: Re: [Talk-ca] NRCan lakes
OMG, a lot of pertinent questions!
You are summarizing questions than were discussed on this list over the last decade. Discussions about canvec/osm data modeling, internal canvec data sources, import problems, edits problems, and artifacts from osm validation tools' history!
Because of that, you cannot assume any coast-to-coast consistency with the problems you have identified, although you can find them almost everywhere.
Here are some clues. Canvec model did not change much over years but the sources used to build the product changed (from federal to provincial/municipal). As far as I know,  canvec.osm product is not maintained anymore, even if its last version is still available. When you find inconsistencies, look at data history. It may help to identify if a problem comes from an initial import, from an adjustment with existing data, from a duplicated erroneous import, or from subsequent edits.
Good mapping!
Daniel
 
Sent from Galaxy S7
From: Hannes Röst <hannesro...@gmx.ch>
Sent: Wednesday, July 8, 2020 11:41:50 AM
To: pierz...@yahoo.fr <pierz...@yahoo.fr>
Cc: Talk-CA OpenStreetMap <talk-ca@openstreetmap.org>
Subject: Re: [Talk-ca] NRCan lakes
 

Dear Pierre
 
Thanks a lot, your explanation of the history is very helpful.  I can also see on the wiki and the mailing list some threads and pages that explain the import but some of the wiki pages are quite old (10 years or so) and its not clear whether they still all apply and contain current policy.
 
In your example it seems that the import produced duplicated ways sometimes where the lake and the multipolygone (inner) were identical.In this case I see that they can be found with the JOSM validator (org.openstreetmap.josm.data.validation.tests.DuplicateWays and can then be merged (Shift-J) but its 4 clicks for each merge so quite some work and a script could potentially fix that automatically.
 

When I look more closely, however, I think this is partially an import artefact and partially a problem in the input data. Take for example the case of https://www.openstreetmap.org/way/129592036 and https://www.openstreetmap.org/way/129592039 which has the same issue (one tagged as "inner" and one as water) and I look in the current CanVec data 031L03 0.3.3 then I only see a single way with 14 nodes at that position. In the same tile I find the ways https://www.openstreetmap.org/way/129592307 and https://www.openstreetmap.org/way/129592315 are duplicated both in OSM as well as in the input CanVec data tile 031L03 0.3.3 (one is inner of wetland, the other inner of wood). I am not sure where this error comes from but it clearly highlights the need for manual fixup of the imported data.
 
> Ici on peut  par exemple ne conserver que le lac (way/60852636) et effacer le doublon pour le role inner (way/60854569) et réviser la relation multipolygone pour y indiquer way/60852636 avec role=inner.
 
Yes I think that is possible with JOSM by selecting both and hitting Shift-J and then making sure to click "Keep" in the relation. But its a lot of work because it is currently done manually and it seems this could easily be done by a script (this was already discussed several years back, especially doing this automatically but nothing seems to have happened [1]).
 
Another issue that I found in the import is with highways: the "almost connected but not connected" ways, luckily they can be found by Osmose but create a ton of warnings: http://osmose.openstreetmap.fr/en/map/#zoom=12&lat=46.0489&lon=-77.5019&item=xxxx&level=1&tags=&fixable=
 
What I also dont understand is differences between CanVec imports, for example looking at the same tile as above ( 031L03 0.3.3 ) there are several waterways that are missing in the CanVec data, for example https://www.openstreetmap.org/way/129591734 (tagged with NRCan-CanVec-8.0) is not present any more in the tiles that I downloaded from [2] - is there some error here, was the stream removed on purpose in the newer CanVec data? In the ESRI and Bing satellite data I can clearly see a feature there in the woods that looks very much like a waterway, so it looks like some sort of stream is there, but not in other images from Maxar (maybe its only part of the year?). So why is it missing in newer CanVec data? How should we deal with these cases in OSM ?
 
Best
 
Hannes
 
1. https://lists.openstreetmap.org/pipermail/talk-ca/2016-September/007225.html
2. https://ftp.maps.canada.ca/pub/nrcan_rncan/vector/osm/
 

Gesendet: Dienstag, 07. Juli 2020 um 12:18 Uhr
Von: "Pierre Béland" <pierz...@yahoo.fr>
An: "Talk-CA OpenStreetMap" <talk-ca@openstreetmap.org>
Cc: "Hannes Röst" <hannesro...@gmx.ch>
Betreff: Re: [Talk-ca] NRCan lakes

Petit rappel pour ceux moins familiers avec les imports Canvec. Il est bon de bien connaître la structure des données et doublons éventuels à corriger. Aussi JOSM est très utile pour repérer les chemins en doublon et corriger.
 
Les développeurs OSM mentionnent régulièrement des multipolygones bois (imports Canvec) très grands et complexes qui causent des problèmes de traitement de données dans la base de données OSM.  Il faut donc éviter de jumeler les multipolygones bois, et plutôt simplifier lorsque possible. 
Aussi, on rencontre souvent des chemins en doublon pour décrire et le lac et les zones à exclure d'un multipolygone. Tobermory Lake (60852636) est un exemple intéressant à ce sujet. Avec JOSM, on clique sur les bords du lac pour voir si des doublons existent.
 
Ici- le lac https://www.openstreetmap.org/way/60852636
- la zone à exclure du multipolygone (role=inner) https://www.openstreetmap.org/way/60854569[https://www.openstreetmap.org/way/60854569]
- le multipolygone https://www.openstreetmap.org/way/946291[https://www.openstreetmap.org/way/946291]

De plus, on retrouve un polygone couvrant une partie du lac pour le marécage adjacent au lac (natural=wetland).
https://www.openstreetmap.org/way/60852071[https://www.openstreetmap.org/way/60852071]
 
Ici on peut  par exemple ne conserver que le lac (way/60852636) et effacer le doublon pour le role inner (way/60854569) et réviser la relation multipolygone pour y indiquer way/60852636 avec role=inner.
 
 
Pierre
 
 

Le mardi 7 juillet 2020 11 h 34 min 08 s UTC−4, James <james2...@gmail.com> a écrit :
 
 

_______________________________________________
Talk-ca mailing list
Talk-ca@openstreetmap.org[mailto:Talk-ca@openstreetmap.org]
https://lists.openstreetmap.org/listinfo/talk-ca

I don't think canvec is updating these things on a regular basis, OSM after corrections are usually more accurate than canvec anyways and doubt would update data from Canvec to fix outdated data 

On Tue., Jul. 7, 2020, 11:27 a.m. Hannes Röst, <hannesro...@gmx.ch[mailto:hannesro...@gmx.ch]> wrote:Dear Adam and Daniel

Thanks a lot, so this answers the question that these are import artefacts and not intended. One question still remains, namely whether we should clean them up and how (joining ways makes sense from the OSM data model but may make a future update based on CANVEC files much harder while adding all ways into a relation would preserve the import but the resulting shape will look funny). My instinct is still to fix the ways unless there is a strong reason against this. One reason I ran into this was trying to match OSM to Wikidata items and of course having 3 ways all called the same name makes this difficult. Let me know what you think

Another issue I found was with nodes such as these: 1279897592, 1279898654 and 1279896951 which also seem to come from an import (see [1] for overpass query). I am not sure whether these are duplicate imports or whether they are supposed to indicate the extent of a feature (most east and most western point) of the channel. The wiki indicates to either map this as "natural=strait" and use either a single node, a line or a multipolygon [2] but not as multiple nodes with the same name. Honestly, in this case its a bit hard to see where the supposed "channel" should be, but connecting the nodes to a line would seem sensible here to me, any thoughts?

Best

Hannes

[1] http://overpass-turbo.eu/map.html?Q=%5Bout%3Ajson%5D%5Btimeout%3A25%5D%3B%0A(%0A%20%20node%5Bname%3D%22Devil%20Island%20Channel%22%5D%3B%0A)%3B%0Aout%20body%3B%0A%3E%3B%0Aout%20skel%20qt%3B[http://overpass-turbo.eu/map.html?Q=%5Bout%3Ajson%5D%5Btimeout%3A25%5D%3B%0A(%0A%20%20node%5Bname%3D%22Devil%20Island%20Channel%22%5D%3B%0A)%3B%0Aout%20body%3B%0A%3E%3B%0Aout%20skel%20qt%3B]
[2] https://wiki.openstreetmap.org/wiki/Tag:natural%3Dstrait#How_to_map[https://wiki.openstreetmap.org/wiki/Tag:natural%3Dstrait#How_to_map]
 

Gesendet: Dienstag, 07. Juli 2020 um 09:56 Uhr
Von: "Adam Martin" <s.adam.mar...@gmail.com[mailto:s.adam.mar...@gmail.com]>
An: "Hannes Röst" <hannesro...@gmx.ch[mailto:hannesro...@gmx.ch]>
Cc: "Talk-CA OpenStreetMap" <talk-ca@openstreetmap.org[mailto:talk-ca@openstreetmap.org]>
Betreff: Re: [Talk-ca] NRCan lakes

As mentioned by Daniel, this is due to the nature of the CANVEC data import.  CANVEC shapefile data is based on tiles and these will chop practically anything into pieces - lakes are just ones of the more noticeable.  I have corrected some of these myself as I've come across them.  Just be careful in cases where the lake pieces are part of different relations in the area - you will need to adjust those to make sure nothing breaks.
 
Adam 

On Tue, Jul 7, 2020 at 2:33 AM Hannes Röst <hannesro...@gmx.ch[mailto:hannesro...@gmx.ch][mailto:hannesro...@gmx.ch[mailto:hannesro...@gmx.ch]]> wrote:Hello

I am a contributor from Toronto and I have a question regarding how to
treat some of the CanVec 6.0 - NRCan imports, specifically for lakes.
I came across this lake here:

https://www.openstreetmap.org/way/69275451[https://www.openstreetmap.org/way/69275451][https://www.openstreetmap.org/way/69275451%5Bhttps://www.openstreetmap.org/way/69275451%5D]
https://www.openstreetmap.org/way/69277932
https://www.openstreetmap.org/way/69745752

Which is strangely split up into 3 parts and I wonder how to proceed:
should we fix this and create a single way out of these 3 parts or is
it beneficial (for comparison to future NRCan database entries) to
keep them that way and create a relation out of the three? Also, does
somebody know why the NRCan dataset does this, is this an import
artefact (splitting into tiles?) and should be corrected when encountered
or is it part of the original dataset?

Best

Hannes Rost

_______________________________________________
Talk-ca mailing list
Talk-ca@openstreetmap.org[mailto:Talk-ca@openstreetmap.org][mailto:Talk-ca@openstreetmap.org[mailto:Talk-ca@openstreetmap.org]]
https://lists.openstreetmap.org/listinfo/talk-ca[https://lists.openstreetmap.org/listinfo/talk-ca]

_______________________________________________
Talk-ca mailing list
Talk-ca@openstreetmap.org[mailto:Talk-ca@openstreetmap.org]
https://lists.openstreetmap.org/listinfo/talk-ca

_______________________________________________
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca
_______________________________________________
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Reply via email to