Re: [Talk-ca] 2020 building import wiki comment by Nate Wessel

[James]

Salut Pierre, peut-être c'est le lien de téléchargement qui fonctionne pas
bien, voici une alternative:
https://www.dropbox.com/s/5ciiex5w67ohwzy/ontario-simplified.tar.xz?dl=0

On Sun, Jan 20, 2019 at 4:19 AM Pierre Béland  wrote:

> Mon problème est dés le téléchargement. Cela m'indique que google ne peut
> effectuer l'analyse d'antivirus et m'offre de télécharger quand même. De
> là, message d'erreur.
> Voici le message avec Chrome
> Ce site ne peut pas fournir de connexion sécurisée
>
> *doc-08-3g-docs.googleusercontent.com
> * a envoyé une réponse
> incorrecte.
>
>- Essayez d'exécuter les diagnostics réseau de Windows.
>
> ERR_SSL_PROTOCOL_ERROR
>
>
> Pierre
>
>
> Le samedi 19 janvier 2019 23 h 10 min 04 s HNE, James 
> a écrit :
>
>
> tar.xz c'est un fichier compressé contenant un fichier geojson(il se ouvre
> bien avec 7zip sur windows, ou n'importe quel program d'archive sur mac ou
> Linux), qui est servie via la Task Manager. Ce n'est pas la version
> originale de Stats Can, tu as raison, c'sst la version minifié sans les
> extras tag de merde qui sont aucunement utile à OSM.
>
> Je travaille sur les fichiers, car quelqu'un a dit les données était de la
> bouse de vache.
>
> On Sat., Jan. 19, 2019, 11:02 p.m. Pierre Béland 
> James,
>
> Je pense que nous travaillons sur deux aspects différents. Tu te concentre
> sur la production / correction des fichiers d'import.
>
> A ma connaissance, tu n'a pas décrit le contenu du fichier
> ontario-simplified.tar.xz.  Je suppose que ce sont les données d'import
> originales où tu apportes divers correctifs.
>
> De mon côté, je propose d'analyser le produit final dans la base OSM et
> mesurer pour les données déja téléchargées quelle proportion des bâtiments
> est avec angles droits (orthogonal), et les superspositions.  Cela
> permettrait simplement de comparer ces données avec ce que l'on retrouve
> ailleurs et rassurer sur la qualité globale de l'import par les différents
> contributeurs.
>
> Normalement, on ne devrait pas retrouver plus de 5% des bâtiments avec
> formes irrégulières,  sinon il faut regarder de plus près et expliquer
> pourquoi.  Mes analyses au cours des derniers 6 mois, incluant révision de
> Butembo en RDC montrent qu'avec une bonne révision on repère beaucoup de
> bâtiments qui ne correspondent pas à ce que l'on observe sur l'imagerie.
>
>
> Je n'ai pas réussi à télécharger le fichier  ontario-simplified.tar.xz à
> partir de Google drive,  ni avec Chrome, ni avec Firefox ni à l'aide d'un
> script wget. Je ne sais si windows 10 peut ajouter des restrictions que
> d'autres OS n'ont pas.
>
> Pierre
>
>
> Le samedi 19 janvier 2019 17 h 01 min 22 s HNE, James 
> a écrit :
>
> Is there no one that will analyse the data I've posted here? 
> https://drive.google.com/file/d/1OK83yrPwMW4nefyu-6JsIInu0meK2rW6/view?usp=sharing
> or are we just email thread warriors?
>
>

-- 
外に遊びに行こう！
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] 2020 building import wiki comment by Nate Wessel

[Pierre Béland via Talk-ca]

Mon problème est dés le téléchargement. Cela m'indique que google ne peut 
effectuer l'analyse d'antivirus et m'offre de télécharger quand même. De là, 
message d'erreur.Voici le message avec Chrome
Ce site ne peut pas fournir de connexion sécurisée

doc-08-3g-docs.googleusercontent.com a envoyé une réponse incorrecte.
   
   - Essayez d'exécuter les diagnostics réseau de Windows.
ERR_SSL_PROTOCOL_ERROR
 
Pierre 
 

Le samedi 19 janvier 2019 23 h 10 min 04 s HNE, James  
a écrit :  
 
 tar.xz c'est un fichier compressé contenant un fichier geojson(il se ouvre 
bien avec 7zip sur windows, ou n'importe quel program d'archive sur mac ou 
Linux), qui est servie via la Task Manager. Ce n'est pas la version originale 
de Stats Can, tu as raison, c'sst la version minifié sans les extras tag de 
merde qui sont aucunement utile à OSM.
Je travaille sur les fichiers, car quelqu'un a dit les données était de la 
bouse de vache.
On Sat., Jan. 19, 2019, 11:02 p.m. Pierre Béland  a 
écrit : 

Is there no one that will analyse the data I've posted here? 
https://drive.google.com/file/d/1OK83yrPwMW4nefyu-6JsIInu0meK2rW6/view?usp=sharing
 or are we just email thread warriors?


  ___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] 2020 building import wiki comment by Nate Wessel

[James]

tar.xz c'est un fichier compressé contenant un fichier geojson(il se ouvre
bien avec 7zip sur windows, ou n'importe quel program d'archive sur mac ou
Linux), qui est servie via la Task Manager. Ce n'est pas la version
originale de Stats Can, tu as raison, c'sst la version minifié sans les
extras tag de merde qui sont aucunement utile à OSM.

Je travaille sur les fichiers, car quelqu'un a dit les données était de la
bouse de vache.

On Sat., Jan. 19, 2019, 11:02 p.m. Pierre Béland  James,
>
> Je pense que nous travaillons sur deux aspects différents. Tu te concentre
> sur la production / correction des fichiers d'import.
>
> A ma connaissance, tu n'a pas décrit le contenu du fichier
> ontario-simplified.tar.xz.  Je suppose que ce sont les données d'import
> originales où tu apportes divers correctifs.
>
> De mon côté, je propose d'analyser le produit final dans la base OSM et
> mesurer pour les données déja téléchargées quelle proportion des bâtiments
> est avec angles droits (orthogonal), et les superspositions.  Cela
> permettrait simplement de comparer ces données avec ce que l'on retrouve
> ailleurs et rassurer sur la qualité globale de l'import par les différents
> contributeurs.
>
> Normalement, on ne devrait pas retrouver plus de 5% des bâtiments avec
> formes irrégulières,  sinon il faut regarder de plus près et expliquer
> pourquoi.  Mes analyses au cours des derniers 6 mois, incluant révision de
> Butembo en RDC montrent qu'avec une bonne révision on repère beaucoup de
> bâtiments qui ne correspondent pas à ce que l'on observe sur l'imagerie.
>
>
> Je n'ai pas réussi à télécharger le fichier  ontario-simplified.tar.xz à
> partir de Google drive,  ni avec Chrome, ni avec Firefox ni à l'aide d'un
> script wget. Je ne sais si windows 10 peut ajouter des restrictions que
> d'autres OS n'ont pas.
>
> Pierre
>
>
> Le samedi 19 janvier 2019 17 h 01 min 22 s HNE, James 
> a écrit :
>
> Is there no one that will analyse the data I've posted here? 
> https://drive.google.com/file/d/1OK83yrPwMW4nefyu-6JsIInu0meK2rW6/view?usp=sharing
> or are we just email thread warriors?
>
>
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] 2020 building import wiki comment by Nate Wessel

[Pierre Béland via Talk-ca]

James,

Je pense que nous travaillons sur deux aspects différents. Tu te concentre sur 
la production / correction des fichiers d'import.
A ma connaissance, tu n'a pas décrit le contenu dufichier 
ontario-simplified.tar.xz.  Je suppose que ce sont les données d'import 
originales où tu apportes divers correctifs.
De mon côté, je propose d'analyser le produit final dans la base OSM et mesurer 
pour les données déja téléchargées quelle proportion des bâtiments est avec 
angles droits (orthogonal), et les superspositions.  Cela permettrait 
simplement de comparer ces données avec ce que l'on retrouve ailleurs et 
rassurer sur la qualité globale de l'import par les différents contributeurs. 

Normalement, on ne devrait pas retrouver plus de 5% des bâtiments avec formes 
irrégulières,  sinon il faut regarder de plus près et expliquer pourquoi.  Mes 
analyses au cours des derniers 6 mois, incluant révision de Butembo en RDC 
montrent qu'avec une bonne révision on repère beaucoup de bâtiments qui ne 
correspondent pas à ce que l'on observe sur l'imagerie.

Je n'ai pas réussi à télécharger le fichier  ontario-simplified.tar.xz à partir 
de Google drive,  ni avec Chrome, ni avec Firefox ni à l'aide d'un script wget. 
Je ne sais si windows 10 peut ajouter des restrictions que d'autres OS n'ont 
pas.

Pierre 


Le samedi 19 janvier 2019 17 h 01 min 22 s HNE, James  a 
écrit : 

Is there no one that will analyse the data I've posted here? 
https://drive.google.com/file/d/1OK83yrPwMW4nefyu-6JsIInu0meK2rW6/view?usp=sharing
 or are we just email thread warriors?

___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] 2020 building import wiki comment by Nate Wessel

[john whelan]

So my interpretation is we save about 9% by simplifying?

However does it mean we probably do need to verify the before and after
manually?

Would someone care to comment on if we would need to manually look at the
before and after and if 9% savings would be significant?

My personal feeling would be at 50% saving it would be well worthwhile but
at a lower figure I'm not sure where the cut off would be.

Thanks John

On Sat, Jan 19, 2019, 6:48 PM James  Resending because these emails are getting over 40KB in size and talk list
> is spazzing out:
> Original:
> 9.263 Average points per feature
> Points:20346517
> Features:2196329
>
> Simplified (20cm): 8.425 Average points per feature
> Points:18504036
> Features:2196329
> ___
> Talk-ca mailing list
> Talk-ca@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk-ca
>
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] 2020 building import wiki comment by Nate Wessel

[James]

Resending because these emails are getting over 40KB in size and talk list
is spazzing out:
Original:
9.263 Average points per feature
Points:20346517
Features:2196329

Simplified (20cm): 8.425 Average points per feature
Points:18504036
Features:2196329
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] 2020 building import wiki comment by Nate Wessel

[john whelan]

It appears the original discussion on talk-ca was missed by many.

There appears to be an issue of local buy-in but I think for the moment we
should first concentrate on the problems we expect to see.

The building outlines we imported in Ottawa predated the Bing imagery I
suspect.

So first off what should be done with buildings in the import that are not
on ESRI or Bing?  Assuming they are being imported by an armchair mapper.

My suggestion is they be ignored.  Realistically we aren't going to capture
every building in Canada but it should give us a reasonable start.

For buildings that already exist I see two possibilities.  First is do not
import, the second import but copy over the history i.e. merge.

I have no strong feelings either way.

For buildings that are imported my suggestion is they should be tagged
building=yes unless there is local knowledge or mapilary or openstreetcam
imagery to support something different.

Should the import be preprocessed to square the buildings or remove small
blobs?

I hear two points of view, one says it should be the other says bring it in
as it is.

My personal gut feel is bring it in as is.  However I don't have strong
views on this.

I'm hearing the import plan should have more detailed instructions.  I
think some were placed on the task manager.

Have I missed anything?

Can we try for consensus or what I have listed?

I suggest we put aside what is local buy in and how we obtain it for the
moment.

Thanks John

On Sat, Jan 19, 2019, 5:31 PM Begin Daniel  It seems that talk-ca works this way - long periods of silence, then a
> burst of emails because something went wrong! I just realized that this
> massive import of buildings started a few weeks ago and I’m surprised.
>
>
>
> I was trying to stay informed about the project because I was worried
> about how the import would deal with the thousands of buildings I have
> modified (or deleted) over the years.
>
>
>
> I then tend to agree with Steve, Andrew and Nate. Something happened too
> fast or went wrong. There is so much to discuss about how to proceed, how
> to deal with existing buildings, how to deal with buildings that do not
> seem to exist on images, etc. I would have liked to participate in the
> discussion. All of this may have been dealt with the import project of
> Ottawa, however, as this particular project did not interest me, I did not
> follow the discussions. I was expecting the new import project to be
> documented and discussed by itself, even if it eventually ends-up as a copy
> paste of Ottawa import’s procedures.
>
>
>
> Communication and clear statements are required to make such large import
> a success and a buy-in of all the community. I think this is why following
> the import guideline is important.
>
>
>
> Daniel
> ___
> Talk-ca mailing list
> Talk-ca@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk-ca
>
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] 2020 building import wiki comment by Nate Wessel

[Begin Daniel]

It seems that talk-ca works this way - long periods of silence, then a burst of 
emails because something went wrong! I just realized that this massive import 
of buildings started a few weeks ago and I'm surprised.

I was trying to stay informed about the project because I was worried about how 
the import would deal with the thousands of buildings I have modified (or 
deleted) over the years.

I then tend to agree with Steve, Andrew and Nate. Something happened too fast 
or went wrong. There is so much to discuss about how to proceed, how to deal 
with existing buildings, how to deal with buildings that do not seem to exist 
on images, etc. I would have liked to participate in the discussion. All of 
this may have been dealt with the import project of Ottawa, however, as this 
particular project did not interest me, I did not follow the discussions. I was 
expecting the new import project to be documented and discussed by itself, even 
if it eventually ends-up as a copy paste of Ottawa import's procedures.

Communication and clear statements are required to make such large import a 
success and a buy-in of all the community. I think this is why following the 
import guideline is important.

Daniel
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] 2020 building import wiki comment by Nate Wessel

[OSM Volunteer stevea]

Yeah, even going to 8 GB (-Xmx8192m on my java launch command line) I hang 
about halfway through opening "Loading json file."  I'll try 16 GB (whew!) and 
QGIS next.

Steve

> On Jan 19, 2019, at 2:17 PM, James  wrote:
> 
> use qgis or launch josm with java in 64bit mode(d64) with memory options 
> (Xms, Xmx), or it will crap out at 3.5GB
> 
> On Sat., Jan. 19, 2019, 5:13 p.m. OSM Volunteer stevea 
>  On Jan 19, 2019, at 2:01 PM, James  wrote:
> > Is there no one that will analyse the data I've posted here? 
> > https://drive.google.com/file/d/1OK83yrPwMW4nefyu-6JsIInu0meK2rW6/view?usp=sharing
> >  or are we just email thread warriors?
> 
> Well, slow down there, cowboy, it is gigabytes of data and I've been busy 
> replying to talk-ca, though my CPU is hot decompressing and opening your file 
> right now.  Though even with JOSM getting assigned multiple gigabytes of heap 
> space (and I've got plenty dozens of GB of RAM), I'm still getting 
> out-of-memory errors on trying to bite-and-chew this monster.
> 
> Maybe some of ARE email thread warriors, but I am not "just" an email thread 
> warrior.
> 
> SteveA
> California


___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] 2020 building import wiki comment by Nate Wessel

[James]

use qgis or launch josm with java in 64bit mode(d64) with memory options
(Xms, Xmx), or it will crap out at 3.5GB

On Sat., Jan. 19, 2019, 5:13 p.m. OSM Volunteer stevea <
stevea...@softworkers.com wrote:

> On Jan 19, 2019, at 2:01 PM, James  wrote:
> > Is there no one that will analyse the data I've posted here?
> https://drive.google.com/file/d/1OK83yrPwMW4nefyu-6JsIInu0meK2rW6/view?usp=sharing
> or are we just email thread warriors?
>
> Well, slow down there, cowboy, it is gigabytes of data and I've been busy
> replying to talk-ca, though my CPU is hot decompressing and opening your
> file right now.  Though even with JOSM getting assigned multiple gigabytes
> of heap space (and I've got plenty dozens of GB of RAM), I'm still getting
> out-of-memory errors on trying to bite-and-chew this monster.
>
> Maybe some of ARE email thread warriors, but I am not "just" an email
> thread warrior.
>
> SteveA
> California
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] 2020 building import wiki comment by Nate Wessel

[Nate Wessel]

Hi James,

I for one will be happy to talk about the technical issues of 
simplification... though I think at the moment it seems to be taking a 
backseat here to bigger issues like building consensus around the 
validity of this import.


Since this is a more technical issue, can I suggest that you share your 
data (original and simplified) along with a description of the 
methodology you used and any relevant stats on the wiki page? I'll be 
curious to see how much shapes have changed vs. how many points were 
removed. Perhaps Some measure of self-intersection between the 
simplified and unsimplified shapes, compared with reductions in the 
average number of nodes per shape? That's just the first thing that 
comes to mind - there may be better ways to assess the quality, 
especially as it varies between municipalities.


Someone also raised a concern about orthogonality (is that a word?) 
which will need to be measured in this context somehow as well.


Thanks,

Nate Wessel
Jack of all trades, Master of Geography, PhD candidate in Urban Planning
NateWessel.com 

On 1/19/19 5:01 PM, James wrote:
Is there no one that will analyse the data I've posted here? 
https://drive.google.com/file/d/1OK83yrPwMW4nefyu-6JsIInu0meK2rW6/view?usp=sharing 
or are we just email thread warriors?


On Sat., Jan. 19, 2019, 4:29 p.m. Pierre Béland via Talk-ca 
mailto:talk-ca@openstreetmap.org> wrote:


Voici une compilation statistique à partir des changesets de OSM
pour la grande région de Toronto
(bbox=-80.8951,42.9142,-78.1046,44.3297). Depuis

Pour respecter la politique de confidentialité de OSM, les
contributeurs ne sont identifiés que selon leur uid.

Depuis décembre, on compte 6 contributeurs, 889 changesets et
6,621,124 objets édités (noeuds, chemins ou relations)

1 immeuble = 5 objets minimum
Le fichier de changeset ne fournit que le nombre d'objets édités
(var num_changes). On ne peut donc à cette étape-ci que
grossièrement estimer le nombre d'immeubles, sans doute plus de 1
million déjà. A partir de Overpass, on ne peut facilement extraire
les données. Je vais télécharger dans PostGIS un fichier récent de
l'Ontario et sélectionner les données éditées depuis le 24
décembre (premier jour avec édition )  de ces 6 contributeurs. 
Mon script Qualité permettra également d'analyser la géométrie des
bâtiments.

lexique
changeset         nombre de changesets (sessions d'édition)
num_changes    nombre d'objets édités dans la session
avg_changes     moyenne, objets édités par changeset

Tableau 1 - Objets édités par contributeur avec moyenne par changeset

uid changesets  num_changes avg_changes % cum
9266764 457 3,439,242   7,526   52.7%
215433  197 1,619,540   8,221   77.6%
5214232 151 815,541 5,401   90.1%
9343732 78  628,032 8,052   99.7%
3016192 3   17,710  5,903   100.0%
1919010 3   1,059   353 100.0%

889 6,521,124   7,335   



Tableau 2 - Objets édités par contributeur - jour avec moyenne par
changeset
uid jourchangesets  num_changes avg_changes
5214232 2018-12-28  3   10,324  3,441
5214232 2019-01-14  11  58,402  5,309
5214232 2019-01-17  9   57,738  6,415
9343732 2019-01-16  7   63,490  9,070
215433  2018-12-29  8   70,562  8,820
5214232 2019-01-16  25  176,942 7,078
5214232 2018-12-23  5   29,279  5,856
9343732 2019-01-14  5   39,000  7,800
5214232 2019-01-12  10  48,329  4,833
5214232 2018-12-24  9   54,888  6,099
215433  2019-01-08  13  77,332  5,949
9266764 2019-01-08  20  156,232 7,812
9266764 2019-01-01  9   67,167  7,463
9266764 2019-01-07  19  154,837 8,149
215433  2019-01-02  30  280,756 9,359
9266764 2018-12-24  11  66,162  6,015
5214232 2019-01-11  2   6,219   3,110
9343732 2019-01-15  10  75,020  7,502
9343732 2019-01-11  11  82,844  7,531
215433  2018-12-30  20  177,602 8,880
9266764 2018-12-31  30  195,300 6,510
9343732 2019-01-12  14  111,821 7,987
215433  2019-01-03  25  200,002 8,000
9266764 2019-01-10  33  260,886 7,906
215433  2019-01-09  17  138,489 8,146
215433  2018-12-31  21  169,903 8,091
5214232 2019-01-09  9   41,664  4,629
9266764 2019-01-13  32  266,247 8,320
9266764 2018-12-29  36  274,357 7,621
9343732 2019-01-10  14  112,606 

Re: [Talk-ca] 2020 building import wiki comment by Nate Wessel

[OSM Volunteer stevea]

On Jan 19, 2019, at 2:01 PM, James  wrote:
> Is there no one that will analyse the data I've posted here? 
> https://drive.google.com/file/d/1OK83yrPwMW4nefyu-6JsIInu0meK2rW6/view?usp=sharing
>  or are we just email thread warriors?

Well, slow down there, cowboy, it is gigabytes of data and I've been busy 
replying to talk-ca, though my CPU is hot decompressing and opening your file 
right now.  Though even with JOSM getting assigned multiple gigabytes of heap 
space (and I've got plenty dozens of GB of RAM), I'm still getting 
out-of-memory errors on trying to bite-and-chew this monster.

Maybe some of ARE email thread warriors, but I am not "just" an email thread 
warrior.

SteveA
California
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] 2020 building import wiki comment by Nate Wessel

[James]

Is there no one that will analyse the data I've posted here?
https://drive.google.com/file/d/1OK83yrPwMW4nefyu-6JsIInu0meK2rW6/view?usp=sharing
or are we just email thread warriors?

On Sat., Jan. 19, 2019, 4:29 p.m. Pierre Béland via Talk-ca <
talk-ca@openstreetmap.org wrote:

> Voici une compilation statistique à partir des changesets de OSM pour la
> grande région de Toronto (bbox=-80.8951,42.9142,-78.1046,44.3297).
> Depuis
>
> Pour respecter la politique de confidentialité de OSM, les contributeurs
> ne sont identifiés que selon leur uid.
>
> Depuis décembre, on compte 6 contributeurs, 889 changesets et 6,621,124
> objets édités (noeuds, chemins ou relations)
>
> 1 immeuble = 5 objets minimum
> Le fichier de changeset ne fournit que le nombre d'objets édités (var
> num_changes). On ne peut donc à cette étape-ci que grossièrement estimer le
> nombre d'immeubles, sans doute plus de 1 million déjà. A partir de
> Overpass, on ne peut facilement extraire les données. Je vais télécharger
> dans PostGIS un fichier récent de l'Ontario et sélectionner les données
> éditées depuis le 24 décembre (premier jour avec édition )  de ces 6
> contributeurs.  Mon script Qualité permettra également d'analyser la
> géométrie des bâtiments.
>
> lexique
> changeset nombre de changesets (sessions d'édition)
> num_changesnombre d'objets édités dans la session
> avg_changes moyenne, objets édités par changeset
>
> Tableau 1 - Objets édités par contributeur avec moyenne par changeset
>
> uid changesets num_changes avg_changes % cum
> 9266764 457 3,439,242 7,526 52.7%
> 215433 197 1,619,540 8,221 77.6%
> 5214232 151 815,541 5,401 90.1%
> 9343732 78 628,032 8,052 99.7%
> 3016192 3 17,710 5,903 100.0%
> 1919010 3 1,059 353 100.0%
>
> 889 6,521,124 7,335
>
>
> Tableau 2 - Objets édités par contributeur - jour avec moyenne par
> changeset
> uid jour changesets num_changes avg_changes
> 5214232 2018-12-28 3 10,324 3,441
> 5214232 2019-01-14 11 58,402 5,309
> 5214232 2019-01-17 9 57,738 6,415
> 9343732 2019-01-16 7 63,490 9,070
> 215433 2018-12-29 8 70,562 8,820
> 5214232 2019-01-16 25 176,942 7,078
> 5214232 2018-12-23 5 29,279 5,856
> 9343732 2019-01-14 5 39,000 7,800
> 5214232 2019-01-12 10 48,329 4,833
> 5214232 2018-12-24 9 54,888 6,099
> 215433 2019-01-08 13 77,332 5,949
> 9266764 2019-01-08 20 156,232 7,812
> 9266764 2019-01-01 9 67,167 7,463
> 9266764 2019-01-07 19 154,837 8,149
> 215433 2019-01-02 30 280,756 9,359
> 9266764 2018-12-24 11 66,162 6,015
> 5214232 2019-01-11 2 6,219 3,110
> 9343732 2019-01-15 10 75,020 7,502
> 9343732 2019-01-11 11 82,844 7,531
> 215433 2018-12-30 20 177,602 8,880
> 9266764 2018-12-31 30 195,300 6,510
> 9343732 2019-01-12 14 111,821 7,987
> 215433 2019-01-03 25 200,002 8,000
> 9266764 2019-01-10 33 260,886 7,906
> 215433 2019-01-09 17 138,489 8,146
> 215433 2018-12-31 21 169,903 8,091
> 5214232 2019-01-09 9 41,664 4,629
> 9266764 2019-01-13 32 266,247 8,320
> 9266764 2018-12-29 36 274,357 7,621
> 9343732 2019-01-10 14 112,606 8,043
> 1919010 2019-01-17 2 973 487
> 9266764 2018-12-26 29 216,701 7,472
> 215433 2019-01-07 12 93,825 7,819
> 5214232 2019-01-10 2 718 359
> 9266764 2019-01-09 25 216,885 8,675
> 5214232 2019-01-08 27 119,761 4,436
> 5214232 2018-12-25 2 10,528 5,264
> 9266764 2019-01-06 18 142,007 7,889
> 215433 2019-01-01 6 58,083 9,681
> 9266764 2019-01-04 7 39,155 5,594
> 1919010 2019-01-16 1 86 86
> 9266764 2019-01-11 22 174,419 7,928
> 9266764 2018-12-25 29 188,931 6,515
> 9266764 2018-12-27 10 67,593 6,759
> 3016192 2018-12-20 3 17,710 5,903
> 215433 2019-01-04 15 128,047 8,536
> 9266764 2018-12-30 14 108,068 7,719
> 9266764 2019-01-15 21 150,419 7,163
> 5214232 2019-01-15 19 116,812 6,148
> 9266764 2019-01-14 17 135,021 7,942
> 9343732 2019-01-13 17 143,251 8,427
> 9266764 2019-01-12 16 120,488 7,531
> 215433 2019-01-06 13 103,233 7,941
> 5214232 2019-01-13 11 57,670 5,243
> 5214232 2019-01-07 4 7,256 1,814
> 215433 2019-01-05 13 99,108 7,624
> 9266764 2019-01-05 4 25,286 6,322
> 9266764 2019-01-16 24 204,397 8,517
> 215433 2019-01-17 2 18,589 9,295
> 215433 2018-12-27 2 4,009 2,005
> 9266764 2019-01-02 13 71,064 5,466
> 9266764 2019-01-03 18 137,620 7,646
> 5214232 2018-12-27 3 19,011 6,337
>
>
> 889 6,521,124 7,335
>
>
> Pierre
>
> ___
> Talk-ca mailing list
> Talk-ca@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk-ca
>
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] 2020 building import wiki comment by Nate Wessel

[Pierre Béland via Talk-ca]

Voici une compilation statistique à partir des changesets de OSM pour la grande 
région de Toronto (bbox=-80.8951,42.9142,-78.1046,44.3297).  Depuis 
Pour respecter la politique de confidentialité de OSM, les contributeurs ne 
sont identifiés que selon leur uid.
Depuis décembre, on compte 6 contributeurs, 889 changesets et 6,621,124 objets 
édités (noeuds, chemins ou relations) 
1 immeuble = 5 objets minimumLe fichier de changeset ne fournit que le nombre 
d'objets édités (var num_changes). On ne peut donc à cette étape-ci que 
grossièrement estimer le nombre d'immeubles, sans doute plus de 1 million déjà. 
A partir de Overpass, on ne peut facilement extraire les données. Je vais 
télécharger dans PostGIS un fichier récent de l'Ontario et sélectionner les 
données éditées depuis le 24 décembre (premier jour avec édition )  de ces 6 
contributeurs.  Mon script Qualité permettra également d'analyser la géométrie 
des bâtiments.
lexique
changeset         nombre de changesets (sessions d'édition)num_changes    
nombre d'objets édités dans la sessionavg_changes     moyenne, objets édités 
par changeset
Tableau 1 - Objets édités par contributeur avec moyenne par changeset

| uid | changesets | num_changes | avg_changes | % cum |
| 9266764 | 457 | 3,439,242 | 7,526 | 52.7% |
| 215433 | 197 | 1,619,540 | 8,221 | 77.6% |
| 5214232 | 151 | 815,541 | 5,401 | 90.1% |
| 9343732 | 78 | 628,032 | 8,052 | 99.7% |
| 3016192 | 3 | 17,710 | 5,903 | 100.0% |
| 1919010 | 3 | 1,059 | 353 | 100.0% |
| 
 | 889 | 6,521,124 | 7,335 | 
 |



Tableau 2 - Objets édités par contributeur - jour avec moyenne par changeset

| uid | jour | changesets | num_changes | avg_changes |
| 5214232 | 2018-12-28 | 3 | 10,324 | 3,441 |
| 5214232 | 2019-01-14 | 11 | 58,402 | 5,309 |
| 5214232 | 2019-01-17 | 9 | 57,738 | 6,415 |
| 9343732 | 2019-01-16 | 7 | 63,490 | 9,070 |
| 215433 | 2018-12-29 | 8 | 70,562 | 8,820 |
| 5214232 | 2019-01-16 | 25 | 176,942 | 7,078 |
| 5214232 | 2018-12-23 | 5 | 29,279 | 5,856 |
| 9343732 | 2019-01-14 | 5 | 39,000 | 7,800 |
| 5214232 | 2019-01-12 | 10 | 48,329 | 4,833 |
| 5214232 | 2018-12-24 | 9 | 54,888 | 6,099 |
| 215433 | 2019-01-08 | 13 | 77,332 | 5,949 |
| 9266764 | 2019-01-08 | 20 | 156,232 | 7,812 |
| 9266764 | 2019-01-01 | 9 | 67,167 | 7,463 |
| 9266764 | 2019-01-07 | 19 | 154,837 | 8,149 |
| 215433 | 2019-01-02 | 30 | 280,756 | 9,359 |
| 9266764 | 2018-12-24 | 11 | 66,162 | 6,015 |
| 5214232 | 2019-01-11 | 2 | 6,219 | 3,110 |
| 9343732 | 2019-01-15 | 10 | 75,020 | 7,502 |
| 9343732 | 2019-01-11 | 11 | 82,844 | 7,531 |
| 215433 | 2018-12-30 | 20 | 177,602 | 8,880 |
| 9266764 | 2018-12-31 | 30 | 195,300 | 6,510 |
| 9343732 | 2019-01-12 | 14 | 111,821 | 7,987 |
| 215433 | 2019-01-03 | 25 | 200,002 | 8,000 |
| 9266764 | 2019-01-10 | 33 | 260,886 | 7,906 |
| 215433 | 2019-01-09 | 17 | 138,489 | 8,146 |
| 215433 | 2018-12-31 | 21 | 169,903 | 8,091 |
| 5214232 | 2019-01-09 | 9 | 41,664 | 4,629 |
| 9266764 | 2019-01-13 | 32 | 266,247 | 8,320 |
| 9266764 | 2018-12-29 | 36 | 274,357 | 7,621 |
| 9343732 | 2019-01-10 | 14 | 112,606 | 8,043 |
| 1919010 | 2019-01-17 | 2 | 973 | 487 |
| 9266764 | 2018-12-26 | 29 | 216,701 | 7,472 |
| 215433 | 2019-01-07 | 12 | 93,825 | 7,819 |
| 5214232 | 2019-01-10 | 2 | 718 | 359 |
| 9266764 | 2019-01-09 | 25 | 216,885 | 8,675 |
| 5214232 | 2019-01-08 | 27 | 119,761 | 4,436 |
| 5214232 | 2018-12-25 | 2 | 10,528 | 5,264 |
| 9266764 | 2019-01-06 | 18 | 142,007 | 7,889 |
| 215433 | 2019-01-01 | 6 | 58,083 | 9,681 |
| 9266764 | 2019-01-04 | 7 | 39,155 | 5,594 |
| 1919010 | 2019-01-16 | 1 | 86 | 86 |
| 9266764 | 2019-01-11 | 22 | 174,419 | 7,928 |
| 9266764 | 2018-12-25 | 29 | 188,931 | 6,515 |
| 9266764 | 2018-12-27 | 10 | 67,593 | 6,759 |
| 3016192 | 2018-12-20 | 3 | 17,710 | 5,903 |
| 215433 | 2019-01-04 | 15 | 128,047 | 8,536 |
| 9266764 | 2018-12-30 | 14 | 108,068 | 7,719 |
| 9266764 | 2019-01-15 | 21 | 150,419 | 7,163 |
| 5214232 | 2019-01-15 | 19 | 116,812 | 6,148 |
| 9266764 | 2019-01-14 | 17 | 135,021 | 7,942 |
| 9343732 | 2019-01-13 | 17 | 143,251 | 8,427 |
| 9266764 | 2019-01-12 | 16 | 120,488 | 7,531 |
| 215433 | 2019-01-06 | 13 | 103,233 | 7,941 |
| 5214232 | 2019-01-13 | 11 | 57,670 | 5,243 |
| 5214232 | 2019-01-07 | 4 | 7,256 | 1,814 |
| 215433 | 2019-01-05 | 13 | 99,108 | 7,624 |
| 9266764 | 2019-01-05 | 4 | 25,286 | 6,322 |
| 9266764 | 2019-01-16 | 24 | 204,397 | 8,517 |
| 215433 | 2019-01-17 | 2 | 18,589 | 9,295 |
| 215433 | 2018-12-27 | 2 | 4,009 | 2,005 |
| 9266764 | 2019-01-02 | 13 | 71,064 | 5,466 |
| 9266764 | 2019-01-03 | 18 | 137,620 | 7,646 |
| 5214232 | 2018-12-27 | 3 | 19,011 | 6,337 |
| 
 | 
 | 889 | 6,521,124 | 7,335 |



 
Pierre 
 
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] 2020 building import wiki comment by Nate Wessel

[Andrew Lester]

I was surprised this morning when I saw that a chunk of buildings had been 
imported in Victoria, BC. The changeset linked to the wiki plan and I then 
checked my email and saw this email chain. 

The "local" (in this case the Canadian community) has not been sufficiently 
consulted. Looking back in the mailing list, there were some tangential 
discussions about some things related to this import (mostly without any final 
consensus), and then a single email stating that the import plan had been 
created and sent off to the imports list 
(https://lists.openstreetmap.org/pipermail/talk-ca/2018-November/008864.html). 
After that, nothing. I can't find any emails related to this import that were 
linked to both the imports and talk-ca list, nor are there any that bring back 
the results from the imports list for those who aren't following that. There 
also wasn't any notification that the import was actually going ahead. I'd 
consider all of this to be a major failure of the "Community Buy-in" section of 
the Import/Guidelines. 

For such a major import, we should be going above-and-beyond to make sure every 
possible aspect has been addressed adequately. The lack of confidence from the 
general OSM community as a result of the botched import in Ottawa and the 
ongoing dislike of our CANVEC imports means we need to treat this import with 
kid gloves. We should be striving to ensure that there's no reason why someone 
could look at this import and find faults with it. That may seem like a lofty 
goal, but we're talking about a building import for the second-largest country 
in the world. 

Once the administrative portions are dealt with and the community has been 
sufficiently consulted, the technical area needs to be looked at. Now that I've 
seen some of the data in action, there are various issues that need to be dealt 
with. Some that became immediately apparent on the data imported in Victoria, 
BC include: 
-A significant number of unsquare buildings (JOSM validator reports this as an 
Other/Building with an almost square angle). Of an estimated 935 buildings in 
this chunk, 692 have almost-square angles. Looking more closely and running the 
JOSM Orthogonalize tool on a sampling of buildings, I believe all of them have 
unsquare angles. This may be the result of rounding errors in data conversion 
and should be fixed in the source data before importing. 
-Inconsistent tagging (some houses are building=yes, some are 
building=detached) 
-A need for simplification (extra nodes in nearly-straight segments that are 
straight in reality) 

I'd suggest the following plan: 
1. Update the tasking manager to indicate in clear terms that this import is on 
hold and no data should be imported at this time. Ideally, the tasking manager 
should be taken down entirely so no data can be imported. 
2. Send a clear, unambiguous email to the talk-ca list indicating that this 
import is being planned and to solicit feedback. 
3. Wait. THIS IS IMPORTANT! The community needs to be given time to see that 
this import is being planned, and to discuss the many aspects related to it. 
For such a major import, silence-as-tacit-acceptance doesn't fly. There are 
local communities out there that need to be brought in to the process. If 
necessary, figure out who the active contributors are in various jurisdictions 
and contact them directly. 
4. Figure out the technical details. It's only after the import had already 
started that people are now talking about conflation and data quality. These 
need to be figured out, a plan documented, and the source data cleaned. Tags 
also need to be clarified. The current wiki plan gives almost no guidance about 
how to actually perform the import. 
5. Only after all of the above has been figured out, let the community know 
that the import is actually going ahead. 

Come on, people. We can do a lot better than this, and definitely should. Let's 
make this a shining example of why imports can be a good thing for OSM, not 
provide fodder for those opposed to them. 

Andrew Lester 
Victoria, BC 


From: "John Whelan"  
To: "Nate Wessel"  
Cc: "talk-ca"  
Sent: Saturday, January 19, 2019 5:07:52 AM 
Subject: Re: [Talk-ca] 2020 building import wiki comment by Nate Wessel 

I'm saying when the matter was first raised on the import mailing list as a 
heads up I made reference to the existing Ottawa pilot and gave a link 
basically saying we would be following the same pattern. There was considerable 
discussion around the Ottawa import plan both on the import plan and talk-ca 
and the Ottawa import which I didn't draw up. Later there was a formal link to 
the data import plan. 

So two stages if you like. This is what we are thinking of doing and this is 
how we intend to proceed. 

Cheerio John 

Nate Wessel wrote on 2019-01-18 11:38 PM: 




John, 


I'm sorry to keep saying this, but I really do not think this is an acceptable 
import ap

Re: [Talk-ca] 2020 building import wiki comment by Nate Wessel

[John Whelan]

I'm saying when the matter was first raised on the import mailing list 
as a heads up I made reference to the existing Ottawa pilot and gave a 
link basically saying we would be following the same pattern.  There was 
considerable discussion around the Ottawa import plan both on the import 
plan and talk-ca and the Ottawa import which I didn't draw up.  Later 
there was a formal link to the data import plan.


So two stages if you like.  This is what we are thinking of doing and 
this is how we intend to proceed.


Cheerio John

Nate Wessel wrote on 2019-01-18 11:38 PM:


John,

I'm sorry to keep saying this, but I really do not think this is an 
acceptable import approval process.


You're saying there was no wiki describing the plan when this went to 
the imports mailing list - only a link to a similar plan with related 
data. You did not follow the import guidelines and you need to go back 
and read that page line by line and follow the procedures that we have 
in place.

https://wiki.openstreetmap.org/wiki/Import/Guidelines
I'll go ahead and add a mention of this plan to the imports catalogue 
to get us started. I'll also add some sections to the wiki and try to 
leave some indication of where things can be better documented.


You may think I'm quibbling over procedural details, but I think this 
process is really important. If we were talking about importing 
buildings in one neighborhood, I would look the other way, but this is 
all of Canada. This is a huge, huge import and we need to take the 
time to do things right, and especially to document the process so 
people can get involved that aren't already.


Best,

Nate Wessel
Jack of all trades, Master of Geography, PhD candidate in Urban Planning
NateWessel.com 

On 1/18/19 3:48 PM, John Whelan wrote:
The import mailing list was pointed to the correct page of the wiki.  
The initial post was to say this is what we were thinking of and 
there was a comment saying we needed to change the comment line.


>There is no mention of this proposed import on the import catalogue


The import process was reviewed by the person who set up the Ottawa 
import did we miss that step on the Ottawa import as well?  Neither 
was it raised as a concern on the import mailing list. I think this 
is very minor and can be corrected.


We learnt a fair bit on the Ottawa import and my expectation is since 
we are using experienced mappers to do the import conflation would be 
either handled by them or the building not imported. We aren't using 
new mappers in a mapathon here and with experienced mappers then I 
think you have to trust them. The world isn't perfect. Think in terms 
of service level.


>There are 2X more nodes than needed to represent the building 
accurately.


The problem with correcting this is you are introducing 
approximations.  This will vary according to the source and this can 
be simplified or corrected once its in OSM. I think this is a 
different issue of a mechanical edit that needs to be considered 
separately.


If we are concerned with database size then I suggest we change the 
instructions to say put the source comment on the change set rather 
than on the building outline.


Cheerio John


Nate Wessel wrote on 2019-01-18 3:06 PM:


John,

You seem to be playing the long game with this data - it sounds like 
you've been working with this a lot longer than I have, and you've 
put in the time and effort to help make this 
actually-quite-incredible dataset available to us. I don't want to 
stop the import from happening - quite the opposite. I just want to 
make sure that the time is taken to do this right. OSM deserves 
that. Your (our) long awaited victory will be the sweeter for our 
patience now.


There are several specific issues I see where the I's are not 
crossed, nor the t's dotted. I've mentioned several already, so I'll 
try to be brief (I really need to get back to working on my 
dissertation).


1) There was extremely limited discussion on the imports mailing 
list. The initial email did not make clear the scope of the project. 
I read the email and did not think twice at it, thinking it was 
entirely about Ottawa. The link in that email was actually to the 
Ottawa import, and not this one, which seems to have been only in 
draft at the time. 
https://lists.openstreetmap.org/pipermail/imports/2018-November/005812.html
As such, this project has NOT been reviewed by the imports list, 
which is a requirement for proceeding with the import.


2) There is no mention of this proposed import on the import 
catalogue (https://wiki.openstreetmap.org/wiki/Import/Catalogue)
which is required in the imports guidelines. I suspect many other 
guidelines have not been followed.


3) The wiki page describing the import is not adequate to assess the 
quality of the data or of the proposed import. See for example: 
https://wiki.openstreetmap.org/wiki/WikiProject_Canada/Canada_Stats_Canada_Building_Outlines_Import/Plan#Risks

Re: [Talk-ca] 2020 building import wiki comment by Nate Wessel

[Nate Wessel]

John,

I'm sorry to keep saying this, but I really do not think this is an 
acceptable import approval process.


You're saying there was no wiki describing the plan when this went to 
the imports mailing list - only a link to a similar plan with related 
data. You did not follow the import guidelines and you need to go back 
and read that page line by line and follow the procedures that we have 
in place.

https://wiki.openstreetmap.org/wiki/Import/Guidelines
I'll go ahead and add a mention of this plan to the imports catalogue to 
get us started. I'll also add some sections to the wiki and try to leave 
some indication of where things can be better documented.


You may think I'm quibbling over procedural details, but I think this 
process is really important. If we were talking about importing 
buildings in one neighborhood, I would look the other way, but this is 
all of Canada. This is a huge, huge import and we need to take the time 
to do things right, and especially to document the process so people can 
get involved that aren't already.


Best,

Nate Wessel
Jack of all trades, Master of Geography, PhD candidate in Urban Planning
NateWessel.com 

On 1/18/19 3:48 PM, John Whelan wrote:
The import mailing list was pointed to the correct page of the wiki.  
The initial post was to say this is what we were thinking of and there 
was a comment saying we needed to change the comment line.


>There is no mention of this proposed import on the import catalogue


The import process was reviewed by the person who set up the Ottawa 
import did we miss that step on the Ottawa import as well?  Neither 
was it raised as a concern on the import mailing list. I think this is 
very minor and can be corrected.


We learnt a fair bit on the Ottawa import and my expectation is since 
we are using experienced mappers to do the import conflation would be 
either handled by them or the building not imported. We aren't using 
new mappers in a mapathon here and with experienced mappers then I 
think you have to trust them. The world isn't perfect. Think in terms 
of service level.


>There are 2X more nodes than needed to represent the building accurately.

The problem with correcting this is you are introducing 
approximations.  This will vary according to the source and this can 
be simplified or corrected once its in OSM. I think this is a 
different issue of a mechanical edit that needs to be considered 
separately.


If we are concerned with database size then I suggest we change the 
instructions to say put the source comment on the change set rather 
than on the building outline.


Cheerio John


Nate Wessel wrote on 2019-01-18 3:06 PM:


John,

You seem to be playing the long game with this data - it sounds like 
you've been working with this a lot longer than I have, and you've 
put in the time and effort to help make this 
actually-quite-incredible dataset available to us. I don't want to 
stop the import from happening - quite the opposite. I just want to 
make sure that the time is taken to do this right. OSM deserves that. 
Your (our) long awaited victory will be the sweeter for our patience 
now.


There are several specific issues I see where the I's are not 
crossed, nor the t's dotted. I've mentioned several already, so I'll 
try to be brief (I really need to get back to working on my 
dissertation).


1) There was extremely limited discussion on the imports mailing 
list. The initial email did not make clear the scope of the project. 
I read the email and did not think twice at it, thinking it was 
entirely about Ottawa. The link in that email was actually to the 
Ottawa import, and not this one, which seems to have been only in 
draft at the time. 
https://lists.openstreetmap.org/pipermail/imports/2018-November/005812.html
As such, this project has NOT been reviewed by the imports list, 
which is a requirement for proceeding with the import.


2) There is no mention of this proposed import on the import 
catalogue (https://wiki.openstreetmap.org/wiki/Import/Catalogue)
which is required in the imports guidelines. I suspect many other 
guidelines have not been followed.


3) The wiki page describing the import is not adequate to assess the 
quality of the data or of the proposed import. See for example: 
https://wiki.openstreetmap.org/wiki/WikiProject_Canada/Canada_Stats_Canada_Building_Outlines_Import/Plan#Risks
The import guidelines call for a description of how conflation will 
be handled. The fact that two of the major importers seem to have a 
substantial disagreement about how to handle existing data indicates 
this was not well discussed and I can see that it isn't well documented.


4) The buildings need to be simplified, quite a bit actually. Most 
buildings have multiple nodes representing straight lines. This 
bloats the database and makes things harder to edit by hand later. 
There are probably 2x more nodes than are needed to represent the 
data accurately, making it 

Re: [Talk-ca] 2020 building import wiki comment by Nate Wessel

[Nate Wessel]

I'm not familiar with the tool, but that is essentially what I'm asking 
for -  nothing all that complicated. We would need to make sure we're 
not losing any valuable detail though, and ensure that topology is 
preserved where buildings share nodes.


Nate Wessel
Jack of all trades, Master of Geography, PhD candidate in Urban Planning
NateWessel.com 

On 1/18/19 4:24 PM, James wrote:
I can run all the shapefiles through qgis simplify tool if this 
resolves the issue...


On Fri., Jan. 18, 2019, 4:08 p.m. Nate Wessel  wrote:


With default settings in JOSM, sure. In the import I was working
on, we used a Douglas-Peucker algorithm with a 20cm threshold
(before the import started) and it worked beautifully. We had many
points that seemed to have been introduced in the shapefiles as
some kind of data artifact - they didn't add any detail to the
shape at all. This procedure removed almost all of them with no
discernible reduction in quality.

Nate Wessel
Jack of all trades, Master of Geography, PhD candidate in Urban
Planning
NateWessel.com 

On 1/18/19 4:03 PM, James wrote:

dare you to run simplify tool on anything remotely round, it will
make it look like garbage

On Fri., Jan. 18, 2019, 3:49 p.m. John Whelan
mailto:jwhelan0...@gmail.com> wrote:

The import mailing list was pointed to the correct page of
the wiki.  The initial post was to say this is what we were
thinking of and there was a comment saying we needed to
change the comment line.

>There is no mention of this proposed import on the import
catalogue


The import process was reviewed by the person who set up the
Ottawa import did we miss that step on the Ottawa import as
well?  Neither was it raised as a concern on the import
mailing list. I think this is very minor and can be corrected.

We learnt a fair bit on the Ottawa import and my expectation
is since we are using experienced mappers to do the import
conflation would be either handled by them or the building
not imported. We aren't using new mappers in a mapathon here
and with experienced mappers then I think you have to trust
them.  The world isn't perfect. Think in terms of service level.

>There are 2X more nodes than needed to represent the
building accurately.

The problem with correcting this is you are introducing
approximations.  This will vary according to the source and
this can be simplified or corrected once its in OSM. I think
this is a different issue of a mechanical edit that needs to
be considered separately.

If we are concerned with database size then I suggest we
change the instructions to say put the source comment on the
change set rather than on the building outline.

Cheerio John


Nate Wessel wrote on 2019-01-18 3:06 PM:


John,

You seem to be playing the long game with this data - it
sounds like you've been working with this a lot longer than
I have, and you've put in the time and effort to help make
this actually-quite-incredible dataset available to us. I
don't want to stop the import from happening - quite the
opposite. I just want to make sure that the time is taken to
do this right. OSM deserves that. Your (our) long awaited
victory will be the sweeter for our patience now.

There are several specific issues I see where the I's are
not crossed, nor the t's dotted. I've mentioned several
already, so I'll try to be brief (I really need to get back
to working on my dissertation).

1) There was extremely limited discussion on the imports
mailing list. The initial email did not make clear the scope
of the project. I read the email and did not think twice at
it, thinking it was entirely about Ottawa. The link in that
email was actually to the Ottawa import, and not this one,
which seems to have been only in draft at the time.

https://lists.openstreetmap.org/pipermail/imports/2018-November/005812.html
As such, this project has NOT been reviewed by the imports
list, which is a requirement for proceeding with the import.

2) There is no mention of this proposed import on the import
catalogue (https://wiki.openstreetmap.org/wiki/Import/Catalogue)
which is required in the imports guidelines. I suspect many
other guidelines have not been followed.

3) The wiki page describing the import is not adequate to
assess the quality of the data or of the proposed import.
See for example:

Re: [Talk-ca] 2020 building import wiki comment by Nate Wessel

[John Whelan]

I agree and we are sensitive to Quebec's position.

I think the hope was we would make the data available and that local 
mappers would be involved in the import over time as happens with 
CANVEC.  One comment I heard early on was this isn't so much an import 
as a marathon.


What seems to have happened is a lot of buildings have been imported 
very quickly.  In rural areas or places where there are few buildings 
this isn't so much of a problem. Certain locations have run in very 
smoothly.


I think the data quality is considerably better than iD and Mapathons 
with new mappers.


The original data files are available on an Open Data portal with a 
license that is compatible with OSM and to be honest we have very little 
control over who can download them or what they do with them.


What we do have is a process that was used in Ottawa and is fairly 
robust. Data quality is very dependent on the individual mappers doing 
the import though.


Looking at the stats I don't think much has been done in Quebec and I 
feel James would be happy to restrict access Quebec in someway if that 
would make you happier for the moment.  It has been set up as a separate 
set of tiles so can be isolated fairly easily.


Could you be nice and chat to the Quebec mappers and sound them out on 
what they would like to do?  The data for Quebec is from Quebec 
municipalities by the way.  Please bear in mind that Microsoft are 
rumoured to be about to release building data for Canada in the same way 
as they have for the US.  This is scanned from images data and I suspect 
the data quality will not be as high as the Municipal data.  I 
understand there are multiple imports going on with the US Microsoft 
building outline data currently.


I seem to recall that Daniel Begin, who I believe is a Quebec mapper, 
made comments on the project in talk-ca some time ago.  I also seem to 
recall it was his suggestion that we made it a single import plan.


Thoughts?

Thanks John

Pierre Béland wrote on 2019-01-18 6:54 PM:

John,

Il y a local et local. Compte-tenu des différences culturelles Québec 
vs Canada en général et que les contributeurs du Québec ne fréquentent 
pratiquement pas cette liste, vous ne devriez pas prendre pour acquis 
que vous représentez cette communauté et pouvez démarrer des projets 
en son nom.



Pierre


Le vendredi 18 janvier 2019 13 h 11 min 37 s HNE, john whelan 
 a écrit :



I know of no other way to contact him but he made an interesting 
comment that the project is on hold in the wiki pending review.


Would he care to comment on who is supposed to be reviewing the project?

My understanding is that the import was raised in talk-ca before it 
commenced for comment and these were generally favourable.  I took 
that as the local mappers to Canada had been consulted and they are 
the "local mappers" authority in this case.


I understand he has concerns about local mappers making decisions but 
in Canada we have been importing similar data through CANVEC for some 
time.  CANVEC data comes from a number of sources including municipal 
data.


Is he suggesting that each of the 3,700 municipalities in Canada 
should form a group of local mappers who can make individual decisions 
on whether their municipal data should be imported and we should end 
up with 3,700 import plans?


Thanks John


___
Talk-ca mailing list
Talk-ca@openstreetmap.org 
https://lists.openstreetmap.org/listinfo/talk-ca


--
Sent from Postbox 

___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] 2020 building import wiki comment by Nate Wessel

[Pierre Béland]

John,
Il y a local et local. Compte-tenu des différences culturelles Québec vs Canada 
en général et que les contributeurs du Québec ne fréquentent pratiquement pas 
cette liste, vous ne devriez pas prendre pour acquis que vous représentez cette 
communauté et pouvez démarrer des projets en son nom.
 
Pierre 
 

Le vendredi 18 janvier 2019 13 h 11 min 37 s HNE, john whelan 
 a écrit :  
 
 I know of no other way to contact him but he made an interesting comment that 
the project is on hold in the wiki pending review.
Would he care to comment on who is supposed to be reviewing the project?
My understanding is that the import was raised in talk-ca before it commenced 
for comment and these were generally favourable.  I took that as the local 
mappers to Canada had been consulted and they are the "local mappers" authority 
in this case.
I understand he has concerns about local mappers making decisions but in Canada 
we have been importing similar data through CANVEC for some time.  CANVEC data 
comes from a number of sources including municipal data.
Is he suggesting that each of the 3,700 municipalities in Canada should form a 
group of local mappers who can make individual decisions on whether their 
municipal data should be imported and we should end up with 3,700 import plans?
Thanks John

___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca
  ___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] 2020 building import wiki comment by Nate Wessel

[John Marshall]

I found the building footprints to be very good. JOSM cleaned up most of
the errors..I'm not sure it would be worth the risk to do more processing.
Sometimes there were crossing ways, usually with  terrace or buildings in
the downtown core that need to be fixed manually. Which took a ton of time.
I always removed all the errors from import buildings before I added the
OSM data.

 I also keeped the city name on the imported data so I could tell which was
the imported data/ OSM data. I removed the city name later before
uploading. If there was already a building in OSM, I used the "Replace
Geometry" unless the building in OSM was better. Sometimes , the local
mapper had done a better job than the city, so I just left it.The building
in the Niagara Region had tons of very good buildings that I just left.

I also tired to fixed other JOSM errors, Like adding road names, missing
tags, spelling error, natural=land ect.  Using Geo Base I added about 300
names to roads mostly around  Muskoka and Goderich. This usually took more
time than adding the buildings.

I would say adding buildings in rural, residential, & Industrial areas
there is low risk for problems. The downtown areas even in small
cities Port Colborne were very time consuming and require an experienced
mapper.. Personally I wouldn't even want to try downtown Toronto.

FYI,  The top 3 CDN OSM mappers are importing the building data.
http://osmstats.neis-one.org/?item=countries=Canada


Cheers

John


On Fri, Jan 18, 2019 at 5:30 PM James  wrote:

> You guys can analyze the simplified version of ontario:
>
> https://drive.google.com/file/d/1OK83yrPwMW4nefyu-6JsIInu0meK2rW6/view?usp=sharing
> If you think it's good, I can simplify the other files and process them
> into mbtiles.
> ___
> Talk-ca mailing list
> Talk-ca@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk-ca
>
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] 2020 building import wiki comment by Nate Wessel

[James]

You guys can analyze the simplified version of ontario:
https://drive.google.com/file/d/1OK83yrPwMW4nefyu-6JsIInu0meK2rW6/view?usp=sharing
If you think it's good, I can simplify the other files and process them
into mbtiles.
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] 2020 building import wiki comment by Nate Wessel

[Yaro Shkvorets]

I'm on board with removing redundant nodes, as long as it doesn't affect
actual geometries much.
Also there are quite a few duplicated nodes in buildings. They can be
removed in JOSM in one click before upload but better to do it at the
source.

On Fri, Jan 18, 2019 at 4:25 PM James  wrote:

> I can run all the shapefiles through qgis simplify tool if this resolves
> the issue...
>
> On Fri., Jan. 18, 2019, 4:08 p.m. Nate Wessel 
>> With default settings in JOSM, sure. In the import I was working on, we
>> used a Douglas-Peucker algorithm with a 20cm threshold (before the import
>> started) and it worked beautifully. We had many points that seemed to have
>> been introduced in the shapefiles as some kind of data artifact - they
>> didn't add any detail to the shape at all. This procedure removed almost
>> all of them with no discernible reduction in quality.
>> Nate Wessel
>> Jack of all trades, Master of Geography, PhD candidate in Urban Planning
>> NateWessel.com 
>>
>> On 1/18/19 4:03 PM, James wrote:
>>
>> dare you to run simplify tool on anything remotely round, it will make it
>> look like garbage
>>
>> On Fri., Jan. 18, 2019, 3:49 p.m. John Whelan > wrote:
>>
>>> The import mailing list was pointed to the correct page of the wiki.
>>> The initial post was to say this is what we were thinking of and there was
>>> a comment saying we needed to change the comment line.
>>>
>>> >There is no mention of this proposed import on the import catalogue
>>>
>>>
>>> The import process was reviewed by the person who set up the Ottawa
>>> import did we miss that step on the Ottawa import as well?  Neither was it
>>> raised as a concern on the import mailing list. I think this is very minor
>>> and can be corrected.
>>>
>>> We learnt a fair bit on the Ottawa import and my expectation is since we
>>> are using experienced mappers to do the import conflation would be either
>>> handled by them or the building not imported. We aren't using new mappers
>>> in a mapathon here and with experienced mappers then I think you have to
>>> trust them.  The world isn't perfect. Think in terms of service level.
>>>
>>> >There are 2X more nodes than needed to represent the building
>>> accurately.
>>>
>>> The problem with correcting this is you are introducing approximations.
>>> This will vary according to the source and this can be simplified or
>>> corrected once its in OSM. I think this is a different issue of a
>>> mechanical edit that needs to be considered separately.
>>>
>>> If we are concerned with database size then I suggest we change the
>>> instructions to say put the source comment on the change set rather than on
>>> the building outline.
>>>
>>> Cheerio John
>>>
>>>
>>> Nate Wessel wrote on 2019-01-18 3:06 PM:
>>>
>>> John,
>>>
>>> You seem to be playing the long game with this data - it sounds like
>>> you've been working with this a lot longer than I have, and you've put in
>>> the time and effort to help make this actually-quite-incredible dataset
>>> available to us. I don't want to stop the import from happening - quite the
>>> opposite. I just want to make sure that the time is taken to do this right.
>>> OSM deserves that. Your (our) long awaited victory will be the sweeter for
>>> our patience now.
>>>
>>> There are several specific issues I see where the I's are not crossed,
>>> nor the t's dotted. I've mentioned several already, so I'll try to be brief
>>> (I really need to get back to working on my dissertation).
>>>
>>> 1) There was extremely limited discussion on the imports mailing list.
>>> The initial email did not make clear the scope of the project. I read the
>>> email and did not think twice at it, thinking it was entirely about Ottawa.
>>> The link in that email was actually to the Ottawa import, and not this one,
>>> which seems to have been only in draft at the time.
>>> https://lists.openstreetmap.org/pipermail/imports/2018-November/005812.html
>>> As such, this project has NOT been reviewed by the imports list, which
>>> is a requirement for proceeding with the import.
>>>
>>> 2) There is no mention of this proposed import on the import catalogue (
>>> https://wiki.openstreetmap.org/wiki/Import/Catalogue)
>>> which is required in the imports guidelines. I suspect many other
>>> guidelines have not been followed.
>>>
>>> 3) The wiki page describing the import is not adequate to assess the
>>> quality of the data or of the proposed import. See for example:
>>> https://wiki.openstreetmap.org/wiki/WikiProject_Canada/Canada_Stats_Canada_Building_Outlines_Import/Plan#Risks
>>> The import guidelines call for a description of how conflation will be
>>> handled. The fact that two of the major importers seem to have a
>>> substantial disagreement about how to handle existing data indicates this
>>> was not well discussed and I can see that it isn't well documented.
>>>
>>> 4) The buildings need to be simplified, quite a bit actually. Most
>>> buildings have multiple nodes 

Re: [Talk-ca] 2020 building import wiki comment by Nate Wessel

[James]

I can run all the shapefiles through qgis simplify tool if this resolves
the issue...

On Fri., Jan. 18, 2019, 4:08 p.m. Nate Wessel  With default settings in JOSM, sure. In the import I was working on, we
> used a Douglas-Peucker algorithm with a 20cm threshold (before the import
> started) and it worked beautifully. We had many points that seemed to have
> been introduced in the shapefiles as some kind of data artifact - they
> didn't add any detail to the shape at all. This procedure removed almost
> all of them with no discernible reduction in quality.
> Nate Wessel
> Jack of all trades, Master of Geography, PhD candidate in Urban Planning
> NateWessel.com 
>
> On 1/18/19 4:03 PM, James wrote:
>
> dare you to run simplify tool on anything remotely round, it will make it
> look like garbage
>
> On Fri., Jan. 18, 2019, 3:49 p.m. John Whelan  wrote:
>
>> The import mailing list was pointed to the correct page of the wiki.  The
>> initial post was to say this is what we were thinking of and there was a
>> comment saying we needed to change the comment line.
>>
>> >There is no mention of this proposed import on the import catalogue
>>
>>
>> The import process was reviewed by the person who set up the Ottawa
>> import did we miss that step on the Ottawa import as well?  Neither was it
>> raised as a concern on the import mailing list. I think this is very minor
>> and can be corrected.
>>
>> We learnt a fair bit on the Ottawa import and my expectation is since we
>> are using experienced mappers to do the import conflation would be either
>> handled by them or the building not imported. We aren't using new mappers
>> in a mapathon here and with experienced mappers then I think you have to
>> trust them.  The world isn't perfect. Think in terms of service level.
>>
>> >There are 2X more nodes than needed to represent the building accurately.
>>
>> The problem with correcting this is you are introducing approximations.
>> This will vary according to the source and this can be simplified or
>> corrected once its in OSM. I think this is a different issue of a
>> mechanical edit that needs to be considered separately.
>>
>> If we are concerned with database size then I suggest we change the
>> instructions to say put the source comment on the change set rather than on
>> the building outline.
>>
>> Cheerio John
>>
>>
>> Nate Wessel wrote on 2019-01-18 3:06 PM:
>>
>> John,
>>
>> You seem to be playing the long game with this data - it sounds like
>> you've been working with this a lot longer than I have, and you've put in
>> the time and effort to help make this actually-quite-incredible dataset
>> available to us. I don't want to stop the import from happening - quite the
>> opposite. I just want to make sure that the time is taken to do this right.
>> OSM deserves that. Your (our) long awaited victory will be the sweeter for
>> our patience now.
>>
>> There are several specific issues I see where the I's are not crossed,
>> nor the t's dotted. I've mentioned several already, so I'll try to be brief
>> (I really need to get back to working on my dissertation).
>>
>> 1) There was extremely limited discussion on the imports mailing list.
>> The initial email did not make clear the scope of the project. I read the
>> email and did not think twice at it, thinking it was entirely about Ottawa.
>> The link in that email was actually to the Ottawa import, and not this one,
>> which seems to have been only in draft at the time.
>> https://lists.openstreetmap.org/pipermail/imports/2018-November/005812.html
>> As such, this project has NOT been reviewed by the imports list, which is
>> a requirement for proceeding with the import.
>>
>> 2) There is no mention of this proposed import on the import catalogue (
>> https://wiki.openstreetmap.org/wiki/Import/Catalogue)
>> which is required in the imports guidelines. I suspect many other
>> guidelines have not been followed.
>>
>> 3) The wiki page describing the import is not adequate to assess the
>> quality of the data or of the proposed import. See for example:
>> https://wiki.openstreetmap.org/wiki/WikiProject_Canada/Canada_Stats_Canada_Building_Outlines_Import/Plan#Risks
>> The import guidelines call for a description of how conflation will be
>> handled. The fact that two of the major importers seem to have a
>> substantial disagreement about how to handle existing data indicates this
>> was not well discussed and I can see that it isn't well documented.
>>
>> 4) The buildings need to be simplified, quite a bit actually. Most
>> buildings have multiple nodes representing straight lines. This bloats the
>> database and makes things harder to edit by hand later. There are probably
>> 2x more nodes than are needed to represent the data accurately, making it
>> harder for editors and data consumers to work with down the road.This is a
>> simple fix that will save countless hours later.
>>
>> ... I could go on, but I think this is plenty sufficient to 

Re: [Talk-ca] 2020 building import wiki comment by Nate Wessel

[Nate Wessel]

With default settings in JOSM, sure. In the import I was working on, we 
used a Douglas-Peucker algorithm with a 20cm threshold (before the 
import started) and it worked beautifully. We had many points that 
seemed to have been introduced in the shapefiles as some kind of data 
artifact - they didn't add any detail to the shape at all. This 
procedure removed almost all of them with no discernible reduction in 
quality.


Nate Wessel
Jack of all trades, Master of Geography, PhD candidate in Urban Planning
NateWessel.com 

On 1/18/19 4:03 PM, James wrote:
dare you to run simplify tool on anything remotely round, it will make 
it look like garbage


On Fri., Jan. 18, 2019, 3:49 p.m. John Whelan  wrote:


The import mailing list was pointed to the correct page of the
wiki. The initial post was to say this is what we were thinking of
and there was a comment saying we needed to change the comment line.

>There is no mention of this proposed import on the import catalogue


The import process was reviewed by the person who set up the
Ottawa import did we miss that step on the Ottawa import as well? 
Neither was it raised as a concern on the import mailing list. I
think this is very minor and can be corrected.

We learnt a fair bit on the Ottawa import and my expectation is
since we are using experienced mappers to do the import conflation
would be either handled by them or the building not imported. We
aren't using new mappers in a mapathon here and with experienced
mappers then I think you have to trust them.  The world isn't
perfect. Think in terms of service level.

>There are 2X more nodes than needed to represent the building
accurately.

The problem with correcting this is you are introducing
approximations.  This will vary according to the source and this
can be simplified or corrected once its in OSM. I think this is a
different issue of a mechanical edit that needs to be considered
separately.

If we are concerned with database size then I suggest we change
the instructions to say put the source comment on the change set
rather than on the building outline.

Cheerio John


Nate Wessel wrote on 2019-01-18 3:06 PM:


John,

You seem to be playing the long game with this data - it sounds
like you've been working with this a lot longer than I have, and
you've put in the time and effort to help make this
actually-quite-incredible dataset available to us. I don't want
to stop the import from happening - quite the opposite. I just
want to make sure that the time is taken to do this right. OSM
deserves that. Your (our) long awaited victory will be the
sweeter for our patience now.

There are several specific issues I see where the I's are not
crossed, nor the t's dotted. I've mentioned several already, so
I'll try to be brief (I really need to get back to working on my
dissertation).

1) There was extremely limited discussion on the imports mailing
list. The initial email did not make clear the scope of the
project. I read the email and did not think twice at it, thinking
it was entirely about Ottawa. The link in that email was actually
to the Ottawa import, and not this one, which seems to have been
only in draft at the time.
https://lists.openstreetmap.org/pipermail/imports/2018-November/005812.html
As such, this project has NOT been reviewed by the imports list,
which is a requirement for proceeding with the import.

2) There is no mention of this proposed import on the import
catalogue (https://wiki.openstreetmap.org/wiki/Import/Catalogue)
which is required in the imports guidelines. I suspect many other
guidelines have not been followed.

3) The wiki page describing the import is not adequate to assess
the quality of the data or of the proposed import. See for
example:

https://wiki.openstreetmap.org/wiki/WikiProject_Canada/Canada_Stats_Canada_Building_Outlines_Import/Plan#Risks
The import guidelines call for a description of how conflation
will be handled. The fact that two of the major importers seem to
have a substantial disagreement about how to handle existing data
indicates this was not well discussed and I can see that it isn't
well documented.

4) The buildings need to be simplified, quite a bit actually.
Most buildings have multiple nodes representing straight lines.
This bloats the database and makes things harder to edit by hand
later. There are probably 2x more nodes than are needed to
represent the data accurately, making it harder for editors and
data consumers to work with down the road.This is a simple fix
that will save countless hours later.

... I could go on, but I think this is plenty sufficient to
justify pressing pause on all this.

Again, I don't in 

Re: [Talk-ca] 2020 building import wiki comment by Nate Wessel

[John Whelan]

James you know I could never resist a dare!

Cheerio John

James wrote on 2019-01-18 4:03 PM:
dare you to run simplify tool on anything remotely round, it will make 
it look like garbage


On Fri., Jan. 18, 2019, 3:49 p.m. John Whelan  wrote:


The import mailing list was pointed to the correct page of the
wiki.  The initial post was to say this is what we were thinking
of and there was a comment saying we needed to change the comment
line.

>There is no mention of this proposed import on the import catalogue


The import process was reviewed by the person who set up the
Ottawa import did we miss that step on the Ottawa import as well? 
Neither was it raised as a concern on the import mailing list. I
think this is very minor and can be corrected.

We learnt a fair bit on the Ottawa import and my expectation is
since we are using experienced mappers to do the import conflation
would be either handled by them or the building not imported. We
aren't using new mappers in a mapathon here and with experienced
mappers then I think you have to trust them.  The world isn't
perfect. Think in terms of service level.

>There are 2X more nodes than needed to represent the building
accurately.

The problem with correcting this is you are introducing
approximations. This will vary according to the source and this
can be simplified or corrected once its in OSM. I think this is a
different issue of a mechanical edit that needs to be considered
separately.

If we are concerned with database size then I suggest we change
the instructions to say put the source comment on the change set
rather than on the building outline.

Cheerio John


Nate Wessel wrote on 2019-01-18 3:06 PM:


John,

You seem to be playing the long game with this data - it sounds
like you've been working with this a lot longer than I have, and
you've put in the time and effort to help make this
actually-quite-incredible dataset available to us. I don't want
to stop the import from happening - quite the opposite. I just
want to make sure that the time is taken to do this right. OSM
deserves that. Your (our) long awaited victory will be the
sweeter for our patience now.

There are several specific issues I see where the I's are not
crossed, nor the t's dotted. I've mentioned several already, so
I'll try to be brief (I really need to get back to working on my
dissertation).

1) There was extremely limited discussion on the imports mailing
list. The initial email did not make clear the scope of the
project. I read the email and did not think twice at it, thinking
it was entirely about Ottawa. The link in that email was actually
to the Ottawa import, and not this one, which seems to have been
only in draft at the time.
https://lists.openstreetmap.org/pipermail/imports/2018-November/005812.html
As such, this project has NOT been reviewed by the imports list,
which is a requirement for proceeding with the import.

2) There is no mention of this proposed import on the import
catalogue (https://wiki.openstreetmap.org/wiki/Import/Catalogue)
which is required in the imports guidelines. I suspect many other
guidelines have not been followed.

3) The wiki page describing the import is not adequate to assess
the quality of the data or of the proposed import. See for
example:

https://wiki.openstreetmap.org/wiki/WikiProject_Canada/Canada_Stats_Canada_Building_Outlines_Import/Plan#Risks
The import guidelines call for a description of how conflation
will be handled. The fact that two of the major importers seem to
have a substantial disagreement about how to handle existing data
indicates this was not well discussed and I can see that it isn't
well documented.

4) The buildings need to be simplified, quite a bit actually.
Most buildings have multiple nodes representing straight lines.
This bloats the database and makes things harder to edit by hand
later. There are probably 2x more nodes than are needed to
represent the data accurately, making it harder for editors and
data consumers to work with down the road.This is a simple fix
that will save countless hours later.

... I could go on, but I think this is plenty sufficient to
justify pressing pause on all this.

Again, I don't in any way want to disrespect the work that has
gone into this effort already. We're all volunteers here and I
know how much time this all takes. However. importing all/most of
the buildings in Canada is a monstrously large task, which will
have to dance around a lot of people's toes. We should expect
this to take a really damn long time if we're going to do it
right. We need to have the patience to learn from experience,
from critique, and from the wisdom 

Re: [Talk-ca] 2020 building import wiki comment by Nate Wessel

[James]

dare you to run simplify tool on anything remotely round, it will make it
look like garbage

On Fri., Jan. 18, 2019, 3:49 p.m. John Whelan  The import mailing list was pointed to the correct page of the wiki.  The
> initial post was to say this is what we were thinking of and there was a
> comment saying we needed to change the comment line.
>
> >There is no mention of this proposed import on the import catalogue
>
>
> The import process was reviewed by the person who set up the Ottawa import
> did we miss that step on the Ottawa import as well?  Neither was it raised
> as a concern on the import mailing list. I think this is very minor and can
> be corrected.
>
> We learnt a fair bit on the Ottawa import and my expectation is since we
> are using experienced mappers to do the import conflation would be either
> handled by them or the building not imported. We aren't using new mappers
> in a mapathon here and with experienced mappers then I think you have to
> trust them.  The world isn't perfect. Think in terms of service level.
>
> >There are 2X more nodes than needed to represent the building accurately.
>
> The problem with correcting this is you are introducing approximations.
> This will vary according to the source and this can be simplified or
> corrected once its in OSM. I think this is a different issue of a
> mechanical edit that needs to be considered separately.
>
> If we are concerned with database size then I suggest we change the
> instructions to say put the source comment on the change set rather than on
> the building outline.
>
> Cheerio John
>
>
> Nate Wessel wrote on 2019-01-18 3:06 PM:
>
> John,
>
> You seem to be playing the long game with this data - it sounds like
> you've been working with this a lot longer than I have, and you've put in
> the time and effort to help make this actually-quite-incredible dataset
> available to us. I don't want to stop the import from happening - quite the
> opposite. I just want to make sure that the time is taken to do this right.
> OSM deserves that. Your (our) long awaited victory will be the sweeter for
> our patience now.
>
> There are several specific issues I see where the I's are not crossed, nor
> the t's dotted. I've mentioned several already, so I'll try to be brief (I
> really need to get back to working on my dissertation).
>
> 1) There was extremely limited discussion on the imports mailing list. The
> initial email did not make clear the scope of the project. I read the email
> and did not think twice at it, thinking it was entirely about Ottawa. The
> link in that email was actually to the Ottawa import, and not this one,
> which seems to have been only in draft at the time.
> https://lists.openstreetmap.org/pipermail/imports/2018-November/005812.html
> As such, this project has NOT been reviewed by the imports list, which is
> a requirement for proceeding with the import.
>
> 2) There is no mention of this proposed import on the import catalogue (
> https://wiki.openstreetmap.org/wiki/Import/Catalogue)
> which is required in the imports guidelines. I suspect many other
> guidelines have not been followed.
>
> 3) The wiki page describing the import is not adequate to assess the
> quality of the data or of the proposed import. See for example:
> https://wiki.openstreetmap.org/wiki/WikiProject_Canada/Canada_Stats_Canada_Building_Outlines_Import/Plan#Risks
> The import guidelines call for a description of how conflation will be
> handled. The fact that two of the major importers seem to have a
> substantial disagreement about how to handle existing data indicates this
> was not well discussed and I can see that it isn't well documented.
>
> 4) The buildings need to be simplified, quite a bit actually. Most
> buildings have multiple nodes representing straight lines. This bloats the
> database and makes things harder to edit by hand later. There are probably
> 2x more nodes than are needed to represent the data accurately, making it
> harder for editors and data consumers to work with down the road.This is a
> simple fix that will save countless hours later.
>
> ... I could go on, but I think this is plenty sufficient to justify
> pressing pause on all this.
>
> Again, I don't in any way want to disrespect the work that has gone into
> this effort already. We're all volunteers here and I know how much time
> this all takes. However. importing all/most of the buildings in Canada is a
> monstrously large task, which will have to dance around a lot of people's
> toes. We should expect this to take a really damn long time if we're going
> to do it right. We need to have the patience to learn from experience, from
> critique, and from the wisdom of the people who've learned from flawed
> imports in the past and have devised guidelines and processes so that we
> can have better experiences with this in the future.
> Nate Wessel
> Jack of all trades, Master of Geography, PhD candidate in Urban Planning
> NateWessel.com 

Re: [Talk-ca] 2020 building import wiki comment by Nate Wessel

[John Whelan]

The import mailing list was pointed to the correct page of the wiki.  
The initial post was to say this is what we were thinking of and there 
was a comment saying we needed to change the comment line.


>There is no mention of this proposed import on the import catalogue


The import process was reviewed by the person who set up the Ottawa 
import did we miss that step on the Ottawa import as well?  Neither was 
it raised as a concern on the import mailing list. I think this is very 
minor and can be corrected.


We learnt a fair bit on the Ottawa import and my expectation is since we 
are using experienced mappers to do the import conflation would be 
either handled by them or the building not imported. We aren't using new 
mappers in a mapathon here and with experienced mappers then I think you 
have to trust them.  The world isn't perfect. Think in terms of service 
level.


>There are 2X more nodes than needed to represent the building accurately.

The problem with correcting this is you are introducing approximations. 
This will vary according to the source and this can be simplified or 
corrected once its in OSM. I think this is a different issue of a 
mechanical edit that needs to be considered separately.


If we are concerned with database size then I suggest we change the 
instructions to say put the source comment on the change set rather than 
on the building outline.


Cheerio John


Nate Wessel wrote on 2019-01-18 3:06 PM:


John,

You seem to be playing the long game with this data - it sounds like 
you've been working with this a lot longer than I have, and you've put 
in the time and effort to help make this actually-quite-incredible 
dataset available to us. I don't want to stop the import from 
happening - quite the opposite. I just want to make sure that the time 
is taken to do this right. OSM deserves that. Your (our) long awaited 
victory will be the sweeter for our patience now.


There are several specific issues I see where the I's are not crossed, 
nor the t's dotted. I've mentioned several already, so I'll try to be 
brief (I really need to get back to working on my dissertation).


1) There was extremely limited discussion on the imports mailing list. 
The initial email did not make clear the scope of the project. I read 
the email and did not think twice at it, thinking it was entirely 
about Ottawa. The link in that email was actually to the Ottawa 
import, and not this one, which seems to have been only in draft at 
the time. 
https://lists.openstreetmap.org/pipermail/imports/2018-November/005812.html
As such, this project has NOT been reviewed by the imports list, which 
is a requirement for proceeding with the import.


2) There is no mention of this proposed import on the import catalogue 
(https://wiki.openstreetmap.org/wiki/Import/Catalogue)
which is required in the imports guidelines. I suspect many other 
guidelines have not been followed.


3) The wiki page describing the import is not adequate to assess the 
quality of the data or of the proposed import. See for example: 
https://wiki.openstreetmap.org/wiki/WikiProject_Canada/Canada_Stats_Canada_Building_Outlines_Import/Plan#Risks
The import guidelines call for a description of how conflation will be 
handled. The fact that two of the major importers seem to have a 
substantial disagreement about how to handle existing data indicates 
this was not well discussed and I can see that it isn't well documented.


4) The buildings need to be simplified, quite a bit actually. Most 
buildings have multiple nodes representing straight lines. This bloats 
the database and makes things harder to edit by hand later. There are 
probably 2x more nodes than are needed to represent the data 
accurately, making it harder for editors and data consumers to work 
with down the road.This is a simple fix that will save countless hours 
later.


... I could go on, but I think this is plenty sufficient to justify 
pressing pause on all this.


Again, I don't in any way want to disrespect the work that has gone 
into this effort already. We're all volunteers here and I know how 
much time this all takes. However. importing all/most of the buildings 
in Canada is a monstrously large task, which will have to dance around 
a lot of people's toes. We should expect this to take a really damn 
long time if we're going to do it right. We need to have the patience 
to learn from experience, from critique, and from the wisdom of the 
people who've learned from flawed imports in the past and have devised 
guidelines and processes so that we can have better experiences with 
this in the future.


Nate Wessel
Jack of all trades, Master of Geography, PhD candidate in Urban Planning
NateWessel.com 

On 1/18/19 2:24 PM, john whelan wrote:
My background is I'm a retired civil servant who has written and 
overseen procurement documents and fairly large procurements. Dotting 
the is and crossing the Ts are my speciality.


There are 

Re: [Talk-ca] 2020 building import wiki comment by Nate Wessel

[Nate Wessel]

John,

You seem to be playing the long game with this data - it sounds like 
you've been working with this a lot longer than I have, and you've put 
in the time and effort to help make this actually-quite-incredible 
dataset available to us. I don't want to stop the import from happening 
- quite the opposite. I just want to make sure that the time is taken to 
do this right. OSM deserves that. Your (our) long awaited victory will 
be the sweeter for our patience now.


There are several specific issues I see where the I's are not crossed, 
nor the t's dotted. I've mentioned several already, so I'll try to be 
brief (I really need to get back to working on my dissertation).


1) There was extremely limited discussion on the imports mailing list. 
The initial email did not make clear the scope of the project. I read 
the email and did not think twice at it, thinking it was entirely about 
Ottawa. The link in that email was actually to the Ottawa import, and 
not this one, which seems to have been only in draft at the time. 
https://lists.openstreetmap.org/pipermail/imports/2018-November/005812.html
As such, this project has NOT been reviewed by the imports list, which 
is a requirement for proceeding with the import.


2) There is no mention of this proposed import on the import catalogue 
(https://wiki.openstreetmap.org/wiki/Import/Catalogue)
which is required in the imports guidelines. I suspect many other 
guidelines have not been followed.


3) The wiki page describing the import is not adequate to assess the 
quality of the data or of the proposed import. See for example: 
https://wiki.openstreetmap.org/wiki/WikiProject_Canada/Canada_Stats_Canada_Building_Outlines_Import/Plan#Risks
The import guidelines call for a description of how conflation will be 
handled. The fact that two of the major importers seem to have a 
substantial disagreement about how to handle existing data indicates 
this was not well discussed and I can see that it isn't well documented.


4) The buildings need to be simplified, quite a bit actually. Most 
buildings have multiple nodes representing straight lines. This bloats 
the database and makes things harder to edit by hand later. There are 
probably 2x more nodes than are needed to represent the data accurately, 
making it harder for editors and data consumers to work with down the 
road.This is a simple fix that will save countless hours later.


... I could go on, but I think this is plenty sufficient to justify 
pressing pause on all this.


Again, I don't in any way want to disrespect the work that has gone into 
this effort already. We're all volunteers here and I know how much time 
this all takes. However. importing all/most of the buildings in Canada 
is a monstrously large task, which will have to dance around a lot of 
people's toes. We should expect this to take a really damn long time if 
we're going to do it right. We need to have the patience to learn from 
experience, from critique, and from the wisdom of the people who've 
learned from flawed imports in the past and have devised guidelines and 
processes so that we can have better experiences with this in the future.


Nate Wessel
Jack of all trades, Master of Geography, PhD candidate in Urban Planning
NateWessel.com 

On 1/18/19 2:24 PM, john whelan wrote:
My background is I'm a retired civil servant who has written and 
overseen procurement documents and fairly large procurements. Dotting 
the is and crossing the Ts are my speciality.


There are two parts to an import.  The first part is the part played 
by the import mailing group.  They confine themselves to is the 
license correct and do you have a reasonable plan.  In this case the 
license is one of the few that has been confirmed by the Legal Working 
Group of OpenStreetMap and as such no questions were raised about it 
on the import mailing list.  We have methodology that has been used 
before successfully with the Ottawa building outline import. There 
were major discussions both on talk-ca and the import mailing group 
before that import took place and we took note of the issues raised 
and addressed them.  The licensing issue goes back about eight years 
to when I was talking to Federal Government Treasury Board and 
explaining their Open Data license did not align with OSM.  That is 
why their license is now known as 2.0.


The second part is the local group makes the decision to import they 
are the authority no one else.


Apparently you were not part of the talk-ca when the discussions took 
place which would have been the time and place to raise concerns.


When the Ottawa import was done there were one or two places where the 
existing buildings and the import overlapped.  In the instructions on 
the import there are instructions to cover this. Specifically there is 
a validation step.  I seem to recall the error rate was of the order 
of 1% and I expect this latest batch to be roughly the same.


If you can identify 

Re: [Talk-ca] 2020 building import wiki comment by Nate Wessel

[Nate Wessel]

Hi John,

As Steve has said, you seem to be the only one suggesting that thousands 
of import committees might need to be formed. Certainly I'm not 
suggesting that.


My understanding of OSM import procedure (and wiki-style projects more 
generally) is that imports should operate in an essentially consensual 
way where possible. The goal is to build consent and bring people on 
board with a project or a change by addressing their concerns in a 
meaningful and respectful way.


I think that I have made some substantive and troubling claims about the 
quality of the data being imported. I've pointed out that this project 
has not followed the import procedures that were produced by a community 
of mappers larger than just those in Canada.


So to respond to your implication, I am in some sense the one reviewing 
the project, just as I would welcome you to find ways that my own 
contributions could be better. If you want my credentials for reviewing 
your work, here they are:


1) I am an active contributor to OSM in Toronto, where I live (and 
elsewhere)


2) I am currently helping to lead a building import in Hamilton County 
Ohio that has better addressed some of the issues I see this import 
struggling with. I can help you do the same.


3) I've been doing research in GIS for a long time now, though I don't 
need that to tell you that the issues I've described are hardly 
insurmountable technically or even all that difficult to fix. It would 
take maybe one day's hard work to get the technical side of this right.


I think Canadian OSMers will agree that we can take a pause to get 
things right on such a massive import. If they don't - if I'm shouted 
down or better, if my critiques are adequately addressed, then I will 
leave you to finish the project in peace. I might even lend a hand if 
all goes well, as I sincerely hope it does :-)


Best,

Nate Wessel
Jack of all trades, Master of Geography, PhD candidate in Urban Planning
NateWessel.com 

On 1/18/19 1:11 PM, john whelan wrote:
I know of no other way to contact him but he made an interesting 
comment that the project is on hold in the wiki pending review.


Would he care to comment on who is supposed to be reviewing the project?

My understanding is that the import was raised in talk-ca before it 
commenced for comment and these were generally favourable. I took that 
as the local mappers to Canada had been consulted and they are the 
"local mappers" authority in this case.


I understand he has concerns about local mappers making decisions but 
in Canada we have been importing similar data through CANVEC for some 
time.  CANVEC data comes from a number of sources including municipal 
data.


Is he suggesting that each of the 3,700 municipalities in Canada 
should form a group of local mappers who can make individual decisions 
on whether their municipal data should be imported and we should end 
up with 3,700 import plans?


Thanks John



___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

[Talk-ca] 2020 building import wiki comment by Nate Wessel

[john whelan]

I know of no other way to contact him but he made an interesting comment
that the project is on hold in the wiki pending review.

Would he care to comment on who is supposed to be reviewing the project?

My understanding is that the import was raised in talk-ca before it
commenced for comment and these were generally favourable.  I took that as
the local mappers to Canada had been consulted and they are the "local
mappers" authority in this case.

I understand he has concerns about local mappers making decisions but in
Canada we have been importing similar data through CANVEC for some time.
CANVEC data comes from a number of sources including municipal data.

Is he suggesting that each of the 3,700 municipalities in Canada should
form a group of local mappers who can make individual decisions on whether
their municipal data should be imported and we should end up with 3,700
import plans?

Thanks John
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca