Re: [Talk-ca] FW: Building Import

[Pierre Béland via Talk-ca]

Voir https://github.com/opendatalabrdc/Documentation/tree/master/topology où 
nous avons entreposé des fichiers geojson du projet de OpenDatalabRDC pour 
consultation.voir par 
exemplehttps://github.com/opendatalabrdc/Documentation/blob/master/topology/topology-irregular-forms-OC_Kampala_hotosm_4360_2018_04_07.geojson
La création d'un répertoire similaire facilliterait la consultations par tous 
des données.  Lors de la consultatin, on clique sur un polygone pour consulter 
les variables d'analyse.
 
Pierre 



 

Le mardi 26 mars 2019 17 h 34 min 24 s HAE, Begin Daniel 
 a écrit :  
 
 Screenshots? A good idea for having everyone seeing the results over 
complicated polygons (I will try keep objective in my selection ;-)

I am working to get it right on multiple adjacent polygons. I'll make the 
results available after I got them.

Daniel

-Original Message-
From: Jarek Piórkowski [mailto:ja...@piorkowski.ca] 
Sent: Tuesday, March 26, 2019 17:19
To: Begin Daniel; talk-ca@openstreetmap.org
Subject: Re: [Talk-ca] FW: Building Import

Hi Daniel,

If you are interested, some more potentially complicated areas around
Golden Horseshoe for testing. Each is roughly one screen on z16. I
don't know some of these as much, you might want to post results as
data files or screenshots for others to also look at to increase
buy-in.

- Spadina Chinatown and Kensington Market in Toronto, small buildings
tight in against each other, many semi-detached or attached, and some
larger ex-industrial buildings: 43.6569,-79.3868,43.6477,-79.4086
- University of Waterloo, with smaller attached residence buildings
that might have somewhat complicated shapes, as well as large
interconnected school buildings: 43.4740,-80.5362,43.4648,-80.5580
- downtown Kitchener, using a variety of grid alignments and some
buildings that might not be square: 43.4562,-80.4782,43.4470,-80.5000
- downtown Hamilton also has streets that aren't at right angles:
43.2619,-79.8572,43.2527,-79.8790
- St. Catharines might also be not square: 43.1640,-79.2322,43.1547,-79.2540
- Unionville, older area of Markham: 43.8717,-79.2993,43.8625,-79.3211

You will notice a trend of downtowns with non-square grids. I'm sure
others will be happy to contribute more examples of areas with
geometries they'd consider tricky. Bigger buildings might be more
likely to not be square if they're built out to max out the available
lot. I imagine only-slightly-non-square grids will be most
challenging...

--Jarek

On Tue, 26 Mar 2019 at 16:49, Begin Daniel  wrote:
>
> As usual, missed the reply all …
>
>
>
> From: jfd...@hotmail.com
> Sent: Tuesday, March 26, 2019 16:26
> To: 'John Whelan'
> Subject: RE: [Talk-ca] Building Import
>
>
>
> It is really kind to consider my background ;-)
>
> You are right regarding the "black box" approach; this is why a large 
> approval from the community is required before I go further.
>
>
>
> Daniel
>
>
>
> From: John Whelan [mailto:jwhelan0...@gmail.com]
> Sent: Tuesday, March 26, 2019 16:04
> To: Begin Daniel
> Cc: Jarek Piórkowski; talk-ca@openstreetmap.org; keith hartley; Alessandro 
> (STATCAN)
> Subject: Re: [Talk-ca] Building Import
>
>
>
> I think my concerns are to do with the "black box" approach.  Knowing your 
> background I trust your work but others might not.
>
> On a technical side I get the impression that cites with buildings that are 
> close to each other are problematical.  I assume that small locations with a 
> population of say under 125,000 this is an insignificant problem?
>
> The other issue is I'd like to either see buy in from Nate or at least some 
> Toronto mappers to get an indication that something will happen at the end of 
> the day as it is a fair chunk of Daniel's time to work out how do the 
> preprocessing.
>
> I think some BC mappers expressed some doubts as well so perhaps they would 
> like to think about if they are happy or would prefer BC to be outside of the 
> import project and express their views.
>
> Out of interest if it does move ahead are we including the Microsoft data for 
> areas where we do not have data from Stats Canada?  If so we will need to 
> amend the project plan.
>
> My personal view is realistically I think having building information even if 
> its a meter or two out is better than not having the building outlines.
>
> What would be nice is if we could have some indication from places such as 
> Manitoba, Alberta, Saskatchewan, Quebec excluding Montreal, Ontario excluding 
> Toronto and the other provinces and territories whether they are happy with 
> importing the buildings either from Stats or Microsoft.
>
> I seem to recall Keith is in Manitoba, so any views other than it wasn't 
> present in the first release from Stats?
>
> Note to Alessandro this is just 

Re: [Talk-ca] FW: Building Import

[Begin Daniel]

Screenshots? A good idea for having everyone seeing the results over 
complicated polygons (I will try keep objective in my selection ;-)

I am working to get it right on multiple adjacent polygons. I'll make the 
results available after I got them.

Daniel

-Original Message-
From: Jarek Piórkowski [mailto:ja...@piorkowski.ca] 
Sent: Tuesday, March 26, 2019 17:19
To: Begin Daniel; talk-ca@openstreetmap.org
Subject: Re: [Talk-ca] FW: Building Import

Hi Daniel,

If you are interested, some more potentially complicated areas around
Golden Horseshoe for testing. Each is roughly one screen on z16. I
don't know some of these as much, you might want to post results as
data files or screenshots for others to also look at to increase
buy-in.

- Spadina Chinatown and Kensington Market in Toronto, small buildings
tight in against each other, many semi-detached or attached, and some
larger ex-industrial buildings: 43.6569,-79.3868,43.6477,-79.4086
- University of Waterloo, with smaller attached residence buildings
that might have somewhat complicated shapes, as well as large
interconnected school buildings: 43.4740,-80.5362,43.4648,-80.5580
- downtown Kitchener, using a variety of grid alignments and some
buildings that might not be square: 43.4562,-80.4782,43.4470,-80.5000
- downtown Hamilton also has streets that aren't at right angles:
43.2619,-79.8572,43.2527,-79.8790
- St. Catharines might also be not square: 43.1640,-79.2322,43.1547,-79.2540
- Unionville, older area of Markham: 43.8717,-79.2993,43.8625,-79.3211

You will notice a trend of downtowns with non-square grids. I'm sure
others will be happy to contribute more examples of areas with
geometries they'd consider tricky. Bigger buildings might be more
likely to not be square if they're built out to max out the available
lot. I imagine only-slightly-non-square grids will be most
challenging...

--Jarek

On Tue, 26 Mar 2019 at 16:49, Begin Daniel  wrote:
>
> As usual, missed the reply all …
>
>
>
> From: jfd...@hotmail.com
> Sent: Tuesday, March 26, 2019 16:26
> To: 'John Whelan'
> Subject: RE: [Talk-ca] Building Import
>
>
>
> It is really kind to consider my background ;-)
>
> You are right regarding the "black box" approach; this is why a large 
> approval from the community is required before I go further.
>
>
>
> Daniel
>
>
>
> From: John Whelan [mailto:jwhelan0...@gmail.com]
> Sent: Tuesday, March 26, 2019 16:04
> To: Begin Daniel
> Cc: Jarek Piórkowski; talk-ca@openstreetmap.org; keith hartley; Alessandro 
> (STATCAN)
> Subject: Re: [Talk-ca] Building Import
>
>
>
> I think my concerns are to do with the "black box" approach.  Knowing your 
> background I trust your work but others might not.
>
> On a technical side I get the impression that cites with buildings that are 
> close to each other are problematical.  I assume that small locations with a 
> population of say under 125,000 this is an insignificant problem?
>
> The other issue is I'd like to either see buy in from Nate or at least some 
> Toronto mappers to get an indication that something will happen at the end of 
> the day as it is a fair chunk of Daniel's time to work out how do the 
> preprocessing.
>
> I think some BC mappers expressed some doubts as well so perhaps they would 
> like to think about if they are happy or would prefer BC to be outside of the 
> import project and express their views.
>
> Out of interest if it does move ahead are we including the Microsoft data for 
> areas where we do not have data from Stats Canada?  If so we will need to 
> amend the project plan.
>
> My personal view is realistically I think having building information even if 
> its a meter or two out is better than not having the building outlines.
>
> What would be nice is if we could have some indication from places such as 
> Manitoba, Alberta, Saskatchewan, Quebec excluding Montreal, Ontario excluding 
> Toronto and the other provinces and territories whether they are happy with 
> importing the buildings either from Stats or Microsoft.
>
> I seem to recall Keith is in Manitoba, so any views other than it wasn't 
> present in the first release from Stats?
>
> Note to Alessandro this is just background stuff.
>
> Thanks
>
> Cheerio John
>
> Begin Daniel wrote on 2019-03-26 3:29 PM:
>
> Jarek,
>
> The area you proposed in quite interesting and will force me to look further 
> at buildings with sharing edges, a concern Pierre also had. I'll be back soon 
> with your area processed.
>
> Daniel
>
>
>
> -Original Message-
>
> From: Begin Daniel [mailto:jfd...@hotmail.com]
>
> Sent: Tuesday, March 26, 2019 14:34
>
> To: Jarek Piórkowski; talk-ca@openstreetmap.org
>
> Subject: Re: [Talk-ca] Buildin

[Talk-ca] FW: Building Import

[Begin Daniel]

As usual, missed the reply all …

From: jfd...@hotmail.com
Sent: Tuesday, March 26, 2019 16:26
To: 'John Whelan'
Subject: RE: [Talk-ca] Building Import

It is really kind to consider my background ;-)
You are right regarding the "black box" approach; this is why a large approval 
from the community is required before I go further.

Daniel

From: John Whelan [mailto:jwhelan0...@gmail.com]
Sent: Tuesday, March 26, 2019 16:04
To: Begin Daniel
Cc: Jarek Piórkowski; talk-ca@openstreetmap.org; keith hartley; Alessandro 
(STATCAN)
Subject: Re: [Talk-ca] Building Import

I think my concerns are to do with the "black box" approach.  Knowing your 
background I trust your work but others might not.

On a technical side I get the impression that cites with buildings that are 
close to each other are problematical.  I assume that small locations with a 
population of say under 125,000 this is an insignificant problem?

The other issue is I'd like to either see buy in from Nate or at least some 
Toronto mappers to get an indication that something will happen at the end of 
the day as it is a fair chunk of Daniel's time to work out how do the 
preprocessing.

I think some BC mappers expressed some doubts as well so perhaps they would 
like to think about if they are happy or would prefer BC to be outside of the 
import project and express their views.

Out of interest if it does move ahead are we including the Microsoft data for 
areas where we do not have data from Stats Canada?  If so we will need to amend 
the project plan.

My personal view is realistically I think having building information even if 
its a meter or two out is better than not having the building outlines.

What would be nice is if we could have some indication from places such as 
Manitoba, Alberta, Saskatchewan, Quebec excluding Montreal, Ontario excluding 
Toronto and the other provinces and territories whether they are happy with 
importing the buildings either from Stats or Microsoft.

I seem to recall Keith is in Manitoba, so any views other than it wasn't 
present in the first release from Stats?

Note to Alessandro this is just background stuff.

Thanks

Cheerio John

Begin Daniel wrote on 2019-03-26 3:29 PM:

Jarek,

The area you proposed in quite interesting and will force me to look further at 
buildings with sharing edges, a concern Pierre also had. I'll be back soon with 
your area processed.

Daniel



-Original Message-

From: Begin Daniel [mailto:jfd...@hotmail.com]

Sent: Tuesday, March 26, 2019 14:34

To: Jarek Piórkowski; 
talk-ca@openstreetmap.org

Subject: Re: [Talk-ca] Building Import



Jarek,

Since it is a one-time process, I expect to be able to process the files if the 
community feels comfortable with it. In the meantime, people are welcome to 
send me the bounding box of an area they would like to examine.



Daniel



-Original Message-

From: Jarek Piórkowski [mailto:ja...@piorkowski.ca]

Sent: Tuesday, March 26, 2019 13:46

To: Begin Daniel; talk-ca@openstreetmap.org

Subject: Re: [Talk-ca] Building Import



On Tue, 26 Mar 2019 at 13:10, Begin Daniel 
 wrote:

There is actually no standard “code” available since I use FME 
(www.safe.com). It is a proprietary ETL application and 
all operations are done using “transformers” 
(https://www.safe.com/transformers/). I can provide you with the workbench I 
developed (a bunch of linked transformers) but you need a license to run it. 
This is why I tried to describe the operations I run on the data in the wiki.



As you did, people may send me coordinates (bounding box) of an area they know 
well. I’ll process the area and send the results back in OSM format. Please, be 
reasonable on the amount of data to process ;-)

Thanks Daniel. Let me know how it looks then!



Coming from an open-source background, the process is unusual to me,

and I have questions about scalability - will you be able to process

and provide updated data files for all of Canada then? - but if others

are comfortable with it then I won't object.



Some general thoughts regarding tooling as raised upthread:



I was initially excited to see building footprints data as they help

two quite distinct purposes:



1. they provide a mostly-automatic source of geometries for the

millions of single-family houses that wouldn't be mapped in the next

decade otherwise



2. they might provide a corrected and fairly accurate source of

geometries in heavily-built-up areas, where GPS signal is not that

reliable and it can be really difficult to get sufficiently accurate

geometries from imagery, whether because it's not sufficiently

high-resolution, two sets of imagery with conflicting offsets (Bing

and Esri are the two best sets in Toronto, and they're off by about

1-2 m on north-south axis from each other - that's not something I can

check with a consumer-grade GPS so I'm 

Re: [Talk-ca] FW: Building Import

[John Whelan]

I like the idea of building a consensus on a building import.

More seriously if we can get any sort of consensus I'll be more than happy.

Cheerio John

Begin Daniel wrote on 2019-03-21 2:40 PM:


Oups, I still miss the “reply all” button

*From:*jfd...@hotmail.com
*Sent:* Thursday, March 21, 2019 14:36
*To:* 'Nate Wessel'
*Subject:* RE: [Talk-ca] Building Import

Hi all,

Concerning the pre-processing, let’s try/check first the 
“orthogonalization” component then, if there is a consensus on the 
validity of the result, we can build on it J


Daniel

*From:*Nate Wessel [mailto:bike...@gmail.com]
*Sent:* Thursday, March 21, 2019 14:30
*To:* John Whelan
*Cc:* talk-ca@openstreetmap.org
*Subject:* Re: [Talk-ca] Building Import

I've specifically and repeatedly requested that the tasking manager be 
taken down while this project is reworked... though that doesn't 
pertain directly to the email I just sent.


Nate Wessel
Jack of all trades, Master of Geography, PhD candidate in Urban Planning
NateWessel.com 

On 3/21/19 2:02 PM, John Whelan wrote:

Nate are you requesting something specific on the Canadian task
manager for Toronto at this time or would you prefer to look
through Daniel's work first?

Thanks

Cheerio John

Nate Wessel wrote on 2019-03-21 1:49 PM:

Daniel,

This is exciting news! After much talk on this list, it seems we
may have some actual progress toward fixing the various data
quality issues. Would you mind sharing some of your code, or a
description of your workflow here or on GitHub or the like so we
can take a look?

One thing you didn't mention which I think will be really
critical, especially in central Toronto: We need to remove
buildings from the import dataset that may already be mapped in
OSM. That is, buildings that overlap with existing buildings. For
this import to make any sense in Central Toronto, we need
conflation to move slowly, and in smaller, more manageable steps.
Buildings that are already mapped should be checked manually at a
later time in batches that a skilled human can manage in less than
an hour. The tasking manager as it's currently set up would have
all of downtown conflated by hand in one task by a single mapper -
a recipe for disaster I'm sure, given how detailed the map is in
that area.

Cheers,

Nate Wessel
Jack of all trades, Master of Geography, PhD candidate in Urban
Planning
NateWessel.com 

On 3/19/19 12:58 PM, Begin Daniel wrote:

Hi all,

As mentioned a few weeks ago, I have almost completed the
development of a clean-up tool for the data to be imported.

So far, it removes nonessential vertices, orthogonalizes
building corners when reasonable and ensures walls’ alignment
within given tolerances. Building footprints that can’t be
processed completely are flagged accordingly, so they could be
examined thoroughly at import time.

Eventually, It should be easy to remove overlapping buildings
(potentially generated from a 3d mapping), but I doubt that
splitting terrace into individual buildings can be done
automatically.

The tool uses some parameters that need to be adjusted. I
would like that those who are interested in this aspect of the
import send me benchmark data that could be problematic. I
will process them to adjust parameters and/or the tool, and I
will send back the results to the sender for a thorough
examination.

I should soon document the process in the “Canada Building
Import” wiki page (in a pre-processing section).

Thought? Comments?

Daniel

___

Talk-ca mailing list

Talk-ca@openstreetmap.org 

https://lists.openstreetmap.org/listinfo/talk-ca

___

Talk-ca mailing list

Talk-ca@openstreetmap.org 

https://lists.openstreetmap.org/listinfo/talk-ca

-- 


Sent from Postbox





___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca


--
Sent from Postbox 

___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

[Talk-ca] FW: Building Import

[Begin Daniel]

Oups, I still miss the "reply all" button

From: jfd...@hotmail.com
Sent: Thursday, March 21, 2019 14:36
To: 'Nate Wessel'
Subject: RE: [Talk-ca] Building Import

Hi all,
Concerning the pre-processing, let's try/check first the "orthogonalization" 
component then, if there is a consensus on the validity of the result, we can 
build on it :)

Daniel

From: Nate Wessel [mailto:bike...@gmail.com]
Sent: Thursday, March 21, 2019 14:30
To: John Whelan
Cc: talk-ca@openstreetmap.org
Subject: Re: [Talk-ca] Building Import


I've specifically and repeatedly requested that the tasking manager be taken 
down while this project is reworked... though that doesn't pertain directly to 
the email I just sent.
Nate Wessel
Jack of all trades, Master of Geography, PhD candidate in Urban Planning
NateWessel.com
On 3/21/19 2:02 PM, John Whelan wrote:
Nate are you requesting something specific on the Canadian task manager for 
Toronto at this time or would you prefer to look through Daniel's work first?

Thanks

Cheerio John

Nate Wessel wrote on 2019-03-21 1:49 PM:

Daniel,

This is exciting news! After much talk on this list, it seems we may have some 
actual progress toward fixing the various data quality issues. Would you mind 
sharing some of your code, or a description of your workflow here or on GitHub 
or the like so we can take a look?

One thing you didn't mention which I think will be really critical, especially 
in central Toronto: We need to remove buildings from the import dataset that 
may already be mapped in OSM. That is, buildings that overlap with existing 
buildings. For this import to make any sense in Central Toronto, we need 
conflation to move slowly, and in smaller, more manageable steps. Buildings 
that are already mapped should be checked manually at a later time in batches 
that a skilled human can manage in less than an hour. The tasking manager as 
it's currently set up would have all of downtown conflated by hand in one task 
by a single mapper - a recipe for disaster I'm sure, given how detailed the map 
is in that area.

Cheers,
Nate Wessel
Jack of all trades, Master of Geography, PhD candidate in Urban Planning
NateWessel.com
On 3/19/19 12:58 PM, Begin Daniel wrote:
Hi all,
As mentioned a few weeks ago, I have almost completed the development of a 
clean-up tool for the data to be imported.
So far, it removes nonessential vertices, orthogonalizes building corners when 
reasonable and ensures walls' alignment within given tolerances. Building 
footprints that can't be processed completely are flagged accordingly, so they 
could be examined thoroughly at import time.
Eventually, It should be easy to remove overlapping buildings (potentially 
generated from a 3d mapping), but I doubt that splitting terrace into 
individual buildings can be done automatically.
The tool uses some parameters that need to be adjusted. I would like that those 
who are interested in this aspect of the import send me benchmark data that 
could be problematic. I will process them to adjust parameters and/or the tool, 
and I will send back the results to the sender for a thorough examination.
I should soon document the process in the "Canada Building Import" wiki page 
(in a pre-processing section).

Thought? Comments?

Daniel



___

Talk-ca mailing list

Talk-ca@openstreetmap.org

https://lists.openstreetmap.org/listinfo/talk-ca

___

Talk-ca mailing list

Talk-ca@openstreetmap.org

https://lists.openstreetmap.org/listinfo/talk-ca

--
Sent from 
Postbox
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca