Bonjour groupe
Looks like we're going in the same direction so far :-)
I agree with Nate regarding the implementation of the task
manager. In my experience, a size of a few blocks would be better
in urban areas, but boring in rural areas. Is it something that
can be adjusted?
Daniel
*From:*Nate Wessel [mailto:[email protected]]
*Sent:* Saturday, January 04, 2020 10:09
*To:* [email protected] <mailto:[email protected]>
*Subject:* Re: [Talk-ca] Importing buildings in Canada
Hi Daniel,
Thank you for all the work you've put into this. I'd like to
offer a couple suggestions and/or clarifications for your
proposed import process, overview though it is.
First, I think it is very important that a tasking manager is set
up on a city/by city basis only, and that only AFTER consensus is
achieved that the import should proceed in that area. I would
really like to avoid seeing the massive nationwide tasking that
was set up the first time around. We should be making it hard for
people to go rogue in regions where consensus for an import
doesn't (yet) exist.
Related to this, though important enough to be a second point in
it's own right, the tasking squares need to be small enough that
a single user can manage them and inspect every single building
in a task. The first round of import used task squares that were
massive, and which couldn't be divided any further past a certain
point. Even in rural areas, it is likely inappropriate to import
areas larger than 1km^2. In central Toronto it would be (and was)
idiotic. An import that doesn't take local scale into account
shouldn't proceed. "Too big to load into JOSM" is about 100x too
big to import in my opinion and is not a good enough benchmark
for import batch sizing.
That is, each import needs to be local, and not just in a
superficial sense.
I'll also add that the issue of conflation doesn't seem to have
been worked out yet except to note that it is an issue. What will
we do with the millions of buildings which will substantially
overlap/duplicate existing buildings or imports? This needs to be
worked out in detail before anything starts up again.
And what needs to be done about already existing low quality
imports? It's good to acknowledge their existence, but what will
be done about them? We've set up a task to clean up some of the
mess in Toronto ( http://tasks.osmcanada.ca/project/168 ) but
this is only the tip of the iceberg.
Again, I thank everyone for their time and effort on this - we
can get this done if we go slow and do it right :-)
Best,
Nate Wessel, PhD
Planner, Cartographer, Transport Nerd
NateWessel.com <https://www.natewessel.com>
On 2020-01-03 3:40 p.m., Daniel @jfd553 wrote:
Bonjour groupe, mes excuses pour ce très long courriel !-)
I have reviewed everything that has been written on the ODB
import (aka Canada Building Import) in Talk-ca and the wiki.
I proposed changes to some wiki pages (via talk tabs) to ease
the discussions about this import and the following. Now, in
order to restart the import, here are some thoughts and a
proposal on how to proceed to complete the task.
*1. Issues with the ODB Data Import*
Many concerns were raised about the import. One major concern
was to obtain local communities’ buy-in in the Canadian
context. Another concern was to improve the quality of the
data prior the import. The following paragraphs intend to
clear most of these concerns.
*1.1. Which data import project?*
According to the import guidelines (steps 3 & 4), a data
import explicitly refers to a single data source (ODB in our
case). Discussions about the availability and quality of
Microsoft or ESRI data, while interesting, are not relevant
as they should be dealt with as other import projects.
*1.2. What has been imported so far?*
According to what I found [1], the ODB import is completed
for 21 municipalities. These imports seem to have kept OSM
content’s history, at least for the samples checked, but many
problems were found. In some case, the imports brought
swimming pools in OSM because they were included in the
dataset (e.g. Moncton). In other cases, importing buildings
with accurate locations (XY) over content mapped from less
accurate imagery resulted in buildings that now overlap the
street network (e.g. Squamish). It means that all these 21
imports need to be carefully re-examined and corrected as
required.
For 12 other municipalities, the import is partial, either
suspended as requested, or because previous imports had
already provided most of the buildings (often from the same
municipal provider). That said the import will definitely
improve OSM accuracy and completeness if done properly.
*2. How should ODB Data be imported?*
I will copy the following paragraphs in the “Canada Building
Import” wiki page [3] for a detailed discussion…
Since the data (ODB, OSM and imagery) differ from one
municipality to another, there can be no imports at the
national or provincial level. We have to work on a municipal
basis and make sure to identify all the problems and the
corrective measures to apply when dealing with issues like
those I identified [1].
*2.1 Importing Locally*
According to the import guidelines (step 2), we must not
import the data without local buy-in. However, and contrarily
to some European country, there is usually no such thing as a
local OSM community in each municipality. However, we may
find a few local mappers from time to time. Working on a
municipal basis should allow identifying these local mappers
before doing the import. I often use this tool [2] to
identify and contact local mappers. Once identified, I
suggest that…
- We contact them to explain our intents by referring to
appropriate wiki pages.
- We wait a week or two to let them respond nothing, that
they have concerns, or wish to help.
- Without negative answers we could proceed to the import.
I first suggest that when a contributor wishes to import ODB
for a given municipality, he first identifies himself as
responsible for the import (we need to create an entry for
each municipality somewhere in the wiki). He can then contact
local mappers, as explain above, and go ahead with the import
once everything settled. For those who already made the
import, I suggest that they review their work since many
issues were detected with some of these imports.
Since there are only a few local OSM communities in Canada,
and because Canada is large, I suggest not limiting the
import of a given municipality to the people of the concerned
province or region.
*2.2 Pre-processing*
Once local mappers have agreed, some pre-processing can be
done if required.
A few months ago, I developed a tool that could be used to
process the data [4]. Concerns were raised because the
application was developed using proprietary software. So I
documented the whole process and algorithms in order to see
courageous coders converting it in open source software. In
the meantime, and as long as I have access to an FME licence,
I could process the data, when necessary, prior to make it
available through the task manager.
Proposed pre-processing [4] includes:
- Reading of original ODB data,
- Removal of near collinear nodes (simplification),
- Orthogonalization of buildings (for corners having near
right angles),
- Tagging of building footprints,
- Providing files in OSM format.
/Proposed tagging:/ In addition to the tags produced by the
orthogonalization process [4] and the source tag (source
<https://wiki.openstreetmap.org/wiki/Key:source>=Statistics
Canada - Open Building Database), the name of the Census
Subdivision provided in ODB data [5] is used to add the
addr:city tag to each building.
The pre-processing requires parameters that are specific to
the data to process. These parameters were estimated on a
municipal basis using actual ODB data. The processing time
increases exponentially according to the number of buildings
so, it may take a couple of days before the data is available
for a given municipality. Currently, the proposed
pre-processing does not convert terrace buildings into
individual houses nor it tags topological errors.
*2.3. Import Process*
After the local mappers, if any, agreed to the import, the
pre-processing completed when required, we can proceed to the
import.
1- Do not bulk import the data! Always use the task manager
(http://tasks.osmcanada.ca/). Select and open a task square
in JOSM. If it’s too big (e.g. too much work or request is
too big to load in JOSM), go back to the task manager and
split the task into smaller squares.
2- Load imagery layer (Bing or ESRI World Imagery) and align
the imagery with ODB data (i.e. create a new image offset) if
necessary because, unless proven otherwise, ODB should be
more accurate (XY) than most available images especially in
hilly areas.
3- Align the existing OSM content to the image (i.e. after
the new offset is applied) if required.
4- Currently step 2 and following as described in the wiki
[2]. I suggest merging the Conflation section [6] here and
reviewing everything to take into account the current proposal.
*References*
[1]
https://wiki.openstreetmap.org/wiki/The_Open_Database_of_Buildings
[2]
https://wiki.openstreetmap.org/wiki/Canada_Building_Import#Import_process
[3] http://resultmaps.neis-one.org(“Overview of OpenStreetMap
Contributors aka who’s around me?”)
[4] https://github.com/jfd553/OrthogonalizingBuildingFootprint
[5]
https://www150.statcan.gc.ca/n1/pub/92-195-x/2011001/geo/csd-sdr/csd-sdr-eng.htm
[6]
https://wiki.openstreetmap.org/wiki/Canada_Building_Import#Conflation
Let’s move ahead!
Daniel
_______________________________________________
Talk-ca mailing list
[email protected] <mailto:[email protected]>
https://lists.openstreetmap.org/listinfo/talk-ca
