Hello all,
Happy to see progress here.
My ongoing question is how to define that a "local" group has
determined that an import can proceed. And more specifically what is
"local"? There are rural and remote parts of Canada which have in the
order of zero active mappers or sense of a local community. How do we
consensus around imports there? Can we get agreement by (part-of)
province/territory where there is not some other group that puts their
hand up? (maybe use admin level 5 boundaries, with holes for big cities)
I just want to make sure we don't stop at doing imports only for big
cities. Buildings are important for the whole country.
On 2020-01-04 10:09 a.m., Nate Wessel wrote:
Hi Daniel,
Thank you for all the work you've put into this. I'd like to offer a
couple suggestions and/or clarifications for your proposed import
process, overview though it is.
First, I think it is very important that a tasking manager is set up
on a city/by city basis only, and that only AFTER consensus is
achieved that the import should proceed in that area. I would really
like to avoid seeing the massive nationwide tasking that was set up
the first time around. We should be making it hard for people to go
rogue in regions where consensus for an import doesn't (yet) exist.
Related to this, though important enough to be a second point in
it's own right, the tasking squares need to be small enough that a
single user can manage them and inspect every single building in a
task. The first round of import used task squares that were massive,
and which couldn't be divided any further past a certain point. Even
in rural areas, it is likely inappropriate to import areas larger
than 1km^2. In central Toronto it would be (and was) idiotic. An
import that doesn't take local scale into account shouldn't proceed.
"Too big to load into JOSM" is about 100x too big to import in my
opinion and is not a good enough benchmark for import batch sizing.
That is, each import needs to be local, and not just in a
superficial sense.
I'll also add that the issue of conflation doesn't seem to have been
worked out yet except to note that it is an issue. What will we do
with the millions of buildings which will substantially
overlap/duplicate existing buildings or imports? This needs to be
worked out in detail before anything starts up again.
And what needs to be done about already existing low quality
imports? It's good to acknowledge their existence, but what will be
done about them? We've set up a task to clean up some of the mess in
Toronto ( http://tasks.osmcanada.ca/project/168 ) but this is only
the tip of the iceberg.
Again, I thank everyone for their time and effort on this - we can
get this done if we go slow and do it right :-)
Best,
Nate Wessel, PhD
Planner, Cartographer, Transport Nerd
NateWessel.com <https://www.natewessel.com>
On 2020-01-03 3:40 p.m., Daniel @jfd553 wrote:
Bonjour groupe, mes excuses pour ce très long courriel !-)
I have reviewed everything that has been written on the ODB import
(aka Canada Building Import) in Talk-ca and the wiki. I proposed
changes to some wiki pages (via talk tabs) to ease the discussions
about this import and the following. Now, in order to restart the
import, here are some thoughts and a proposal on how to proceed to
complete the task.
*1. Issues with the ODB Data Import*
Many concerns were raised about the import. One major concern was
to obtain local communities’ buy-in in the Canadian context.
Another concern was to improve the quality of the data prior the
import. The following paragraphs intend to clear most of these
concerns.
*1.1. Which data import project?*
According to the import guidelines (steps 3 & 4), a data import
explicitly refers to a single data source (ODB in our case).
Discussions about the availability and quality of Microsoft or ESRI
data, while interesting, are not relevant as they should be dealt
with as other import projects.
*1.2. What has been imported so far?*
According to what I found [1], the ODB import is completed for 21
municipalities. These imports seem to have kept OSM content’s
history, at least for the samples checked, but many problems were
found. In some case, the imports brought swimming pools in OSM
because they were included in the dataset (e.g. Moncton). In other
cases, importing buildings with accurate locations (XY) over
content mapped from less accurate imagery resulted in buildings
that now overlap the street network (e.g. Squamish). It means that
all these 21 imports need to be carefully re-examined and corrected
as required.
For 12 other municipalities, the import is partial, either
suspended as requested, or because previous imports had already
provided most of the buildings (often from the same municipal
provider). That said the import will definitely improve OSM
accuracy and completeness if done properly.
*2. How should ODB Data be imported?*
I will copy the following paragraphs in the “Canada Building
Import” wiki page [3] for a detailed discussion…
Since the data (ODB, OSM and imagery) differ from one municipality
to another, there can be no imports at the national or provincial
level. We have to work on a municipal basis and make sure to
identify all the problems and the corrective measures to apply when
dealing with issues like those I identified [1].
*2.1 Importing Locally*
According to the import guidelines (step 2), we must not import the
data without local buy-in. However, and contrarily to some European
country, there is usually no such thing as a local OSM community in
each municipality. However, we may find a few local mappers from
time to time. Working on a municipal basis should allow identifying
these local mappers before doing the import. I often use this tool
[2] to identify and contact local mappers. Once identified, I
suggest that…
- We contact them to explain our intents by referring to
appropriate wiki pages.
- We wait a week or two to let them respond nothing, that they have
concerns, or wish to help.
- Without negative answers we could proceed to the import.
I first suggest that when a contributor wishes to import ODB for a
given municipality, he first identifies himself as responsible for
the import (we need to create an entry for each municipality
somewhere in the wiki). He can then contact local mappers, as
explain above, and go ahead with the import once everything
settled. For those who already made the import, I suggest that they
review their work since many issues were detected with some of
these imports.
Since there are only a few local OSM communities in Canada, and
because Canada is large, I suggest not limiting the import of a
given municipality to the people of the concerned province or region.
*2.2 Pre-processing*
Once local mappers have agreed, some pre-processing can be done if
required.
A few months ago, I developed a tool that could be used to process
the data [4]. Concerns were raised because the application was
developed using proprietary software. So I documented the whole
process and algorithms in order to see courageous coders converting
it in open source software. In the meantime, and as long as I have
access to an FME licence, I could process the data, when necessary,
prior to make it available through the task manager.
Proposed pre-processing [4] includes:
- Reading of original ODB data,
- Removal of near collinear nodes (simplification),
- Orthogonalization of buildings (for corners having near right
angles),
- Tagging of building footprints,
- Providing files in OSM format.
/Proposed tagging:/ In addition to the tags produced by the
orthogonalization process [4] and the source tag (source
<https://wiki.openstreetmap.org/wiki/Key:source>=Statistics Canada
- Open Building Database), the name of the Census Subdivision
provided in ODB data [5] is used to add the addr:city tag to each
building.
The pre-processing requires parameters that are specific to the
data to process. These parameters were estimated on a municipal
basis using actual ODB data. The processing time increases
exponentially according to the number of buildings so, it may take
a couple of days before the data is available for a given
municipality. Currently, the proposed pre-processing does not
convert terrace buildings into individual houses nor it tags
topological errors.
*2.3. Import Process*
After the local mappers, if any, agreed to the import, the
pre-processing completed when required, we can proceed to the import.
1- Do not bulk import the data! Always use the task manager
(http://tasks.osmcanada.ca/). Select and open a task square in
JOSM. If it’s too big (e.g. too much work or request is too big to
load in JOSM), go back to the task manager and split the task into
smaller squares.
2- Load imagery layer (Bing or ESRI World Imagery) and align the
imagery with ODB data (i.e. create a new image offset) if necessary
because, unless proven otherwise, ODB should be more accurate (XY)
than most available images especially in hilly areas.
3- Align the existing OSM content to the image (i.e. after the new
offset is applied) if required.
4- Currently step 2 and following as described in the wiki [2]. I
suggest merging the Conflation section [6] here and reviewing
everything to take into account the current proposal.
*References*
[1]
https://wiki.openstreetmap.org/wiki/The_Open_Database_of_Buildings__
[2]
https://wiki.openstreetmap.org/wiki/Canada_Building_Import#Import_process
[3] http://resultmaps.neis-one.org(“Overview of OpenStreetMap
Contributors aka who’s around me?”)
[4] https://github.com/jfd553/OrthogonalizingBuildingFootprint__
[5]
https://www150.statcan.gc.ca/n1/pub/92-195-x/2011001/geo/csd-sdr/csd-sdr-eng.htm
[6]
https://wiki.openstreetmap.org/wiki/Canada_Building_Import#Conflation
Let’s move ahead!
Daniel
_______________________________________________
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca
N�n�r����)em�h�yhiם�w^���
_______________________________________________
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca
