Re: [Talk-ca] Canvec Reverts

[James]

I agree that forest that are way too costly in time should be removed, but
they should also be replaced with something(better) not just mass removed,
oh well we'll get to it later kind of thing

On Sep 1, 2016 8:05 PM, "Sam Dyck"  wrote:

>  I took a break to make supper, so I'm going to have to respond
> collectively.
>
> - Sammuell and sammuell_imports are my accounts. Both the forest and the
> lake are from Canvec data imported together as in the same tile. I Imported
> the whole thing (I would never import over existing data that carelessly),
> and take responsibility for that.
>
> - As my subsequent email showed, I missed the part about water/forest
> overlaps in Paul's email. Both and other people on this list have explained
> our feelings about this, but I will accept DWGs decision.
>
> - Per Michael's suggestion: This is constructive, but I do not feel that
> it is feasible because of the tiled structure of Canvec. To try and shift
> the forest layer over would make the problem worse.
>
> - Removal of the forest layer may be the best solution here. However there
> are many places in where the data matches up perfectly. Speaking only of
> areas where I am the primary importer, most of the forests in Ontario west
> of Thunder Bay would have to go, which is unfortunate because some have
> been manually corrected, though not enough to save them. Much (but
> certainly not all) of the forests in Manitoba could be saved with minimal
> effort.
>
> - If/when this is done we could look at ways to restore these forests. In
> areas that are mostly forest it shouldn't be too difficult to fill them
> using large multipolygons.
>
> Sam
>
> ___
> Talk-ca mailing list
> Talk-ca@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk-ca
>
>
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] Canvec Reverts

[Sam Dyck]

 I took a break to make supper, so I'm going to have to respond
collectively.

- Sammuell and sammuell_imports are my accounts. Both the forest and the
lake are from Canvec data imported together as in the same tile. I Imported
the whole thing (I would never import over existing data that carelessly),
and take responsibility for that.

- As my subsequent email showed, I missed the part about water/forest
overlaps in Paul's email. Both and other people on this list have explained
our feelings about this, but I will accept DWGs decision.

- Per Michael's suggestion: This is constructive, but I do not feel that it
is feasible because of the tiled structure of Canvec. To try and shift the
forest layer over would make the problem worse.

- Removal of the forest layer may be the best solution here. However there
are many places in where the data matches up perfectly. Speaking only of
areas where I am the primary importer, most of the forests in Ontario west
of Thunder Bay would have to go, which is unfortunate because some have
been manually corrected, though not enough to save them. Much (but
certainly not all) of the forests in Manitoba could be saved with minimal
effort.

- If/when this is done we could look at ways to restore these forests. In
areas that are mostly forest it shouldn't be too difficult to fill them
using large multipolygons.

Sam
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] CanVec Reverts

[Alan Richards]

I agree with Frederik here. New imports need to make sure they follow the
import guidelines, including using a dedicated import account.

The wiki information could all be cleaned up, consolidated, and then new
users would know the current status and process for importing new
information.

For cleaning up the existing information, I like the idea of getting some
MapRoulette tasks going, or maybe writing some new validations in osmlint,
osmose, etc.
Certainly in the BC and Ontario data I've seen, it's not just the forest
that is problematic, but also the water layer. Creeks occasionally flow the
wrong way (a bad very old import I suspect), lakes and wetlands are
mixed/overlaid.

Alan (alarobric)

On Thu, Sep 1, 2016 at 4:30 PM, john whelan  wrote:

> >I think a super good first step would be try and ensure that future
> imports are done diligently and don't introduce new issues.
>
> I think that is a reasonable way forward, and I concur with the rest of
> your post.
>
> I think we need to identify which parts of CANVEC are giving concern, each
> province is a different mixture of data sources, I suspect it is the forest
> and land use that are the most problematic.
>
> The older tiles had a problem in that a highway would reach the edge of
> the tile and there would be a matching highway on the next tile but there
> was no connection.  I spent many a happy hour, too many of them, merging
> nodes so routing would work.  I think the cities have been cleaned up but
> more rural areas still have the odd one or two to do.
>
> Probably what would make a lot of sense as a next step is to grab the
> provincial OSM dumps, chop them up into manageable portions then load them
> up into JOSM and run a modern validation on them.
>
> Cheerio John
>
> On 1 September 2016 at 19:13, Frederik Ramm  wrote:
>
>> Andrew,
>>
>> On 09/02/2016 12:47 AM, Andrew Lester wrote:
>> > If people from outside of Canada have decided that our data is so bad
>> > that it needs to be completely wiped out in its entirety, then I guess
>> > we're going to have to do something drastic to try to prevent this.
>>
>> I think a super good first step would be try and ensure that future
>> imports are done diligently and don't introduce new issues. (This might
>> be the "better documentation" step that Paul mentioned.) It really
>> shouldn't be too hard to detect whether your planned import causes
>> overlapping lakes and forests, but there needs to be an agreement that
>> these things matter and that you cannot simply upload "because if CanVec
>> says that forest and water overlap then this must be true".
>>
>> Then one could take stock of existing issues and make a plan on how to
>> fix them.
>>
>> Whether fixing existing issues will necessitate the wholesale removal of
>> some imports is something that should be decided down the line; I know
>> too little about CanVec imports to say whether some problems are
>> systemic in the data source, or certain regions, or just introduced by
>> clumsy importers. Any large-scale removal of imported data (perhaps to
>> replace it with new, better-imported data) would also have to take into
>> account potential manual work that has been performed on the imported by
>> mappers with local knowledge and it would be sad to lose that.
>>
>> Bye
>> Frederik
>>
>> --
>> Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"
>>
>> ___
>> Talk-ca mailing list
>> Talk-ca@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/talk-ca
>>
>
>
> ___
> Talk-ca mailing list
> Talk-ca@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk-ca
>
>
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] CanVec Reverts

[john whelan]

>I think a super good first step would be try and ensure that future
imports are done diligently and don't introduce new issues.

I think that is a reasonable way forward, and I concur with the rest of
your post.

I think we need to identify which parts of CANVEC are giving concern, each
province is a different mixture of data sources, I suspect it is the forest
and land use that are the most problematic.

The older tiles had a problem in that a highway would reach the edge of the
tile and there would be a matching highway on the next tile but there was
no connection.  I spent many a happy hour, too many of them, merging nodes
so routing would work.  I think the cities have been cleaned up but more
rural areas still have the odd one or two to do.

Probably what would make a lot of sense as a next step is to grab the
provincial OSM dumps, chop them up into manageable portions then load them
up into JOSM and run a modern validation on them.

Cheerio John

On 1 September 2016 at 19:13, Frederik Ramm  wrote:

> Andrew,
>
> On 09/02/2016 12:47 AM, Andrew Lester wrote:
> > If people from outside of Canada have decided that our data is so bad
> > that it needs to be completely wiped out in its entirety, then I guess
> > we're going to have to do something drastic to try to prevent this.
>
> I think a super good first step would be try and ensure that future
> imports are done diligently and don't introduce new issues. (This might
> be the "better documentation" step that Paul mentioned.) It really
> shouldn't be too hard to detect whether your planned import causes
> overlapping lakes and forests, but there needs to be an agreement that
> these things matter and that you cannot simply upload "because if CanVec
> says that forest and water overlap then this must be true".
>
> Then one could take stock of existing issues and make a plan on how to
> fix them.
>
> Whether fixing existing issues will necessitate the wholesale removal of
> some imports is something that should be decided down the line; I know
> too little about CanVec imports to say whether some problems are
> systemic in the data source, or certain regions, or just introduced by
> clumsy importers. Any large-scale removal of imported data (perhaps to
> replace it with new, better-imported data) would also have to take into
> account potential manual work that has been performed on the imported by
> mappers with local knowledge and it would be sad to lose that.
>
> Bye
> Frederik
>
> --
> Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"
>
> ___
> Talk-ca mailing list
> Talk-ca@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk-ca
>
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] CanVec Reverts

[Frederik Ramm]

Andrew,

On 09/02/2016 12:47 AM, Andrew Lester wrote:
> If people from outside of Canada have decided that our data is so bad
> that it needs to be completely wiped out in its entirety, then I guess
> we're going to have to do something drastic to try to prevent this.

I think a super good first step would be try and ensure that future
imports are done diligently and don't introduce new issues. (This might
be the "better documentation" step that Paul mentioned.) It really
shouldn't be too hard to detect whether your planned import causes
overlapping lakes and forests, but there needs to be an agreement that
these things matter and that you cannot simply upload "because if CanVec
says that forest and water overlap then this must be true".

Then one could take stock of existing issues and make a plan on how to
fix them.

Whether fixing existing issues will necessitate the wholesale removal of
some imports is something that should be decided down the line; I know
too little about CanVec imports to say whether some problems are
systemic in the data source, or certain regions, or just introduced by
clumsy importers. Any large-scale removal of imported data (perhaps to
replace it with new, better-imported data) would also have to take into
account potential manual work that has been performed on the imported by
mappers with local knowledge and it would be sad to lose that.

Bye
Frederik

-- 
Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"

___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] CanVec Reverts

[Andrew Lester]

If people from outside of Canada have decided that our data is so bad that it 
needs to be completely wiped out in its entirety, then I guess we're going to 
have to do something drastic to try to prevent this. 

@Michael, Frederik, Paul: would it be good enough for us to wipe out any and 
all CanVec forest data (or even ALL forest data because some could have been 
derived from CanVec)? This seems to be the biggest cause for concern. If not, 
what do you think we need to do to prevent all CanVec data from being wiped 
out? Wiping out all CanVec data would effectively clear out the majority of the 
Canadian map and really isn't an option in our minds. Do we need to get rid of 
all forest data and then go on a cooperative fixing blitz (maybe using 
MapRoulette or Tasking Manager) to fix every single JOSM validator error across 
the country? In short, if we're doing things so wrong, what can we do to make 
things right other than have a German revert all of our changesets so we can 
start from scratch? 

Andrew 
Victoria, BC, Canada 

- Original Message -

From: "Sam Dyck" <samueld...@gmail.com> 
To: "Talk-CA OpenStreetMap" <talk-ca@openstreetmap.org> 
Sent: Thursday, September 1, 2016 2:38:38 PM 
Subject: Re: [Talk-ca] CanVec Reverts 


After reading Paul's email again, its possible that what Nakaner is doing is in 
line with Paul's suggestion, if unnecessarily confrontational. I tried to play 
around in JOSM to see if I could get the forest polygons to a point where 
Nakaner would leave us alone by mercilessly deleting all of the inner ways in 
the forest multipolygons, but because of the way things are structured around 
rivers that would be several hours worth of work for one tile. Given this 
perhaps the only solution is to bulk delete all Canvec forest data. As someone 
who actually finds the forest data useful this would be extremely unfortunate, 
but if it allows us to continue imports without excessive external scrutiny 
then I am willing to except it. (apologies for the English only emails, my 
French writing skills are sadly lacking) 



On Thu, Sep 1, 2016 at 4:20 PM, Pierre Béland < pierz...@yahoo.fr > wrote: 




L'idéal préparer Une réponse standard indiquant que l'Import Canvec par la 
communauté OSM Canada est itératif et nous nous assurons collectivement 
d'améliorer les données. Voir page Wiki Import Canvec et venir discuter sur 
Talk-Ca si vous avez d'autres questions. 




Pierre 








De : Sam Dyck < samueld...@gmail.com > 
À : Talk-CA OpenStreetMap < talk-ca@openstreetmap.org > 
Envoyé le : jeudi 1 Septembre 2016 17h06 
Objet : Re: [Talk-ca] CanVec Reverts 






I received the following changeset comment from Nakaner for a Canvec import 
(changeset 
38158126) at 15:55 Central Time (20:55 UTC): 

"This changeset has uploaded data which does not fit to each other. There is an 
offset between the water areas and the forest areas. Example: 
https://www.openstreetmap.org/ way/406539219 
Could you please fix this?" 
I believe the given what we have just spent the last 24 hours discussing this 
request is unreasonable and the issue is not significant. Thoughts? 
Sam 

___ 
Talk-ca mailing list 
Talk-ca@openstreetmap.org 
https://lists.openstreetmap.org/listinfo/talk-ca 






___ 
Talk-ca mailing list 
Talk-ca@openstreetmap.org 
https://lists.openstreetmap.org/listinfo/talk-ca 

___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] CanVec Reverts

[Frederik Ramm]

Sam,

On 09/01/2016 11:26 PM, Frederik Ramm wrote:
>> I believe the given what we have just spent the last 24 hours discussing
>> this request is unreasonable and the issue is not significant. Thoughts?
> 
> An importer who imports data into OSM that doesn't match up with already
> existing data and doesn't notice it is careless and should do a better job.
> 
> An importer who is asked to fix non-matching data he has accidentally
> imported and responds that the request is "unreasonable" should have his
> account taken away.

It appears I have mistakenly thought that you, Sam, and the import user
"sammuell" were one and the same. Apologies. I still think that if the
importer cannot be bothered to fix it, the data should be removed until
someone has the time to do it right (rather than keeping bad data in OSM
until someone can fix it).

Bye
Frederik

-- 
Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"

___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] CanVec Reverts

[Sam Dyck]

After reading Paul's email again, its possible that what Nakaner is doing
is in line with Paul's suggestion, if unnecessarily confrontational. I
tried to play around in JOSM to see if I could get the forest polygons to a
point where Nakaner would leave us alone by mercilessly deleting all of the
inner ways in the forest multipolygons, but because of the way things are
structured around rivers that would be several hours worth of work for one
tile. Given this perhaps the only solution is to bulk delete all Canvec
forest data. As someone who actually finds the forest data useful this
would be extremely unfortunate, but if it allows us to continue imports
without excessive external scrutiny then I am willing to except it.
(apologies for the English only emails, my French writing skills are sadly
lacking)

On Thu, Sep 1, 2016 at 4:20 PM, Pierre Béland <pierz...@yahoo.fr> wrote:

> L'idéal préparer Une réponse standard indiquant que l'Import Canvec par la
> communauté OSM Canada est itératif et nous nous assurons collectivement
> d'améliorer les données. Voir page Wiki Import Canvec et venir discuter sur
> Talk-Ca si vous avez d'autres questions.
>
>
> Pierre
>
>
> --
> *De :* Sam Dyck <samueld...@gmail.com>
> *À :* Talk-CA OpenStreetMap <talk-ca@openstreetmap.org>
> *Envoyé le :* jeudi 1 Septembre 2016 17h06
> *Objet :* Re: [Talk-ca] CanVec Reverts
>
> I received the following changeset comment from Nakaner for a Canvec
> import (changeset
> 38158126) at 15:55 Central Time (20:55 UTC):
> "This changeset has uploaded data which does not fit to each other. There
> is an offset between the water areas and the forest areas. Example: 
> https://www.openstreetmap.org/
> way/406539219 <https://www.openstreetmap.org/way/406539219>
> Could you please fix this?"
> I believe the given what we have just spent the last 24 hours discussing
> this request is unreasonable and the issue is not significant. Thoughts?
> Sam
>
> ___
> Talk-ca mailing list
> Talk-ca@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk-ca
>
>
>
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] CanVec Reverts

[Michael Reichert]

Hi Sam,

Am 01.09.2016 um 23:06 schrieb Sam Dyck:
> I received the following changeset comment from Nakaner for a Canvec import
> (changeset
> 38158126) at 15:55 Central Time (20:55 UTC):
> 
> "This changeset has uploaded data which does not fit to each other. There
> is an offset between the water areas and the forest areas. Example:
> https://www.openstreetmap.org/way/406539219
> 
> Could you please fix this?"
> 
> I believe the given what we have just spent the last 24 hours discussing
> this request is unreasonable and the issue is not significant. Thoughts?

If you have a proper look at the whole area around, you will see that
there is a systematic offset between the water "layer" and the forest
layer. The forest layer should be moved about 10 to 30 metres to
northwest. I wonder how such an error can be overseen during the
preparation of this tile.

Other examples:
https://www.openstreetmap.org/way/402043390
https://www.openstreetmap.org/#map=16/49.3997/-87.4329

Maybe the original data was traced from different aerial imagery and
that's why there is an offset which is not constant.

Best regards

Michael


-- 
Per E-Mail kommuniziere ich bevorzugt GPG-verschlüsselt. (Mailinglisten
ausgenommen)
I prefer GPG encryption of emails. (does not apply on mailing lists)



signature.asc
Description: OpenPGP digital signature
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] CanVec Reverts

[Frederik Ramm]

Hi,

On 2016-09-01 21:06:57, Sam Dyck wrote:
> I believe the given what we have just spent the last 24 hours discussing
> this request is unreasonable and the issue is not significant. Thoughts?

An importer who imports data into OSM that doesn't match up with already
existing data and doesn't notice it is careless and should do a better job.

An importer who is asked to fix non-matching data he has accidentally
imported and responds that the request is "unreasonable" should have his
account taken away.

(This is my personal opinion. DWG is more nuanced but Paul Norman's
message which you have read says clearly that the importer has the
responsibility to see that the data matches up with existing stuff.)

Either https://www.openstreetmap.org/way/406539219 is wrong, or the
forest around it is wrong. And to quote someone else from this thread,
the result is certainly not aesthetically pleasing either. You should
find out which, and take appropriate steps. I am speechless to hear that
not only are you importing broken data, but you also consider it
unreasonable to fix it.

If you can't be bothered to care for quality when importing, then don't,
and wait for someone more responsible to come along.

Bye
Frederik

-- 
Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"

___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] CanVec Reverts

[Sam Dyck]

I received the following changeset comment from Nakaner for a Canvec import
(changeset
38158126) at 15:55 Central Time (20:55 UTC):

"This changeset has uploaded data which does not fit to each other. There
is an offset between the water areas and the forest areas. Example:
https://www.openstreetmap.org/way/406539219

Could you please fix this?"

I believe the given what we have just spent the last 24 hours discussing
this request is unreasonable and the issue is not significant. Thoughts?

Sam
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] CanVec Reverts

[James]

>From what Rps333 told me in person he had worked many hours on that
changeset and was frustrated when someone reverted before fixes could be
applied. Could you revert the revert?

On Sep 1, 2016 4:09 PM, "Paul Norman"  wrote:

> Hello,
>
> Multiple people have referred this issue and changeset 39517002 to the
> OpenStreetMap Foundation Data Working Group about this issue. The Data
> Working Group has responsibility for the resolution of disputes beyond
> the normal means of the community, as well as some responsibilities
> concerning imports. I am replying here rather than the changeset as it
> should reach everyone involved.
>
> CanVec Quality
> ==
>
> The CanVec import is one that was started long ago so parts of the
> documentation are lacking and some aspects are unclear.[1] CanVec, like
> any other import, has certain hard requirements, including the use of a
> dedicated user account.
>
> In particular it is expected that imported CanVec data is
>
> - Integrated with existing data, particularly at NTS/CanVec tile edges
>
> - Valid
>
> - Internally consistent with both the newly imported CanVec and existing
>   OSM data, particularly to avoid overlapping landuse like forest and
>   water, forest and residential, or wetlands in what is obviously a
>   residential subdivision.
>
> - Uploaded in small enough parts that the changesets make sense. This
>   means never uploading more than 50k objects at once, and typically
>   fewer than 10k.
>
> - Not duplicating existing OSM data. Evaluating what data is better
>   generally requires experience with both CanVec and OSM data in the
>   *region being mapped*
>
> It is not required that CanVec data is compared an external source like
> Bing imagery, but this is helpful, particularly when resolving problems.
>
> We encourage the Canadian community to develop better documentation for
> people importing CanVec to achieve this and to remove outdated
> documentation.[2]
>
> Reverting
> =
>
> Advance permission is not required for reverts, nor for normal mapping
> activities. At the same time, users are expected to be responsible,
> particularly when using tools for reverting which allow large-scale
> changes where other users may disagree with them.
>
> Where there are problems with an import reverting is an option, but
> just one of many, and often not the appropriate first action. Unless
> there are legal problems[3] or fatal problems with the import it is
> preferable if the original importer can fix the problems in a timely
> manner. There was every indication this was going to happen in this case.
>
> The revert of 39517002 was inappropriate and counter-productive. New
> actions like this revert may lead to further Data Working Group
> involvement and potentially blocks. If the Canadian community needs help
> reverting 41749133 and 41756737, the Data Working Group can revert those
> changesets.
>
> While not going into depth on the changeset discussion at this time, I
> want to remind everyone involved that OpenStreetMap is a crowd-sourcing
> project, which inherently involves working with other people. This
> requires good communication, which was absent here.
>
> Paul Norman
>
> For the OpenStreetMap Foundation Data Working Group
>
> [1]: Except for the CanVec license, which is ODbL and possibly CC BY-SA
>
> [2]: Much of the CanVec documentation is outdated, which makes it
>  difficult to know what is relevant. A good start would be removing
>  outdated documentation.
>
> [3]: Please refer cases of large-scale infringement to
> d...@osmfoundation.org
>  so we can "redact" the content to remove it from publicly viewable
>  history.
>
>
> ___
> Talk-ca mailing list
> Talk-ca@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk-ca
>
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] CanVec Reverts

[john whelan]

Canvec data has been imported for many years so why the sudden challenge
and change?  Are you willing to reimport the CANVEC data or are you intent
on leaving a blank area in the map?

Cheerio John

On 1 September 2016 at 15:41, Michael Reichert  wrote:

> Hi James,
>
> Am 2016-09-01 um 15:04 schrieb James:
> > To blatantly toss discussions of the
> > past whether to import CanVec or not into OSM and *that was approved*, …
>
> Could you point me to the discussion at the Imports mailing list? (link
> to the archive of the mailing list)
>
> I am not against importing CanVec data, I am just against importing
> CanVec data in a bad way.
>
> Best regards
>
> Michael
>
> --
> Per E-Mail kommuniziere ich bevorzugt GPG-verschlüsselt. (Mailinglisten
> ausgenommen)
> I prefer GPG encryption of emails. (does not apply on mailing lists)
>
>
>
> ___
> Talk-ca mailing list
> Talk-ca@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk-ca
>
>
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] CanVec Reverts

[Paul Norman]

Hello,

Multiple people have referred this issue and changeset 39517002 to the
OpenStreetMap Foundation Data Working Group about this issue. The Data
Working Group has responsibility for the resolution of disputes beyond
the normal means of the community, as well as some responsibilities
concerning imports. I am replying here rather than the changeset as it
should reach everyone involved.

CanVec Quality
==

The CanVec import is one that was started long ago so parts of the
documentation are lacking and some aspects are unclear.[1] CanVec, like
any other import, has certain hard requirements, including the use of a
dedicated user account.

In particular it is expected that imported CanVec data is

- Integrated with existing data, particularly at NTS/CanVec tile edges

- Valid

- Internally consistent with both the newly imported CanVec and existing
  OSM data, particularly to avoid overlapping landuse like forest and
  water, forest and residential, or wetlands in what is obviously a
  residential subdivision.

- Uploaded in small enough parts that the changesets make sense. This
  means never uploading more than 50k objects at once, and typically
  fewer than 10k.

- Not duplicating existing OSM data. Evaluating what data is better
  generally requires experience with both CanVec and OSM data in the
  *region being mapped*

It is not required that CanVec data is compared an external source like
Bing imagery, but this is helpful, particularly when resolving problems.

We encourage the Canadian community to develop better documentation for
people importing CanVec to achieve this and to remove outdated
documentation.[2]

Reverting
=

Advance permission is not required for reverts, nor for normal mapping
activities. At the same time, users are expected to be responsible,
particularly when using tools for reverting which allow large-scale
changes where other users may disagree with them.

Where there are problems with an import reverting is an option, but
just one of many, and often not the appropriate first action. Unless
there are legal problems[3] or fatal problems with the import it is
preferable if the original importer can fix the problems in a timely
manner. There was every indication this was going to happen in this case.

The revert of 39517002 was inappropriate and counter-productive. New
actions like this revert may lead to further Data Working Group
involvement and potentially blocks. If the Canadian community needs help
reverting 41749133 and 41756737, the Data Working Group can revert those
changesets.

While not going into depth on the changeset discussion at this time, I
want to remind everyone involved that OpenStreetMap is a crowd-sourcing
project, which inherently involves working with other people. This
requires good communication, which was absent here.

Paul Norman

For the OpenStreetMap Foundation Data Working Group

[1]: Except for the CanVec license, which is ODbL and possibly CC BY-SA

[2]: Much of the CanVec documentation is outdated, which makes it
 difficult to know what is relevant. A good start would be removing
 outdated documentation.

[3]: Please refer cases of large-scale infringement to 
d...@osmfoundation.org

 so we can "redact" the content to remove it from publicly viewable
 history.


___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] CanVec Reverts

[Begin Daniel]

Hum 
First discussions about importing NRCan data can be find here...
https://lists.openstreetmap.org/pipermail/talk-ca/2008-November/000228.html

They are talking about importing GeoBase - which is exactly the same content 
that is found in Canvec since Canvec is the merging of GeoBase layers.

If you look at the date (November 2008), the import mailing list even did not 
exist.

We can go that way but I do not think it the point here. We must talk about 
deleting data that was imported in good faith, without leaving the community 
enough time to correct them. 

Daniel

-Original Message-
From: Michael Reichert [mailto:naka...@gmx.net] 
Sent: Thursday, 1 September, 2016 15:41
To: James
Cc: talk-ca@openstreetmap.org
Subject: Re: [Talk-ca] CanVec Reverts

Hi James,

Am 2016-09-01 um 15:04 schrieb James:
> To blatantly toss discussions of the
> past whether to import CanVec or not into OSM and *that was approved*, 
> …

Could you point me to the discussion at the Imports mailing list? (link to the 
archive of the mailing list)

I am not against importing CanVec data, I am just against importing CanVec data 
in a bad way.

Best regards

Michael

--
Per E-Mail kommuniziere ich bevorzugt GPG-verschlüsselt. (Mailinglisten
ausgenommen)
I prefer GPG encryption of emails. (does not apply on mailing lists)


___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] CanVec Reverts

[Darren Wiebe]

I'm usually just a lurker on these lists but this has dragged me out of my
cave.

Michael,

I'm glad you care about the quality of the map, I do too.  I welcome you to
take an constructive approach to working with these problems.  Canada has
an area of 9,984,670 square kilometers with a population density of only
8.8.  That presents us with challenges that wouldn't apply to a country
with an area of 357,022 square kilometers and a population density of
593.  Rather
than handing out an ultimatum to the Canadian mapping community how about
you work with us?  We share your concerns in regards to data quality but
your unilateral reversion of commits without communication or cooperation
is damaging to the map.

I find your comment "if it has not been touched for a few weeks" comment to
be insulting and it shows a complete lack of understanding of the realities
on the ground.  I'm personally working at improving the map in my area
(County of Vermilion River, Alberta, Canada) and you are more than welcome
to constructively help. However, just because I haven't touched something
for a couple of weeks doesn't mean I've forgotten about it.  It means I was
on holidays or got busy with other things in the office.

Have you ever tried importing data using JOSM?  I've spent hours poring
over a few square miles of countryside and trying to get everything cleaned
up and merged.  Then, when I actually import, the data gets split into much
smaller sizes.  Or maybe I get some of it imported and then have to fix a
bug I missed in something else.  To somebody like you it looks like I just
dumped a bunch of data in when in reality I've been picking away at it for
a few days.

Darren Wiebe



On Thu, Sep 1, 2016 at 8:49 AM, Sam Dyck  wrote:

> Micheal
>
> Thanks for contacting us. I must object strongly to your use of the Worst
> of OSM example and generally assumption that the data is broken if it
> doesn't line up. I checked multiple commercial imagery providers before I
> found a digitalglobe image that covered the area during the summer. There
> is a large patch of sand between the vegetation-filled area and the coast.
> As for the boundary, that comes from another official source, I think it is
> supposed to be spaced off of the coastline, though I don't remember exactly
> how they calculated it, we would likely need a constitutional change to
> make it line up with the coast. Just because things don't match up does not
> mean that the data is wrong. Nature doesn't always translate into nice,
> clean maps.
>
> Sam
>
> -Original Message-
> From: Michael Reichert [mailto:naka...@gmx.net]
> Sent: Thursday, 1 September, 2016 01:39
> To: talk-ca@openstreetmap.org
> Subject: [Talk-ca] CanVec Reverts
>
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Hi,
>
> unfortunately posting via Gmane does not seem to work (the website is down
> but NNTP still works), that's why I have to start a new thread. :-(
>
> Am Tue, 30 Aug 2016 21:41:21 -0500 schrieb Sam Dyck:
> > After reading through the changeset discussion, I discovered that one
> > of my imports in Northern Manitoba made Worst of OSM.
> > (http://worstofosm.tumblr.com/post/22180046353/dear-
> > openstreetmap-isnt-it-strange-how-the). As someone who spends a some
> > time amount of time in some of relatively unpopulated areas of Canada
> > and makes an effort to check the quality of Canvec data (which is
> > usually pretty good), I do agree that it is impossible to do
> > everything to the same level of quality that we would provide in
> > Toronto or Timmins or even small prairie towns.
>
> First of all, it is ok that an import takes a few years and therefore
> creates ugly green rectancles on the map. If an import is "unavoidable"
> :-), a manual import is the best thing that can be happen. But if someone
> uploads a changeset without a manual review beforehand, he counteracts the
> aim of a manual import: addind good data to OpenStreetMap. That's what I am
> mainly fighting against. If a users uploads much more than 100 objects per
> minute [1], you can be sure that he has not done any manual review. A
> manual review by myself confirmed this these. I am fighting against such
> changesets/users.
>
> A good imports must be reviewed *before* it is being uploaded. The review
> contains:
> - - Run JOSM validator, fix all warnings and errors. This includes all
> warnings regarding validity of areas. (you can argue if all warnings about
> "deprecated" tagging have to be fixed)
> - - Compare the data with available imagery. Is the forest really a forest
> or is another tag more appropiate? Right-click on a Bing tile at JOSM and
> have a look how old/recent the imagery is.
> - - Check if CanVec data fits to itself.
> http://worstofosm.tumblr.com/post/22180046353/dear-openstreetmap-isnt-
> it-strange-how-the
> 
> - - Check if there has been any other data 

Re: [Talk-ca] CanVec Reverts

[Sam Dyck]

Micheal

Thanks for contacting us. I must object strongly to your use of the Worst
of OSM example and generally assumption that the data is broken if it
doesn't line up. I checked multiple commercial imagery providers before I
found a digitalglobe image that covered the area during the summer. There
is a large patch of sand between the vegetation-filled area and the coast.
As for the boundary, that comes from another official source, I think it is
supposed to be spaced off of the coastline, though I don't remember exactly
how they calculated it, we would likely need a constitutional change to
make it line up with the coast. Just because things don't match up does not
mean that the data is wrong. Nature doesn't always translate into nice,
clean maps.

Sam

-Original Message-
From: Michael Reichert [mailto:naka...@gmx.net]
Sent: Thursday, 1 September, 2016 01:39
To: talk-ca@openstreetmap.org
Subject: [Talk-ca] CanVec Reverts

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Hi,

unfortunately posting via Gmane does not seem to work (the website is down
but NNTP still works), that's why I have to start a new thread. :-(

Am Tue, 30 Aug 2016 21:41:21 -0500 schrieb Sam Dyck:
> After reading through the changeset discussion, I discovered that one
> of my imports in Northern Manitoba made Worst of OSM.
> (http://worstofosm.tumblr.com/post/22180046353/dear-
> openstreetmap-isnt-it-strange-how-the). As someone who spends a some
> time amount of time in some of relatively unpopulated areas of Canada
> and makes an effort to check the quality of Canvec data (which is
> usually pretty good), I do agree that it is impossible to do
> everything to the same level of quality that we would provide in
> Toronto or Timmins or even small prairie towns.

First of all, it is ok that an import takes a few years and therefore
creates ugly green rectancles on the map. If an import is "unavoidable"
:-), a manual import is the best thing that can be happen. But if someone
uploads a changeset without a manual review beforehand, he counteracts the
aim of a manual import: addind good data to OpenStreetMap. That's what I am
mainly fighting against. If a users uploads much more than 100 objects per
minute [1], you can be sure that he has not done any manual review. A
manual review by myself confirmed this these. I am fighting against such
changesets/users.

A good imports must be reviewed *before* it is being uploaded. The review
contains:
- - Run JOSM validator, fix all warnings and errors. This includes all
warnings regarding validity of areas. (you can argue if all warnings about
"deprecated" tagging have to be fixed)
- - Compare the data with available imagery. Is the forest really a forest
or is another tag more appropiate? Right-click on a Bing tile at JOSM and
have a look how old/recent the imagery is.
- - Check if CanVec data fits to itself.
http://worstofosm.tumblr.com/post/22180046353/dear-openstreetmap-isnt-
it-strange-how-the

- - Check if there has been any other data before. If yes, adapt the either
the CanVec data or the old data.
https://wiki.openstreetmap.org/wiki/File:Import-Fails-Powerlines-Not-Ins
ide-Cutting.png

https://www.openstreetmap.org/way/439631732
- - Ways should not overlap with other ways if it is not necessary. The
outer ring of a lake should also be inner member of the forest
multipolygon. Maybe the program which created the OSM files should be
imprved?
- - Keep the history.
https://wiki.openstreetmap.org/wiki/Good_practice#Keep_the_history

If a tile has been imported without being checked manually and no
post-upload fixes have been done (i.e. upload without any checks), I will
not shrink from reverting it. If a tile has been uploaded to OSM without a
review and if it has not been fixed within a month, it is worthless and can
easily be reimported at a later time if someone has the time to check and
fix it.

For the future, I will abstain from reverting changesets which have been
imported before September 1, 2016 and whose users are currently doing the
fixes that should already have been done. But if I come across an imported
tile of low quality which has not been touched for a few weeks and is full
of errors, it is just a question of time until it is reverte d.

Best regards

Michael

[1] I had a look on a few of my changesets which added a large number of
buildings to OSM. The fastest changeset contained about 60 objects per
minute and was full of missing buildings as I later detected while
collecting the housenumbers and usage of the buildings.
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] Canvec reverts

[Loïc Haméon]

ne wrong, how to edit and
>>> how to fix them. :-/
>>>
>>> > 2) A addr:interp way may be split in 2 parts. 2 consequences:
>>> > - the interpolation way become useless because it's now 2 different
>>> objects.
>>> > - the mid-point becomes 2 superposed nodes. Useless duplication.
>>> >
>>> > 3) A grid tile has a fixed size. It may be appropriate for unpopulated
>>> areas
>>> > but it is too large for urban areas where editors work at a high zoom
>>> level
>>> > (17 and up). It's easy to damage a relation without knowing it.
>>> >
>>> > But there are other problems (not related to the grid):
>>> > 4) the relations seem to be designed to be stand-alone. Thus the
>>> relation
>>> > borders don't share a way. They use 2 superposed ways. Useless
>>> duplication.
>>> > It's very confusing and we frequently alter the wrong way.
>>> >
>>> > 5) lakes are represented by 2 superposed identical objects, one for
>>> the hole
>>> > and one for the lake. If the hole happen to be on top, the name will
>>> not be
>>> > displayed. It's an unjustifiable complexity for the casual user.
>>> > I've also seen triplicated contour (one for the lake, on for the inner
>>> role
>>> > and one for the outer role)
>>> >
>>> > Yes, all these quirks can be fixed manually but it's time-consuming
>>> and error-
>>> > prone.
>>>
>>> What about reverting the tiles which have all these issues and seem to
>>> be uploaded with too few checks beforehand, run a better version of the
>>> preprocessor on the CanVec raw data and reimport them again? (That
>>> causes a loss of OSM history but an import changeset is not as much
>>> valueable than a manual changeset)
>>>
>>> > Ideally, the contour of a forest must not split any object and it's not
>>> > possible with a grid.
>>> > The sole advantage of a grid IMHO is to simplify the CanVec exports.
>>>
>>> What about replacing the grid by less artificial borders, e.g. roads,
>>> rivers, powerlines etc. If an area has too few of theses objects,
>>> artifical borders could be automatically drawn which are optimized to
>>> cut as few objects as possible into two parts.
>>>
>>> > Some years ago I would have proposed that someone write a guide "How
>>> to fix a
>>> > CanVec import". But now I would rather propose that someone write a
>>> "How to
>>> > pre-process a CanVec export before importing it into OSM".
>>>
>>> +1
>>>
>>> All ongoing changesets which import CanVec data should either use an
>>> improved version of the preprocessor or should undergo the manual
>>> quality checks I described in my other emails and the changeset comments.
>>>
>>> Best regards
>>>
>>> Michael
>>>
>>>
>>> --
>>> Per E-Mail kommuniziere ich bevorzugt GPG-verschlüsselt. (Mailinglisten
>>> ausgenommen)
>>> I prefer GPG encryption of emails. (does not apply on mailing lists)
>>>
>>>
>>>
>>> ___
>>> Talk-ca mailing list
>>> Talk-ca@openstreetmap.org
>>> https://lists.openstreetmap.org/listinfo/talk-ca
>>>
>>>
>>
>> ___
>> Talk-ca mailing list
>> Talk-ca@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/talk-ca
>>
>>
>
> -- Forwarded message --
> From: Begin Daniel <jfd...@hotmail.com>
> To: Michael Reichert <naka...@gmx.net>, Begin Daniel <jfd...@hotmail.com>,
> "talk-ca@openstreetmap.org" <talk-ca@openstreetmap.org>
> Cc:
> Date: Thu, 1 Sep 2016 13:04:30 +
> Subject: Re: [Talk-ca] CanVec Reverts
> I understand your point. You might be right about the time between
> changesets (even though it may depends on if the users is working with
> layers), but I maintain my points about the time it may take within a
> changeset.
> Daniel
>
> -Original Message-
> From: Michael Reichert [mailto:naka...@gmx.net]
> Sent: Thursday, 1 September, 2016 08:37
> To: Begin Daniel; talk-ca@openstreetmap.org
> Subject: Re: [Talk-ca] CanVec Reverts
>
> Hi Daniel,
>
> Am 2016-09-01 um 12:26 schrieb Begin Daniel:
> > Furthermore, I hope you will not use you 100 objects per minute to
> >

Re: [Talk-ca] CanVec Reverts

[Begin Daniel]

I understand your point. You might be right about the time between changesets 
(even though it may depends on if the users is working with layers), but I 
maintain my points about the time it may take within a changeset.
Daniel

-Original Message-
From: Michael Reichert [mailto:naka...@gmx.net] 
Sent: Thursday, 1 September, 2016 08:37
To: Begin Daniel; talk-ca@openstreetmap.org
Subject: Re: [Talk-ca] CanVec Reverts

Hi Daniel,

Am 2016-09-01 um 12:26 schrieb Begin Daniel:
> Furthermore, I hope you will not use you 100 objects per minute to 
> decide whether or not you will delete a changeset. I think this 
> threshold is value doesn't' apply (see below)
> 
> Daniel
> 
> About the100 objects threshold.
> From my experience, if I load a Canvec tile in JOSM, make all the necessary 
> corrections and then import the result to OSM, I throw up to 25K objects to 
> the database within five minutes.  As far as I know, the timestamps attached 
> to the changeset and to the objects is generated by the OSM database when 
> receiving the data. The five minutes it takes to upload the data to the 
> database (5K objects per minute) do not reflect the time I spent editing the 
> data prior to the upload.

That's the base of my calculation I did with Rps333's changesets:

changeset   start   end object count

3951757119:30:5319:32:564311
3951768619:35:3019:41:1211724
3951794419:45:1519:47:274963
3951814719:53:2520:04:5519286

As you see, he took less than three minutes minutes after uploading
39517571 to prepare 39517686. You cannot check such an amount of data very well 
within that time.

Best regards

Michael


--
Per E-Mail kommuniziere ich bevorzugt GPG-verschlüsselt. (Mailinglisten
ausgenommen)
I prefer GPG encryption of emails. (does not apply on mailing lists)

___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] CanVec Reverts

[Michael Reichert]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Hi,

Am 2016-09-01 um 13:22 schrieb john whelan:
> In many parts of the country there are no local mappers for many 
> miles or kilometers if you prefer.  We do have some very 
> experienced GIS people around and I'm under the impression that we 
> really do know what we are doing.

This is no carte blanche to import any dataset you can get. The import
discussion exists to ensure that an import is done the right way, i.e.
importing good data in a good way. If the users who import a dataset
base on an import disussion/proposal do not care about the quality,
they violate the proposal and therefore the policy because the import
becomes an import which has not been approved.

Best regards

Michael


- -- 
Per E-Mail kommuniziere ich bevorzugt GPG-verschlüsselt.
(Mailinglisten ausgenommen)
I prefer GPG encryption of emails. (does not apply on mailing lists)
-BEGIN PGP SIGNATURE-

iQIcBAEBCAAGBQJXyCKEAAoJEB87G9rMCMyIqvgQAK58kt5ULsvv0nf0MnCm6ahM
Iq8Kub3bLS2ZE93kVlgUl43cEJ6pzr3/XS0fd4hbRPuybektD0N9kNumm/KVZ2Pn
2/fZvOpai3YcVRAiSMZ2HUHaZnz9CYI56xsFYJcE8+RXcmTBXbp1WBmbqd4j7ceu
+IRdkrxweAQEDymlYtfI1rd1gA+CJBWnfcRr7Fq1CUNi2yI4M4U697qxtK1TdQuW
4Y+SmiDlUvGJLos0qjzucXs3zatY5SELY3/sTZ4iS0Emla8m5Eq1Wqo09FCGngrT
Yov1gQQBuMlUl80BsILM6beAKfhYq0hYgje77VzONmZYN79mMzkXHD3IJS2/lVfZ
r4pU2BL6bDTYues0diPefNbCvTi0SkLAOJccsy4+6OLWQ5B9WJgW8yT3KTbv6N0T
25yuaEGBdbLcs4dacZj1PdMKy2W4EgTy2UQKadlc4l4DAVJYyuaLifx0ij8n5pgs
hepMWWTh0B5WyIckDGVMBUk9awQFu1IN7gm5zJWgTap6Kz3m0O0TbvOGtaXjod7C
teFq+MmZ7wf/ocGc0AMCweZgJUBKiTu+tcKYrFUQq4f6XSCblzYiSJA95NlRNGJh
BiLVgu/K7nJ1fYgPhN9wSVGgP21HRs80X2ZVtGLbTL/hZTjSDKNLd8sS6tT+86Ie
ek7VLR9LDoAeihcRu3zU
=Ea1G
-END PGP SIGNATURE-

___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] CanVec Reverts

[James]

What tells you he didnt prepare batches outside the upload time(offline)?
Your logic is skewed

On Sep 1, 2016 8:39 AM, "Michael Reichert"  wrote:

> Hi Daniel,
>
> Am 2016-09-01 um 12:26 schrieb Begin Daniel:
> > Furthermore, I hope you will not use you 100 objects per minute to
> decide whether or not you will delete a changeset. I think this threshold
> is value doesn't' apply (see below)
> >
> > Daniel
> >
> > About the100 objects threshold.
> > From my experience, if I load a Canvec tile in JOSM, make all the
> necessary corrections and then import the result to OSM, I throw up to 25K
> objects to the database within five minutes.  As far as I know, the
> timestamps attached to the changeset and to the objects is generated by the
> OSM database when receiving the data. The five minutes it takes to upload
> the data to the database (5K objects per minute) do not reflect the time I
> spent editing the data prior to the upload.
>
> That's the base of my calculation I did with Rps333's changesets:
>
> changeset   start   end object count
> 
> 3951757119:30:5319:32:564311
> 3951768619:35:3019:41:1211724
> 3951794419:45:1519:47:274963
> 3951814719:53:2520:04:5519286
>
> As you see, he took less than three minutes minutes after uploading
> 39517571 to prepare 39517686. You cannot check such an amount of data
> very well within that time.
>
> Best regards
>
> Michael
>
>
> --
> Per E-Mail kommuniziere ich bevorzugt GPG-verschlüsselt. (Mailinglisten
> ausgenommen)
> I prefer GPG encryption of emails. (does not apply on mailing lists)
>
>
> ___
> Talk-ca mailing list
> Talk-ca@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk-ca
>
>
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] CanVec Reverts

[Michael Reichert]

Hi Daniel,

Am 2016-09-01 um 12:26 schrieb Begin Daniel:
> Furthermore, I hope you will not use you 100 objects per minute to decide 
> whether or not you will delete a changeset. I think this threshold is value 
> doesn't' apply (see below)
> 
> Daniel
> 
> About the100 objects threshold.
> From my experience, if I load a Canvec tile in JOSM, make all the necessary 
> corrections and then import the result to OSM, I throw up to 25K objects to 
> the database within five minutes.  As far as I know, the timestamps attached 
> to the changeset and to the objects is generated by the OSM database when 
> receiving the data. The five minutes it takes to upload the data to the 
> database (5K objects per minute) do not reflect the time I spent editing the 
> data prior to the upload.

That's the base of my calculation I did with Rps333's changesets:

changeset   start   end object count

3951757119:30:5319:32:564311
3951768619:35:3019:41:1211724
3951794419:45:1519:47:274963
3951814719:53:2520:04:5519286

As you see, he took less than three minutes minutes after uploading
39517571 to prepare 39517686. You cannot check such an amount of data
very well within that time.

Best regards

Michael


-- 
Per E-Mail kommuniziere ich bevorzugt GPG-verschlüsselt. (Mailinglisten
ausgenommen)
I prefer GPG encryption of emails. (does not apply on mailing lists)



signature.asc
Description: OpenPGP digital signature
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] CanVec Reverts

[john whelan]

Note to Michael Reichert

I think you should respect the views of the local community.

We know the data isn't perfect but for the most part its good data and as
Daniel has said when I've imported the work is done in JOSM off line over a
period of time so the time apparently taken doesn't really indicate this.

In many parts of the country there are no local mappers for many miles or
kilometers if you prefer.  We do have some very experienced GIS people
around and I'm under the impression that we really do know what we are
doing.

Cheerio John

On 1 September 2016 at 06:26, Begin Daniel  wrote:

> Thank for contacting the Canadian community Michael,
> You provided us with a short but useful reminder of current rules we
> should apply when importing data (or even just making standard edits).
>
> However, I understand from your last paragraph that you will keep deleting
> changesets. I was hoping your email started a discussion on best practices
> that we could be put in place in our context since adjusting Canvec data to
> the latest rules is a daunting task. I rather feel it is an ultimatum.
>
> Do your future actions will apply to the imports made a few months, a few
> years ago, which are 'full of errors' and for which nobody seems to care?
> Are you going to check with concerned contributors (old/future imports) if
> they bother or not to see their work deleted before you do it?
>
> Furthermore, I hope you will not use you 100 objects per minute to decide
> whether or not you will delete a changeset. I think this threshold is value
> doesn't' apply (see below)
>
> Daniel
>
> About the100 objects threshold.
> From my experience, if I load a Canvec tile in JOSM, make all the
> necessary corrections and then import the result to OSM, I throw up to 25K
> objects to the database within five minutes.  As far as I know, the
> timestamps attached to the changeset and to the objects is generated by the
> OSM database when receiving the data. The five minutes it takes to upload
> the data to the database (5K objects per minute) do not reflect the time I
> spent editing the data prior to the upload.
>
> -Original Message-
> From: Michael Reichert [mailto:naka...@gmx.net]
> Sent: Thursday, 1 September, 2016 01:39
> To: talk-ca@openstreetmap.org
> Subject: [Talk-ca] CanVec Reverts
>
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Hi,
>
> unfortunately posting via Gmane does not seem to work (the website is down
> but NNTP still works), that's why I have to start a new thread. :-(
>
> Am Tue, 30 Aug 2016 21:41:21 -0500 schrieb Sam Dyck:
> > After reading through the changeset discussion, I discovered that one
> > of my imports in Northern Manitoba made Worst of OSM.
> > (http://worstofosm.tumblr.com/post/22180046353/dear-
> > openstreetmap-isnt-it-strange-how-the). As someone who spends a some
> > time amount of time in some of relatively unpopulated areas of Canada
> > and makes an effort to check the quality of Canvec data (which is
> > usually pretty good), I do agree that it is impossible to do
> > everything to the same level of quality that we would provide in
> > Toronto or Timmins or even small prairie towns.
>
> First of all, it is ok that an import takes a few years and therefore
> creates ugly green rectancles on the map. If an import is "unavoidable"
> :-), a manual import is the best thing that can be happen. But if someone
> uploads a changeset without a manual review beforehand, he counteracts the
> aim of a manual import: addind good data to OpenStreetMap. That's what I am
> mainly fighting against. If a users uploads much more than 100 objects per
> minute [1], you can be sure that he has not done any manual review. A
> manual review by myself confirmed this these. I am fighting against such
> changesets/users.
>
> A good imports must be reviewed *before* it is being uploaded. The review
> contains:
> - - Run JOSM validator, fix all warnings and errors. This includes all
> warnings regarding validity of areas. (you can argue if all warnings about
> "deprecated" tagging have to be fixed)
> - - Compare the data with available imagery. Is the forest really a forest
> or is another tag more appropiate? Right-click on a Bing tile at JOSM and
> have a look how old/recent the imagery is.
> - - Check if CanVec data fits to itself.
> http://worstofosm.tumblr.com/post/22180046353/dear-openstreetmap-isnt-
> it-strange-how-the
> - - Check if there has been any other data before. If yes, adapt the
> either the CanVec data or the old data.
> https://wiki.openstreetmap.org/wiki/File:Import-Fails-Powerlines-Not-Ins
> ide-Cutting.png
> https://www.openstreetmap.org/way/439631732
> - - Ways should not overlap with other ways if it is not necessary. The
> outer ring of a lake should also be inner member of the forest
> multipolygon. Maybe the program which created the OSM files should be
> imprved?
> - - Keep the history.
> 

Re: [Talk-ca] CanVec Reverts

[Begin Daniel]

Thank for contacting the Canadian community Michael, 
You provided us with a short but useful reminder of current rules we should 
apply when importing data (or even just making standard edits).

However, I understand from your last paragraph that you will keep deleting 
changesets. I was hoping your email started a discussion on best practices that 
we could be put in place in our context since adjusting Canvec data to the 
latest rules is a daunting task. I rather feel it is an ultimatum.  

Do your future actions will apply to the imports made a few months, a few years 
ago, which are 'full of errors' and for which nobody seems to care? Are you 
going to check with concerned contributors (old/future imports) if they bother 
or not to see their work deleted before you do it? 

Furthermore, I hope you will not use you 100 objects per minute to decide 
whether or not you will delete a changeset. I think this threshold is value 
doesn't' apply (see below)

Daniel

About the100 objects threshold.
From my experience, if I load a Canvec tile in JOSM, make all the necessary 
corrections and then import the result to OSM, I throw up to 25K objects to the 
database within five minutes.  As far as I know, the timestamps attached to the 
changeset and to the objects is generated by the OSM database when receiving 
the data. The five minutes it takes to upload the data to the database (5K 
objects per minute) do not reflect the time I spent editing the data prior to 
the upload.

-Original Message-
From: Michael Reichert [mailto:naka...@gmx.net] 
Sent: Thursday, 1 September, 2016 01:39
To: talk-ca@openstreetmap.org
Subject: [Talk-ca] CanVec Reverts

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Hi,

unfortunately posting via Gmane does not seem to work (the website is down but 
NNTP still works), that's why I have to start a new thread. :-(

Am Tue, 30 Aug 2016 21:41:21 -0500 schrieb Sam Dyck:
> After reading through the changeset discussion, I discovered that one 
> of my imports in Northern Manitoba made Worst of OSM.
> (http://worstofosm.tumblr.com/post/22180046353/dear-
> openstreetmap-isnt-it-strange-how-the). As someone who spends a some 
> time amount of time in some of relatively unpopulated areas of Canada 
> and makes an effort to check the quality of Canvec data (which is 
> usually pretty good), I do agree that it is impossible to do 
> everything to the same level of quality that we would provide in 
> Toronto or Timmins or even small prairie towns.

First of all, it is ok that an import takes a few years and therefore creates 
ugly green rectancles on the map. If an import is "unavoidable"
:-), a manual import is the best thing that can be happen. But if someone 
uploads a changeset without a manual review beforehand, he counteracts the aim 
of a manual import: addind good data to OpenStreetMap. That's what I am mainly 
fighting against. If a users uploads much more than 100 objects per minute [1], 
you can be sure that he has not done any manual review. A manual review by 
myself confirmed this these. I am fighting against such changesets/users.

A good imports must be reviewed *before* it is being uploaded. The review 
contains:
- - Run JOSM validator, fix all warnings and errors. This includes all warnings 
regarding validity of areas. (you can argue if all warnings about "deprecated" 
tagging have to be fixed)
- - Compare the data with available imagery. Is the forest really a forest or 
is another tag more appropiate? Right-click on a Bing tile at JOSM and have a 
look how old/recent the imagery is.
- - Check if CanVec data fits to itself.
http://worstofosm.tumblr.com/post/22180046353/dear-openstreetmap-isnt-
it-strange-how-the
- - Check if there has been any other data before. If yes, adapt the either the 
CanVec data or the old data.
https://wiki.openstreetmap.org/wiki/File:Import-Fails-Powerlines-Not-Ins
ide-Cutting.png
https://www.openstreetmap.org/way/439631732
- - Ways should not overlap with other ways if it is not necessary. The outer 
ring of a lake should also be inner member of the forest multipolygon. Maybe 
the program which created the OSM files should be imprved?
- - Keep the history.
https://wiki.openstreetmap.org/wiki/Good_practice#Keep_the_history

If a tile has been imported without being checked manually and no post-upload 
fixes have been done (i.e. upload without any checks), I will not shrink from 
reverting it. If a tile has been uploaded to OSM without a review and if it has 
not been fixed within a month, it is worthless and can easily be reimported at 
a later time if someone has the time to check and fix it.

For the future, I will abstain from reverting changesets which have been 
imported before September 1, 2016 and whose users are currently doing the fixes 
that should already have been done. But if I come across an imported tile of 
low quality which has not been touched for a few weeks and is full of errors, 
it is just a question of time until it is 

Re: [Talk-ca] CanVec Reverts

[James]

See that's where I have an issue with your revert logic, if errors are
mostly corrected and say theres only 1 or 2 warnings, you want to revert
the whole thing which is very bad for: 1. The community and 2. The
database. Let me explain: there are many hours that go into merging down
CanVec ways and you want to undo all that work. Why is it bad for the
database? It makes it bigger everytine you do stuff like that, an ID has to
be allocated to every node, way and relation and by you "deleting"
everything, they remain in the database as history which is bad as the full
history planet file is already retardedly big(200+GB) and then when someone
redoes the work you undid, new IDs are allocated which doubles the data
size just for you being anal about perfection. Instead of reverting it for
1-2 warnings FIX IT! You dont have nice bing satelite imagery to
reference(which is the case with most of northern Canada)? Too bad you want
to be anal. Also this is a reminder that bing imagery can be 1. Really
really really stale(10-15 years old as taking high res photos of trees
doesnt return on investment) and 2. 1-20m offset. Instead of reverting the
data that isnt perfect, why not spend your time correcting it instead of
work being done: Canada isn't Germany.

Ich wünsche einen guten Tag!

On Sep 1, 2016 1:40 AM, "Michael Reichert"  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Hi,
>
> unfortunately posting via Gmane does not seem to work (the website is
> down but NNTP still works), that's why I have to start a new thread. :-(
>
> Am Tue, 30 Aug 2016 21:41:21 -0500 schrieb Sam Dyck:
> > After reading through the changeset discussion, I discovered that
> > one of my imports in Northern Manitoba made Worst of OSM.
> > (http://worstofosm.tumblr.com/post/22180046353/dear-
> > openstreetmap-isnt-it-strange-how-the). As someone who spends a
> > some time amount of time in some of relatively unpopulated areas of
> > Canada and makes an effort to check the quality of Canvec data
> > (which is usually pretty good), I do agree that it is impossible to
> > do everything to the same level of quality that we would provide in
> > Toronto or Timmins or even small prairie towns.
>
> First of all, it is ok that an import takes a few years and therefore
> creates ugly green rectancles on the map. If an import is "unavoidable"
> :-), a manual import is the best thing that can be happen. But if
> someone uploads a changeset without a manual review beforehand, he
> counteracts the aim of a manual import: addind good data to
> OpenStreetMap. That's what I am mainly fighting against. If a users
> uploads much more than 100 objects per minute [1], you can be sure that
> he has not done any manual review. A manual review by myself confirmed
> this these. I am fighting against such changesets/users.
>
> A good imports must be reviewed *before* it is being uploaded. The
> review contains:
> - - Run JOSM validator, fix all warnings and errors. This includes all
> warnings regarding validity of areas. (you can argue if all warnings
> about "deprecated" tagging have to be fixed)
> - - Compare the data with available imagery. Is the forest really a forest
> or is another tag more appropiate? Right-click on a Bing tile at JOSM
> and have a look how old/recent the imagery is.
> - - Check if CanVec data fits to itself.
> http://worstofosm.tumblr.com/post/22180046353/dear-openstreetmap-isnt-
> it-strange-how-the
> - - Check if there has been any other data before. If yes, adapt the
> either the CanVec data or the old data.
> https://wiki.openstreetmap.org/wiki/File:Import-Fails-Powerlines-Not-Ins
> ide-Cutting.png
> https://www.openstreetmap.org/way/439631732
> - - Ways should not overlap with other ways if it is not necessary. The
> outer ring of a lake should also be inner member of the forest
> multipolygon. Maybe the program which created the OSM files should be
> imprved?
> - - Keep the history.
> https://wiki.openstreetmap.org/wiki/Good_practice#Keep_the_history
>
> If a tile has been imported without being checked manually and no
> post-upload fixes have been done (i.e. upload without any checks), I
> will not shrink from reverting it. If a tile has been uploaded to OSM
> without a review and if it has not been fixed within a month, it is
> worthless and can easily be reimported at a later time if someone has
> the time to check and fix it.
>
> For the future, I will abstain from reverting changesets which have been
> imported before September 1, 2016 and whose users are currently doing
> the fixes that should already have been done. But if I come across an
> imported tile of low quality which has not been touched for a few weeks
> and is full of errors, it is just a question of time until it is reverte
> d.
>
> Best regards
>
> Michael
>
>
>
> [1] I had a look on a few of my changesets which added a large number of
> buildings to OSM. The fastest changeset contained about 60 objects per
> minute and was full