Re: [OSM-talk] Announcing Daylight Map Distribution

[Marc M.]

Mikel Maron wrote :

> "you don't really know what you're talking about" is a rude thing to say.

it why I said "I get the impression that..."

> And expressing a belief that FB is not capable of doing this to a small, or 
> even large degree, is not based on any factual inquiry of what they've done.

my "factual inquiry" is :
- for some change, checking it if it's valid take more time than doing
it (somebody delete a shop in a changeset with other stuffs and without
a comment about the shop : is the shop closed or not ?)
- FB has a hard time dealing with community feedback.
Either it is voluntary or it underestimates the resources needed.
It is not reasonable to think that a company can afford to do "more"
when it fails to ack previous feedback properly.

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Announcing Daylight Map Distribution

[Mateusz Konieczny via talk]

Mar 10, 2020, 23:17 by m...@teczno.com:

> things we filtered out individually because we found a problem with them
>
Once this is separated out it may be interesting to use.

Especially if one will be able to filter by error/heuristic type.

Maybe some will report serious issues not detected by current QA
tools like JOSM validator and Osmose.
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Announcing Daylight Map Distribution

[80hnhtv4agou--- via talk]

how do i fix a local mistake, as a local, that goes on for miles.

  
>Tuesday, March 10, 2020 5:13 AM -05:00 from Volker Schmidt :
> 
> 
>  Fixing stuff in OSM purely from imagery may not be good.
>
>A local mapper who sees something may add it before any satellite imagery has 
>it.
>
>If you then 'fix' this back to the satellite imagery you will have committed 
>an error,
>and that error may dissuade our most important resource from ever making any 
>further changes- the local mapper.
>
>Be very careful!
 
I second this last line !
 
I am observing an influx of mixed-quality remote edits from Amazon Logistics in 
my area.
I expect this Facebook operation to produce much more changes or potential 
changes (=suspected errors).
What we need for both cases and similar ones in the future is a way of being 
able to identify such changes, which by their nature will be armchair-mapping 
efforts.
I do not have a specific proposal, but I would appreciate a tool that helps me, 
as local mapper,  find these edits, and, more importantly we need a new 
approach to organise digesting these massive distributed armchair-mapping 
interventions on OSM data.
I don't realistically think that banning these activities is good for OSM. Not 
dealing in a systematic way with it at all presents, however, a big risk of 
deteriorating the map for two reasons:
(1) bad armchair edits by Amazon and Facebook (and others)
(2) demotivating non-armchair mappers
 
I repeat I do not have a proposal how to handle that. My main concern is that 
the required work for locally checking even only those edits that need checking 
(I am assuming that at least FB has good algorithms to sort out the 
dead-certain corrections beforehand. I am more sceptical with Amazon's changing 
local access tagging to, essentially, "yes" everywhere they have delivered 
something by delivery van. I came across a good number of them, and in most 
cases they were at least dubious)
 
Volker
(Padova, Italy)
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk  
 
 
 ___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Announcing Daylight Map Distribution

[Mikel Maron]

I heard Mapbox is working on this and divide data spatially not as
a sequence of changeset. My impression would be that this way
you could produce a "nice looking map" but most likely it will
break for routing purposes in most horrible ways where ways suddenly
are not connected anymore as some changeset inbetween has been
withdrawn/rejected.

Mapbox Streets Review groups data for review by feature type and spatial 
proximity, for a single day. There’s some complexities but it does maintain the 
routing graph.
I understand Facebook does something similar, but yes their use case does not 
involve routing but only visible map. Nothing wrong with that, fit the process 
to purpose.

> So i'd guess the way you and IIRC Mapbox try to solve the vandalism/bad
edit issue is a labour and machine learning intensive task which you cant win. 
Once you eliminate changesets you fall behind and you pile up inconsistencie
That’s not the case. It is labor intensive, but with well designed processes 
it’s manageable, and you can stay on pace and consistent.
I’d say one issue is not missing problems but being overly conservative, and 
flagging false positives. So flags in OSMCha from Mapbox shouldn’t be 
interpreted as a definite problem, but a suspicion. That’s by design, but it 
would be good to get even more accurate.
> So i'd love to hear more thoughts about long term ideas how to solvethis in a 
>collaborative manner. OSMCha is probably not the final solution but currently 
>it brings together analysis, be it human
or machine learning in a transparent way, not that it currently has an impact 
on the main OSM database.
100%. OSM and OSM validation needs to be collaborative to work. One idea, 
OSMCha could be more integrated into OSM.org, could provide more insightful 
insight in the history view. 
Mikel

On Tuesday, March 10, 2020, 5:52 PM, Florian Lohoff  wrote:

On Mon, Mar 09, 2020 at 05:08:10PM -0700, Michal Migurski wrote:
> Hi everyone,
> 
> I’m writing to let you know about a new OpenStreetMap project Facebook
> just released. It’s called Daylight Map Distribution. Daylight is a
> complete, downloadable preview of OpenStreetMap data we plan to start
> using in a number of our public maps:

I think its a humble approach to try to identify (un)intentional bad
edits. A lot of people try to deal with this.

I am doing a lot of QA myself and i look at OSMCha changesets in 
my greater surrounding on a daily basis.

I fail to see a sane technical way of producing consistent map data
out of some intermingled data of which some changesets have been 
flagged/removed.

I heard Mapbox is working on this and divide data spatially not as
a sequence of changeset. My impression would be that this way
you could produce a "nice looking map" but most likely it will
break for routing purposes in most horrible ways where ways suddenly
are not connected anymore as some changeset inbetween has been
withdrawn/rejected.

So i'd guess the way you and IIRC Mapbox try to solve the vandalism/bad
edit issue is a labour and machine learning intensive task which you
cant win. Once you eliminate changesets you fall behind and you
pile up inconsistencies. This is, i guess, the reason for
your "one shot" dump of your current internal state.

So from my perspective the vandalism/bad edit issue will only be
fixable if we have some review process (Not that i would suggest one)
for strictly sequential changesets where review must be in order
and a once rejected/withdrawn changeset can only be requeued not
put in that sequential place again. And even then you'll see 
vandalism sneak by with innocent looking edits or intentional
3rd party validation.

So i'd love to hear more thoughts about long term ideas how to solve
this in a collaborative manner. OSMCha is probably not the final
solution but currently it brings together analysis, be it human
or machine learning in a transparent way, not that it currently
has an impact on the main OSM database.

Flo
-- 
Florian Lohoff                                                f...@zz.de
        UTF-8 Test: The  ran after a , but the  ran 
away___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk



___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Announcing Daylight Map Distribution

[stevea]

On Mar 10, 2020, at 3:17 PM, Michal Migurski  wrote:
> Thanks Mikel for your followup!
> 
> In response to your recommendation that we publish what *didn't* make it 
> through the filter process, we released an OSC diff file:
> 
>   
> https://daylight-map-distribution.s3.amazonaws.com/pbf/planet-2020-03-06_v0.1.osc.bz2
> 
> At this early stage of trying to determine what’s useful for the community, 
> it’s pretty raw. The OSC does not yet differentiate between things we 
> filtered out categorically because we don't show them on our maps, vs. things 
> we filtered out individually because we found a problem with them. Individual 
> tree nodes are an example of the former: they don't get shown or labeled so 
> they’re not in Daylight.
> 
> I appreciate everyone’s questions about this data release. The FB team behind 
> Daylight and our other mapping efforts is well aware that OSM is a tough and 
> curious community!
> 
> -mike.

I realize that OSMCha (as Mapbox describes its use) and the compressed planet 
file extract (as Facebook publishes and begins to describe its use) are 
(respectively) ways to "flag changesets/features with reasons" and are 
characterizable as in "early stages of trying to determine what's useful for 
the community (pretty raw)."

I'd call both of these a reasonable start, especially as Mapbox's pipeline 
seems a bit further along in its development and Facebook's data is in an early 
state, still needing more feedback from the community.

I encourage the community to provide this feedback, perhaps on an 
ongoing/continual basis, and thank both Mapbox and Facebook (well, Mikel and 
Michal) for their participation and good communication here.

Good QA improvement can and does work:  in a project like OSM with corporate 
participants endeavoring on major sub-projects, it is usually based on very 
open communication with very wide community like this.  Yay!  (And conversely, 
as these get better established and "tightened up" into what the community 
expects and can participate in improving, the volume knob here can be turned 
down as these discussions don't have to be quite so public).

SteveA
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Announcing Daylight Map Distribution

[Michal Migurski]

Thanks Mikel for your followup!

In response to your recommendation that we publish what *didn't* make it 
through the filter process, we released an OSC diff file:


https://daylight-map-distribution.s3.amazonaws.com/pbf/planet-2020-03-06_v0.1.osc.bz2
 


At this early stage of trying to determine what’s useful for the community, 
it’s pretty raw. The OSC does not yet differentiate between things we filtered 
out categorically because we don't show them on our maps, vs. things we 
filtered out individually because we found a problem with them. Individual tree 
nodes are an example of the former: they don't get shown or labeled so they’re 
not in Daylight.

I appreciate everyone’s questions about this data release. The FB team behind 
Daylight and our other mapping efforts is well aware that OSM is a tough and 
curious community!

-mike.


michal migurski- contact info and pgp key:
sf/cahttp://mike.teczno.com/contact.html

> On Mar 10, 2020, at 1:20 PM, Mikel Maron  wrote:
> 
> Thanks Mike and Facebook for doing this. I commented on your diary post, but 
> also adding to the coversation here. It's great to have this insight out and 
> available. There's a good tradition of downstream data processing and 
> redistribution in the community (you could call them packages I supposed) -- 
> from GeoFabrik's regional and country downloads, to OSMQATiles, etc.
> 
> 
> In this case (and I focused on this when we spoke), I'm not sure that the 
> most valuable thing to distribute is what made it through Facebook filters, 
> but rather what didn't make it through and why. That insight is valuable to 
> identify problems that need fixing on a faster basis, notify local 
> communities and other editors, and to build up a corpus of understanding of 
> what problematic edits in OSM look like.
> 
> 
> The most actionable way to do this distribution will be through OSMCha. 
> Through the OSMCha API, you can flag changesets/features with reasons, and 
> can be set up so that any reason tag by Facebook has a "Facebook:" prefix.
> 
> 
> This is what Mapbox has set up. The Mapbox Streets Review team looks at edits 
> every day, and problems are flagged and surfaced in OSMCha. You can see all 
> of this [with this OSMCha 
> filter](https://osmcha.org/?aoi=083b147b-a72c-4026-9db5-b70761a6795c). You'll 
> see the most recent flag as about 3 days ago -- that's the typical time 
> between OSM edit and review / publishing in Mapbox Streets.
> 
> 
> Adding in Facebook flagged problems to OSMCha would provide even stronger 
> signal of problems, and hope to explore implementing it with you all.
> 
> -Mikel
> 
> * Mikel Maron * +14152835207 @mikel s:mikelmaron
> 
> 
> 
> 
> 
> 
> On Monday, March 9, 2020, 08:10:29 PM EDT, Michal Migurski  
> wrote: 
> 
> 
> 
> 
> 
> Hi everyone,
> 
> I’m writing to let you know about a new OpenStreetMap project Facebook just 
> released. It’s called Daylight Map Distribution. Daylight is a complete, 
> downloadable preview of OpenStreetMap data we plan to start using in a number 
> of our public maps:
> 
> https://www.openstreetmap.org/user/migurski/diary/392416
> 
> Facebook uses maps to let our users find friends, businesses, groups and 
> more. OpenStreetMap (OSM) has a substantial global footprint of map data 
> built and maintained by a dedicated community of global mappers and it’s a 
> natural choice for us. Every day, OSM receives millions of contributions from 
> the community. Some of these contributions may have intentional and 
> unintentional edits that are incompatible with our needs. Our mapping teams 
> work to scrub these contributions for consistency and quality. 
> 
> What’s Included in the Daylight Map Distribution:
> 
> • A PBF planet file composed of 100% OSM data, released under the terms 
> of the Open Database License.
> • Only those edits which have been validated to contain no malicious 
> vandalism or unintentional errors so we can show them in our display maps
> 
> This is just an initial first release, and we’re looking for feedback from 
> the community to decide what would be useful to release in the future and how 
> frequently. I’d be interested to hear any response you might have about it!
> 
> -mike.
> 
> 
> michal migurski- contact info and pgp key:
> sf/cahttp://mike.teczno.com/contact.html
> 
> 
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk
> 

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Announcing Daylight Map Distribution

[Florian Lohoff]

On Mon, Mar 09, 2020 at 05:08:10PM -0700, Michal Migurski wrote:
> Hi everyone,
> 
> I’m writing to let you know about a new OpenStreetMap project Facebook
> just released. It’s called Daylight Map Distribution. Daylight is a
> complete, downloadable preview of OpenStreetMap data we plan to start
> using in a number of our public maps:

I think its a humble approach to try to identify (un)intentional bad
edits. A lot of people try to deal with this.

I am doing a lot of QA myself and i look at OSMCha changesets in 
my greater surrounding on a daily basis.

I fail to see a sane technical way of producing consistent map data
out of some intermingled data of which some changesets have been 
flagged/removed.

I heard Mapbox is working on this and divide data spatially not as
a sequence of changeset. My impression would be that this way
you could produce a "nice looking map" but most likely it will
break for routing purposes in most horrible ways where ways suddenly
are not connected anymore as some changeset inbetween has been
withdrawn/rejected.

So i'd guess the way you and IIRC Mapbox try to solve the vandalism/bad
edit issue is a labour and machine learning intensive task which you
cant win. Once you eliminate changesets you fall behind and you
pile up inconsistencies. This is, i guess, the reason for
your "one shot" dump of your current internal state.

So from my perspective the vandalism/bad edit issue will only be
fixable if we have some review process (Not that i would suggest one)
for strictly sequential changesets where review must be in order
and a once rejected/withdrawn changeset can only be requeued not
put in that sequential place again. And even then you'll see 
vandalism sneak by with innocent looking edits or intentional
3rd party validation.

So i'd love to hear more thoughts about long term ideas how to solve
this in a collaborative manner. OSMCha is probably not the final
solution but currently it brings together analysis, be it human
or machine learning in a transparent way, not that it currently
has an impact on the main OSM database.

Flo
-- 
Florian Lohoff f...@zz.de
UTF-8 Test: The  ran after a , but the  ran away


signature.asc
Description: PGP signature
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Announcing Daylight Map Distribution

[stevea]

Similar to what Mikel described as "what Mapbox has set up," I humbly suggest 
that Facebook offer the wider OSM community (here on OSM-talk is a good place 
to do so) something similar.  As we (here) better understand what, exactly, 
Facebook's QA processes are as they use (and improve) OSM data, the wider OSM 
community can offer a critique of them, suggest more robust improvements to 
workflow (if we see that there are any to offer) and everybody wins:  both OSM 
and Facebook together.  If "better data" are the ultimate goal (and aren't 
they?!) there should be no argument with these suggestions.  Facebook might say 
"our QA process are proprietary" (and they'd be correct), but in the spirit of 
"Open" being OSM's first name, I hope not.

I say this as a professional software and data quality 
scientist/engineer/analyst (since the 1980s) and long-time contributor to OSM 
(for most of its history).  It is nearly always true that improvements to 
"quality process" can be made, especially as these are opened up to the wider 
scrutiny of a knowledgable and experienced community, as we are here.  Indeed, 
"quality process improvement" is itself correctly a near-constant process.  
Whether at the level of individual contributor or corporate behemoth, such 
wider scrutiny in a project like OSM should be par for the course (expected, 
"business as usual").

SteveA
California


___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Announcing Daylight Map Distribution

[Sören Reinecke via talk]

> But "you don't really know what you're talking about" is a rude thing to say. And expressing a belief that FB is not capable of doing this to a small, or even large degree, is not based on any factual inquiry of what they've done.Thank you. We cannot really know how Facebook is working on Quality Assurance. Original Message Subject: Re: [OSM-talk] Announcing Daylight Map DistributionFrom: Mikel Maron To: talk@openstreetmap.org,"Marc M." CC: It's fair to point out that "no malicious vandalism or unintentional errors" is overdoing it, no one can claim 100% on this. But "you don't really know what you're talking about" is a rude thing to say. And expressing a belief that FB is not capable of doing this to a small, or even large degree, is not based on any factual inquiry of what they've done.* Mikel Maron * +14152835207 @mikel s:mikelmaronOn Tuesday, March 10, 2020, 04:06:05 PM EDT, Marc M.  wrote: Michal Migurski wrote> Only those edits which have been validated to contain no malicious vandalism or unintentional errors so we can show them in our display mapsI get the impression that you don't really know what you're talkingabout. There are many "complaints" that FB does not respond enoughtto community feedback. So to believe that FB is capable of validatingeven the slightest change is naive at best.I spend a lot of time checking every changeset in 2 comfort zones, Iregularly detect anomalies, but to know if it is correct or not takes time.Regards,Marc___talk mailing listtalk@openstreetmap.orghttps://lists.openstreetmap.org/listinfo/talk___talk mailing listtalk@openstreetmap.orghttps://lists.openstreetmap.org/listinfo/talk___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Announcing Daylight Map Distribution

[Mikel Maron]

Thanks Mike and Facebook for doing this. I commented on your diary post, but 
also adding to the coversation here. It's great to have this insight out and 
available. There's a good tradition of downstream data processing and 
redistribution in the community (you could call them packages I supposed) -- 
from GeoFabrik's regional and country downloads, to OSMQATiles, etc.


In this case (and I focused on this when we spoke), I'm not sure that the most 
valuable thing to distribute is what made it through Facebook filters, but 
rather what didn't make it through and why. That insight is valuable to 
identify problems that need fixing on a faster basis, notify local communities 
and other editors, and to build up a corpus of understanding of what 
problematic edits in OSM look like.


The most actionable way to do this distribution will be through OSMCha. Through 
the OSMCha API, you can flag changesets/features with reasons, and can be set 
up so that any reason tag by Facebook has a "Facebook:" prefix.


This is what Mapbox has set up. The Mapbox Streets Review team looks at edits 
every day, and problems are flagged and surfaced in OSMCha. You can see all of 
this [with this OSMCha 
filter](https://osmcha.org/?aoi=083b147b-a72c-4026-9db5-b70761a6795c). You'll 
see the most recent flag as about 3 days ago -- that's the typical time between 
OSM edit and review / publishing in Mapbox Streets.


Adding in Facebook flagged problems to OSMCha would provide even stronger 
signal of problems, and hope to explore implementing it with you all.

-Mikel

* Mikel Maron * +14152835207 @mikel s:mikelmaron






On Monday, March 9, 2020, 08:10:29 PM EDT, Michal Migurski  
wrote: 





Hi everyone,

I’m writing to let you know about a new OpenStreetMap project Facebook just 
released. It’s called Daylight Map Distribution. Daylight is a complete, 
downloadable preview of OpenStreetMap data we plan to start using in a number 
of our public maps:

    https://www.openstreetmap.org/user/migurski/diary/392416

Facebook uses maps to let our users find friends, businesses, groups and more. 
OpenStreetMap (OSM) has a substantial global footprint of map data built and 
maintained by a dedicated community of global mappers and it’s a natural choice 
for us. Every day, OSM receives millions of contributions from the community. 
Some of these contributions may have intentional and unintentional edits that 
are incompatible with our needs. Our mapping teams work to scrub these 
contributions for consistency and quality. 

What’s Included in the Daylight Map Distribution:

    • A PBF planet file composed of 100% OSM data, released under the terms of 
the Open Database License.
    • Only those edits which have been validated to contain no malicious 
vandalism or unintentional errors so we can show them in our display maps

This is just an initial first release, and we’re looking for feedback from the 
community to decide what would be useful to release in the future and how 
frequently. I’d be interested to hear any response you might have about it!

-mike.


michal migurski- contact info and pgp key:
sf/ca            http://mike.teczno.com/contact.html


___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Announcing Daylight Map Distribution

[Mikel Maron]

It's fair to point out that "no malicious vandalism or unintentional errors" is 
overdoing it, no one can claim 100% on this. 

But "you don't really know what you're talking about" is a rude thing to say. 
And expressing a belief that FB is not capable of doing this to a small, or 
even large degree, is not based on any factual inquiry of what they've done.

* Mikel Maron * +14152835207 @mikel s:mikelmaron






On Tuesday, March 10, 2020, 04:06:05 PM EDT, Marc M. 
 wrote: 





Michal Migurski wrote
> Only those edits which have been validated to contain no malicious vandalism 
> or unintentional errors so we can show them in our display maps

I get the impression that you don't really know what you're talking
about. There are many "complaints" that FB does not respond enought
to community feedback. So to believe that FB is capable of validating
even the slightest change is naive at best.
I spend a lot of time checking every changeset in 2 comfort zones, I
regularly detect anomalies, but to know if it is correct or not takes time.

Regards,
Marc


___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Announcing Daylight Map Distribution

[Marc M.]

Michal Migurski wrote
> Only those edits which have been validated to contain no malicious vandalism 
> or unintentional errors so we can show them in our display maps

I get the impression that you don't really know what you're talking
about. There are many "complaints" that FB does not respond enought
to community feedback. So to believe that FB is capable of validating
even the slightest change is naive at best.
I spend a lot of time checking every changeset in 2 comfort zones, I
regularly detect anomalies, but to know if it is correct or not takes time.

Regards,
Marc

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Announcing Daylight Map Distribution

[Alan Mackie]

I would personally prefer it if a certain bit of software said "GPX traces"
or similar for the public traces rather than "survey" when autofilling
sources, but I never remember to raise it in the bug tracker.

On Tue, 10 Mar 2020, 18:09 Volker Schmidt,  wrote:

> My personal approach: when I map I routinely use three sources in
> parallel: survey,  GPX track, Mapillary images, satellite photos (picking
> the one with the most up-to -date pictures in the area, and aligning them
> to the Italian PCN2006 pictures, which are by our experience the
> best-aligned pictures available here).
> As a consequence (I am not always consistent, to be honest) I would have
> something like "source=survey; GPX tracks; Mapillary; Esri Images aligned
> with PCN2006"
> But my mapping is often not anything near to armchair mapping, I am using
> the images in addition to the other tools.
> I would not consider the fact that sattellite images are used, on its own
> as an indication that the date need to taken with caution.
>
>
> On Tue, 10 Mar 2020 at 15:54, Sören Reinecke 
> wrote:
>
>> Yes, I thought also about this and planned it to integrate in my concept.
>>
>>
>>  Original Message 
>> Subject: Re: [OSM-talk] Announcing Daylight Map Distribution
>> From: Joseph Eisenberg
>> To: Sören Reinecke
>> CC: Volker Schmidt ,talk@openstreetmap.org
>>
>>
>> My understanding is that the common way to describe armchair mapping,
>> based on aerial imagery, is to identify the imagery source. So I often
>> write:
>>
>> Changeset Comment: "Added and adjusted streams and rivers near Oksibil
>> with ESRI"
>> Changeset Source: "Esri world imagery"
>>
>> This makes it clear that I used Esri imagery to map the streams and
>> rivers, right?
>>
>> - Joseph Eisenberg
>>
>> On 3/10/20, Sören Reinecke via talk wrote:
>> > Hey
>> >
>> > some ideas about identifying such changes:
>> >
>> >
>> > Example changeset comment where a mapper did armchair mapping:
>> > Data updated, added amenity=restaurant
>> > #armchair
>> >
>> > In addition if the mapper works for a company:
>> > #
>> > e.g. #facebook
>> > #amazon
>> > #microsoft
>> > #apple
>> >
>> > Example changeset comment where a mapper did a survey and added data as
>> > (s)he saw it (from the ground):
>> > #survey
>> >
>> >
>> >
>> > This way we can organize our changes and Facebook and other companies
>> and
>> > the community as well know how to validate and can distinguish
>> changesets
>> > from another. I could create a wikipage where we think about this
>> "changeset
>> > governance"
>> >
>> > Cheers
>> >
>> > Sören Reinecke alias Valor Naram
>> >
>> >
>> >  Original Message 
>> > Subject: Re: [OSM-talk] Announcing Daylight Map Distribution
>> > From: Volker Schmidt
>> > To: talk@openstreetmap.org
>> > CC:
>> >
>> >
>> >>
>> >>
>> >>> Fixing stuff in OSM purely from imagery may not be good.
>> >>>
>> >>> A local mapper who sees something may add it before any satellite
>> imagery
>> >>> has it.
>> >>>
>> >>> If you then 'fix' this back to the satellite imagery you will have
>> >>> committed an error,
>> >>> and that error may dissuade our most important resource from ever
>> making
>> >>> any further changes- the local mapper.
>> >>>
>> >>> Be very careful!
>> >>
>> >>
>> >> I second this last line !
>> >>
>> >> I am observing an influx of mixed-quality remote edits from Amazon
>> >> Logistics in my area.
>> >> I expect this Facebook operation to produce much more changes or
>> potential
>> >> changes (=suspected errors).
>> >> What we need for both cases and similar ones in the future is a way of
>> >> being able to identify such changes, which by their nature will be
>> >> armchair-mapping efforts.
>> >> I do not have a specific proposal, but I would appreciate a tool that
>> >> helps me, as local mapper,  find these edits, and, more importantly we
>> >> need a new approach to organise digesting these massive distributed
>> >> armchair-mapping interventions on 

Re: [OSM-talk] Announcing Daylight Map Distribution

[Volker Schmidt]

My personal approach: when I map I routinely use three sources in parallel:
survey,  GPX track, Mapillary images, satellite photos (picking the one
with the most up-to -date pictures in the area, and aligning them to the
Italian PCN2006 pictures, which are by our experience the best-aligned
pictures available here).
As a consequence (I am not always consistent, to be honest) I would have
something like "source=survey; GPX tracks; Mapillary; Esri Images aligned
with PCN2006"
But my mapping is often not anything near to armchair mapping, I am using
the images in addition to the other tools.
I would not consider the fact that sattellite images are used, on its own
as an indication that the date need to taken with caution.


On Tue, 10 Mar 2020 at 15:54, Sören Reinecke 
wrote:

> Yes, I thought also about this and planned it to integrate in my concept.
>
>
>  Original Message --------
> Subject: Re: [OSM-talk] Announcing Daylight Map Distribution
> From: Joseph Eisenberg
> To: Sören Reinecke
> CC: Volker Schmidt ,talk@openstreetmap.org
>
>
> My understanding is that the common way to describe armchair mapping,
> based on aerial imagery, is to identify the imagery source. So I often
> write:
>
> Changeset Comment: "Added and adjusted streams and rivers near Oksibil
> with ESRI"
> Changeset Source: "Esri world imagery"
>
> This makes it clear that I used Esri imagery to map the streams and
> rivers, right?
>
> - Joseph Eisenberg
>
> On 3/10/20, Sören Reinecke via talk wrote:
> > Hey
> >
> > some ideas about identifying such changes:
> >
> >
> > Example changeset comment where a mapper did armchair mapping:
> > Data updated, added amenity=restaurant
> > #armchair
> >
> > In addition if the mapper works for a company:
> > #
> > e.g. #facebook
> > #amazon
> > #microsoft
> > #apple
> >
> > Example changeset comment where a mapper did a survey and added data as
> > (s)he saw it (from the ground):
> > #survey
> >
> >
> >
> > This way we can organize our changes and Facebook and other companies and
> > the community as well know how to validate and can distinguish changesets
> > from another. I could create a wikipage where we think about this
> "changeset
> > governance"
> >
> > Cheers
> >
> > Sören Reinecke alias Valor Naram
> >
> >
> >  Original Message 
> > Subject: Re: [OSM-talk] Announcing Daylight Map Distribution
> > From: Volker Schmidt
> > To: talk@openstreetmap.org
> > CC:
> >
> >
> >>
> >>
> >>> Fixing stuff in OSM purely from imagery may not be good.
> >>>
> >>> A local mapper who sees something may add it before any satellite
> imagery
> >>> has it.
> >>>
> >>> If you then 'fix' this back to the satellite imagery you will have
> >>> committed an error,
> >>> and that error may dissuade our most important resource from ever
> making
> >>> any further changes- the local mapper.
> >>>
> >>> Be very careful!
> >>
> >>
> >> I second this last line !
> >>
> >> I am observing an influx of mixed-quality remote edits from Amazon
> >> Logistics in my area.
> >> I expect this Facebook operation to produce much more changes or
> potential
> >> changes (=suspected errors).
> >> What we need for both cases and similar ones in the future is a way of
> >> being able to identify such changes, which by their nature will be
> >> armchair-mapping efforts.
> >> I do not have a specific proposal, but I would appreciate a tool that
> >> helps me, as local mapper,  find these edits, and, more importantly we
> >> need a new approach to organise digesting these massive distributed
> >> armchair-mapping interventions on OSM data.
> >> I don't realistically think that banning these activities is good for
> OSM.
> >> Not dealing in a systematic way with it at all presents, however, a big
> >> risk of deteriorating the map for two reasons:
> >> (1) bad armchair edits by Amazon and Facebook (and others)
> >> (2) demotivating non-armchair mappers
> >>
> >> I repeat I do not have a proposal how to handle that. My main concern is
> >> that the required work for locally checking even only those edits that
> >> need checking (I am assuming that at least FB has good algorithms to
> sort
> >> out the dead-certain corrections beforehand. I am more sceptical with
> >> Amazon's changing local access tagging to, essentially, "yes" everywhere
> >> they have delivered something by delivery van. I came across a good
> number
> >> of them, and in most cases they were at least dubious)
> >>
> >> Volker
> >> (Padova, Italy)
> >
>
>
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Announcing Daylight Map Distribution

[Michal Migurski]

Thanks for your feedback, Christoph.

The “100%” is accurate because there’s no data released in Daylight that hasn't 
previously passed through the OSM.org  database. When we 
observe errors and fix them manually, we make our edits to OpenStreetMap like 
any other editor.

I’m not sure I fully understand your second question, hopefully you can 
elaborate!

-mike.


michal migurski- contact info and pgp key:
sf/cahttp://mike.teczno.com/contact.html

> On Mar 10, 2020, at 3:44 AM, Christoph Hormann  wrote:
> 
> On Tuesday 10 March 2020, Michal Migurski wrote:
>> 
>>  • A PBF planet file composed of 100% OSM data, released under the
>> terms of the Open Database License. • Only those edits which have
>> been validated to contain no malicious vandalism or unintentional
>> errors so we can show them in our display maps
> 
> Could someone maybe do an analysis of the diff regarding numbers of 
> features removed/changed/added for various types of objects?
> 
> Regarding
> 
>>  • A PBF planet file composed of 100% OSM data
> 
> that is probably an incorrect characterization because any time you 
> modify OSM data without uploading the results to OSM what you get is no 
> more 100% OSM data.
> 
> Thinking this further - the real question is if there is other data used 
> in production of the maps using this that constitutes a derivative 
> database according to the ODbL or in other words:  Does Facebook claim 
> that this is the only derivative database they are using?
> 
> -- 
> Christoph Hormann
> http://www.imagico.de/
> 
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Announcing Daylight Map Distribution

[Sören Reinecke via talk]

Yes, I thought also about this and planned it to integrate in my concept. Original Message Subject: Re: [OSM-talk] Announcing Daylight Map DistributionFrom: Joseph Eisenberg To: Sören Reinecke CC: Volker Schmidt ,talk@openstreetmap.orgMy understanding is that the common way to describe armchair mapping,based on aerial imagery, is to identify the imagery source. So I oftenwrite:Changeset Comment: "Added and adjusted streams and rivers near Oksibilwith ESRI"Changeset Source: "Esri world imagery"This makes it clear that I used Esri imagery to map the streams andrivers, right?- Joseph EisenbergOn 3/10/20, Sören Reinecke via talk  wrote:> Hey>> some ideas about identifying such changes:>>> Example changeset comment where a mapper did armchair mapping:> Data updated, added amenity=restaurant> #armchair>> In addition if the mapper works for a company:> #> e.g. #facebook> #amazon> #microsoft> #apple>> Example changeset comment where a mapper did a survey and added data as> (s)he saw it (from the ground):> #survey>>>> This way we can organize our changes and Facebook and other companies and> the community as well know how to validate and can distinguish changesets> from another. I could create a wikipage where we think about this "changeset> governance">> Cheers>> Sören Reinecke alias Valor Naram>>>  Original Message > Subject: Re: [OSM-talk] Announcing Daylight Map Distribution> From: Volker Schmidt> To: talk@openstreetmap.org> CC:>>>>>>>>> Fixing stuff in OSM purely from imagery may not be good.>>>>>> A local mapper who sees something may add it before any satellite imagery>>> has it.>>>>>> If you then 'fix' this back to the satellite imagery you will have>>> committed an error,>>> and that error may dissuade our most important resource from ever making>>> any further changes- the local mapper.>>>>>> Be very careful!>>>>>> I second this last line !>>>> I am observing an influx of mixed-quality remote edits from Amazon>> Logistics in my area.>> I expect this Facebook operation to produce much more changes or potential>> changes (=suspected errors).>> What we need for both cases and similar ones in the future is a way of>> being able to identify such changes, which by their nature will be>> armchair-mapping efforts.>> I do not have a specific proposal, but I would appreciate a tool that>> helps me, as local mapper,  find these edits, and, more importantly we>> need a new approach to organise digesting these massive distributed>> armchair-mapping interventions on OSM data.>> I don't realistically think that banning these activities is good for OSM.>> Not dealing in a systematic way with it at all presents, however, a big>> risk of deteriorating the map for two reasons:>> (1) bad armchair edits by Amazon and Facebook (and others)>> (2) demotivating non-armchair mappers>>>> I repeat I do not have a proposal how to handle that. My main concern is>> that the required work for locally checking even only those edits that>> need checking (I am assuming that at least FB has good algorithms to sort>> out the dead-certain corrections beforehand. I am more sceptical with>> Amazon's changing local access tagging to, essentially, "yes" everywhere>> they have delivered something by delivery van. I came across a good number>> of them, and in most cases they were at least dubious)>>>> Volker>> (Padova, Italy)>___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Announcing Daylight Map Distribution

[Mario Frasca]

that would be VERY nice, if you managed to achieve this.

On 10/03/2020 06:46, Sören Reinecke via talk wrote:

In addition if the mapper works for a company:
#
e.g. #facebook
#amazon
#microsoft
#apple


I've been asking about the '#apple' hashtag, for quite a while, straight 
from Andrew Wiseman, but I don't seem to be able to make myself 
understood, or heard.


they have their team, you can find their edits matching changeset 
timestamp with historic team composition, if you really insist.


___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Announcing Daylight Map Distribution

[Joseph Eisenberg]

My understanding is that the common way to describe armchair mapping,
based on aerial imagery, is to identify the imagery source. So I often
write:

Changeset Comment: "Added and adjusted streams and rivers near Oksibil
with ESRI"
Changeset Source: "Esri world imagery"

This makes it clear that I used Esri imagery to map the streams and
rivers, right?

- Joseph Eisenberg

On 3/10/20, Sören Reinecke via talk  wrote:
> Hey
>
> some ideas about identifying such changes:
>
>
> Example changeset comment where a mapper did armchair mapping:
> Data updated, added amenity=restaurant
> #armchair
>
> In addition if the mapper works for a company:
> #
> e.g. #facebook
> #amazon
> #microsoft
> #apple
>
> Example changeset comment where a mapper did a survey and added data as
> (s)he saw it (from the ground):
> #survey
>
>
>
> This way we can organize our changes and Facebook and other companies and
> the community as well know how to validate and can distinguish changesets
> from another. I could create a wikipage where we think about this "changeset
> governance"
>
> Cheers
>
> Sören Reinecke alias Valor Naram
>
>
>  Original Message 
> Subject: Re: [OSM-talk] Announcing Daylight Map Distribution
> From: Volker Schmidt
> To: talk@openstreetmap.org
> CC:
>
>
>>
>>
>>> Fixing stuff in OSM purely from imagery may not be good.
>>>
>>> A local mapper who sees something may add it before any satellite imagery
>>> has it.
>>>
>>> If you then 'fix' this back to the satellite imagery you will have
>>> committed an error,
>>> and that error may dissuade our most important resource from ever making
>>> any further changes- the local mapper.
>>>
>>> Be very careful!
>>
>>
>> I second this last line !
>>
>> I am observing an influx of mixed-quality remote edits from Amazon
>> Logistics in my area.
>> I expect this Facebook operation to produce much more changes or potential
>> changes (=suspected errors).
>> What we need for both cases and similar ones in the future is a way of
>> being able to identify such changes, which by their nature will be
>> armchair-mapping efforts.
>> I do not have a specific proposal, but I would appreciate a tool that
>> helps me, as local mapper,  find these edits, and, more importantly we
>> need a new approach to organise digesting these massive distributed
>> armchair-mapping interventions on OSM data.
>> I don't realistically think that banning these activities is good for OSM.
>> Not dealing in a systematic way with it at all presents, however, a big
>> risk of deteriorating the map for two reasons:
>> (1) bad armchair edits by Amazon and Facebook (and others)
>> (2) demotivating non-armchair mappers
>>
>> I repeat I do not have a proposal how to handle that. My main concern is
>> that the required work for locally checking even only those edits that
>> need checking (I am assuming that at least FB has good algorithms to sort
>> out the dead-certain corrections beforehand. I am more sceptical with
>> Amazon's changing local access tagging to, essentially, "yes" everywhere
>> they have delivered something by delivery van. I came across a good number
>> of them, and in most cases they were at least dubious)
>>
>> Volker
>> (Padova, Italy)
>

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Announcing Daylight Map Distribution

[Andrew Harvey]

On Tue, 10 Mar 2020 at 21:14, Volker Schmidt  wrote:

> I expect this Facebook operation to produce much more changes or potential
> changes (=suspected errors).
> What we need for both cases and similar ones in the future is a way of
> being able to identify such changes, which by their nature will be
> armchair-mapping efforts.
>

OSMCha has a ticket for integrating with OSM teams
https://github.com/mapbox/osmcha-frontend/issues/390 it would allow you to
open up OSMCha and filter for changesets by a given editing team in a given
area for you to review. There is a PR implementing this in OSMCha but it's
yet to be merged.
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Announcing Daylight Map Distribution

[Sören Reinecke via talk]

Heysome ideas about identifying such changes:Example changeset comment where a mapper did armchair mapping:Data updated, added amenity=restaurant#armchairIn addition if the mapper works for a company:#e.g. #facebook#amazon#microsoft#appleExample changeset comment where a mapper did a survey and added data as (s)he saw it (from the ground):#surveyThis way we can organize our changes and Facebook and other companies and the community as well know how to validate and can distinguish changesets from another. I could create a wikipage where we think about this "changeset governance"CheersSören Reinecke alias Valor Naram Original Message Subject: Re: [OSM-talk] Announcing Daylight Map DistributionFrom: Volker Schmidt To: talk@openstreetmap.orgCC: Fixing stuff in OSM purely from imagery may not be good.

A local mapper who sees something may add it before any satellite imagery has it.

If you then 'fix' this back to the satellite imagery you will have committed an error,
and that error may dissuade our most important resource from ever making any further changes- the local mapper.

Be very careful!I second this last line !I am observing an influx of mixed-quality remote edits from Amazon Logistics in my area. I expect this Facebook operation to produce much more changes or potential changes (=suspected errors).What we need for both cases and similar ones in the future is a way of being able to identify such changes, which by their nature will be armchair-mapping efforts.I do not have a specific proposal, but I would appreciate a tool that helps me, as local mapper,  find these edits, and, more importantly we need a new approach to organise digesting these massive distributed armchair-mapping interventions on OSM data.I don't realistically think that banning these activities is good for OSM. Not dealing in a systematic way with it at all presents, however, a big risk of deteriorating the map for two reasons: (1) bad armchair edits by Amazon and Facebook (and others)(2) demotivating non-armchair mappersI repeat I do not have a proposal how to handle that. My main concern is that the required work for locally checking even only those edits that need checking (I am assuming that at least FB has good algorithms to sort out the dead-certain corrections beforehand. I am more sceptical with Amazon's changing local access tagging to, essentially, "yes" everywhere they have delivered something by delivery van. I came across a good number of them, and in most cases they were at least dubious)Volker(Padova, Italy)
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Announcing Daylight Map Distribution

[Florimond Berthoux]

My proposal :

survey > aerial imagery.

So its the aerial imagery mapper who has to check that its aerial imageries
are more recent than the last changes made with survey sources.
That’s something I do often when I see weird differences.
(So we put the load on the big companies instead of the free local mappers
;)

Regards.

Le mar. 10 mars 2020 à 11:14, Volker Schmidt  a écrit :

>
>
> Fixing stuff in OSM purely from imagery may not be good.
>>
>> A local mapper who sees something may add it before any satellite imagery
>> has it.
>>
>> If you then 'fix' this back to the satellite imagery you will have
>> committed an error,
>> and that error may dissuade our most important resource from ever making
>> any further changes- the local mapper.
>>
>> Be very careful!
>>
>
> I second this last line !
>
> I am observing an influx of mixed-quality remote edits from Amazon
> Logistics in my area.
> I expect this Facebook operation to produce much more changes or potential
> changes (=suspected errors).
> What we need for both cases and similar ones in the future is a way of
> being able to identify such changes, which by their nature will be
> armchair-mapping efforts.
> I do not have a specific proposal, but I would appreciate a tool that
> helps me, as local mapper,  find these edits, and, more importantly we need
> a new approach to organise digesting these massive distributed
> armchair-mapping interventions on OSM data.
> I don't realistically think that banning these activities is good for OSM.
> Not dealing in a systematic way with it at all presents, however, a big
> risk of deteriorating the map for two reasons:
> (1) bad armchair edits by Amazon and Facebook (and others)
> (2) demotivating non-armchair mappers
>
> I repeat I do not have a proposal how to handle that. My main concern is
> that the required work for locally checking even only those edits that need
> checking (I am assuming that at least FB has good algorithms to sort out
> the dead-certain corrections beforehand. I am more sceptical with Amazon's
> changing local access tagging to, essentially, "yes" everywhere they have
> delivered something by delivery van. I came across a good number of them,
> and in most cases they were at least dubious)
>
> Volker
> (Padova, Italy)
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk
>


-- 
Florimond Berthoux
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Announcing Daylight Map Distribution

[Christoph Hormann]

On Tuesday 10 March 2020, Michal Migurski wrote:
>
>   • A PBF planet file composed of 100% OSM data, released under the
> terms of the Open Database License. • Only those edits which have
> been validated to contain no malicious vandalism or unintentional
> errors so we can show them in our display maps

Could someone maybe do an analysis of the diff regarding numbers of 
features removed/changed/added for various types of objects?

Regarding

>   • A PBF planet file composed of 100% OSM data

that is probably an incorrect characterization because any time you 
modify OSM data without uploading the results to OSM what you get is no 
more 100% OSM data.

Thinking this further - the real question is if there is other data used 
in production of the maps using this that constitutes a derivative 
database according to the ODbL or in other words:  Does Facebook claim 
that this is the only derivative database they are using?

-- 
Christoph Hormann
http://www.imagico.de/

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Announcing Daylight Map Distribution

[Volker Schmidt]

Fixing stuff in OSM purely from imagery may not be good.
>
> A local mapper who sees something may add it before any satellite imagery
> has it.
>
> If you then 'fix' this back to the satellite imagery you will have
> committed an error,
> and that error may dissuade our most important resource from ever making
> any further changes- the local mapper.
>
> Be very careful!
>

I second this last line !

I am observing an influx of mixed-quality remote edits from Amazon
Logistics in my area.
I expect this Facebook operation to produce much more changes or potential
changes (=suspected errors).
What we need for both cases and similar ones in the future is a way of
being able to identify such changes, which by their nature will be
armchair-mapping efforts.
I do not have a specific proposal, but I would appreciate a tool that helps
me, as local mapper,  find these edits, and, more importantly we need a new
approach to organise digesting these massive distributed armchair-mapping
interventions on OSM data.
I don't realistically think that banning these activities is good for OSM.
Not dealing in a systematic way with it at all presents, however, a big
risk of deteriorating the map for two reasons:
(1) bad armchair edits by Amazon and Facebook (and others)
(2) demotivating non-armchair mappers

I repeat I do not have a proposal how to handle that. My main concern is
that the required work for locally checking even only those edits that need
checking (I am assuming that at least FB has good algorithms to sort out
the dead-certain corrections beforehand. I am more sceptical with Amazon's
changing local access tagging to, essentially, "yes" everywhere they have
delivered something by delivery van. I came across a good number of them,
and in most cases they were at least dubious)

Volker
(Padova, Italy)
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Announcing Daylight Map Distribution

[Warin]

Fixing stuff in OSM purely from imagery may not be good.

A local mapper who sees something may add it before any satellite imagery has 
it.

If you then 'fix' this back to the satellite imagery you will have committed an 
error,
and that error may dissuade our most important resource from ever making any 
further changes- the local mapper.

Be very careful!


On 10/3/20 1:11 pm, Michal Migurski wrote:

Hi Joseph, thanks for your questions!

We are unsure yet about the public release schedule we want to commit to, 
because a lot of it depends on community feedback which we’re getting now. I 
hope we will do this again!

We do correct OSM upstream for the errors that we find. When our human review 
process catches a map error, we do two things:

1) Hold it back from release to our display maps
2) Fix the error upstream in OSM.org

Ideally, the fix subsequently passes our process on the next round.

-mike.


On Mar 9, 2020, at 5:47 PM, Joseph Eisenberg  wrote:

Interesting! That sounds like a huge amount of work, if you are
validating every changeset in the globe, though so far there is no
schedule to release this frequently.

Thank you for following the Open Database license by releasing this to
the public.

Is facebook planning to create new planet files like this on a
frequent basis, for internal use? If so, those should be released to
the public on a regular schedule, if I understand the ODbL correctly.

Can I confirm that facebook is editing Openstreetmap data to fix the
errors which are found? Or is this only happening in some cases?

The diary post gave links that explain how this was made (both from
September 2019):
- https://engineering.fb.com/ml-applications/mars/ "MaRS: How
Facebook keeps maps current and accurate"

"We calculate the absolute diff between the two OSM versions (from
option 1 above).
We then split this diff into a set of LoChas that can be individually
applied (from option 2)."
...
"At a high level, a map change is vetted by a reviewer,"

- 
https://2019.stateofthemap.us/program/sun/keepin-it-fresh-and-good-continuous-ingestion-of-osm-data-at-facebook.html
"“Keepin’ it fresh (and good)!” - Continuous Ingestion of OSM Data at
Facebook":

This is a video presentation, so I haven't seen it all, but the
abstract says: "we have created an automated ingestion and integrity
framework for OSM data that allows us to selectively update parts of
the map instead of doing a full snapshot change all at once.

"Decomposing a large set of changes in this way gives us the
flexibility to rapidly ingest our own additions to the map, focus on
geographical areas of importance to downstream products, and allows us
to quickly apply hotfixes whenever egregious problems do arise.

"With millions of tiny changes happening every week, we have created a
system that is built on per-feature approval and preprocessing, that
allows us to ingest changes at scale, while creating rules to
automatically process logical changesets and enforce integrity
constraints (e.g. anti-vandalism, anti-profanity etc.).

"Due to the contextual nature of some of the changes in OpenStreetMap,
the system combines Human Approval, necessary for highly visible
features such as names of large administrative areas, with Automated
AI/ML-based approval: for example, using computer vision techniques to
reconcile newly created features against satellite imagery ground
truth, or applying NLP techniques to determine whether other
user-visible string changes are sensible and valid. These components
are combined to create a continuous ingest-validate-deploy cycle for
OSM map data."

Lot's of buzz words there! But it sounds like it is a combination of
computer algorithms and human checking for vandalism and errors.

- Joseph Eisenberg

On 3/10/20, Michal Migurski  wrote:

Hi everyone,

I’m writing to let you know about a new OpenStreetMap project Facebook just
released. It’s called Daylight Map Distribution. Daylight is a complete,
downloadable preview of OpenStreetMap data we plan to start using in a
number of our public maps:

https://www.openstreetmap.org/user/migurski/diary/392416

Facebook uses maps to let our users find friends, businesses, groups and
more. OpenStreetMap (OSM) has a substantial global footprint of map data
built and maintained by a dedicated community of global mappers and it’s a
natural choice for us. Every day, OSM receives millions of contributions
from the community. Some of these contributions may have intentional and
unintentional edits that are incompatible with our needs. Our mapping teams
work to scrub these contributions for consistency and quality.

What’s Included in the Daylight Map Distribution:

• A PBF planet file composed of 100% OSM data, released under the terms 
of
the Open Database License.
• Only those edits which have been validated to contain no malicious
vandalism or unintentional errors so we can show them in our display maps

This is just an initial first release, and we’re 

Re: [OSM-talk] Announcing Daylight Map Distribution

[Michal Migurski]

Hi Joseph, thanks for your questions!

We are unsure yet about the public release schedule we want to commit to, 
because a lot of it depends on community feedback which we’re getting now. I 
hope we will do this again!

We do correct OSM upstream for the errors that we find. When our human review 
process catches a map error, we do two things:

1) Hold it back from release to our display maps 
2) Fix the error upstream in OSM.org

Ideally, the fix subsequently passes our process on the next round.

-mike.

> On Mar 9, 2020, at 5:47 PM, Joseph Eisenberg  
> wrote:
> 
> Interesting! That sounds like a huge amount of work, if you are
> validating every changeset in the globe, though so far there is no
> schedule to release this frequently.
> 
> Thank you for following the Open Database license by releasing this to
> the public.
> 
> Is facebook planning to create new planet files like this on a
> frequent basis, for internal use? If so, those should be released to
> the public on a regular schedule, if I understand the ODbL correctly.
> 
> Can I confirm that facebook is editing Openstreetmap data to fix the
> errors which are found? Or is this only happening in some cases?
> 
> The diary post gave links that explain how this was made (both from
> September 2019):
> - https://engineering.fb.com/ml-applications/mars/ "MaRS: How
> Facebook keeps maps current and accurate"
> 
> "We calculate the absolute diff between the two OSM versions (from
> option 1 above).
> We then split this diff into a set of LoChas that can be individually
> applied (from option 2)."
> ...
> "At a high level, a map change is vetted by a reviewer,"
> 
> - 
> https://2019.stateofthemap.us/program/sun/keepin-it-fresh-and-good-continuous-ingestion-of-osm-data-at-facebook.html
> "“Keepin’ it fresh (and good)!” - Continuous Ingestion of OSM Data at
> Facebook":
> 
> This is a video presentation, so I haven't seen it all, but the
> abstract says: "we have created an automated ingestion and integrity
> framework for OSM data that allows us to selectively update parts of
> the map instead of doing a full snapshot change all at once.
> 
> "Decomposing a large set of changes in this way gives us the
> flexibility to rapidly ingest our own additions to the map, focus on
> geographical areas of importance to downstream products, and allows us
> to quickly apply hotfixes whenever egregious problems do arise.
> 
> "With millions of tiny changes happening every week, we have created a
> system that is built on per-feature approval and preprocessing, that
> allows us to ingest changes at scale, while creating rules to
> automatically process logical changesets and enforce integrity
> constraints (e.g. anti-vandalism, anti-profanity etc.).
> 
> "Due to the contextual nature of some of the changes in OpenStreetMap,
> the system combines Human Approval, necessary for highly visible
> features such as names of large administrative areas, with Automated
> AI/ML-based approval: for example, using computer vision techniques to
> reconcile newly created features against satellite imagery ground
> truth, or applying NLP techniques to determine whether other
> user-visible string changes are sensible and valid. These components
> are combined to create a continuous ingest-validate-deploy cycle for
> OSM map data."
> 
> Lot's of buzz words there! But it sounds like it is a combination of
> computer algorithms and human checking for vandalism and errors.
> 
> - Joseph Eisenberg
> 
> On 3/10/20, Michal Migurski  wrote:
>> Hi everyone,
>> 
>> I’m writing to let you know about a new OpenStreetMap project Facebook just
>> released. It’s called Daylight Map Distribution. Daylight is a complete,
>> downloadable preview of OpenStreetMap data we plan to start using in a
>> number of our public maps:
>> 
>>  https://www.openstreetmap.org/user/migurski/diary/392416
>> 
>> Facebook uses maps to let our users find friends, businesses, groups and
>> more. OpenStreetMap (OSM) has a substantial global footprint of map data
>> built and maintained by a dedicated community of global mappers and it’s a
>> natural choice for us. Every day, OSM receives millions of contributions
>> from the community. Some of these contributions may have intentional and
>> unintentional edits that are incompatible with our needs. Our mapping teams
>> work to scrub these contributions for consistency and quality.
>> 
>> What’s Included in the Daylight Map Distribution:
>> 
>>  • A PBF planet file composed of 100% OSM data, released under the terms 
>> of
>> the Open Database License.
>>  • Only those edits which have been validated to contain no malicious
>> vandalism or unintentional errors so we can show them in our display maps
>> 
>> This is just an initial first release, and we’re looking for feedback from
>> the community to decide what would be useful to release in the future and
>> how frequently. I’d be interested to hear any response you might have about
>> it!
>> 
>> -mike.
>> 

Re: [OSM-talk] Announcing Daylight Map Distribution

[Joseph Eisenberg]

Interesting! That sounds like a huge amount of work, if you are
validating every changeset in the globe, though so far there is no
schedule to release this frequently.

Thank you for following the Open Database license by releasing this to
the public.

Is facebook planning to create new planet files like this on a
frequent basis, for internal use? If so, those should be released to
the public on a regular schedule, if I understand the ODbL correctly.

Can I confirm that facebook is editing Openstreetmap data to fix the
errors which are found? Or is this only happening in some cases?

The diary post gave links that explain how this was made (both from
September 2019):
 - https://engineering.fb.com/ml-applications/mars/ "MaRS: How
Facebook keeps maps current and accurate"

"We calculate the absolute diff between the two OSM versions (from
option 1 above).
We then split this diff into a set of LoChas that can be individually
applied (from option 2)."
...
"At a high level, a map change is vetted by a reviewer,"

 - 
https://2019.stateofthemap.us/program/sun/keepin-it-fresh-and-good-continuous-ingestion-of-osm-data-at-facebook.html
"“Keepin’ it fresh (and good)!” - Continuous Ingestion of OSM Data at
Facebook":

This is a video presentation, so I haven't seen it all, but the
abstract says: "we have created an automated ingestion and integrity
framework for OSM data that allows us to selectively update parts of
the map instead of doing a full snapshot change all at once.

"Decomposing a large set of changes in this way gives us the
flexibility to rapidly ingest our own additions to the map, focus on
geographical areas of importance to downstream products, and allows us
to quickly apply hotfixes whenever egregious problems do arise.

"With millions of tiny changes happening every week, we have created a
system that is built on per-feature approval and preprocessing, that
allows us to ingest changes at scale, while creating rules to
automatically process logical changesets and enforce integrity
constraints (e.g. anti-vandalism, anti-profanity etc.).

"Due to the contextual nature of some of the changes in OpenStreetMap,
the system combines Human Approval, necessary for highly visible
features such as names of large administrative areas, with Automated
AI/ML-based approval: for example, using computer vision techniques to
reconcile newly created features against satellite imagery ground
truth, or applying NLP techniques to determine whether other
user-visible string changes are sensible and valid. These components
are combined to create a continuous ingest-validate-deploy cycle for
OSM map data."

Lot's of buzz words there! But it sounds like it is a combination of
computer algorithms and human checking for vandalism and errors.

- Joseph Eisenberg

On 3/10/20, Michal Migurski  wrote:
> Hi everyone,
>
> I’m writing to let you know about a new OpenStreetMap project Facebook just
> released. It’s called Daylight Map Distribution. Daylight is a complete,
> downloadable preview of OpenStreetMap data we plan to start using in a
> number of our public maps:
>
>   https://www.openstreetmap.org/user/migurski/diary/392416
>
> Facebook uses maps to let our users find friends, businesses, groups and
> more. OpenStreetMap (OSM) has a substantial global footprint of map data
> built and maintained by a dedicated community of global mappers and it’s a
> natural choice for us. Every day, OSM receives millions of contributions
> from the community. Some of these contributions may have intentional and
> unintentional edits that are incompatible with our needs. Our mapping teams
> work to scrub these contributions for consistency and quality.
>
> What’s Included in the Daylight Map Distribution:
>
>   • A PBF planet file composed of 100% OSM data, released under the terms 
> of
> the Open Database License.
>   • Only those edits which have been validated to contain no malicious
> vandalism or unintentional errors so we can show them in our display maps
>
> This is just an initial first release, and we’re looking for feedback from
> the community to decide what would be useful to release in the future and
> how frequently. I’d be interested to hear any response you might have about
> it!
>
> -mike.
>
> 
> michal migurski- contact info and pgp key:
> sf/cahttp://mike.teczno.com/contact.html
>
>
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk
>

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

[OSM-talk] Announcing Daylight Map Distribution

[Michal Migurski]

Hi everyone,

I’m writing to let you know about a new OpenStreetMap project Facebook just 
released. It’s called Daylight Map Distribution. Daylight is a complete, 
downloadable preview of OpenStreetMap data we plan to start using in a number 
of our public maps:

https://www.openstreetmap.org/user/migurski/diary/392416

Facebook uses maps to let our users find friends, businesses, groups and more. 
OpenStreetMap (OSM) has a substantial global footprint of map data built and 
maintained by a dedicated community of global mappers and it’s a natural choice 
for us. Every day, OSM receives millions of contributions from the community. 
Some of these contributions may have intentional and unintentional edits that 
are incompatible with our needs. Our mapping teams work to scrub these 
contributions for consistency and quality. 

What’s Included in the Daylight Map Distribution:

• A PBF planet file composed of 100% OSM data, released under the terms 
of the Open Database License.
• Only those edits which have been validated to contain no malicious 
vandalism or unintentional errors so we can show them in our display maps

This is just an initial first release, and we’re looking for feedback from the 
community to decide what would be useful to release in the future and how 
frequently. I’d be interested to hear any response you might have about it!

-mike.


michal migurski- contact info and pgp key:
sf/cahttp://mike.teczno.com/contact.html


___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk