Re: [Talk-us] [OSM-talk] Redacting 75, 000 street names contributed by user chdr
On Thu, Oct 12, 2017 at 3:17 PM, Kevin Kenny wrote: > With that in hand, I can probably finish up New Jersey this week. Noo Joisey is done. ___ Talk-us mailing list Talk-us@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] [OSM-talk] Redacting 75, 000 street names contributed by user chdr
On Sat, Oct 7, 2017 at 7:47 PM, Frederik Ramm wrote:
> http://www.remote.org/frederik/tmp/chdr.details
>
> A new list (CSV file) with way id, coordinates, and country/state/county
> information. I've eliminated all objects that have been reported to be
> ok, and plan to remove or change the names on these remaining ones. (To
> avoid misunderstandings: There's a column in the file that says what I
> plan to do, either "change to XYZ" or "delete", but that does NOT mean
> "delete the object", just "delete the name tag"!)
Thanks for taking care of this. Could I make a suggestion for future work
of this kind: add a note:redaction (or some similar key) with value
identifying the particular redaction that the object belongs to? At Max's
suggestion, I was doing Overpass queries with the set of way ID's
looking for ones that were still last modified by 'woodpeck_repair',
but I realize that if the ways had an identifiable tag, I could easily
hack up a reusable script to say, "give me the next object from
this redaction" - and remove the tag when the object is re-uploaded.
Simply having a tag like "note:redaction=chdr_20171008" on
the redacted way would do it.
I may be too much of an old woman here, worrying about identifying
objects from the wrong repair. It appears that for this particular
incident,
way(newer:"2017-10-07T00:00:00Z")(user:"woodpeck_repair")({{bbox}});
is a perfectly workable Overpass query for "tell me the work to do
in this particular bbox".
With that in hand, I can probably finish up New Jersey this week.
___
Talk-us mailing list
Talk-us@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] [OSM-talk] Redacting 75, 000 street names contributed by user chdr
Thanks again to Max Erickson for setting me up with that Overpass query. New York State is done. It was enough of a chore that I think I'm going to decree that the rest of the world is Not My Problem. ___ Talk-us mailing list Talk-us@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] [OSM-talk] Redacting 75, 000 street names contributed by user chdr
On Tue, Oct 10, 2017 at 7:58 AM, Max Erickson wrote: > I reviewed about 40 ways in New York. Here's an Overpass script for > finding the ways that have not been changed since the redaction: > > https://gist.github.com/maxerickson/e02651cce99af983949b91f8d471fb23 > > The ways are clustered quite a lot. Thanks, Max. The script was just what I needed. I wrapped a little bit of Tcl code around it so that it would command JOSM to load the ways, one at a time, and was able to blitz through about three hundred of them last night. I'm afraid that I didn't remember to hit 'Upload' often enough, so some of the change sets are not as well clustered geographically as they ought to be. (Sorry about that, everyone!) The clustering of the ways goes back to the TIGER import, which, if I recall correctly, was done by census tract. It looked in the aerials as if the roads that needed to remain nameless were all service ways, and I tagged them accordingly. There were also some motorway_link and similar things. I left tiger:reviewed=no on most of them, because even though I usually verified existence on aerials, I did NOT attempt to verify any other attributes beside the name. I left a note or two on the ways where the 'name' in OSM already matched TIGER 2017 and I had no third source to check. ___ Talk-us mailing list Talk-us@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] [OSM-talk] Redacting 75, 000 street names contributed by user chdr
Just a heads up for everybody- I'll handle the redactions in the Dallas-Fort Worth area of Texas (Collin, Dallas, Denton, Ellis, Tarrant, Johnson, and Wise Counties) Thanks, Andrew Matheny On Tue, Oct 10, 2017 at 6:58 AM, Max Erickson wrote: > I reviewed about 40 ways in New York. Here's an Overpass script for > finding the ways that have not been changed since the redaction: > > https://gist.github.com/maxerickson/e02651cce99af983949b91f8d471fb23 > > The ways are clustered quite a lot. > > > Max > > ___ > Talk-us mailing list > Talk-us@openstreetmap.org > https://lists.openstreetmap.org/listinfo/talk-us > ___ Talk-us mailing list Talk-us@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] [OSM-talk] Redacting 75, 000 street names contributed by user chdr
I reviewed about 40 ways in New York. Here's an Overpass script for finding the ways that have not been changed since the redaction: https://gist.github.com/maxerickson/e02651cce99af983949b91f8d471fb23 The ways are clustered quite a lot. Max ___ Talk-us mailing list Talk-us@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] [OSM-talk] Redacting 75, 000 street names contributed by user chdr
On Sat, Oct 7, 2017 at 7:47 PM, Frederik Ramm wrote: > http://www.remote.org/frederik/tmp/chdr.details > > A new list (CSV file) with way id, coordinates, and country/state/county > information. I've eliminated all objects that have been reported to be > ok, and plan to remove or change the names on these remaining ones. (To > avoid misunderstandings: There's a column in the file that says what I > plan to do, either "change to XYZ" or "delete", but that does NOT mean > "delete the object", just "delete the name tag"!) Thanks, Frederik, this makes things easier. I'm getting started on the New York morass. This evening, did the ways with ID's 5610940-5714662 New York has a big enough pile of ways that this probably could use an OSM task. I don't think I'm going to have time to finish in very short order. I've been sweeping up some TIGER turds as I go. It's not just renaming. For what it's worth http://maps.nyc.gov/doitt/nycitymap/ is a clean reference for the New York City boroughs. The underlying data are all in NYC Planimetry, which is effectively in the public domain as per Local Law 11 of 2012, section 23-502 https://www1.nyc.gov/site/doitt/initiatives/open-data-law.page I used that to sort out some questionable TIGER 2017 data. Kevin ___ Talk-us mailing list Talk-us@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] [OSM-talk] Redacting 75, 000 street names contributed by user chdr
FYI, I reviewed the ways with redacted names in California (all in San Diego County) and where possible set the names per Tiger 2017 data. In most cases the names set by chdr matched the Tiger names but there were some exceptions. There are a roads that did not have names showing in the Tiger 2017 overlay image layer in JOSM. Unfortunately there was no Mapillary imagery for guidance on those so they were left unnamed. Tod > On Oct 7, 2017, at 4:47 PM, Frederik Ramm wrote: > > Hi, > > On 27.09.2017 21:49, Martijn van Exel wrote: >> That is helpful. Let us know when you have re-executed the analysis and >> posted the results. > > http://www.remote.org/frederik/tmp/chdr.details > > A new list (CSV file) with way id, coordinates, and country/state/county > information. I've eliminated all objects that have been reported to be > ok, and plan to remove or change the names on these remaining ones. (To > avoid misunderstandings: There's a column in the file that says what I > plan to do, either "change to XYZ" or "delete", but that does NOT mean > "delete the object", just "delete the name tag"!) > > I'll start doing that in ~ 20 hours from now. > > I'll then redact the versions that carried the "bad" name. > > The redaction will also affect a few historic objects that *used* to > have a "bad" name and where the name has meanwhile been changed again, > or where the object has been deleted; these redactions will be of little > consequence. > > Bye > Frederik > > -- > Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" > > ___ > Talk-us mailing list > Talk-us@openstreetmap.org > https://lists.openstreetmap.org/listinfo/talk-us ___ Talk-us mailing list Talk-us@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] [OSM-talk] Redacting 75, 000 street names contributed by user chdr
Hi, On 27.09.2017 21:49, Martijn van Exel wrote: > That is helpful. Let us know when you have re-executed the analysis and > posted the results. http://www.remote.org/frederik/tmp/chdr.details A new list (CSV file) with way id, coordinates, and country/state/county information. I've eliminated all objects that have been reported to be ok, and plan to remove or change the names on these remaining ones. (To avoid misunderstandings: There's a column in the file that says what I plan to do, either "change to XYZ" or "delete", but that does NOT mean "delete the object", just "delete the name tag"!) I'll start doing that in ~ 20 hours from now. I'll then redact the versions that carried the "bad" name. The redaction will also affect a few historic objects that *used* to have a "bad" name and where the name has meanwhile been changed again, or where the object has been deleted; these redactions will be of little consequence. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ Talk-us mailing list Talk-us@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] [OSM-talk] Redacting 75, 000 street names contributed by user chdr
Hi,
On 08/27/2017 08:51 PM, Mikel Maron wrote:
> Also, Frederik, I think your script picked up false positives. Spot
> checked in DC, and these are expansions of both the street and the
> quadrant ("St NW" -> "Street Northwest"(. Can we fix the script and
> regen the list?
I have modified my "name equality rule" to consider "N" equal to "North"
etc., also it will ignore case, whitespace, and as before the usual
street type expansions (St->Street etc).
This brings the number of problematic objects down by around 5500, and
practically all of them are in the US. However, I noticed that I forgot
to account for "Saint"->"St", and will re-do the numbers yet again
before publishing an updated list.
I think the best course of action would be:
1. Wait a while, until various communities (potentially pointed to this
conversation via the widely-read weekly new roundup) have had the time
to check whether my automated assessment of which names count as
"contributed" by chdr is correct. Mikel has found the issue above and I
fixed it; it is quite possible that there are others.
2. Run the redaction, and remove all names contributed by chdr. At
present it looks as if less than 10% of these objects had a different
name before; more than 90% had not name at all. Perhaps it is indeed
best to remove the name in these cases as well instead of reverting to
the old name.
3. Load the IDs of all affected objects in a MapRoulette task or
similar, so people can check the names by survey, or from different
sources. (I assume that, as Simon pointed out, open data will not be
available for all countries affected. I fear that, with MapRoulette
geared towards armchair mapping, there might be a temptation for people
to yet again fill in the blanks from inadmissible sources. Maybe we
should limit the use of MapRoulette to countries where we know that open
sources exist, and use fixme tags or notes for other countries?)
I think that would be cleaner than verifying the names ahead of time.
Also it would create an audit trail - from the object history, you could
then see that the name was removed for copyright reasons, and you could
then see that user XYZ has added a new name. If it should later turn out
that this name was also copied from an indadmissible source, we know
that user XYZ is at fault, whereas people creating lists with
independently verified names is not something that would give us such a
recording.
I must apologize for not having given a time frame in my initial email;
there's absolutely no reason to panic. This matter has been sitting idle
for years, and a few more weeks won't kill us. We can sort this out
calmly and then do the right thing.
Bye
Frederik
--
Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33"
___
Talk-us mailing list
Talk-us@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] [OSM-talk] Redacting 75, 000 street names contributed by user chdr
On Sun, Aug 27, 2017 at 11:51 AM, Mikel Maron wrote: > > we can find a good workflow for that. I wasn't expecting the community > to start working on this pre-redaction but if people prefer that to > fixing issues later... > > Absolutely, let's do this! > > On Sun, Aug 27, 2017 at 10:55 AM, Martijn van Exel > wrote: > > Happy to help. All we'd need for MapRoulette is a list of locations and a > proper description of the work we'd expect people to do. Anyone can create > the challenge but I'd be happy to do it. > Martijn > > > Change set # 51502902 Here's what I did and the plan: * I selected out Arizona USA. * I used the magic of regular expressions to change the way id in the file Frederik provided to a josm remote control command. * The remote control command allowed me to load all the data into one file. I changed the name tag to chdr_USA_AZ_name_fixup_required. The new name creates a great search target. * I made no other changes before uploading the changeset. * I would expect that the DWG would redact the potential name issues from all the prior versions. * Next a MapRoulette challenge can be constructed with the chdr_USA_AZ_name_fixup_required search target. The challenge will be localized to Arizona because of the name change. * The Tiger 2015 layer will be a good source to use name correction in the challenge. Others without the Tiger name source will not be so lucky. The questions and observations: * chdr is the last modifier of 2980 out of 5696 ways, 52%. These ways would appear like the first and last introduction of the potential damage. * The highest way version on one example way was 24. Does this mean that the redaction would have to be performed in all the 24 revisions? * I came in at 1685 out of 5969 ways, 30%. Even though I can rationalize and observe that chdr's street name matches the Tiger 2015 layer, I cannot prove or disprove this. I'd rather make a proactive resolution to the issue than wait for additional edits on these ways. * It took a long time for the name change to complete in JOSM once the last remote control command was issued. DWG you are up. Please redact the name issue in the prior versions of the ways in Change set # 51502902. Once the DWG is finished, it would be great if Martijn could assist me develop a MapRoulette challenge. Regards, Greg ___ Talk-us mailing list Talk-us@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] [OSM-talk] Redacting 75, 000 street names contributed by user chdr
On Sun, Aug 27, 2017 at 12:01 PM, Nicolás Alvarez wrote: > I don't understand what people mean with 'verifying' objects. We're > not trying to find factually-incorrect data. The data is legally > tainted. It's questionable whether looking at the current names > imported from GMaps, comparing to another source, seeing they match > and marking them as "verified" will legally change anything. > And it's > impossible to know if people are really verifying anything or just > blindly marking them as verified. > > Nicolás is there any chance you can refrain from slimming the community like that? > I think the only clean way to solve this is to redact and then re-map > from legal sources. > If you are in another country than the US and Canada, then you may not have a second legal source and you would make this statement. In my case, I have had a series of tiger name layers work from over the years.. So what Fredrick wants to do based on his list is to wipe out my work with his purposed blind revert. As an example, here is way https://www.openstreetmap.org/way/103141172 . I cleaned up the geometry two years ago. I added access tags and what not. I make it a habit of removing all tiger tags when I am finished cleaning up and verifying names with the tiger layer. Way 103141172 is on Fredrick's list. If he performs is revert, then I am going to have to go back and add the name back that I have already checked on. Hence, I do not agree with either of your statements. ___ Talk-us mailing list Talk-us@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] [OSM-talk] Redacting 75, 000 street names contributed by user chdr
> we can find a good workflow for that. I wasn't expecting the community to
>start working on this pre-redaction but if people prefer that to fixing issues
>later...
Absolutely, let's do this!
Also, Frederik, I think your script picked up false positives. Spot checked in
DC, and these are expansions of both the street and the quadrant ("St NW" ->
"Street Northwest"(. Can we fix the script and regen the list?
http://www.openstreetmap.org/way/109419946/historyhttp://www.openstreetmap.org/way/109431926/history
http://www.openstreetmap.org/way/109431927/history
-Mikel
* Mikel Maron * +14152835207 @mikel s:mikelmaron
On Sunday, August 27, 2017 2:45 PM, Greg Morgan
wrote:
On Sun, Aug 27, 2017 at 10:55 AM, Martijn van Exel
wrote:
Happy to help. All we'd need for MapRoulette is a list of locations and a
proper description of the work we'd expect people to do. Anyone can create the
challenge but I'd be happy to do it.Martijn
Martijn,
I'd would be great if you can break this down to an area. For example, I have
a list of Arizona streets. I'd prefer to work on this as an Arizona challenge
verses one big chdr challenge.
Please Advise,Greg
___
talk mailing list
t...@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk
___
Talk-us mailing list
Talk-us@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-us
