Re: [OSM-dev] Useful post-bot visualisation?
Hi, On 07/20/2012 10:51 PM, Frederik Ramm wrote: I'd like to replace it with something that displays where the bot has actually edited something. Now released, and announced on talk list. production URL is http://tools.geofabrik.de/osmi/?view=redactionbotlon=7.84268lat=48.78466zoom=5 * The overview layer is wrong, it actually shows the old WTFE overview and not the bot-edited-or-deleted overview, so you have to zoom in to see details. Now correct; * The various buttons on the right don't necessarily make sense (why call Potlatch for a deleted node etc.) Should be correct; * Updates aren't yet automated; the plan is to be relatively quick with dropping stuff from the map as soon as it is edited by someone. Currently updates ~ once in four hours, faster turnarounds on the horizon. Stuff edited by the bot and later by someone else, will stay around in yellow for reference. * It would be great to have a way of clearing the deleted-object markers from the map. Still to do. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Useful post-bot visualisation?
On Mon, Jul 23, 2012 at 12:17 PM, Frederik Ramm frede...@remote.org wrote: Stuff edited by the bot and later by someone else, will stay around in yellow for reference. Yellow is for ways or nodes last modified by the bot, and not touched since, isnt ? The aim is still dropping stuff from the map as soon as it is edited by someone. or is it a special layer for reference ? Pieren ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Useful post-bot visualisation?
On Tue, Jul 24, 2012 at 1:04 AM, Pieren pier...@gmail.com wrote: Yellow is for ways or nodes last modified by the bot, and not touched since, isnt ? The aim is still dropping stuff from the map as soon as it is edited by someone. or is it a special layer for reference Sorry, forget my previous message. I see now the superseded and modified layers although the colours range is not really appropriate. Pieren ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Useful post-bot visualisation?
On 21.07.2012 01:54, Frederik Ramm wrote: 1. The bot used a blacklist of changesets, changesets that were nominally done by agreers but in which they used unacceptable sources (mostly, the so-called copypaste remapping). OSMI did not use these lists, so if someone whitewashed data by copypasting then it would have been shown ok on OSMI but later caught by the bot. i think i can rule that out for my neighborhood, i marked quite a few things as odbl=clean there, and i still can see quite a few green nodes and ways in the license inspector view the missing way 33744987 was created by an agreer in 2009 and was removed by the redaction bot ... i'd expected that in the worst case it would have been reverted to this initial version from 2009 i'm also not aware of anybody else having edited in this special area 2-3 blocks around my home lately ... 2. The bot knows about knock-on effects, where a way is deleted if all, or all but one, of its nodes have to be deleted. This is a straight way between just two nodes, and both of them were created anonymously ... one of them has only been touched by agreers afterwards and the other one was only once touched by a disagreer (towi on version 7 of node 43641809) who didn't change any actual data on them though (same position, no tags changed) This is something where OSMI would have marked the nodes red but treated the way as if it was safe. The nodes were not marked by OSMI in any way, still exist (not touched by the redaction bot), other ways using them are still connected to them (e.g. way 5739722 is starting on node 43664769, has been created anonymously in 2007 and was modified by agreers only afterwards, not touched by the redaction bot at all) So i'm still curious regarding the reasons for the removal of way 33744987, on the other hand it probably isn't too important to spent much more time on this anymore ... i've restored the way already and i'm not aware of any other cases like this. If this were something that happened frequently then it might be worth to look into it though as some of these could rightfully be restored? (probably not automatically anymore though as this might conflict with manual fixes already?) -- hartmut ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Useful post-bot visualisation?
Hi, On 17.07.2012 09:48, Frederik Ramm wrote: I'd like to replace it with something that displays where the bot has actually edited something. First preview version is here: http://tools.geofabrik.de/osmi/debug.html?view=botlon=8.55688lat=48.88608zoom=5 It shows stuff edited by the bot and stuff deleted by the bot. Stuff edited by the bot is shown in its current geometry; stuff deleted by the bot is shown with its old geometry (what else) and when clicked, an excerpt of the deleted object's tags is shown. I assume that this will draw criticism from some (who fear this is an invitation to infringe a decliner's copyright) but I think it is important to be able to get a glimpse of exactly *what* has been deleted so that you can decide whether you want to go there and survey or whether that was an un-important (or long replaced) detail. Things that don't work yet: * The overview layer is wrong, it actually shows the old WTFE overview and not the bot-edited-or-deleted overview, so you have to zoom in to see details. * The various buttons on the right don't necessarily make sense (why call Potlatch for a deleted node etc.) * Updates aren't yet automated; the plan is to be relatively quick with dropping stuff from the map as soon as it is edited by someone. * It would be great to have a way of clearing the deleted-object markers from the map. Once these things are fixed, or at least a significant portion, I'll make the view available on the non-debug OSMI, support tile access, and announce more widely. Comments are welcome. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Useful post-bot visualisation?
on 20.07.2012 22:51, Frederik Ramm wrote: Comments are welcome. looks very (!) promising. Thanks for the work! Looking very forward for the live/production version. One hint: maybe the tools should not remove objects touched by someone from its display: I checked Helgoland aftzer the redaction bot. And: e.g. a few streets had been only left as way with no tags. So I reconnected them at the crossings to other ways according to aerials. And I marked them with highway=road as temporary sollution (don't know the local situation). And I added a fixme=yes tag (together with a note=). = maybe the checker should leave the objects with a fixme tag displayed but in another color. Just as a suggestion (and if it is not too complex to realize). In that special case, other persons could argue: * higway=road is found by other checkers = use them * fixme=yes is found by other checkers = use them Both arguments are valid. But the fixme in the end is caused by the bot = so it would match to that display... Best regards, Michael. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Useful post-bot visualisation?
On 07/20/2012 10:51 PM, Frederik Ramm wrote: Hi, On 17.07.2012 09:48, Frederik Ramm wrote: I'd like to replace it with something that displays where the bot has actually edited something. First preview version is here: http://tools.geofabrik.de/osmi/debug.html?view=botlon=8.55688lat=48.88608zoom=5 ok, this now somewhat helps to explain why a street in my neighborhood vanished even though both the old inspector and the cleanmap showed it as OK, but it still leaves me with a few open questions: Bot Inspector shows: http://tools.geofabrik.de/osmi/debug.html?view=botlon=8.51666lat=52.02539zoom=17 There are two things that i don't understand: 1) The way itself was created by an agreer, five further edits seem to have been redacted. http://osm.mapki.com/history/way.php?id=33744987 I'm pretty sure though that at least the max_speed=30 was added by me just recently? 2) Both endpoints of the way were added by an anonymous nodecision user: http://osm.mapki.com/history/node.php?id=43664769 http://osm.mapki.com/history/node.php?id=43641809 I understand that this caused the way to disappear even though the way itself was OKish, i do not understand why both the cleanmap and the OSMI License Change view showed the way as ok, and OSMI did not complain about the end nodes either? http://tools.geofabrik.de/osmi/?view=wtfelon=8.51738lat=52.02510zoom=17 -- hartmut ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Useful post-bot visualisation?
Am 21.07.2012 00:11, schrieb Michael Kugelmann: on 20.07.2012 22:51, Frederik Ramm wrote: Comments are welcome. looks very (!) promising. Thanks for the work! Looking very forward for the live/production version. One hint: maybe the tools should not remove objects touched by someone from its display: I checked Helgoland aftzer the redaction bot. And: e.g. a few streets had been only left as way with no tags. So I reconnected them at the crossings to other ways according to aerials. And I marked them with highway=road as temporary sollution (don't know the local situation). And I added a fixme=yes tag (together with a note=). = maybe the checker should leave the objects with a fixme tag displayed but in another color. In future please use fixme= directly. With your solution others have to investigate more than one tag to see what you mean by the fixme. Even the tools - at least OSMI and keepright - show the content of the fixme tag directly, but not the content of the note tag. regards Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Useful post-bot visualisation?
Hi, On 21.07.2012 00:15, Hartmut Holzgraefe wrote: ok, this now somewhat helps to explain why a street in my neighborhood vanished even though both the old inspector and the cleanmap showed it as OK, but it still leaves me with a few open questions: I can't say what cleanmap does but the old license view in OSM differed from what the bot actually did in at least two important points: 1. The bot used a blacklist of changesets, changesets that were nominally done by agreers but in which they used unacceptable sources (mostly, the so-called copypaste remapping). OSMI did not use these lists, so if someone whitewashed data by copypasting then it would have been shown ok on OSMI but later caught by the bot. (I assume that not all cases of whitewashing will have been caught, and we will have to use the bot a couple more times in the future to get rid of whitewashed data when it is discovered.) 2. The bot knows about knock-on effects, where a way is deleted if all, or all but one, of its nodes have to be deleted. This is something where OSMI would have marked the nodes red but treated the way as if it was safe. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Useful post-bot visualisation?
On 07/20/2012 10:51 PM, Frederik Ramm wrote: Hi, On 17.07.2012 09:48, Frederik Ramm wrote: I'd like to replace it with something that displays where the bot has actually edited something. First preview version is here: http://tools.geofabrik.de/osmi/debug.html?view=botlon=8.55688lat=48.88608zoom=5 Thank you, It works very well to see where work needs to be done. --- m.v.g., Cartinus ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Useful post-bot visualisation?
Am 2012-07-17 09:48, schrieb Frederik Ramm: However that would not account for stuff deleted by the bot. To highlight that, I would have to access pre-bot information which is also possible. But then again, stuff edited by the bot would vanish from the map once someone else has modified it, whereas stuff that I highlight because it has been deleted by the bot would remain there forever. I think having a static layer showing exactly what the bot deleted would be extremely useful. (Both in terms of nodes/ways deleted, and nodes/ways where tags were deleted.) Stuff last edited by the bot does not seem too useful for me - if the bot deletes a lot of useful information from a node, and then someone adds something small and unrelated, the node would disappear from the list. I think the only proper way would be, as you suggested, some kind of interface where you can manually mark the elements as replaced/fixed, for elements deleted or edited by the bot. Maybe something customized? I don't know how much effort that would require, but it seems rather straightforward (a list of change IDs with associated boolean flags, plus a changelog containing user, timestamp, id and new state). Kind regards, Jan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Useful post-bot visualisation?
Does it only work on an OSM file data source or will it work on an apidb or pgsnapshot source too? The plugin should work with all Osmosis data sources, though I haven't tested that yet. - how to best extract bot changesets only Given a list of changesets I have a newly-announced (http://lists.openstreetmap.org/pipermail/dev/2012-July/025307.html) tool that can generate .osc files for them without having to issue /download requests for each from the API. I also have a local mirror of replication diffs so I could generate these without being limited by network speed. Thanks for pointing that out. I had read your mail, but thought it might not be the right thing to download lots of changesets. But with a little tweaking - e.g. getting the time span from bot logs, changing the filter - and a local diff mirror it might well be. - similar to what Frederik mentioned, the result would be static an not take remapping into account If it were re-run on a more recent extract would it take remapping into account? Basically, the tool is merely showing changes. I'm thinking about adding a filter to the OpenLayers viewer. With that I can think of the following ways how might actually be somewhat helpful for remapping: a) With a post-bot extract and re-runs you won't see the bot changes anymore, but could monitor remappings by filtering all changes where the bot is the user of the old version. b) With a pre-bot extract and using all changes, remapped bot changes of non-deleted objects would be removed (I guess) by simplifying the changefile on re-runs, remaining bot changes could be filtered by user of the new version. Deleted objects will remain but you could see created objects at the same position when showing all changes. With increasing number of changes the map will probably get quite slow over time. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Useful post-bot visualisation?
Hi, I'll be switching off the OSMI license change layer soon because it is of limited use in areas where the license change bot has already been. I'd like to replace it with something that displays where the bot has actually edited something. It would be trivial to simply take a current planet file and look: What object has been last modified by the bot? And draw those. Something like the TIGER edited map. However that would not account for stuff deleted by the bot. To highlight that, I would have to access pre-bot information which is also possible. But then again, stuff edited by the bot would vanish from the map once someone else has modified it, whereas stuff that I highlight because it has been deleted by the bot would remain there forever. Unless one would do it Javascript-based - something like a special OpenStreetBugs version where each object deleted by the bot would be a bug and you could close that once you have verified that no damage is caused. But the OSB platform is really not suitable for large amounts of data. So should we just ignore deleted things and make a stuff last edited by the bot map, or what would you say? Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Useful post-bot visualisation?
Hi, FYI there is already such analyser for France, thanks to osmose tool : http://osmose.openstreetmap.fr/map/?item=7060 Results are still partial (limited to few regions such as Haute-Normandie and Languedoc-Roussillon), as the analyser is running at this very moment. Later on, analysis is performed every 48hrs Stats are available here : http://osmose.openstreetmap.fr/utils/info.py?item=7060 Unfortunately, coverage should be limited to France, Benelux and Swiss Bye 2012/7/17 Frederik Ramm frede...@remote.org So should we just ignore deleted things and make a stuff last edited by the bot map, or what would you say? Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 __**_ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.**org/listinfo/devhttp://lists.openstreetmap.org/listinfo/dev -- ab_fab http://wiki.openstreetmap.org/wiki/User:Ab_fab Il n'y a pas de pas perdus ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Useful post-bot visualisation?
On Tue, Jul 17, 2012 at 9:48 AM, Frederik Ramm frede...@remote.org wrote: So should we just ignore deleted things and make a stuff last edited by the bot map, or what would you say? Thank you Frederik. I called for such worlwide service some days ago. The first priority is the modified elements. We have seen many roads where the geometry is affected, sometimes deeply. Imo, highlighting the deleted things is more a nice to have. Pieren ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Useful post-bot visualisation?
Hi, I'm about to develop - amongst other things - an interactive local change file viewer. It consists of an Osmosis plugin to augment a OSC file (currently filters references to a separate OSM file) and an OpenLayers vector map. This is intended as a desktop tool for monitoring a specific area. So I'm not sure if it will really be useful for visualizing redaction bot changes. Anyway, as a preview I uploaded an example of what I have right now: http://norbertrenner.de/osm/preview/oscviewer.html To be used for analyzing bot changes I see these challenges: - this currently is not a service, but a tool that everyone would need to execute, which is not trivial because of the data needed - needs an extract with data before bot start - how to best extract bot changesets only - similar to what Frederik mentioned, the result would be static an not take remapping into account Limitations are: - no relation support yet - no clustering yet I will work on publishing an early access version soon. Norbert ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Useful post-bot visualisation?
From: ikonor [mailto:iko...@gmx.de] Sent: Tuesday, July 17, 2012 2:31 PM To: dev@openstreetmap.org Subject: Re: [OSM-dev] Useful post-bot visualisation? Hi, I'm about to develop - amongst other things - an interactive local change file viewer. It consists of an Osmosis plugin to augment a OSC file (currently filters references to a separate OSM file) and an OpenLayers vector map. Does it only work on an OSM file data source or will it work on an apidb or pgsnapshot source too? This is intended as a desktop tool for monitoring a specific area. So I'm not sure if it will really be useful for visualizing redaction bot changes. Anyway, as a preview I uploaded an example of what I have right now: http://norbertrenner.de/osm/preview/oscviewer.html Looks good! To be used for analyzing bot changes I see these challenges: - this currently is not a service, but a tool that everyone would need to execute, which is not trivial because of the data needed - needs an extract with data before bot start - how to best extract bot changesets only Given a list of changesets I have a newly-announced (http://lists.openstreetmap.org/pipermail/dev/2012-July/025307.html) tool that can generate .osc files for them without having to issue /download requests for each from the API. I also have a local mirror of replication diffs so I could generate these without being limited by network speed. - similar to what Frederik mentioned, the result would be static an not take remapping into account If it were re-run on a more recent extract would it take remapping into account? ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
