Re: [talk-au] Updating the validator plugin...
--- On Sat, 4/7/09, Rick Peterson ausr...@iinet.net.au wrote: The 'addr:postcode' key is in the presets because it's the recommended way of entering a postcode (should you want to) when applying street numbering along a way. It doesn't really apply when providing a postcode for a suburb or group of suburbs as is the case with the ABS bulk import. What is, if anything, used for place markers or other types of non-address specific markers? None of the keys developed for the ABS bulk import (or other special projects in Australia) need to be included in the presets menu within JOSM. This menu is a feature for the convenience of JOSM editors when entering commonly used keys/key combinations. Not just Australia, the tiger data set has a bunch of other keys that would be well worth ignoring, there is probably a lot of attribution type keys that need to exist for legal purposes, that aren't considered valid but shouldn't cause warnings. This is where the flaw in the current plugin programming is revealed. The coder probably thought he was being quite clever reusing the preset data to validate the current information. The preset menu is trying to be used for more than it was originally designed. The people managing this plugin may tell you that you'll always get errors, because not all keys that are validly used are also in the presets. There is a big diff here between an actual valid warning/error/info alert and just noise, the noise should be removed so the validator plugin can be useful. What they don't realise is that once you start ignoring error warnings, it can quickly become a one way street and the value of the plugin can be diminished to the point where you may as well not run it at all... I mean why would you, if you ignore everything? Well earlier this week it took me a bit to figure out why unnamed streets weren't showing and I'm not exactly a newbie when it comes to computers, so it's a perfect example to illustrate your point. This is a great way to address the issue! The tag checking should have been implemented similar to this in the first place, not saying my code style or method to solve this problem is perfect, but it does come closer than just relying on presets as the be all and end all. Just as external files are used by the validator plugin for: I tried to code within the existing frame work so the code was more likely to make it into the trunk code. 2. A list appending only specially used keys to be used in conjunction with the existing presets method. I've gone with this, the code already exists to validate against the presets so we might as well use that information, as well as building a secondary list to supplement it. If there are any warnings still being triggers, let me know the key name or the specific key pair if the key can't equal anything and I'll include it in the list. Which brings me back to postal_code, should this be ignored or what should we do about this specifically? ___ Talk-au mailing list Talk-au@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-au
Re: [talk-au] Updating the validator plugin...
I just added all the religious denominations to the list, however this isn't technically correct since certain denominations are only valid with certain religions. To do this cleanly I'll need to expand the code + data to cope with this type of multi-type matching. ___ Talk-au mailing list Talk-au@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-au
Re: [talk-au] Updating the validator plugin...
John Smith wrote: What is, if anything, used for place markers or other types of non-address specific markers? I'm not sure that I understand? You're probably aware that the 'place' key is used for countries, states, cities, towns, villages, suburbs etc http://wiki.openstreetmap.org/wiki/Map_Features#Places The 'postal_code' key also makes an appearance nearby in the next table down. What type of non-address thing are you trying to find a key for? If this is about the keys for the ABS bulk import, they seem to be either; 1. Taken from the Map Features page and part of JOSMs presets 2. Taken from the Map Features page and not part of JOSMs presets 3. Not part of the Map Features page and not part of JOSMs presets (made up to suit particular data that was included) I've found another example that may relate to a bulk import using data from the Geographical Names Board of NSW to insert a single Suburb node with the suburb name and some reference keys. In this case, they've been careful to use the 'ref:*=' key which I think is ignored in validator plugin by default. The 'making up' of keys isn't a problem as the need requires, it's more a matter of finding a practical way to remove them from warnings in validator plugin where they are widely used, such as a bulk import that affects all of Australia, or one that places a node in every suburb or town in NSW. Given the response from JOSM developers to comments in your ticket about removing unnamed roundabouts from the validator warnings, I'm starting to think that an Australian config file is going to be the solution, addressing the differences in editing styles and key usage here as opposed to Germany where the program seems to be largely developed. Which brings me back to postal_code, should this be ignored or what should we do about this specifically? My view is that the 'postal_code' key should be ignored, together with any other key used for this bulk import as listed on the specific ABS import wiki page. Cheers, Rick :) ___ Talk-au mailing list Talk-au@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-au
Re: [talk-au] Updating the validator plugin...
--- On Sun, 5/7/09, Rick Peterson ausr...@iinet.net.au wrote: The 'postal_code' key also makes an appearance nearby in the next table down. Yea I noticed that after and since it was listed decided to add it to the ignore cfg file I was building, which has been incorporated into SVN, not sure on time frames of when the next update will be produced or if it is automatically. Given the response from JOSM developers to comments in your ticket about removing unnamed roundabouts from the validator warnings, I'm starting to think that an Australian config file is going to be the solution, addressing the differences in editing styles and key usage here as opposed to Germany where the program seems to be largely developed. Yes, there seems to be a couple of cultural/language/country differences I'm haggling over in the bug list that may need some extra input from others, such as naming junction=roundabouts. http://josm.openstreetmap.de/ticket/2792 I suspect this issue is going to be split, either based on cultural influences or something as different countries have opted for different things. I'm not sure what the best way to handle it, grouping config options by rough geo-cords might be a solution. ___ Talk-au mailing list Talk-au@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-au
Re: [talk-au] Updating the validator plugin...
John Smith wrote: Yes, there seems to be a couple of cultural/language/country differences I'm haggling over in the bug list that may need some extra input from others, such as naming junction=roundabouts. http://josm.openstreetmap.de/ticket/2792 I suspect this issue is going to be split, either based on cultural influences or something as different countries have opted for different things. I'm not sure what the best way to handle it, grouping config options by rough geo-cords might be a solution. I was taken with stoecker's comment in one of your tickets: "We need to find a way to have different checking sets for different countries. Something like a country selector button which sets different defaults in the preferences." I've lost track of which issues have been solved and gone to SVN, and which remain outstanding. I'm guessing the unnamed roundabouts issue remains, however I think you've developed a separate ignore file that's been incorporated - http://trac.openstreetmap.org/browser/applications/editors/josm/plugins/validator/ignoretags.cfg?rev=16319 This results in the ABS keys and lots of other keys/key combinations being ignored in the plugin, even though they are not in the preset list. A different ticket addresses adding certain agreed keys to the preset menu, but is yet to be actioned. Other code changes outside of the ignore file address boundary ways, turning circles, no exits and abandoned railway tags. Perhaps you could clarify the changes that have been incorporated, and those that remain outstanding? Great work John, thanks for your efforts, this will make the plugin so much more useful for Australian mappers by highlighting real errors that need addressing. Cheers, Rick :) ___ Talk-au mailing list Talk-au@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-au
Re: [talk-au] Updating the validator plugin...
--- On Sun, 5/7/09, Rick Peterson ausr...@iinet.net.au wrote: A different ticket addresses adding certain agreed keys to the preset menu, but is yet to be actioned. I was never really worried about this issue, I was misled down this path when all I wanted was a way to have them not show warnings, one patch I coded has been accepted, the second patch extends on this and I don't think it has been incorporated yet. Other code changes outside of the ignore file address boundary ways, turning circles, no exits and abandoned railway tags. This has also been accepted. #2794 Perhaps you could clarify the changes that have been incorporated, and those that remain outstanding? I think you have summed it up already. Great work John, thanks for your efforts, this will make the plugin so much more useful for Australian mappers by highlighting real errors that need addressing. Well since I also added some US tags to the ignore list it won't just be Aussies that benefit but the OSM community as a whole, and others now have a way of getting tags ignored too. As for the junction=roundabout issue, I came up with a solution to it as well, it can still throw a warning, but it will be in it's own tag group and the group can be ignored, this effectively ends up with the same result, it should also keep the JOSM guys in Germany happy and should get the patch accepted. ___ Talk-au mailing list Talk-au@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-au
Re: [talk-au] Updating the validator plugin...
--- On Sun, 5/7/09, Rick Peterson ausr...@iinet.net.au wrote: Perhaps you could clarify the changes that have been incorporated, and those that remain outstanding? A couple more patches have been accepted and are available for update, however when I went to update I got a message telling me I'd need to upgrade to JOSM version 1725 or later. Patch for Not show an error on junction=roundabout when it is unnamed #2792 has also now been accepted, this patch shows warnings as Unnamed Junction which can as a group be ignored. Judging by comments I think this is the best outcome as junctions in Germany at least seem to be named or have ref tags. Patch for Several errors shown for railway=abandoned #2793 was also accepted, this patch won't show errors about railway=abandoned such as missing railway crossings and nodes ending near ways. Patch for Invalid Way end node near other way error #2794 was also accepted, so if a street is a dead end (noexit=yes) or highway=turning_circle or barrier=* and it ends near another way it won't show up as a warning. As for bugs #2803 and #2807, I was misled that this was a prefix issue, when really it was a lack of code in the validator plugin causing warnings for valid tags to show, both code patches I supplied have been applied. The only outstanding bugs I can think of are: #2806, highway=unknown not showing a warning when unnamed. #2830, false positive triggers by certain combinations of tags when natural=coastline and the direction of the arrows basically telling the user they're wrong when they were right. ___ Talk-au mailing list Talk-au@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-au
Re: [talk-au] Updating the validator plugin...
--- On Sat, 4/7/09, Rick Peterson ausr...@iinet.net.au wrote: That's why I suggested suppressing these kind of validation errors when an ABS boundary way is involved for a while until a considered (and practical) method of combining them is sorted out. Yup, I found a similar situation and have fixed this in the latest version of the validator.jar and submitted patches back to JOSM bug tracker. ___ Talk-au mailing list Talk-au@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-au
Re: [talk-au] Updating the validator plugin...
I just built a new version of the validator plugin, I enabled unclassified roads to be shown as a warning if they had no name, I updated this to exclude access=private bug+patch: http://josm.openstreetmap.de/ticket/2806 I also coded a cleaner way to exclude junction=roundabout bug+patch: http://josm.openstreetmap.de/ticket/2792 updated validator.jar: http://sharebee.com/7b79f445 ___ Talk-au mailing list Talk-au@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-au
Re: [talk-au] Updating the validator plugin...
John Smith wrote: submitted a bug about all the false positives about the "Presets do not contain property key" validation errors The validator informational (Other) warning says: "Presets do not contain property key" This led me to http://josm.openstreetmap.de/wiki/TaggingPresets , since it says that it is the 'presets' that do not contain the keys. This webpage is about having presets added via an XML file linked in josm/preferences/map settings. This will add (or replace??) those items to the Presets menu in JOSM's main menu. Because the ABS related keys, and other keys used in Australia do not appear in http://josm.openstreetmap.de/browser/trunk/presets/presets.xml I'm wondering that if we add a separate preset file that includes these things (even though I can't see why it would be necessary to have such specialist keys available in the Preset menu) will those errors disappear in Validator ? Just a thought ... Rick :) ___ Talk-au mailing list Talk-au@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-au
Re: [talk-au] Updating the validator plugin...
--- On Sat, 4/7/09, Rick Peterson ausr...@iinet.net.au wrote: I'm wondering that if we add a separate preset file that includes these things (even though I can't see why it would be necessary to have such specialist keys available in the Preset menu) will those errors disappear in Validator ? It looks like the validator plugin uses the presets to match against, however I'm starting to agree that these shouldn't be presets but they shouldn't be showing up as errors either. I'll see if I can hack something together as a general ignore list, even if they aren't presets. ___ Talk-au mailing list Talk-au@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-au
Re: [talk-au] Updating the validator plugin...
John Smith wrote: It looks like the validator plugin uses the presets to match against, however I'm starting to agree that these shouldn't be presets but they shouldn't be showing up as errors either. I'll see if I can hack something together as a general ignore list, even if they aren't presets. For reference, the keys are listed here http://wiki.openstreetmap.org/wiki/Import/Catalogue/ABS_Data#OSM_Representation I recently linked this page to the WikiProject_Australia page ... the beginnings of an Other Information and Reading section. ___ Talk-au mailing list Talk-au@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-au
Re: [talk-au] Updating the validator plugin...
I ended up splitting JOSM ticket #2803 into 2 tickets, the first is about keys not being ignored that probably shouldn't show up as warnings like au.gov.abs:*. The second ticket, #2807, is about missing presets that are showing up as warnings when they are actually valid and probably should be presets. eg: * Value 'secondary_link' for key 'highway' not in presets * Value 'wood' for key 'surface' not in presets * Value 'mall' for key 'shop' not in presets * Value 'dirt' for key 'surface' not in presets * Value 'waste_disposal' for key 'amenity' not in presets * Key 'construction' not in presets * Key 'loc_name' not in presets * Key 'attribution' not in presets * Key 'admin_level' not in presets * Key 'old_name' not in presets * Key 'usage' not in presets I don't think this is a complete list so please update the respective tickets if there are presets that should be added, or tags that shouldn't generate warnings. ___ Talk-au mailing list Talk-au@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-au
Re: [talk-au] Updating the validator plugin...
I just fixed bug #2794, about a highway=turning_circle not being considered terminal, and so if it's near another way it will throw a warning still. Patch is attached to the bug report. validator.jar: http://sharebee.com/46ba48d0 Now all I need is some kind of dead_end tag to search/filter so they don't show up as unconnected, suggestions? ___ Talk-au mailing list Talk-au@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-au
Re: [talk-au] Updating the validator plugin...
Now all I need is some kind of dead_end tag to search/filter so they don't show up as unconnected, suggestions? Never mind... noexit=yes ___ Talk-au mailing list Talk-au@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-au
Re: [talk-au] Updating the validator plugin...
John Smith wrote: ... I have a new version uploaded that ignores a lot of the ABS tags, there is still some missing preset tags but that's a job for someone else really. http://sharebee.com/856f6789 Good job John, that version of the validator.jar clears all of the ABS key/preset errors that are attributed to the ABS boundary ways. I see only the "postal_code" key remains in the Validator warning, but that is showing against the ABS Postcode relation and not the ways on the screen. The standard presets do use a postcode key, but in the Karlsruhe Schema format of "addr:postcode". The 'way end nodes near other way' boundary ways are involved have gone too .. the unnamed highway=*/junction=roundabout combination is no longer showing as an error you're on fire !! ...my JOSM has never looked so uncluttered! Thank you :) ...now, if I could just do something about those pesky 'highway without a reference' errors when highway=tertiary ... but I'll save that for another day ;) Hope your improvements are incorporated into the standard release so that all Aussie mappers can enjoy your good work and see the real errors that JOSM validator will throw up that need correcting. Thanks again :) ___ Talk-au mailing list Talk-au@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-au
Re: [talk-au] Updating the validator plugin...
--- On Sat, 4/7/09, Rick Peterson ausr...@iinet.net.au wrote: I see only the postal_code key remains in the Validator warning, but that is showing against the ABS Postcode relation and not the ways on the screen. The standard presets do use a postcode key, but in the Karlsruhe Schema format of addr:postcode. Hmmm so it really is an error and they should have entered it in the same way that people were already using when they did a batch import... I'm tempted not to do anything about this to be honest. ...now, if I could just do something about those pesky 'highway without a reference' errors when highway=tertiary ... but I'll save that for another day ;) Just ignore the whole group, they are valid error messages so I don't think I should mess with it, but you are free to ignore them all with very little effort. Hope your improvements are incorporated into the standard release so that all Aussie mappers can enjoy your good work and see the real errors that JOSM validator will throw up that need correcting. Well I've posted patches back on all the changes I've made, you could always me too, but don't just say me too obviously :) a bug to show support for it, the more people that seem to want the same issue fixed the more likely it will be fixed. At least that's how these things work in theory ;) ___ Talk-au mailing list Talk-au@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-au
Re: [talk-au] Updating the validator plugin...
the screen. The standard presets do use a postcode key, but in the Karlsruhe Schema format of addr:postcode. What is the right thing to do here? Also why did the bulk import vary from what everyone else is doing? ___ Talk-au mailing list Talk-au@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-au
Re: [talk-au] Updating the validator plugin...
--- On Sat, 4/7/09, John Smith delta_foxt...@yahoo.com wrote: I just fixed bug #2794, about a highway=turning_circle not being considered terminal, and so if it's near another way it will throw a warning still. Patch is attached to the bug report. I found another corner case where the way ends because of a barrier close to another way, either the way really continues and turns into access=private or ends with some kind of fence. validator.jar: http://sharebee.com/e6551cc7 ___ Talk-au mailing list Talk-au@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-au
Re: [talk-au] Updating the validator plugin...
John Smith wrote: Rick: Karlsruhe Schema format of "addr:postcode". John: What is the right thing to do here? Also why did the bulk import vary from what everyone else is doing? The 'addr:postcode' key is in the presets because it's the recommended way of entering a postcode (should you want to) when applying street numbering along a way. It doesn't really apply when providing a postcode for a suburb or group of suburbs as is the case with the ABS bulk import. I suspect that the key 'postal_code' which does appear in the Map Features page, but as somewhat deprecated, was selected to use in the bulk import because it's a key name that seems appropriate for this particular role. None of the keys developed for the ABS bulk import (or other special projects in Australia) need to be included in the presets menu within JOSM. This menu is a feature for the convenience of JOSM editors when entering commonly used keys/key combinations. Most of these special use keys show as a warnings in the plugin because they aren't in the presets, however they don't belong in the presets. This is where the flaw in the current plugin programming is revealed. The preset menu is trying to be used for more than it was originally designed. The people managing this plugin may tell you that you'll always get errors, because not all keys that are validly used are also in the presets. What they don't realise is that once you start ignoring error warnings, it can quickly become a one way street and the value of the plugin can be diminished to the point where you may as well not run it at all... I mean why would you, if you ignore everything? I think the validator plugin has great value, as it will regularly show genuine errors (sometimes many) that need correcting that I would otherwise not notice. Currently, you just have to be prepared to wade through all the false-positives to get to the real errors, and that is what I was looking to address by mentioning the ABS data import related errors in the first place. John Smith wrote: Being able to have arbitary keys and key pairs ignored that aren't in the preset list is a good idea, I've taken this a step further and given the plugin the ability to download a list of keys and keypairs to ignore or exempt from warnings. This is a great way to address the issue! Just as external files are used by the validator plugin for: spelling - http://svn.openstreetmap.org/applications/utils/planet.osm/java/speller/words.cfg and tagging - http://svn.openstreetmap.org/applications/editors/josm/plugins/validator/tagchecker.cfg so too should there be an external file listing keys that are validly used either internationally, or in specific areas such as the ABS keys used here in Australia. This could be structured in two ways: 1. A list that includes all of the commonly used keys and any special purpose keys, thus returning the preset menu to its original role 2. A list appending only specially used keys to be used in conjunction with the existing presets method. I'm totally behind this initiative to improve the usability and reduction in false positives in validator plugin. Cheers Rick :) ___ Talk-au mailing list Talk-au@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-au
Re: [talk-au] Updating the validator plugin...
John Smith wrote: Forgot to ask, is there any more situations that isn't validly shown up as a warning/error but should be, or errors that shouldn't be that are shown up? The ABS boundary data import throws up a LOT of warnings and information notes in JOSM validator. Some might simply ignore the errors! Others might do some 'interesting' adjustments in an effort to clear the errors. Some of the Validation errors warn that keys are not part of the presets. That seems simple enough, add those keys to the presets. Others are more complicated to address, such as way end node near other way warnings. These types of warnings will continue until a recommended approach is developed to properly incorporate nodes of the boundary ways that coincide with other ways existing in the data at the same location (streets, streams, etc). Making changes to boundary data may also have implications for updating that data in the future. Perhaps, for the moment, the best approach is to remove any boundary related warnings from showing in JOSM validator until a solution that has been properly considered and that can be managed by JOSM users of all skill levels can be developed. (I hate to think how some users might be adjusting the data at the moment to clear the errors they are being presented) Cheers, Rick :) ___ Talk-au mailing list Talk-au@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-au
Re: [talk-au] Updating the validator plugin...
--- On Fri, 3/7/09, Rick Peterson ausr...@iinet.net.au wrote: The ABS boundary data import throws up a LOT of warnings and information notes in JOSM validator. Some might simply ignore the errors! Others might do some 'interesting' adjustments in an effort to clear the errors. I just ignored those warnings, but yea they should be fixed. Others are more complicated to address, such as way end node near other way warnings. Can you give me an example of this, I haven't see that occur, in fact when I marked an ABS boundary as highway=unclassified it didn't show up as a warning about being unnamed. Perhaps, for the moment, the best approach is to remove any boundary related warnings from showing in JOSM validator until a solution that has been properly considered and that can be managed by JOSM users of That isn't really something I think I could hope to get incorporated into the main code base, the changes I'm making at present are corner cases but for something like that I think I'd be over stepping bounds and should be added to their bug tracking system as a feature request. ___ Talk-au mailing list Talk-au@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-au
Re: [talk-au] Updating the validator plugin...
--- On Fri, 3/7/09, John Smith delta_foxt...@yahoo.com wrote: occur, in fact when I marked an ABS boundary as highway=unclassified it didn't show up as a warning about being unnamed. Actually looking through code it seems no highway=unclassified will throw up an unnamed error. ___ Talk-au mailing list Talk-au@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-au
Re: [talk-au] Updating the validator plugin...
I submitted a bug about all the false positives I'm getting, feel free to add to the list: http://josm.openstreetmap.de/ticket/2803 ___ Talk-au mailing list Talk-au@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-au
Re: [talk-au] Updating the validator plugin...
--- On Thu, 2/7/09, John Smith delta_foxt...@yahoo.com wrote: I also want to get it to ignore any railways that are abandoned since there is no track to connect with, and any level crossings have been ripped out etc. Well I managed to solve that issue, now I just have unconnected ways being an issue in this situation: http://www.openstreetmap.org/?lat=-26.15111lon=152.67372zoom=17layers=B000FTF What was railway line has been turned into an unpaved road and the validator plugin is throwing up an error about unconnected way where the barrier=gate is. ___ Talk-au mailing list Talk-au@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-au
Re: [talk-au] Updating the validator plugin...
--- On Thu, 2/7/09, John Smith delta_foxt...@yahoo.com wrote: What was railway line has been turned into an unpaved road and the validator plugin is throwing up an error about unconnected way where the barrier=gate is. I think I have solved that issue as well... new version up http://sharebee.com/b3f5e675 The changes in this version are a little more substantial, patch attached. validator.patch.gz Description: GNU Zip compressed data ___ Talk-au mailing list Talk-au@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-au
