Re: [OSM-dev] Disallowing certain characters in tag keys
On Tue, Oct 19, 2010 at 11:52:09AM +0100, Andy Allan wrote: On Tue, Oct 19, 2010 at 10:25 AM, Jochen Topf joc...@remote.org wrote: On Tue, Oct 19, 2010 at 10:06:15AM +0100, Tom Hughes wrote: On 16/10/10 19:44, Jochen Topf wrote: I am currently fighting some issues where tags with strange characters in them need to be represented in a URL for Taginfo. Lots of other websites probably will have similar issues. Characters like /, ?,, etc. have special meaning in URLs so if they appear in tags I can't have those tags in URLs. Sometimes escaping characters as %XX helps, sometimes not. And those problems are not confined to web pages and URLs only. Special characters that need escaping are often a problem. I really don't understand the problem here - as far as I know all characters can be used in URLs so long as they are properly escaped. If your server software is not coping with that for some reason then I think it's a bug. That might well be a bug. But those bugs creep up all the time, because these things are hard to do and because the specs are not as clear as they should be. I am not saying these things can't be done right, but wouldn't it be nice if we can get rid of that problem instead of everybody writing software for OSM having to make sure all those cases are handled properly? As a test I just created a file called '+?#;%.html' in an apache served directory and then asked Firefox to fetch: http://server/%3c%3e%26%2b%3f%23%3b%25.html and it was retrieved just fine. And now try the same thing again creating a filename with a '/' in it and see whether it works this time. It doesn't, because '/' is special for Unix (and HTTP) and you need to create a directory with the first part of your name and then the second as file. If you would actually want to create one file for every key in the OSM database in your filesystem, you'd have a problem. You example basically proves my point. :-) No, it really doesn't. Obviously I haven't made my point clear enough. I am saying, those special characters don't work like normal characters in many cases. They have special meanings. For instance as directory separators. Or in URLs or HTML code or programming languages. So whenever you do anything where those characters can appear, you have to take special care that your code doesn't break. And programmers are notoriously bad at taking that special care. Let's put it this way - there is a subset[1] of unicode code points that is valid for both keys and values. If you find any characters emitted by OSM that lie outwith that range, then do let us know[3] But we've taken great care to permit all other code points in both keys and values alike, since we've no idea when someone is going to need them. Your example of why we need (and presumably ) is actually great example to undermine your point. Its really a case a weighting the different cases. On the one hand it makes sense to allow everything, because you never know what you will need. But on the other hand it makes sense to restrict what you allow to make handling easier. We have restricted the number of characters in keys and values for instance. There are certainly cases where it would be nice to have more characters, but for practical reasons they are restricted. We have put in a restriction that a key can only appear once on an object. Thats also for practical purposes. I am arguing that there are other things we can do to make working with OSM-tags more convenient, for what I think, no extra cost. Look at what happend with email addresses: You can have nearly every ASCII character in email addresses, spaces and double quotes are allowed for instance, but you have to escape them in the right way. Real mail programs can handle that generally. But most scripts tha people write don't. The result is that in practice you can't use all those characters in email addresses, because they work only half the time. If you send programmers to the RFC and ask them to implement it properly, they can't figure out how to do that and give up. And each one implements his own system, each having his own list of characters that work and that don't work. The end-result is a rather small list of characters that always work and some that work sometimes. (See the details at http://www.remote.org/jochen/mail/info/chars.html ) I argue that if we disallow some characters we can actually expect developers to implement our spec, if we leave the spec open too much, people will ignore the difficult parts. If too many programs don't work with the difficult bits those tags will in practice not be usable anyway, so why not forbid them outright and all have an easier life? Some of these characters need escaping for particular purposes. If you find a unicode character that cannot be URLencoded[2], then do let us know. Or if you find another encoding scenario which can only encode a
[OSM-dev] FYI - job posting
I don't know anything about this, but just in case anyone is interested... http://forum.openstreetmap.org/viewtopic.php?id=9662 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Renderd + expire_list
I found the a previous version of the render_expired does work for me, so it wasn't an issue in my setup as much as code changes. Looking at git hub: This commit broke my intended behavior. http://github.com/openstreetmap/mod_tile/commit/1f3d2ddab6bb3ef5b492bf73089c61e718a22ee5 as far as I can tell there are 3 modes of operation. 1.) -d, --delete-from=ZOOM when expiring tiles of ZOOM or higher, delete them instead of re-rendering (default is off) (Which seems to delete the metal files affected) 2.) -T, --touch-from=ZOOM when expiring tiles of ZOOM or higher, touch them instead of re-rendering (default is off) (Which seems to update the timestamp to be Jan 1 00:00 2000, either ways a time in the past which would force a re-render on the next view by the user). 3.) The last condition checked is if doRender is true, it would render, but the variable is never set to true, so the program always just falls through, and never does anything. 3b. There is a condition that checks if minZoom touchFrom or minZoom deleteFrom to enable the rendering flag. I was going to submit a patch... but I wanted to understand the intended purpose of these changes. it seems like the program flow now is to only render in combination with touchFrom or deleteFrom. I'd like an option that simply given a list of tiles to re-render them. I presume that touch/delete has some advantages in regards with rendering them on demand via apache's mod_tile. Would adding an option such as --render-from=ZOOM be acceptable? that simply takes the list of tiles and re-renders all the meta files in the list? ie. cat expire.list| render_expired --render-from=10 -z 10 -Z 13 --socket=/var/run/renderd/renderd.sock On Mon, Oct 18, 2010 at 10:10 AM, Samir Faci (Dev) d...@esamir.com wrote: I'm having some issues with the expire_list method. I wanted to run this by everyone and see if there's anything Im doing that's obviously wrong. I downloaded 48 hours of changes using osmosis which generated a changes.osc file. 1.) write the changes to the database. osm2pgsql -U osmuser -H localhost -d gis -e10 -o /tiles/sandbox/expire.list --append --slim /OSMbin/osmosis/change.osc Everything looks good so far. 2.) Now, at this point ideally, I'd like to just re-render the tiles in my expire.list. my usual command would be something like this: cat /tiles/sandbox/expire.list | render_expired -v -all --socket=/var/run/renderd/renderd.sock --min-zoom=10 --max-zoom=13 which issues a lot of warnings about: tile being already requested Looking at the output of renderd in the foreground shows me that no tiles are being rendered. I was following/reading the http://wiki.openstreetmap.org/wiki/Tile_expire_methods as a reference. as far as I can tell there are 2 options --touch-from (touch instead of render ) which I don't believe is what I want, but also complains about already requested --delete-from (delete instead of render) which is definitely not what I want.. and the last option is the default behavior which is not working. I should be on svn r23377 (git:f68f5dee3f50eaf12a6f7efbdcb9a6a9dda800b6 ) of mod_tile , I can try to update to latest commit but it seems like it should be working. Any thoughts? -- Samir Faci *insert title* fortune | cowsay -f /usr/share/cows/tux.cow -- -- Samir Faci *insert title* fortune | cowsay -f /usr/share/cows/tux.cow ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Renderd + expire_list
On 20-10-2010 15:51, Samir Faci (Dev) wrote: I was going to submit a patch... but I wanted to understand the intended purpose of these changes. The intended purpose was to remove the need to setup rendering threads with renderd, when they wouldn't be used anyway. it seems like the program flow now is to only render in combination with touchFrom or deleteFrom. That's a bug. The default behaviour is to rerender tiles, and only when a suitable combination of min-zoom and touch-from/delete-from is given, to switch to those exclusively. I just committed a fix that does more explicit checking of these combinations. So, for example: Expire tiles z9 and up by setting their timestamp to the past. This causes mod_tile to issue a rerender the next time the expired tile is requested: --min-zoom=9 --touch-from=9 Rerender tiles z8-11 and touch z12+. The lower zooms take much longer to render, so this rerenders those right away, while delaying rendering the higher zooms until someone actually looks at them. --min-zoom=8 --touch-from=12 If you give no min-zoom/max-zoom (default is 0 and 18 respectively), and no touch-from/delete-from, it will go through all zooms and rerender them, based on the input file. -- Lennard ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [josm-dev] [OSM-talk] JOSM shp file plugin
Hi, On 10/20/10 09:52, Sam Vekemans wrote: I'm wondering if it is in the works to have a JOSM plugin where it can automatically convert shp files (ie. run shp-to-osm.jar in the background), No, that would only encourage people to do even more mindless imports than we have already. There's not a day gone past without somebody, somewhere, importing data without talking to anyone first - overwriting existing data, creating duplicate nodes, ignoring license questions, not thinking about updates, and whatnot. Imports are a complex topic and they require a lot of thought and discussion. Making imports easier will enable everyone to go through the steps technically without necessarily having the proper understanding of OSM that is required. Any editor offering a simple shp import functionality would soon be regarded as the software that causes more harm than good. Calls for banning that editor would ensue. Bye Frederik ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] [OSM-talk] JOSM shp file plugin
Hi, On 10/20/10 15:42, Tobias Wendorff wrote: Any editor offering a simple shp import functionality would soon be regarded as the software that causes more harm than good. Calls for banning that editor would ensue. Guns don't kill people... ... and still people call for banning them, q.e.d. ;) Bye Frederik ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
