[OSM-talk] OSM Server Side Script
Hello, the project I've been working on the last few month now into some kind of beta status. So I you would like a reverse gazetter or a download an area of the size of a city, have a look at http://78.46.81.38 In particular, this might be relevant to the topic Advanced multipolygons - do we need area types? How well are they supported? of the London Hack Weekend http://wiki.openstreetmap.org/wiki/London_Hack_Weekend The idea behind the story is to have a server where one can obtain derived data as a web service. Areas are a standard example of that kind of things: the borders are as ways of use on their own, but they also define the area. In the OSM database, you find only the borders represented as ways and a relation declaring which borders constitute a certain area. So every application must figure out the areas on its own and has to rewrite the code and spent possibly substantial computation time (think of calculating a nation's borders on a mobile phone) on that. That's where the OSM Server Side Script server comes into the game: the derived data gets accessible to any application just with a single query, and the mappers still only need to edit and declare the independent data. And even the rules can be edited by the user as explained in documentation: http://78.46.81.38/#section.rule_example So in the long term, we may also do things like preparing the data for routing, deriving residental areas as desired here http://lists.openstreetmap.org/pipermail/dev/2009-February/014175.html or apply the machine readable version of the wiki as proposed here http://wiki.openstreetmap.org/wiki/Machine-readable_Map_Feature_list to detect conflicting objects in the database. There's a lot of work to do left. So I would like to get some feedback what to do first. And maybe there's even somebody who would like to join the project :) Some issues I see so far features: * A spatially intrinsic query for ways: At the moment, you only can query for nodes and then get the back references to get the data for an area. However, this would not include ways that cross an area without having a node inside of it. So this enhancement of the area-query would make it possible to include also those ways. * Mixed queries with spatial and tag-based criteria: an example would be to find all motorways in Germany. * Restriction of the output: If the size of the data is relevant (think of a mobile phone as a client), the server could omit certain useless tags (like the frequent created_by tag to reduce file size or processing complexity. * Or other things that come into your mind ... basics: * Proceed with the documentation: at the moment, the documentation is reduced to the essential things. And I don't even know whether the documentation is helpful or not. * Make the source code of the server accessible: The code is a bunch of C++ source files along with some bash scripts. It is quite a mess at the moment. And I'm even not sure whether I should place it in the OSM SVN or not. I would be grateful for every kind of feedback. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Is Xapi working?
This happened over the weekend too. The request this time was http://osmxapi.hypercube.telascience.org/api/0.6/node[amenity=studio] You can try to use the OSM Server Side Script server. Just try on the command line wget -O studio.nosm --post-data=query type=\node\has-kv k=\amenity\ v=\studio\//queryprint mode=\body\/ http://78.46.81.38/api/interpreter (all on a single line) or run in you favourite browser http://78.46.81.38/api/interpreter?data=%3Cquery%20type=%22node%22%3E%3Chas-kv%20k=%22amenity%22%20v=%22studio%22/%3E%3C/query%3E%3Cprint%20mode=%22body%22/%3E (entire URL on a single line) This returns a gzipped file with all results. The service is less up-to-date (designed to be 4 to 6 hours behind) and does not contain edit metadata (timestamp, uid of editor, version) but depending on your needs it still might help. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Is Xapi working?
is there any other way to get OSM data without going to the main server? there are no other caches, right? There is a whole ecosystem of servers providing OSM data http://wiki.openstreetmap.org/wiki/Planet.osm There are a couple of sources for excerpts or diff files listed on this page. http://wiki.openstreetmap.org/wiki/ROMA http://wiki.openstreetmap.org/wiki/TRAPI These two services are optimised for queries to make a map of the data. They are intended to be only some minutes behind the main server but don't offer all the tags. So you should not use the data for further editing. http://wiki.openstreetmap.org/wiki/XAPI This is the well known alternative to the main API. It's also intended to be only minutes behind the main server. It has an extended API with still a concise syntax. The data is usable for editing. The only tag that is filtered out is created_by - this tag can safely be ignored. http://wiki.openstreetmap.org/wiki/OSM_Server_Side_Script This one is very recent and still in a state of playground. It is intended to serve particular complex queries beyond the scope of XAPI. It also offers (almost) the complete functionality of XAPI. It is some hours behind the main API. It does not serve data that can be used for editing, in particular it does not provide version information. If somebody asks for version number support (or other metadata), I'll start to implement that. There has been no demand so far. There may be other storage servers, the category Data storage on the wiki is not yet written. But at least, there are plenty of alternatives to download data from the main server. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] OSM TrustPoints
Hello, Goal Prevent creation of new sock puppet accounts for potential acts of vandalism on larger scale and spam. Gradually give trust to users, and give them additional privileges. It should not be in the way when new users want to contribute normally. It should not encourage competition so that itself doesn't become an abuse target. Well, in the way described, it will conflict with a couple of legitimate use cases. Is there a real vandalism problem on the map? Although I've been working for almost a year with a lot of the map data, I have not seen any piece of intentional vandalism. * allow larger daily bbox for changes A couple of weeks ago, I discovered that the subway stations in Roma are incomplete, so I added stations to my best knowledge. Or, an even more opposed case: A couple of day ago somebody added arabic names to all the coutries. I would consider this as a legitimate use, but it invokes almost the entire planet. * allow more daily edits (number of affected nodes, ways...) Even something simple like the bus route I've added yesterday might easily touch more than a hundred elements. On the other hand, you could damage significantly the map with less than a hundred destructive edits. * Regular editing activity I personally do a lot of mapping in my holidays, so a pattern like massive activity from time to time will appear from the system's point of view. On the other hand, it is easy to tune the amount of activity of a malicious bot to any pattern expected by the server as regular. * Track uploads There might be a lot of use cases, e.g. naming things, adding POIs, bus routes and so on that require no GPX data at all. * Regular activity in other systems? (mailing lists, forums, wiki, svn repository, diary/blog, trac...) Perhaps totally different systems shouln't be mixed - one who can program or is very vocal doesn't necessarily yet know how to map well and shouldn't be trusted with enormous imports and vice versa) This makes contributing even more complicated. Why should I be forced to use these frills? There might be users who can't contribute something useful in one or more of the above systems (What should a non-programmer contribute to the SVN Repository? Should every user write wiki pages that nobody wants to read, just to display honest acitivity?) or just aren't interested in certain tools (not everybody wants to blog). * Rated by other users (manualy: reverted changesets, reported spam in diary, getting comments to a diary entry while not being flagged as spam...) ? Oh, sounds like the eBay reputation system. Do we really want to have an intervention by lawyers which comments are admissible and which aren't? An implementation of the whole idea would not only take a lot of ressources of the implementers, but also affect significanty normal users. The way mappers and other users use the system in a legitimate way is so widespread that any reputation system would end up bullying a smaller or bigger fraction of the users. Remember that the API has intentionally been kept small to make usage simple. On the other hand, there is no strong need to prevent vandalism because there is only sparse vandalism. So I would suggest to keep the whole concept away from the map. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] is_in and similar tags
Could someone[1] setup a web-service where you send it a lat/lon and it returns a list of all boundaries that point is within? So just one website imports the boundary data instead of everyone having to know how to do the 'is within' search[2]. I think you might be able to do this with http://wiki.openstreetmap.org/wiki/OSM_Server_Side_Script Yes. To appeal to [1], replace in the URL (in one line) http://78.46.81.38/api/interpreter?data=%3Ccoord- query%20lat=%2251.0%22%20lon=%227.0%22/%3E%3Cprint%20mode=%22body%22/%3E the values 51.0 (latitude) and 7.0 (longitude) by the respective values. Then save the file to disk and you receive an OSM-alike file with the areas that cover the given location. Another, maybe more convenient way would be (command line in one line) wget -O - --post-data=coord-query lat=\51.0\ lon=\7.0\/print mode=\body\/ http://78.46.81.38/api/interpreter | gunzip The details are explained at http://78.46.81.38/#section.reverse_gazetteer Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Finding what country something is in (new website)
Does the script also take boundaries in relations into account? I'm a little puzzled by http://dev.openstreetmap.org/~ojw/WhatCountry/?lat=42.8145lon=20.365 which is inside Kosovo with two relations as border, #1057;#1088;#1073;#1080;#1112;#1072;, admin_level 2, which is seen and Kosovo, admin_level 3, which is not seen. Two boundary relations is also the way to map the Australian example. Basicallly, the OSM3S takes into account any relation that has a tag with key admin_level (no matter what value) and name (no matter what value). Then it tries to make one or several polygons from the way members of the relation. If the way members constitute proper polygons, an area is made from these. The tagging of the ways doesn't matter. If not, you can spot the problems by a query like id-query type=relation ref=53295/ report/ Just send this by a post request like wget -O - --post-data=id-query type=\relation\ ref=\53295\/report/ http://78.46.81.38/api/interpreter or just paste the query in an arbitrary form on http://78.46.81.38/ Concerning the Kosovo example, there is something odd at http://www.openstreetmap.org/?lat=42.8362124lon=20.3513993zoom=16 and http://www.openstreetmap.org/?lat=42.8362313lon=20.351468zoom=16 Concerning Australia, a query like coord-query lat=-34.7758269 lon=149.6918631/ print mode=body/ does find relation 80500 which represents Australia. So please specify where in Australia the script fails. Then I'll try to fix it as fast as possible. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Finding what country something is in (new website)
Dear OJ, I put a wrapper around the rather excellent http://wiki.openstreetmap.org/wiki/OSM_Server_Side_Script which can tell you which town/county/state/country something is in: http://dev.openstreetmap.org/~ojw/WhatCountry/?lat=51.51lon=-0.05 - which replies that the specified numbers are in Tower Hamlets and London and the UK It does mean you can get all the admin levels for a place using just one line of PHP: $MyArray = explode(\n, file_get_contents(sprintf(http://dev.openstreetmap.org/~ojw/WhatCountry/?l at=%flon=%f, 51.51, -0.05))); (so $MyArray[1] would then contain the country name. Apparently this is ISO 3166-1) first of all, thank you for concise way of getting country information. After some playing around, I get some error messages with http://dev.openstreetmap.org/~ojw/WhatCountry/?lat=-34.7758269lon=149.6918631 (should be somewhere in Australia) ---8--- br / bWarning/b: Invalid argument supplied for foreach() in b/home/ojw/public_html/WhatCountry/index.php/b on line b45/bbr / ---8--- If the problem is on the OSM3S side, I'll try to fix things as fast as possible. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Finding what country something is in (new website)
There is really something broken, compare : http://dev.openstreetmap.org/~ojw/WhatCountry/?lat=51.8478lon=9.0282lang= demode=raw and http://dev.openstreetmap.org/~ojw/WhatCountry//?lat=51.894lon=9.1909mode= raw Both queries are responded from cache, but from different times. While the data of the former is from 2009-07-31 15h00 UTC, the latter is from 2009-08-01 03h00 UTC. In the meantime, somebody has edited holes into the border of Nordrhein-Westfalen. Based on the data of 2009-08-01 08h00, there are holes at http://www.openstreetmap.org/?lat=50.9780863lon=5.9195152zoom=16 http://www.openstreetmap.org/?lat=50.9576158lon=6.0061315zoom=16 http://www.openstreetmap.org/?lat=50.9479lon=6.0151zoom=16 and http://www.openstreetmap.org/?lat=50.9781246lon=5.9194663zoom=16 At least the second one results from a refinement of borders where and old piece of border (way 35960115) has not been removed. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Finding what country something is in (new website)
Something that would also be very cool would be if the script told you all polygons or multipolygons you're in regardless of whether they are a relation or normal polygon, and you could filter the result for country boundaries or other type of areas. It could for example tell you you're in a building in a school area in a residential area in a county in a province in a country on an island. It's a question of a proper specification. Then you can add the rules by yourself. Please have a look at http://78.46.81.38/#section.rule_example Currently, the areas are created based on the two rules osm-script name=Area::Create_from_admin_level query type=relation has-kv k=admin_level/ has-kv k=name/ /query foreach into=rel union recurse type=relation-way from=rel/ recurse type=way-node/ /union make-area pivot=rel into=odd/ detect-odd-nodes into=odd/ foreach from=odd into=i unionitem set=i/item set=rel//union conflictIn item set=rel/, the item set=i/ is contained in an odd number of segments./conflict /foreach /foreach /osm-script and osm-script name=Area::Create_from_multipolygon query type=relation has-kv k=type v=multipolygon/ has-kv k=name/ /query foreach into=rel union recurse type=relation-way from=rel/ recurse type=way-node/ /union make-area pivot=rel into=odd/ detect-odd-nodes into=odd/ foreach from=odd into=i unionitem set=i/item set=rel//union conflictIn item set=rel/, the item set=i/ is contained in an odd number of segments./conflict /foreach /foreach /osm-script These rules translate as follows: Consider every relation that has a tag with key admin_level and a tag with key name. Create a polygon from all the member ways. If this fails, attach a message In relation $Rel, the node $Node is contained in an odd number of segments to this relation. and Consider every relation that has a tag with key type value multipolygon and a tag with key name. Create a polygon from all the member ways. If this fails ... Thus, if you think of a rule like Consider every way that has a tag with key type and value multipolygon and a tag with key name. Create a polygon from this way. If this fails ... this translates to osm-script name=Area::Create_from_multipolygon query type=way has-kv k=type v=multipolygon/ has-kv k=name/ /query foreach into=way union item set=way/ recurse type=way-node from=way/ /union make-area pivot=way into=odd/ detect-odd-nodes into=odd/ foreach from=odd into=i unionitem set=i/item set=way//union conflictIn item set=rel/, the item set=i/ is contained in an odd number of segments./conflict /foreach /foreach /osm-script You just can submit the rule (or any other rule) as described on http://78.46.81.38/#section.rule_example and some hours later it should be processed. Feel free to ask if you have any questions. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Finding what country something is in (new website)
There is still something wrong here : http://dev.openstreetmap.org/~ojw/WhatCountry//?lat=51.894lon=9.1909 Thank you for submitting the bug. Unfortunately, it revealed a larger fault. However, I've added a temporary patch such that the area should work in about 3 hours (22h00 UTC). Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Tagging vague, ill-defined, or unfriendly paths
I suppose this brings up all the stuff about path tagging again, but, how do people in general tag vague, ill-defined countryside paths? The sort of things I'm talking about are either very narrow and occasionally hard to follow paths through woods, or, firebreaks in forests where there is some evidence of foot use but there isn't a nice path as such. I use http://wiki.openstreetmap.org/wiki/Tag:highway=path/Examples and have concluded to use highway=path, wheelchair=no The first tag classifies the way as being an unpaved and small path while the second clarifies that you can't use it for anything on wheels. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] XAPI down?
Hello, Is XAPI working? When I try http://www.informationfreeway.org/api/0.6/map?bbox=-83.56,42.17,-83.5599,42 .1701, I get redirected to http://osmxapi.hypercube.telascience.org/api/0.6/map?bbox=-83.56,42.17,-83. 5599,42.1701which turns to be an empty page. The bounding box you have specified _is_ empty. By the way, it is very small, only 11 x 7 meters. Are you sure that you have specified the right bounds? Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] OSMXAPI stable?
But if the XAPI is not working I have to cancel that. Or is there a similar service I can use for that? You can use OSM3S. Just send a post request with content http://78.46.81.38/api/poi?bbox=15.5337,48.1732,15.7161,48.2362amenity=restaurant You should receive a gzip-compressed osm file. This request is a short cut for the request to http://78.46.81.38/api/interpreter with POST data: osm-script timeout=180 query type=node bbox-query w=15.5337 s=48.1732 e=15.7161 n=48.2362/ has-kv k=amenity v=restaurant/ /query print mode=body/ /osm-script A comprehensive (but not yet complete) documentation of OSM3S can be found at http://78.46.81.38 Feel free to experiment with other queries or ask for other short cuts. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Computing the 12-mile line
I just tagged up the one I found in the database, I attempted to use GIS software to create a section where it misses a scottish island, but failed after 2-3 days of playing, I can't recall who put the data in originally. I'd be interested in seeing any code put into svn for others to use. On 20/02/2009, Adrian Frith adrian at frith.co.za wrote: For those of you who have been adding the 12-mile territorial waters line: did you calculate that data by offsetting the coastline/baseline? And if so, how did you do it? I mean: what software did you use, and how? I've generated the source code with a tool handwritten in C++. You can find the source code in the file sweep-brim.c on the site http://wmaz.math.uni-wuppertal.de/olbricht/osm/ The code itself is a mess and far from being user friendly, I wrote it merely to use it by myself. I stopped to use it because it fails on quite a lot of coastline errors. But I would be glad to reanimate it if you would like to use it or understand it. If you would like to get a particular piece of coastline, just ask for it. The basic idea of how to use it: Feed it a file of coastlines and a file with the resprective nodes. You can produce these files with osmosis by filtering for ways tagged as natural coastline and the nodes they point at. Or you can use the script coastline-extractor from the same site with a .bz2-file as the only argument, then use filter-nodes-bbox (compiled from filter-nodes-bbox.c) with a bounding box for the four arguments. Finally you can feed the two files as arguments to sweep-brim. The compiling flags for filter-nodes-bbox.c and sweep-brim.c are given in the file coastline-extractor. Unfortenatley, you then have to manually dig the right component from all the produced data - the tool produces also junk data for the onshore side of a segment and it is pretty hard to find and truncate the correct brim automatically. I've done it by hand. Be aware that the whole process is quite slow. The extraction of the coastline can take some hours. The calculation of the brim should take some minutes. Then the upload might take quite a lot of time - the coastlines of France and Spain were a weekend and the United Kingdom another weekend. The basic idea of how it works: Image the planet covered with equidistant scanlines along the latitudes (the distance between to scanlines is currently 0.001 degrees of latitude). Now you add for every segment of every way from the coastline all the intervals of points on each scanline that are less than 12 nm from any point of that way (simplified: take into account only the endpoints of the segment and the intersections with the scanlines - the inaccuracies won't matter compared to the scanline gaps or the projection correction). After the tool has taken the union of all those intervals, it spans a way from interval border to interval border. I will give more detailed explanations or run the tool on a particular region if you would like to. But I'm abroad for some days from tomorrow morning. So please ask, I'll try to answer on thursday evening. There is also a tool mentioned on Regards, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] License plan
Everything is up for debate. For me, this license change resembles the EULA story with openSuse, see http://zonker.opensuse.org/2008/11/26/opensuse-sports-a-new-license-ding-dong-the-eulas-dead/ and http://www.linuxjournal.com/content/opensuse-ends-eula At least in Germany, this EULA story might had more impact on openSuse than the cooperation of Microsoft and Novell. And it started as a clash of cultures when Novell changed the Suse pages from the Suse way of organizing a site to the Novell way of organizing a site. A lot of end users have been trained to the following way of perceiving: a screen mask that consists of several pages of scrollable text and then two buttons Yes or Abort means We never warrant that any part of this software works. But we always let you pay again when you do something we haven't planned. no matter what's actually written in the text. For a lot of people who are not primarly interested in law, this is what commercial means. So I would like to suggest the following: 1. Create a message like --- We are trying to get out of the caveats and flaws of copyright law and therefore need a new license. The final draft can be found at http://www.opendatacommons.org/licenses/odbl/ and http://www.opendatacommons.org/licenses/fil/ For non-law-experts, this means http://wiki.openstreetmap.org/wiki/Open_Data_Licence/Use_Cases --- 2. When a useful version of that message exists, request for as many translations as possible. Even doing here on talk@ would be a good place. 3. After some days, make the thing available at every user login. 4. Don't start the license commit itself at most a month after this message has been announced. At least for those who perceive Yes-Abort-pages that way, this would much more look like the behaviour of an open project. And what to users who do not log in with a browser? Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Getting boundaries from XAPI
Hello, you can try a HTTP POST request to OSM3S via http://78.46.81.38/api/interpreter with union id-query type=relation ref=359761/ recurse type=relation-way/ recurse type=way-node/ /union print mode=body/ E.g. paste the XML into a textfile req.xml and run wget --post-file=req.xml http://78.46.81.38/api/interpreter The result is a gzipped XML file. Or just open http://78.46.81.38/ and paste the request into an arbitrary text field. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Fwd: Nav4All navigation shut down by Navteq
Hi, You make this sound as if this is about the freedom of the new mappers. But they are, even today, free to follow any ruleset, cheatsheet, or book that they want to use. It's just that they don't get a guarantee that everyone else is using the same ruleset but that's ok - there might be rulesets much too complex for a newcomer, or the newcomer ruleset for rural Peru might be different from the one for urban Japan. Trying to make them all the same will needlessly reduce OSM's richness. These rulesets are unlikely to be devised by the same body; it would be too complex and the result would be less than optimal for everyone involved. We could have support for local tagging guides in a future version of the database without much effort. See http://wiki.openstreetmap.org/wiki/API_v0.7#Classes This would allow an editor to suggest tagging schemes with respect to the area where the mapping takes place. Mappers can explicitly tell what their tagging means. The advantage over hard-coded click-buttons is that it can be used across different editors. The advantage over the wiki is that it is maintained by those who really map. If different mappers want to use slightly and subtle different tagging schemes, they just can do without rants. But a simple postprocessing server can for any defined purpose still automatically derive a consistent tagging. Thus we could have rules to check minimum data quality without forcing the would into a overly complex, ill-fitting tagging scheme. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Boundary relation with 5000 members?
Dear Julio, Some examples of coastline boundaries that I found in Europe: http://osm.org/go/xVvgL5p-- http://osm.org/go/xX2ApwZz- http://osm.org/go/eq...@o- If it is not the right way to do it or there is a better way to do it, please let me know. Most countries in Europe (e.g. UK, Belgium, the Netherlands, Germany and others) have the territorial waters as national boundaries. There had been a longer discussion some time ago http://lists.openstreetmap.org/pipermail/dev/2008-October/012291.html http://lists.openstreetmap.org/pipermail/dev/2008-October/012340.html It sums up to: - the correct boundary would be the baseline, see http://en.wikipedia.org/wiki/Territorial_waters but nobody uses it, because it is difficult to obtain. - the territorial waters line is much simpler than the coastline: it has 10-100 times fewer nodes and it is much smoother. - the territorial waters often have the look and feel of the boundary: a ferry to an island near the coast doesn't have passport controls like when you cross a national border For this reason I would strongly encourage you to use the territorial waters like most countries in Europe. If you need a tool the generate these territorial water lines automatically, have a look at sweep-brim.c in http://wmaz.math.uni-wuppertal.de/olbricht/osm/osm-boundaries-source.tgz Feel free to ask if you need any help. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Detailed tagging scheme for railways - India
I recently got addicted to mapping the railway network in India in detail using landsat imagery. Good news for the map :) This is really great work. Since i wasnt able to find any other railway mapping project which specified detailed mapping conventions, i came up with my own scheme which i've compiled here: http://wiki.openstreetmap.org/wiki/India/Railways What about http://wiki.openstreetmap.org/wiki/Public_Transport#Railways http://wiki.openstreetmap.org/wiki/Railways ? In general, the two schemes are quite compatible, so there's not much to worry about. Some remarks (which may or may not apply to railways in India): - railway=halt is at least in Europe already frequently used with a different meaning: station designates stations where trains can begin or terminate. halt means (usually smaller) stations where trains only stop but legally can't begin or end. To make things worse, most mappers map either all stations as station or they map all small stations (regardless of their legal status) as halt. You have a clear definition of the feature (not regularly served), thus I'd strongly encourage you to use a different value for it. Rails within a city (which usually serve for mass transit within the city) should be tagged as railway=tram or railway=light_rail. Please add this to the wiki page to prevent somebody else from mapping them on error as railway=rail or something else. There is also a mailing list for Public transport. Please subscribe to the list and repost the mail there: http://lists.openstreetmap.org/listinfo/talk-transit There are more railway addicts. Hence, you are likely to get more feedback. If you get bored by mapping only the railway network, you can also start mapping the (regular) services on it: http://wiki.openstreetmap.org/wiki/User:Oxomoa/Public_transport_schema is a scheme for mapping those. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] How to extract national borders?
What's the best way to get an OSM file with the national borders? I tried using XAPI to look for ../*[admin_level=2], or .../*[admin_level=2][boundary=administrative] or relation[admin_level=2], but all took so long that I stopped them and were producing OSM files that were hunderds of MB big. So I wonder if I'm doing something wrong, or if there is a better way. You are tying to get all the nation boundary data from all over the world with this query. This *is* that much data. You can try to send osm-script timeout=180 query type=relation has-kv k=admin_level v=2/ has-kv k=name v=Ireland/ /query union item/ recurse type=relation-way/ recurse type=way-node/ /union print mode=body/ /osm-script to http://78.46.81.38/interpreter/ . You can do this as follows - go to http://78.46.81.38/ and paste the above into one of the text fields. - use wget: save the above to a file foo.txt and run wget -O - --post-file=foo.xml http://78.46.81.38/api/interpreter | gunzip foo.osm (command in a single line) Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] Planet.osm
Hello everybody, does anybody know why there is no planet.osm from this week, .i.e. there is no file planet-20100616.osm.bz2 at http://planet.openstreetmap.org/ ? Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] What could we do to make this licences discussion more inclusive?
I've split this from the original thread before it derails the one it was in any further, and cc'd legal-talk. [...] What could we (you/me/LWG) do to make this more inclusive? Just some bullet points at first, explanation follows: - There is no tool yet to see the impact of the relicensing to the data. But this is the key need for those who are rather interested in the data than the legalese. Please develop the tool first or leave sufficient time to let develop such a tool. - Please present a sound and complete technical solution to disentangle the data between the relicensed and the not relicensed. - Be prepared on a successive per-region move to the license. The communities in different parts of the world are at different pace. I don't think that the mappers in general are annoyed about that somebody works on legal issues. But don't forget that one of the key features of the project is the message: Care for the data and the applications - we promise you won't be affected by legal trouble. Thus, I would consider the license as a technical detail, like the change from API v0.5 to API v0.6. Now, if the API change would have damaged an unknown amount of data at unknown places, if would have been never done. This is because those responsible for the API change were aware that the new API is a mean, not and end. Legal things are less logical than technical things, thus everybody would accept more collateral damage. But still, I would expect good faith from the LWG: it is technical feasible to preview the impact of the license change on the data with an appropriate tool. Some suggestions - Have another read-only mirror that contains only the already relicensed data. This would allow to render a map with the ODbL-avaiable. Thus, the data loss or not-loss gets easily visible. We only need another server and a list of all user-ids that have so far relicensed, and about 4 weeks to make everything working. - Don't use an extra server, but make the relicensing data available via the main API. This needs much more brainpower, would save a server and prevents the user-id list from being published. I would estimate this takes at least 8 weeks to develop. I would volunteer to do option 1 if I get time until the end of the year. Maybe somebody else could offer this faster. Then, the algorithm unbroken chain of history of ODbL users is close to nonsense. An easy exploit would be a bot, possible camouflaged by different user accounts, that systematically deletes and re-inserts every object. Then, all data would have unbroken chain of history but won't have in general. Note that massive delete and re-create takes place from time to time, e.g. when imports and synced with pre-existing data. I claim more time to first get a more elaborate algorithm for the data move decision, so please remove the fixed timings from the plan. And, of course, things like translating messages into foreign languages and back, explaining the licensing issues at all to mappers in foreign systems of legislation and so on takes time. Indeed much more time than to implement a license within the special legal system it was designed for. I don't find the issues addressed in the implementation plan at all. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] [OSM-dev] Overpass API: new big server, IPv6
Roland, Is the Overpass server source code available? Yes, it is, licensed as GPL. Each installer consists of the source code. The most recent stable version is http://www.overpass-api.de/misc/osm-3s_v0.6.94.tar.gz You can also get the latest development version at http://gitorious.org/~drol/osm3s/drol-osm3s Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] [OSM-dev] Overpass API: new big server, IPv6
Side issue re. Overpass API How might it compare with the osm2pgsql route (in performance, and memory usage for the import process) for running a local API for a fairly small area, and with limited data, e.g. highways and wood/water feature polygons in the UK? In the default settings, Overpass API imports the planet without exceeding 1GB of RAM. It can be tuned to need less than that, probably around 256 MB, by changing the settings before compiling. This would make the import slower, but a smaller area may more than compensate for that. The whole planet (20 GB gzip compressed XML) needs 24 to 48 hours in a standard setting. An extract of Northrine-Westphalia (300 MB gzip compressed XML) takes about 10 minutes. Assuming the UK around 1 GB gzip XML, it should take between 30 minutes and 2 hours. An important disadvantage might be that Overpass API has no functionality to clip bounding boxes after updates. Thus using the minute updates would mean that the database gets filled more and more with irrelevant data. Best regards, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] Night of the living maps 07.02.2012 - Avoid API congestion
Dear JOSM users, Mapping events have in the past often triggered an API block from the admins due to too much download traffic. See e.g. http://wiki.openstreetmap.org/wiki/API_usage_policy To avoid that, I've just started the JOSM plugin mirrored_download. This plugin allows to download OSM data into JOSM from both instances of the Overpass API and thus avoids download requests to the main API at all. As an extra bonus to all plugins users, Overpass API is almost always faster than the main API. As both Overpass servers have still only a load of 5%-15% even at peak hours, there is much room for more download traffic. Just install the plugin (under preferences plugins mirrored_download), restart JOSM and then use Mirrored download Download from OSM... instead of File Download from OSM If you run a readonly API yourself, feel free to add that one in the JOSM SVN at josm/plugins/mirrored_download/src/mirrored_download/UrlSelectionDialog.java after line 101. Best regards, Roland Olbricht ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Night of the living maps 07.02.2012 - Avoid API congestion
And here are some problems: Thank you, Toby, for reporting these issues. I've just released a new version which solves the problems: 1) Activating this plugin seems to overwrite the default JOSM download action so the only way to start using the normal API again is to remove the plugin which doesn't seem entirely good. Fixed. 2) after I bring up the URL dialog, there is no way to dismiss it. I can also bring up multiple copies at the same time Fixed. 3) downloading via this plugin does not trigger JOSM's Draw boundaries of downloaded data feature. I've fixed it in the way that the plugin uses the requested bounding box to draw boundaries. Suggestion: Why require source changes and a recompile to change the URL? Just make the JComboBox editable. Done. I think I'll embed these URLs somehow in the settings, but I haven't yet figured out how. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Night of the living maps 07.02.2012 - Avoid API congestion
Better yet, maybe make it an advanced setting string (maybe up to 5 of them or something?) so that user choices are persisted across restarts. It has taken some days, but now in the plugin mirrored_downlaod the selected URL is saved as user setting and persists over a JOSM restart. Best regards, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] Overpass API: version 0.6.96
Dear all, the new version 0.6.96 of Overpass API is available. See http//wiki.openstreetmap.org/OSM3S/versions The most important change is the introduction of a new, more concise query language. You can now search with concise one-line-links (and, like before, HTTP GET or POST requests). An example with XML output is: http://overpass.osm.rambler.ru/cgi/interpreter?data=node[name%3DRosental]%3Bout%3Btarget=openlayerszoom=12lat=50.72lon=7.1 The second most import change is the introduction of JSON output: http://overpass.osm.rambler.ru/cgi/interpreter?data=[out%3Djson];node[name%3DRosental]%3Bout%3Btarget=openlayerszoom=12lat=50.72lon=7.1 And for the sake of convenience, you can also get the result as a map with ugly markers: http://overpass.osm.rambler.ru/cgi/convert?data=node[name%3DRosental]%3Bout%3Btarget=openlayerszoom=12lat=50.72lon=7.1 Please note that this is a functionality in a very early state. It needs at the moment hard wired coordinates for positioning of the map, and it does neither support multiple layers nor less ugly markers. Along with the new query language comes a new language guide: http://wiki.openstreetmap.org/wiki/Overpass_API/Language_Guide The impatient may want to read at least the sections http://wiki.openstreetmap.org/wiki/Overpass_API/Language_Guide#About_the_syntax http://wiki.openstreetmap.org/wiki/Overpass_API/Language_Guide#About_the_links http://wiki.openstreetmap.org/wiki/Overpass_API/Language_Guide#Sample_map_calls The next step will be a wizard to create from OSM element-ids such concise queries like above. Those can serve as stable long-time links and replace element-ids in the wiki. Best regards, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Creating a subset of OSM and storing it in Postgis tables
My current thinking is that I need to: 1. download the weekly England, Scotland Wales excerpts from geofabrik 2. merge the 3 files 3. filter out what I don't need You can replace steps one to three by doing a wget from Overpass. Please try e.g.: wget -O natural.osm http://overpass.osm.rambler.ru/cgi/interpreter?data=[timeout:900];node(50,-10,61,2) [natural];out; This returns all nodes that have a tag with key natural. You may need to fine-tune the bounding box (50,-10,61,2), but I think, it should roughly match. It may take some time to download, but it is perfectly well within normal server usage. If you want to combine several keys in one query, you can enclose node(50,-10,61,2)[natural]; in parentheses ( ); and write multiple. E.g. wget -O multiple.osm http://overpass.osm.rambler.ru/cgi/interpreter?data=[timeout:900]; (node(50,-10,61,2)[natural];node(50,-10,61,2)[historic];);out; If you want also ways and relations, you can try wget -O multiple.osm http://overpass.osm.rambler.ru/cgi/interpreter?data=[timeout:900]; (way(50,-10,61,2)[natural];node(w);rel(50,-10,61,2)[natural];node(r)- .x;way(r);node(w););out; All commands are in one line. More information is at http://wiki.openstreetmap.org/wiki/Overpass_API/Language_Guide Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] (OSM-dev) Overpass API questions
Dear Alan, This means, from 30 April on, you can get with the above query what you intended to get. The update for this change is not yet operational. The software is, in contrast to the testing system, painfully slow, and I'm still figuring out what has gone wrong. I'm sorry for the delay. I'll inform you when the update gets operational. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] Overpass API: new version 0.6.98
Hello everybody, I proudly announce version 0.6.98 of Overpass API. It brings a couple of improvements and new feature that have been asked for in the last two months. You don't need to worry that there are no planet files at the moment. Now you can set up your own instance of Overpass API by just downloading an existing instance, of compressed size 15 GB (or 25 GB with meta data): This only takes 4 to 8 hours, and then you have already ready-made database when others still download the planet file. See http://overpass-api.de/no_frills.html#clone and http://overpass-api.de/no_frills.html for complete installation instructions. Of course, also the query language Overpass QL has made progress: For memberships in ways and relations, some extra recurse operators are available: and collect the ways and nodes that are members of the given relations or ways. and collect the ways and relations that have the given nodes, ways or relatios as members. Thus, a bbox query simply gets http://overpass.osm.rambler.ru/cgi/interpreter?data=(node(50.74,7.05,50.75,7.06);;);out; To get also meta-relations on relations in this bbox, just use http://overpass.osm.rambler.ru/cgi/interpreter?data=(node(50.74,7.05,50.75,7.06);;);out; On the other hand, the recurse-full query just means to add ; (or ; if you want to also resolve meta relations): http://overpass.osm.rambler.ru/cgi/interpreter?data=(rel[ref=63] [network=VRS];;);out; The negation clauses have changed their meaning, as the meaning in version 0.6.97 caused more confusion than benefit. A query like http://overpass.osm.rambler.ru/cgi/interpreter?data=(way(50.74,7.05,50.75,7.06) [highway!=residential];;);out; will now find every way that hasn't a tag with key highway and value residential. If you want all ways that have a tag highway but not with value residential, you can query http://overpass.osm.rambler.ru/cgi/interpreter?data=(way(50.74,7.05,50.75,7.06) [highway][highway!=residential];;);out; Of course, you can also query for all ways that don't have a tag highway at all: http://overpass.osm.rambler.ru/cgi/interpreter?data=(way(50.74,7.05,50.75,7.06) [highway!~.];;);out; These changes are already documented in detail in the language guide: http://wiki.openstreetmap.org/wiki/Overpass_API/Language_Guide For the other changes, most notably an optimization of the database layout and a new transaction singularization system, see the changelog http://wiki.openstreetmap.org/wiki/Overpass_API/versions Happy data mining, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Redaction process is hogging up the tile rendering
Hi, I made some edits in Iceland yesterday before midnight UTC and the changes have yet to be drawn (around noon UTC). I think the redaction process is causing a major backlog in the tile rendering queue. Backlogs have usually been worked on and finished during the night but that was not the case last night. It is even worse. Due to a software incompability between Osmosis and the redaction bot, no updates will go at all to the map. This is not a problem of the rendering server backlog. It is a problem of the minute diff generation. It may take up to a month until we see the currently made edits. This affects all data comsumers, the mapnik map included, because nobody gets at the moment data updates. Best regards, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Redaction process is hogging up the tile rendering
Dear Richard, This is not a problem of the rendering server backlog. It is a problem of the minute diff generation. ...which has now been fixed. :) Thank you very much. Now it works fine, great work. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] ODbL-Planet
Hello everybody, the first ODbL planet has arrived. A big thank you to all who have contributed. To lift some load off planet.openstreetmap.org, I'll make my copy of the planet accessible (for some days) on http://overpass-api.de/misc/planet-latest.osm.bz2 Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] ODbL-Planet
The page 'http://planet.openstreetmap.org/' still says planet created 2 weeks ago. Is it The new planet? Yes it is. It is a copy of http://planet.openstreetmap.org/planet/2012/planet-120912.osm.bz2 Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Searching OSM
Hi Phil, Is there a way to search for a node type where it is on a way of a type, or types? An example would be gates on trunk or primary roads. Yes, there is a way. Please paste way(50.6,7.0,50.8,7.3)[highway=tertiary];(node(w)[barrier];);out skel; into the Convert Form at http://www.overpass-api.de/query_form.html and select to OpenLayers overlay and have some seconds patience. If you want to edit the data instead, you can download it with way(50.6,7.0,50.8,7.3)[highway=tertiary];(node(w)[barrier];);out meta; in the upper form and then open it in JOSM. The (50.6,7.0,50.8,7.3) is the bounding box (lat, lon, lat, lon) and can of course be changed, as well as tertiary to the tag value you actually want. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] Overpass API v0.7: Polygons, Areas and more
Dear all,
Overpass API 0.7 has finally reached its long expected version 0.7:
http://wiki.openstreetmap.org/wiki/Overpass_API/versions
There are essentially three major changes:
In addition to bounding boxes, now also polygons can be used to cut out a
region for download.
http://wiki.openstreetmap.org/wiki/Overpass_API/Overpass_QL#By_Polygon
A simple example is a part of my home town in Germany:
http://overpass-api.de/api/interpreter?data=(node(poly:50.7+7.1+50.7+7.12+50.71+7.11);;);out;
More general, almost all examples from
http://wiki.openstreetmap.org/wiki/Overpass_API/Language_Guide#Sample_map_calls
can be adapted to the polygon variant. The only restriction is that polygons
can only be used as borders for nodes. Please use one of the various recurse
statements to get ways and relations from it.
If you want to use a polygon like the output of rel2poly as a boundary, you can
convert it with the following script:
#!/usr/bin/env bash
echo -n '(node(poly:'
awk '{ if ($1 != 1 $1 != polygon $1 != END) printf $2 $1 ; }'
echo ');;);out;'
and then send the output (let's call it request.txt) with
wget --post-file=request.txt http://overpass-api.de/api/interpreter
The second change improves area handling.
http://wiki.openstreetmap.org/wiki/Overpass_API/Overpass_QL#By_Area
http://wiki.openstreetmap.org/wiki/Overpass_API/Overpass_QL#By_Tag
http://wiki.openstreetmap.org/wiki/Overpass_API/Overpass_QL#Query_for_Areas
If you want to download a city (with well-formed boundary), for example the
medium German town Alfter, you can just use
http://overpass-api.de/api/interpreter?area[name=Alfter;];(node(area);;);out;
Again, the (area) clause is yet restricted to nodes, but the Map Call Examples
http://wiki.openstreetmap.org/wiki/Overpass_API/Language_Guide#Sample_map_calls
show how to get the complete data of the desired flavor from this.
In the back direction, you can use the improved coord query to almost do
reverse Geocoding:
http://overpass-api.de/api/interpreter?data=is_in(50.75,7.21);out;
tells you in which city and country I currently are (latitude 50.75, longitude
7.21). For those, who prefer JSON, the same thing in JSON:
http://overpass-api.de/api/interpreter?data=[out:json];is_in(50.75,7.21);out;
(works also with all other examples)
Do you want to know where else on the world are placed named Birlinghoven?
Just call
http://overpass-api.de/api/interpreter?data=[out:json];node[name~Röttgen];foreach(out;is_in;out;);
or more concise
http://overpass-api.de/api/interpreter?data=[out:json];node[name~Röttgen];foreach(out;is_in;area._[admin_level~6|8];out;);
And, chaining operators makes that possible also for streets:
http://overpass-api.de/api/interpreter?data=[out:json];way[name~Elberfelder
Straße];foreach(out;;is_in;area._[admin_level~6|8];out;);
... unless you are hit by way too much results :)
The third change is different handling of HTTP headers. This is a highly
technical matter, mostly to enable proper CORS. Additionally, syntactically
malformed requests now get a 400 Bad Request reply to conform to standards.
Please see
http://www.overpass-api.de/command_line.html#headers
I'm happy that I have been able to address with this extensions several feature
requests. The next version will exclusively care on a proper migration to
64-bit node ids. As Overpass API uses unsigned integers, I have about three
years time to do that, but I'm confident to be faster :)
I expect that be done before Christmas and the next round of feature requests
to be implemented in January.
Happy querying,
Roland
___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] public_transport=platform not rendered
Hi, In April 2011 a new schema was approved for tagging bus stops. I've been holding off for 1,5 years, but now I started to retag highway=bus_stop to public_transport=platform. Please see https://help.openstreetmap.org/questions/14628/where-should-i-tag- highwaybus_stop-on-stop_positions-or-platforms I you are willing to help with public transport tagging, you can add value by repairing broken route relations. This happens quite often due to bugs in Potlatch or otherwise careless mappers. Repaired route relations make the data better. Retagging by hand does not help. Retagging by script is vandalism and has kindled all kinds of conflicts in the community and is, when detected, reverted. Please do not retag. Remember: the wiki is not normaitve, even less after years without change. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Role of the Wiki
Hi, In general, is there a method to when the wiki is or is not relevant? Yes, please have a look at http://taginfo.openstreetmap.org/ If a tag appears there in quite large numbers, it is relevant. Otherwise it is not relevant. Whatever the wiki tells you is rather irrelevant. What's the role of the wiki as a source of information in the OSM community? You need to be careful. Beside the people who do real mapping and often have left an initially useful description in the wiki, a lot of people care for their pet wiki page, with mixed results. Thus, the wiki is often far from reality, both leaving open real world problems and devoting broad space to things that simply never happen in reality. In general: the wiki is only descriptive, but often it sounds normative. It is a good idea to - use tags or tag keys that have been used quite often http://taginfo.openstreetmap.org/ - search the wiki for keywords of the thing to tag - read the relevant pages and take them as advice, not as a law - if the pages don't make sense to you or don't match, ask at http://help.openstreetmap.org - add an additional, new tag if the often used tags don't describe the situation appropriately I'm hoping it's relevant as OSM continues to grow - reading through old email archives isn't super efficient (or clarifying). The general outline is - discussion happens on the mailing list - decision taking is ultimately done by the inidividual mapper - documentation happens at the wiki Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Role of the Wiki
Hi, what if I tag it wrong? There is no right or wrong tagging. When you tag, you tell all data users a message. And either they understand you or they don't. There are things that are easy to tell like street names, because this is completely formal. There are things that are not formal at all and difficult to tell (e.g. street importance, although displayed prominently on the map). And things that could be formal but being formalized different in different regions because they are factual different in different regions (e.g. speed limit systematics, seasonal infrastructure, may bicycles run against a one-way direction?, various facets of public transport). That's the point where the wiki could show it's value: make a systematic tagging for your region and then explain the subtle details of this tagging in a dedicated wiki page such that a foreigner can make sense of the tagging. What if a new way of tagging something gets approved? The moment it becomes approved there will be probably 0 objects with the new tags, and if someone follows this principle (from what I see, this is exactly what happens) the new tags won't be used, rendering useless the discussion and approving of the new tags*. That's exactly the problem. The proposal concept errorneously conveys the idea that tagging-schemes are somehow designed and standardized and then afterwards applied to real world. This fails notoriously inside and outside OSM because reality quite often tends to not fit into any formal scheme. The best practice is to first observe the facts in the real. Then tell it other mappers and data consumers. This includes tagging it to enable other mappers to understand the local situation as well as afterwards documenting it on the wiki. This may or may not end up with the discovery that a similar problem has already appeared elsewhere. That is the helpful process of the discussion. If the a retagging is necessary at all, then the discussion process will turn out which old tag shall be mapped on which new tag. The actual mapping is then done very easily by software. So the important message is: The wiki comes into play only after observing reality and tagging it in some way, even if that tagging is not the final way. Hence a proposal having 0 known examples is not a transitional state, but very likely useless. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] Overpass API v0.7.1
Hello everybody, A new version of Overpass API http://wiki.openstreetmap.org/wiki/Overpass_API is available. Please read http://wiki.openstreetmap.org/wiki/Overpass_API/versions#Overpass_API_v0.7.1 for details. The most important thing is that this version accomodates 64-bit node ids. So the expected overflow over node id 2^31 in February 2013 will not harm Overpass API. The second most import thing is that the fair usage policy needs a subtle adjustment. You don't need to do anything, the server helps you automatically to comply to the policy. Some users (I assume a malformed JavaScript application) have sent a sequence of queries with very large bounding boxes within a few seconds. This used up all server ressources and harmed other users, and at the same time did not return useful results (due to oversize). A sample from the logfile: [07/Dec/2012:20:41:20 0100] runtime: 181, return size: 341, query string: /api/xapi?node[man_made=surveillance] [bbox=-73.6312499955,-21.20088405484794,102.4312499776,79.67900280484885] [07/Dec/2012:20:41:21 0100] runtime: 180, return size: 341, query string: /api/xapi?node[man_made=surveillance] [bbox=-29.61562499774,17.868367821866197,58.41562499785,69.46927728884238] [07/Dec/2012:20:41:22 0100] runtime: 180, query string: /api/xapi?node[man_made=surveillance] [bbox=-7.60781249978,35.566567573194945,36.4078124989,61.23135136351583] To avoid this, now queries from the same host (based on IP address) are serialized (executed one after another) and queries waiting for this reason for more than 15 seconds are rejected with HTTP code 429. If you have sent a runaway query and therefore receive 429 for subsequent queries, you can cancel the runaway query with http://overpass-api.de/api/kill_my_queries Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] House numbers
Dear all, house numbers are one of the areas where in OSM a lot of things are still to do even in well mapped regions. To support mapping efforts, there are two new tools available http://overpass-api.de/api/hausnummern shows the already existing house numbers. This allows to plan the next mapping tour more centered around poorly covered regions. These are shown on zoom levels 16 to 18. For example a mixed situation in the potential SoTM city Birmingham: http://overpass- api.de/api/hausnummern?zoom=16lat=52.48185lon=-1.91794layers=BT (link in one line) http://qa.poole.ch/ shows buildings that don't have a house number. This is helpful to spot forgotten addresses. The same part of Birmingham: http://qa.poole.ch/?zoom=16lat=52.48185lon=-1.91794layers=FTB0 Cheers and happy mapping, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] OpenStreetMap Future Look
Dear Jeff, have you ever though about organizing a market-wide vote whether Pepsi Coke or Coca Coke is preferred? And put up a vision that Coke A shall finally surrender? Not? Please just step back for a moment and take into account to possibility that OSM is more like a market and less like an organization. As you like evidence, let's go through the key elements. The mappers or users or stakeholders or simply the somehow involved persons: - In an organization, we have one or few distinct forms of membership - In a market, there is no clear distinction between a market player and a non-market player. You don't need a permission to buy a Coke and you can do so only once every decade, you only need a credit card. Our mappers contribute on very distinct levels of activity, and registration is commonly seen as a technical necessity (like the credit card). For example, it is likely not a membership because for a lot of deceded people there accounts will simply left off untouch, not somehow deleted. Different measures of active contributor are established and they all give different numbers. In particular, when voting was discussed around the license change, it was a very broad consensus that no selection of people was legit as a voting body. This sounds very much like a market, not like an organization. The same is right for tools development: Mapnik and all the other tools you mentioned have all been developed without a strategic vision and without formal permission from whomever. Again, sounds more like a market than an organization. You miss the flow of money? It's not a market of money and goods but rather of data and ideas. The key difference is redundancy: On a market, you get what you want when you find a supplier for it, regardless whether your demand conincides with the demand of the majority or not. The greek concept of agora fits well. In an organization, you need some kind of majority (might be your boss only or in a more democratic case, a majority by numbers) to steamroll down the minority's will. This is not how OSM ever worked or not how OSM shall ever work in the future. It is how Google and Apple work but exactly what most of us dislike on those companies. The OSMF sees themself rather like a regulating body for this market-like agora, not as the market itself. Now, as you won't expect FTC to have a vision which products have to be sold more, please don't abuse OSMF to formulate such a vision. Maybe we can add a clarifying statement to the OSMF mission statement if you have misunderstood it? Best regards, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] OpenStreetMap Future Look
Dear Pawel, And that's why TTT list moves so slowly. Please tell people the truth, you actively contribute to impede the Top Ten Tasks. Let's take a look at http://wiki.openstreetmap.org/wiki/Top_Ten_Tasks#Clickable_POIs_on_the_frontpage http://wiki.openstreetmap.org/wiki/Top_Ten_Tasks#POI_inspection_tool_on_the_frontpage There are several solutions that can be used off the shelves, including http://overpass-api.de/open_layers_popup.html Have you followed EWG discussions about the main issues from that list? Now, let's look at EWG minutes of October 15th http://www.osmfoundation.org/wiki/Working_Group_Minutes/EWG_2012-10-15 I offered to solve one of the tasks, including to care for the becessary support, and a couple of people liked to ides to have the problem solved. 18:17:18 drol I suggest to use now the Overpass Popup feature, because it is ready to use. ... 18:18:08 zere but, as drol suggests, there's already working code for doing this by click interception and db query ... 18:33:45 drol Installation means: just copy the JavaScript code. You can configure the categories there. I also voluteer to configure it in the way the community wants. 18:33:46 zere ok. let's put it to a poll. the question is: should we try for the overpass-style approach (hopefully quickly)? (the alternative being to go the vector-tiles route) +1 / -1, please. ... 18:34:33 ppawel -1 [altogether mixed result] The task was then postponed for indefinite time, because you promised to do some work you have not done since October. I have, by the way, done that myself, too, in the past; on several occasions I was approached by someone who wanted additions coded for JOSM or other OSM related tools and I built them and added them to the code base. In at least one situation I had an idea myself and approached a company working with OSM and asked if they'd be interested in funding it. I've never asked for, or received money directly from OSMF though. I don't think you appreciate the complexity of the OSM main website and related services. JOSM and standalone tools and scripts are just single purpose tools which are rather easy to code (although of course require a lot of effort). There are no user-driven scalability, point of failure, hardware, security and integration challenges involved. There are several stable working installations of rendering chains out there, including CycleMaps, the German openstreetmap servers and others. There is more than one installations of nominatim. It's not rocket science, in partciular not from a programming point of view. It's a matter of long time care and responsibility, and that's exactly the point for which the admin team deserves acknowledgement. I do acknowledge that reliability, carried out by responsible humans, not by some magic super-software. By contrast, your list user-driven scalability, point of failure, hardware, security and integration challenges, 21st century web site are just buzzwords. For example, security http://en.wikipedia.org/wiki/Information_Security has a well-defined meaning that already includes availablity which is the reason to do scalability and avoid designing a single point of failure. In particular, one virtue of security exactly to prevent overwhelming complexity is to divide and conquer. Adding features not only to the main site but even intermix them with the core system (the main OSM DB) makes the task indeed difficult. But this is due to bad design, not because the task is difficult. My two cents. Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] OpenStreetMap Future Look
Dear Jeff, Has a decision been made that that *is* the routing engine that will be added to OSM? If so, great. I look forward to it. Has that been publicized in the community? Please have a look into the wiki: http://wiki.openstreetmap.org/wiki/Top_Ten_Tasks#Routing_frontend The demonstration instance shows that several routing engines are offered, including OSRM. I would call that the location where I expect that information. Or in the metaphore, we haven't taken a decision for Pepsi, but rather offer Coke, Pepsi and even Club Mate alongside :) The formal process to add yet another routing engine is to actually write it and then to talk to User:Amm. Data supported by numbers, external studies, some employment of the scientific method that include evaluation of alternatives or the absence of what has been done, rather than long speculations in email. Hey, good idea. Go for it. If you want data, please write an email to Pascal Neis. He has conducted several studies about OSM and is surely happy for new ideas. The other road would be to throw money after consultants. That are the 70% of money on Wikipedia Frederik has mentioned. If you have money leftover, feel free to ask a consultant of your choice and publish the result here (and in the wiki). Would you otherwise really want to divert money from OSMF for development and hardware and to consultants? Before this gets into a cultural gap: In Germany consultants appear to be primarly paid to convey the opinion of the funder, because the funder doesn't want to tell his opinion openly and directly. A typical example is when management wants to cut away jobs. Open-ended research is done rather in the scientiic community. This may be different in your home country. That said, your first reaction to the suggestion of adding routing to the home page is negative. Then, later, you describe one routing effort you've been working on as good - and - it sounds like someone's already made a decision to add it to OSM. I think you have misunderstood Frederik. Every routing engine is welcome, and so is every additional editor, rendering engine and so on. The whole point is that a no stage OSMF needs to approve, fund or manage any particular project. Frederik, I and probably a lot of other people see this as feature and asset both of OSM and OSMF, not as a flaw. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] OpenStreetMap Future Look
Dear Pawel, EWG meetings are one-hour sessions where random people like me can say stuff. It's not the greatest way to discuss major features like clickable POI's - there is simply no time. So a short -1 to describe someone's months of work can be hurtful - I know I would be hurt if that was how OWL or other work I'm passionate about was treated. So I hereby apologize if that is how and why Roland felt. Thank you for calming down. I'm myself sorry that my mail has sounded like an accusation. You have very well described what I have felt at that point, and I wanted to described my feelings. I apologize that my mail made you feel accused. I agree that we should learn from it that IRC is a difficult place to discuss major features. Let's just all come together and hug. +1 Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] OpenStreetMap Future Look
Dear Matt, Ideally people from the ecosystem would be willing to write some code to integrate their cool projects into the main site. That is clearly not happening. sure, ideally. it doesn't happen often and there are a wide range of reasons for it, often simply that integration into the site requires completely different skills from implementing the original cool project, or that it seems too complex or time-consuming to do so. finding out why and trying to improve the situation are parts of why EWG was set up. but, as you said before, sometimes it's not the greatest way to discuss major features. but surely better than not discussing them at all? Thank you. For me it is new insight that writing more code for the Rails Port is an issue. I've just added a clarifying remark to the wiki, please feel free to clarify it further. In particular, would you appreciate a rails branch with the POI layer to faciliate a later integration? Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] POI display on osm.org
Dear all, have you ever been annoyed that Mapnik doesn't render a name for a street or a pub, although you are interested in? The POI click feature for osm.org now has a public prototype: http://overpass.apis.dev.openstreetmap.org/ Just click on the map somewhere and all the nearby named items are shown with their tags. It follows the idea http://wiki.openstreetmap.org/wiki/Top_Ten_Tasks#POI_inspection_tool_on_the_frontpage Before this moves to the main site, I would like to improve the usability as much as possible. So I'm grateful for all feedback. For example: Are there places or zoom levels where interesting points are missing? Or places where you get too much? Would you like some other formatting, more or less headlines, icons or whatever to easier categorize the results? Would you like to see something different than the list of tags? Have you other observations or suggestions? For example, Pawel made some technical observations that will give rise to improved speed for way-bbox queries on future version of Overpass API. A few details: You may see three kinds of results. If there are few results, they are shown immediately. If there are more results, they are shown as expandable headlines. If there are no results or way too much results, an error message is shown. The range depends on the zoom level, compensating for mouse position inaccuracies. It is currently about 20 meters on zoom level 18 and doubles with every zoom level. As an extra feature, if an element has a tag value starting with http://;, the headline of the element gets a hyperlink to the URL written in this tag value. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] POI display on osm.org
One thing about tags: You might want to consider hiding some. Especially well known import related tags. I'm thinking specifically of the tiger:* tags we have here in the US but there might be others. They are obviously a formatting nightmare but they are also not really *our* data so excluding them from such a display might make sense. http://i.imgur.com/p9amuhk.png Thank you very much for the example. There are even more problematic tags, for example the various name:* tags on locations. So one result of this beta is that the default view should not be based on tags, but rather some processed result. Let's see whether a view of all tags on request is helpful, because it would still help to give clear feedback to contributers. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] POI display on osm.org
- It would be nice to have a transparent circle where you click your mouse, with the diameter of those 20 meters on zoom 18 So you know what area the baloon is showing. - When you put the URL in the headline, put it on the website tag too. Before I read your mail to the end, I tried to click on the URL and was annoyed it wasn't made into a url. Or at least, put the underline below the headline, so it's visible that it's a link. -When you hover over an item in the baloon, a marker could show up on the map (a circle for a POI, a line for a street) -In the future, we should make a service that gives meaning to key-value pairs. For example, oneway=yes and oneway=-1 are just oneway, and oneway=no is twoway. Thank you for the multiple suggestions. I've tried to collect them at http://wiki.openstreetmap.org/wiki/POI_display along with the other suggestions made so far. Please feel free to edit that further. Best regards, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] POI display on osm.org
Hello Peter, thank you for the feedback. I've moved all but 4) to http://wiki.openstreetmap.org/wiki/POI_display to not loose the suggestions. 4) especially for POIs not displayed on the map it might be useful to show an additional icon, as there's a missing link between poi nearby and the map. What exactly do you mean here? Having the icons in the bubble? This is indeed a good idea, although not all feature classes have already icons. Having the icons on the map is unlikely possible, because there might be simply not enough space on the Mapnik map. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] POI display on osm.org
Hello Peter, Having the icons on the map is unlikely possible, because there might be simply not enough space on the Mapnik map. If I get a popup saying something about 3 restaurants, but I only see one or two, that looks strange and I have no idea, where the third one might be on the map. It may also discourage the mapper if the third restaurant is not shown on the map. That gets us closer to the problem: there are two different conversions involved here: The first is to get the reality into the database. This is at the heart of the project but essentially a problem of human motivation. A technical solution thus has to motivate mappers and give feedback. This is what the prototype is about: It should show the content of the database as verbatim as possible yet human-readable and -presentable. In particular, the choice which of the three restaurants is the least important is subjective, thus not possible for a machine on a general base, and thus the prototype must show all three restaurants here. The second is to make of the database content a visually appealing map. This is essentially a problem of design. Leaving out features for one or another reason is in general a good choice of design. A tooltip mechamism here is helpful, but on purpose a different task of the Top Ten Tasks. I think a perfect solution here would be to use one or another form of clustering whenever rendering conflicts occur. In particular this requires a straightforward possibilty for the user to easliy hide or show object categories and is thus a completely different approach than a slippy map. A vector map would go in the right direction. A complete solution here is sadly still some years away. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] POI display on osm.org
There are of course feature requests/suggestions.
Thank you for the response. Some of them have already been suggested, and I
will try to collect them all on
http://wiki.openstreetmap.org/wiki/POI_display
1) Make it more obvious for people that the POIs can be clicked on. It
doesn't have to be on the POIs themselves so it might fall outside of
your technical solution. Or maybe lift the solution from Wikipedia's POI
map since it's already been done (if license permits).
The solution from openlink.org looks like a very good approach to this. But I
don't think this would be acceptable to be shown by default on the main map.
Maybe it can be made an optional overlay.
2) Internationalization would be great.
Yes, indeed.
3) Rather than displaying the tags and their values, translate them into
user-friendly strings. A complete list of tags could just clutter the
UI. Like if one click's on the border of Reykjavík (capital of Iceland)
and choose Reykjavík. It's mainly a list of the city's name in other
languages, which has very limited use.
Yes. I have noted this particular use case of tags as an example for tags to
be treated in a special way.
4) Display information when clicking on buildings. Not just about the
POIs themselves, also the construction year and such. Maybe present the
complete address within the country if available. I'd think the general
public would like that very much.
I fear we don't have very much construction dates. But a decent presentation
may help to show this and other data.
It is an open question whether we can the data make so appealing that the
general public likes it and at the same time the transformation so
straightforward that an average mapper can control the transformation process.
This is highly desirable but a huge job. Please don't forget that Changemonger
has needed a lot of effort to make sense of changesets.
5) Link to the Wikipedia entry if there is one, with priority to the
UI's language of choice. This has been done before, I think, in the
Wikipedia POI map. You could maybe use the same api to get the correct
language.
I thought that the wikipedia link is present explicitly in the tags, isn't it?
The question is whether it is possible to link to a different language
6) And of course make the code configurable in the backend so others can
implement it easily on their OSM sites. :)
The prototype is on purpose almost perfectly decoupled. Just copy the file
https://github.com/drolbr/openstreetmap-
website/blob/master/app/assets/javascripts/popuplayer.js
and insert a line
map.on('click', popupLayer.onMapClick);
More configurability unfortunately doesn't make sense at the moment, as most
things that could be configured may get another implementation with other
configuration options. Whenever code gets more final, it will also get
appropriate configuration options.
Cheers,
Roland
___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] POI display on osm.org
A)I having problems when selecting any (large) area. For example a stadium, park, shopping centre, hospital, school, etc where they are amenity, etc as an area. Is that because I am not hitting the actual nodes? Or maybe you are not covering areas properly? Thank you for pointing this out. The areas are only found at their border, because in fact the ways that make up their border are found. To compensate for this, the search radius for ways is twice as big than the search radius for nodes. I'll experiment with also including areas. The area mechanism of Overpass API can be used for this, and ironically covers the complex cases with relations better than the simple cases with ways. The difficult thing here is to decide what makes up an area and what not. Not every closed way is an area, so there has to be some tag selection. Another thing is a certain lag behind for the areas of about 24 hours, but that can be improved with some effort on the server side. B) wikipedia links. It has been said already but even then there is some confusion. The recommended way for tagging wikipedia links is to use wikipedia=language:page title and not wikipedia=http://en.wikipedia.com/blah Thank you for clarifying this. While link buiding including escaping is clearly feasile, I'm not yet sure about the language selection. Does the interwiki mechanism expose an API for this? I think it would be best to first link always to the primary language of the object and later implement language selection. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] Overpass API: new version 0.7.2
Dear all,
a new version of Overpass API, version 0.7.2 has been released and deployed on
http://overpass-api.de/
and
http://overpass.osm.rambler.ru/
Details of the improvements are listed on
http://wiki.openstreetmap.org/wiki/Overpass_API/versions#Overpass_API_v0.7.2
In general, thing are getting faster, and areas are now almost ubiquious. This
lets the POI inspection tool
http://overpass.apis.dev.openstreetmap.org/
show more complete results.
One interesting thing for end users might be that is now easy to produce a
list of all streets in a city or other area. For example, for Grenoble in
France just call
http://overpass-api.de/api/interpreter?data=area[name=Grenoble];way(area)
[highway][name];out;
save the result in a file e.g. streets.osm and filter them with
grep 'tag k=name' streets.osm | sort | uniq | awk '{ s=substr($0,22);
print substr(s,1,index(s,\)-1); }' streets.txt
(both the URL and the command line in one line).
Happy querying,
Roland
___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] Overpass API: new version 0.7.3
Dear all, a new version of the Overpass API has just been released and delopyed on overpass-api.de. Some more details: http://wiki.openstreetmap.org/wiki/Overpass_API/versions#Overpass_API_v0.7.3 Examples for the improved functionality will follow in the next days. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] osm.org POI display: next beta
Dear all, the beta for a popup POI display has got its first round of improvements. See http://overpass.apis.dev.openstreetmap.org/ for the live demo. The popup should now be faster: The data is loaded in several chunks, and the first chunk should arrive after at most 300ms. A lot of work has gone to properly handle left clicks vs. double clicks. As this works around some design limitations in Leaflet, I would be happy about situations where this workaround still goes wrong to refine it. Comments are welcome here or on http://wiki.openstreetmap.org/wiki/POI_display The found objects now get a classification. This is currently text, icons should be used in the long run. This is also where I need your help: Categories may still be imprecise, so Im grateful for examples where objects need another classification. The classification is currently done in function classifyElement(element) in https://github.com/drolbr/openstreetmap-website/blob/master/app/assets/_javascript_s/popuplayer.js Comments are welcome as pull requests or again here or on http://wiki.openstreetmap.org/wiki/POI_display Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] osm.org POI display: next beta
Hi, are these links alright? They seem to link to gmx.net and redirect to the given URL meaning I get a 404. Has your mail client done something strange with the URLs? Yes, it has. The have set up a new webmail interface. I'm sorry, I had errorneously sent this email as HTML mail instead of text mail. The corrected mail: --- Dear all, the beta for a popup POI display has got its first round of improvements. See http://overpass.apis.dev.openstreetmap.org/ for the live demo. The popup should now be faster: The data is loaded in several chunks, and the first chunk should arrive after at most 300ms. A lot of work has gone to properly handle left clicks vs. double clicks. As this works around some design limitations in Leaflet, I would be happy about situations where this workaround still goes wrong to refine it. Comments are welcome here or on http://wiki.openstreetmap.org/wiki/POI_display The found objects now get a classification. This is currently text, icons should be used in the long run. This is also where I need your help: Categories may still be imprecise, so I'm grateful for examples where objects need another classification. The classification is currently done in function classifyElement(element) in https://github.com/drolbr/openstreetmap-website/blob/master/app/assets/javascripts/popuplayer.js Comments are welcome as pull requests or again here or on http://wiki.openstreetmap.org/wiki/POI_display Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] osm.org POI display: next beta
Hi, I don't know what is planned and by whom, just two things to inform the discussion - The EWG and I have considered to run an Overpass instance on the openstreetmap servers. This may eventually happen in the future, but for the current beta page the overpass-api.de server instance has sufficient ressources and it guaranteed to run until at least 2015, as Frederik has explained. Knowing from the beta how much ressources are necessary for an osm.org instance later on will help to choose the right hardware and a sustainable usage policy. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Area tags in Overpass API
Hi all, Why just building=yes? I do my best to be more descriptive on the type of building that it is, such as whether it is residential, industrial, commercial, retail etc. Thank you for the feedback. This helps to make a more complete list of tags. So I assume a better criterion would be has building, but different from building=no? Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Area tags in Overpass API
Hi all, I'm also curious about the motivation for considering only a strictly limited subset of the areas in the database in the first place. Presumably this is because maintaining a large number of areas would be detrimental to performance? First of all, thank you for taking it here. This allows for a in-depth discussion. The performance was the initial reason to use a whitelist at all. However, the performance has since been a lot improved, and it could be, with some hours of programming effort, improved even more. The reason to choose objects with name was that all use cases so far have been variations of the question Where am I?. And without a name or other distinctive tag value, the area has been considered to be of little use. The deeper reason is that I would like in general to let Overpass API be tag agnostic. The areas are one of the few places where I haven't found a way to get areas without tag evaluation. And there is even a future driven reason to encourage feedback: In the long term, Overpass API will produce GeoJSON to allow vector rendering. The main roadblocker for this is the distinction between linestrings and polygons. I assume that an established list of tags that designate areas are the best way to achieve this with long-term stable semantics. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Area tags in Overpass API
Dear all, [1] http://wiki.openstreetmap.org/wiki/Overpass_turbo/Polygon_Features I have now adopted this list. Thus, there are now much more areas available from overpass-api.de. The rambler instance has not been enlarged so far, but will follow in a couple of days. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] Replication down?
Dear all, http://planet.openstreetmap.org/replication/minute/ got stuck at 02:17 UTC. Can somebody please look what has happened? Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Read-only for next few hours
Hi Grant, OpenStreetMap is now back online. All services are returned to normal. Thank you for the quick fix :) Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] Overpass API 0.7.4: the difference operator
Dear Co-mappers, The new release 0.7.4 of Overpass API has just been deployed on http://overpass-api.de/ The Rambler instance will continue to run version 0.7.3 for some days in case of unexpected flaws in the new version. A lot of minor bugs have been fixed. More important, the query for ways on small bounding boxes is now more efficient. This speeds up http://overpass.apis.dev.openstreetmap.org/ the beta prototype for a popup overlay on the main page. I've also made two extensions in the syntax: The use of Global Bounding Boxes will be subject of a later mail to talk@ when the newest version of the JOSM plugin mirrored_download has been deployed. Now I talk about the difference operator: It simplifies the kind of searches every object that has property (or is an) X, but hasn't property (or is an) Y. For example, all nodes that have a value for maxheight but aren't part of a street (a way with tag highway): http://overpass-turbo.eu/s/Ib // New in Overpass 0.7.4: the difference operator [bbox:50.6,7.0,50.8,7.3]; ( node[maxheight]; // All nodes witha value for maxheight - (way[highway];;); // that aren't part of any kind of street ); out; For the very common case has tag X, but not tag Y I suggest to carry on with [...!~...]. This is more efficient than the difference operator: http://overpass-turbo.eu/s/Ic // New in Overpass 0.7.4: the difference operator // // If the negation only covers that a certain tag is absent, you may prefer // the tag negation operator. This operator is faster than difference. [bbox:50.7,7.0,50.75,7.1]; ( way[highway=residential][name!~'.']; // ways with tag highway, but without tag name ; ); out; All details are given in the wiki: http://wiki.openstreetmap.org/wiki/Overpass_API/versions Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] Edit filtered data
Hello everybody, as proposed now an application for the global bbox feature of Overpass API v0.7.4: Edit filtered objects: http://wiki.openstreetmap.org/wiki/Overpass_API/Language_Guide#Edit_filtered_data For example, one could check for all shops in Bonn whether they have a wheelmap tag or not and add the missing tags. Basically, the global bounding box feature http://wiki.openstreetmap.org/wiki/Overpass_API/Overpass_QL#Global_Bbox has two effects: - The given bounding box will be applied to all subqueries - The values will be returned in an extra element embounds/em in the beginning of the response. Thus the two queries ( way[shop=supermarket](50.6,7.0,50.8,7.3); ; node[shop=supermarket](50.6,7.0,50.8,7.3);); out; and [bbox:50.6,7.0,50.8,7.3]; ( way[shop=supermarket]; ; node[shop=supermarket];); out; are equal. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] XAPI and using a home server
Well, there could perhaps be another solution, like running your own XAPI server - the minutely diffs are usually less than 100Kb, so the required bandwidth to download from planet.openstreetmap.org would be less than 2 Kb/second in average. But the question is - how large would be the planet database on disk (how large would it get once you import the planet dump) Triggered by your request, I've made a deployable version of OSM3S. It needs only modest hardware requirements (1 GB RAM and 40 GB hard disk space for the entire world). See http://wiki.openstreetmap.org/wiki/OSM3S/install It has a different syntax than XAPI and partly different capabilities, but if you work with map data and don't need metadata about users, you likely can use it. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] XAPI and using a home server
Hey, thats look interesting! I'm almost tempted to kill my osmosis planet update. Any experience with daily diffs ? Yes, they should work. Please run again ./update_database --db-dir=YOUR_DB_DIR/ DIFF_FILE or in the more probable case that the diff file is compressed bunzip2 YOUR_PLANETFILE | ./update_database --db-dir=YOUR_DB_DIR/ Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] adding multiple relations (bus routes) to one road
Am Mittwoch, 6. Juli 2011 04:54:37 schrieb Robin Paulson: hi, I'm currently adding a lot of bus routes to roads in central Auckland. problem is, it's getting hard to manage. I've successfully mapped a full public transport network with 50 lines and 1500 stops. I've used JOSM with the public transport plugin. See http://wiki.openstreetmap.org/wiki/JOSM/Plugins/public_transport Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] XAPI compability layer
Overpass API has now a XAPI compability layer. It can answer the more common XAPI queries, details are explained on the wiki: http://wiki.openstreetmap.org/wiki/Overpass_API#XAPI_Compability_Layer Example queries are http://www.overpass-api.de/api/xapi?map?bbox=7.1,51.2,7.2,51.3 or http://www.overpass-api.de/api/xapi?*[bbox=7.1,51.2,7.2,51.3][name=Weststraße] Feel free to produce rather a lot of load. Although this is still a development server, I need a realistic load test. The server should have a primitive yet working protection against overload, so you cannot harm anybody. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] XAPI compability layer
It looks like you're missing the expand ways step. For example, a query like this: Thank you for reporting this. It should be fixed now. Do I understand it right, this does not apply to relations? Perhaps it's not meant to do that (the XML root isn't osm) but it would be useful to have to make it closer to XAPI. I assume that most tools don't need the osm tag in particular. The reason for the differing tag is that there might be elements (areas, error and status messages, the copyleft remark) which are not present in osm files. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] Overpass API: new version 0.6.93, now with meta data
Hello everybody, I proudly present the new version 0.6.93 of Overpass API. Overpass API now also provides the OSM meta data (timestamp, version, changeset id, user name, and user id). This allows to use the data directly in e.g. JOSM, including re-upload. It is the most-requested feature at the moment. Other features are a hardening of the software against file errors. The print/ statement now allows an attribute limit to limit the size of the response. And a reworked planet import should now work about faster than before. Read more information on http://wiki.openstreetmap.org/wiki/Overpass_API For the next version, which should be 0.7, I'll enable bounding boxes also directly for ways and relations. Furthermore, the scripting language will get some clean-up around the query statement and a concise semantic suitable for effective GET requests. I hope, I'll complete that version in September. Some details about the meta data: As expected, this is twice as much more data (65 GB instead of 35 GB) and makes the server an order of magnitude slower. To mitigate the effects on those that don't need meta data, the feature must be explicitly requested. In OSM script, this is done via setting the respective print/ statement to mode=meta. In the XAPI compatibility layer, add the directive [@meta]. The meta data give rise to the following special keys inside the XAPI compatibility layer: [@meta] lets the database include meta data (version, timestamp, changeset id, user name and user id). [@user=Roland Olbricht], [@uid=65282] restricts the data to data last edited by the user. He or she can be identified by name or by user id. [@newer=2011-08-01] restricts the data to only those data last edited after the given date. This is only possible in combination with another conditional. The corresponding tags in the scripting language are: print mode=meta/ instead of print/ for the print statements that shall print meta data. user name=Roland Olbricht/, user uid=65282/ can be used as a standalone statement and as a sub-statement of a query statement. newer than=2011-08-01/ can only be used as sub-statement of a query statement. The [@changeset] special key is not realized. I want to keep up the possibility to restart at any time for a recent Planet.osm, and that Planet.osm won't contain elements that have been deleted meanwhile. Even worse, the same is true for an old version referred in the changeset if the element has been changed since then. A so much crippled response isn't useful any more. These problems do have also an impact on the user and newer queries, but I estimate them to be still useful. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] The temporal dimension (was: Re: Overpass API: new version 0.6.93 ...)
[@newer=2011-08-01] restricts the data to only those data last edited after the given date. This is only possible in combination with another conditional. Why wasn't something like [@timestamp2011-08-01] used? Well, at first, because there is no [@older=..] or [@timestamp..] . By the way, to correct myself, it should be something like [@newer=2011-08-01T09:15:00Z], because the simplistic parser needs a full date. The [@older=..] in turn doesn't exist, because it doesn't make sense in the current state of affairs. See below. [@newer=..] shall have a humble look and feel, because it is a very humble solution. To spark the discussion, imagine the following data items: - Element A hasn't changed since January. - Element B has been re-uploaded without changes yesterday. - Element C has been deleted yesterday. - Element D has substantially changed its meaning yesterday such that it is now out of scope. (To make this clearer: think e.g. you are searching for bridges and D is a way that has been split such that one half is no longer a bridge.) - Element E has been created yesterday morning but then disappeared yesterday evening, due to a bad edit. Now what would you expect from the [@newer=24-hours-ago] operator to find? A good implementation should produce surely C and D, maybe E and/or B. Actually, Overpass API yields B and only B, which is unsatisfactory. The reason for this: At the moment, Overpass API represents at any moment the data as it would be visible in a fictional Planet.osm, patched from the last Planet.osm by the diff files applied so far. And a Planet.osm would also only show B. This is closely related to discussions of the type How do I find deleted elements?. What are other options? (Please mind: All of them are thoughts, not even vapour ware) Overpass API could become a full blown history server. This would allow to give sane answers to [@timestamp..], [@timestamp..], and a create a diff option, produce search results from a certain time in the past and a lot more of amazing thing. But this has at least four downsides: 1. This partly breaks with the OSM data model. For example, a way can change its geometry without getting literally changed itself: just move the underlying nodes. The OSM element doesn't recognize this, the database must recognize this for consistent data delivery. The way might for example have entered or left the bounding box you have searched for. The issue popped up at some time in a discussion about the undo features of Potlatch, but I don't have a link to that. Another thing is that such a server would mix CC-BY-SA and ODBL data at a certain time in the future, which is a unnecessary legal hassle. Most likely, I would be slow enough on development to roll out the software after the license change :) In any case, this may produce a flame war on details, which I exactly don't want to get the project into, and I'm not diplomat enough to avoid this. 2. Hourly, daily, weekly diffs are incompatible, and even the Planet.osm and minute updates need diligent analysis. Note that in all the diffs, multiple changes will be collapsed into a single diff. Thus, an element like E above might never appear on the server. I'm not sure whether the minute updates are guaranteed to contain all changes, but changes reverted in less than a minute might be acceptable to lose. The full history Planet.osm could replace an ordinary Planet.osm, but mind that it is an order of magnitude bigger. 3. This all could multiply the hardware requirements. I'm simply not sure what the current server can handle. For this reason, I started with the Planet.osm meta data, which already doubled the data amount from roughly 35 GB to roughly 65 GB. With history data, I expect rather 100 GB to 150 GB. The impact on the query times can probably be kept under control (if we keep the historic data apart from the current data), but the data updates in that case will become much slower. 4. And it will need a lot of programming effort. Just alone the necessary documentation to make clear all of the decisions necessary in point 1. and 2. takes weeks. Implementation and testing will be the same or even more effort, depending on how much tricks are necessary to keep the system responsive. While it is challenging, I don't see the massive demand in comparison to other features that would be postponed in that case. A second option would be to produce some kind of feed, where you need to subscribe to get changes. This can be realized quite easily, because the way and relation updater already receive some kind of internal feed to update their geometries from their members. So you subscribe with an arbitrary query, e.g. a bounding box or a certain tag or a combination of both, and get all changes concerning that query roughly every few hours, without the assertion of being complete. But I don't expect that much users would be interested in such a service
Re: [OSM-talk] Overpass API problem with underscores
When I open http://www.overpass-api.de/api/xapi?*[highway=trunk_link][@meta][bbox=-77,3 8.9,-76.9,39] in JOSM I get a bunch of extraneous relations. Thank you for reporting this. The positive and the negative examples have been very useful to track down the bug. It is fixed now on the server. It will also be fixed in the upcoming new release 0.6.94 next week, so I won't issue a separate patch for this for 0.6.93. This does not happen for highway=trunk, so the problem is with the underscore. By the way, it was not due to the underscore, but rather because there exists in the entire Planet a relation with highway=trunk but no relation with highway=trunk_link (only ways). Best regards, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] Overpass API: new version 0.6.93, a collection of bugfixes
Dear all, Overpass API goes a huge step forward for more stability. Or, to avoid corporate speech, fixing a couple of bugs kept me busy the last five weeks :) All known bugs are fixed at the moment, so I hope, I can develop the features in the next weeks. First of all, thanks to all that have sent bug reports. In particular, two deeper bugs which could potentially affect the database content have been fixed. A further, shallow but annoying bug, the forgotten escaping of user names, has also been fixed. You can find the full list of fixed bugs on http://wiki.openstreetmap.org/wiki/OSM3S/versions#Overpass_API_v0.6.94 As a first visible step of the feature progress towards version 0.7, the query statement can now have almost any combination of substatements. A further detailed planning showed that there is still a long way to version 0.7. Planned features are: - A more concise query language, as terse as XAPI, but as flexible as the native Overpass API language. - Some useful extensions, in detail it will support: -- getting transitively all referred elements of one or more relations -- selecting all but one value for a given key in a query -- enabling around, bbox and area to handle way and relation geometries I don't make promises on the release date this time, but I plan to release a version every four to six weeks, with version numbers indication whether the progress is small or large. So a 0.6.95 or 0.7 may appear next month, depending on whether all features have been implemented. Best regards, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Making iD the default editor on osm.org
What I deeply regret is that OSM website still does not offer to everyone, including newcomers, a fast and easy way to revert an edit once it is saved. Work is under way. Please stay tuned an have an eye on the workshop The geometry and data of change at the SOTM. In more detail: As opposed to wikipedia, a typical edit usually gets non-trivally interdependend both with other edits as well as the different parts of an edit to each other. So a revert function that just reverts entire changesets unless they have other dependencies is of limited use, because most changesets that do significant harm get quickly interdepend on those changes intended to repair the harm. You can argue that more people would do wholesale reverts instead of repairing if they were easier, but a lot of the interesting cases have changesets that mix damage and merit. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] Annoucement: Poup layer beta reworked
Dear all, the Popup extension for the osm.org main site has just got a major rework. The feedback from the previous version has been incorporated and resulted in a couple of new features. I'll introdurce them by example: http://overpass.apis.dev.openstreetmap.org/#map=14/64.1289/-21.8902 Please click on the large building Kringlan. A blue box and a popup with all the named features from the box appears. The page indicator [1] [2] [3] at the bottom shows that there are more features to explore within the blue box. Please click on details of Lyf heilsa: You will see both a link to the website tagged on the element and the adress of the element. The details are derived from well-known tags like here the addr:* family and website. To show further evaluated tags, you can go to page [3] and click Ísland: You will get a Wikipedia entry and a list of all known translations. These are made from the name:* family. In all cases, you can see all tagged details of an object on the [tags] switch. If there are multiple objects tagged exactly the same way, then they are grouped together. This happens for example on page [2] with the streets called Kringlan. To allow further inspection of the objects, you can click on each and get the respective webpage for the OSM object. What I would in particular ask you for: A lot of objects still show Other object or unexpected object classes. This is because the list on https://github.com/drolbr/openstreetmap- website/blob/master/app/assets/javascripts/tagprocessor.js is yet to be completed. If you find a category that should appear there, please add it to the feedback page http://wiki.openstreetmap.org/wiki/POI_display#Missing_classification Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Annoucement: Poup layer beta reworked
Hi Jason, Without some context it is hard to make any comments. What are the goals/user scenarios of the feature The feature shows all elements that have a well-understood meaning by their tagging. So the mapper gets a basic feedback what of his edits have made sense in the commonly accepted semantics. Additionally it is useful to get quick information about what exists in a certain location, in particular at any zoom level. not covered by the show data function? The data function is essentially a read-only view like an in-place editor. So it is only useful for me if I don't want to log in at that moment. It cares about raw data, without any processing. It is in particular deeply hidden, bceause it works in dense areas only on the highest zoom level, and then only with patience. For example http://www.openstreetmap.org/#map=19/50.73183/7.0 still takes around 10 seconds to load, and it is difficult to select the right object. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Annoucement: Poup layer beta reworked
Hi Stefan, I see following solutions: 1. adapt height of the popup to the remaining space (down to at least 2 items, currently it's 7) 2. leave as is but make default height of the popup smaller, perhaps to 5 items 3. pan map down automatically until popup fits into map window Actually, I think a solution 4. enhance Leaflet such that popups on the upper part of the viewport open downwards. is the preferrable solution and likely attractive for most other Leaflet popup users. I've no idea whether that is technically feasible, but 1 and 2 cannot solve the problem if I get just close enough to the upper brim, although they are good ideas otherwise. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Annoucement: Poup layer beta reworked
Hallo Eric, Take a look at my Rrose plugin for Leaflet, it may do what you need. Thank you. Yes, this is exactly what I have searched for. -- Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] SOTM 2013 - Final programme and end of mates rates
Hi Rob, Due to high demand we are getting close to being full. As such, we will be closing the mates rates discounted tickets at 12 noon BST this Friday. Be quick if you want to get the cheaper entry rates! My wife is meanwhil planning to be also on Saturday evening in Birmingham, but she will not attend the conference. What option can I book for her if she just would like to accompany me to the dinner? Best regards, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Annoucement: Poup layer beta reworked
Hi My local Supermarket: http://overpass.apis.dev.openstreetmap.org/#map=19/51.38116/-2.36842 If you click on it it display other objects (Bus Stop Road) before the Supermarket. Is this expected behaviour?, if so, why? It is accepted behaviour. I would love to have some fixed order and would love even more to have all the classes sorted into categories, but it is quite a lot and rather subjective. So I again ask you for help: I've copied the list of known categories to http://wiki.openstreetmap.org/wiki/POI_display and would like you to reorder the different tags by priority and maybe classes (probably Streets, Public transport, Boundaries, but maybe also other classes). Form doesn't matter, I'll bring it by hand into working JavaScript code. I've just no justified opinion whether a leisure=pitch is more important than a shop=chemist, and more opinions help to fix some order. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Annoucement: Poup layer beta reworked
Is it possible to hide them in an collapsible list ? Technically, yes. Current suggestions to handle large tag lists are: - Don't show tag lists at all but refer to the object's page - Clip them after some entries - Show them with a vertical scrollbar in a decent viewport (5 lines or so) - Show them in a dropdown box - paginate them All five apporaches have their merits and downsides, and I tend to test more than one approach. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] SOTM 2013 videos
Hi, I've added a reworked version of my talk with the properly rendered sildes: http://www.youtube.com/watch?v=LxWEHQ2DpI0 Have fun, Roland ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Overpass or XAPI API servers for 60, 000 node queries?
In the past I could generally get the query to complete by running it once, cancelling it, then running it a second time (maybe the 2nd time more data was cached). Now I get either infinite timeout or a server rejection. The query is: osm-script timeout=990 element-limit=1073741824 query type=node has-kv k=amenity v=drinking_water/ /query print/ /osm-script I'm glad that you bring up the whole subject. It's a challenging query for Overpass API. Malcolm has already sketched the long-term solution: operating from a SSD will largely solve the performance limitations, but currently I'm not aware of any cost-effective hosting options with large SSDs (more than 300 GB). Please do not use a cancel-restart strategy. This may double the load because not in all cases the Apache server cancels the abandoned query. It is possible that this strategy causes the rejection because the server rejects concurrent requests from the same IP. In fact, if a query is already running a second query from the same IP then the new request is first postponed for up to 15 seconds. It is immediately started if the former request completes during this time. If the former request doesn't complete within 15 seconds then the newer request is finally rejected. I suggest the following strategy: Please prefer the Rambler instance. It is for this use case probably the fastest. Then, feel free to use a much larger timeout. On a test run, the query had taken 12 minutes, so with a safety margin factor, please try 3600 seconds instead of 990. On the other hand, please omit the element-limit (and thus use the default of 512M instead). That's much more than necessary and the server is rather picky on RAM consuming requests. If the request is still rejected, please retry every few minutes. Peaks most often only last for less than an hour but aren't that regular. And trying a request doesn't hurt anybody. Cheers, Roland ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Creating a single better maintained list of XAPI/Overpass servers
Separately, there seem to be a number of slightly different lists of public XAPI/Overpass servers: https://wiki.openstreetmap.org/wiki/Jxapi#Overpass_API https://wiki.openstreetmap.org/wiki/Overpass_API http://wiki.openstreetmap.org/wiki/Overpass_API/status http://wiki.openstreetmap.org/wiki/Pt-br:Xapi Would it make sense to direct everyone to: http://wiki.openstreetmap.org/wiki/Platform_Status[http://wiki.openstreetmap.org/wiki/Platform_Status] These pages serve different purposes. I document the status of the two servers that I maintain in concise form on Platform_Status and in a detailed form on Overpass_API/status. The other instances work well but I don't maintain them personally so I cannot write status reports for them. So an accurate information should point out that the osm.fr instance, the Rambler instance and overpass-api.de all three are active and welcome requests, but only for the latter two the status is tracked in the wiki. Concerning the XAPI instances: The list on Pt-br:Xapi is plainly outdated and should be removed. These servers were listed two years ago on the English Xapi page but have been purged meanwhile. The jxapi instance exists since some time, but I don't know to what standards it is maintained. AFAIK no other Xapi-only instances are operational today. Cheers, Roland ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Overpass or XAPI API servers for 60, 000 node queries?
Dear Bryce, I've cross-checked the Rambler instance. I'm sorry it indeed doesn't work on that instance. It looks like an element on the network, most likely Nginx, disconnects any connection if no data is sent for ten minutes. A similar thing happened on overpass-api.de after 20 to 30 minutes. In that case, that Apache log indicates that the client has aborted the connection, but the client still waits. Looks like again something on the network did break the inactive connection. Cheers, Roland ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
[OSM-talk] Gzip on Overpass API
Dear all, To improve the average speed of larger requests on overpass-api.de, I have activated gzip compression by the Apache configuration. This is standard conforming, tests show no difficulties. However, weird errors can never be ruled out. So if you encounter odd behaviour with overpass-api.de since this evening, please send me an email to investigate the matter further. Cheers, Roland ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
[OSM-talk] Planned downtime for Overpass API
Dear all, the server hardware of overpass-api.de will be enhanced Wednesday morning. For this purpose, in the time window between 05h00 UTC and 11h00 UTC a hopefully much shorter downtime is scheduled. The Overpass instance overpass.osm.rambler.ru is planned to be fully operational, so please switch during the break to that instance. The work will add a SSD to the disks of the server. This should, after reconfiguring the server, speed-up queries significantly (I hope for at least a factor of 2). I would like to thank FOSSGIS for funding this server enhancement. Best regards, Roland ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Planned downtime for Overpass API
Dear all, overpass-api.de is back to normal operations. Best regards, Roland ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
[OSM-talk] Overpass API v0.7.50 almost done
Dear all, first of all a big thank you to all who have contributed to the SSD funding or to the general FOSSGIS funding for Overpass API. I will publish before SotM-EU 2014 a new stable Overpass API release, the first since about a year. To avoid confusion, I would like to sketch what it does and what it doesn't and how this relates to the running service. A proper documentation will be written along with the workshop slides for the SotM-EU. In particular, I ask you to test the attic feature to detect bugs now such that they are fixed in the database re-run in during July. == Summary for the impatient == For current data, the Overpass API is and will be completely reliable. Most new features for current data will be postponed to the next version, because they don't affect the urgent database rebuild. For Augmented Diffs, I will change the mode of operations, mostly because the current approach has intrinsic stability problems. The last bigger incident happened on 18th April this year. Currently, the new Augmented Diff mode can be tested, and the old one should be continued until end of July. Attic data is a completely new feature. The underlying data is basically available since the license change in September 2012. But due to two detected and possible other undetected bugs, attic data before 02nd June 2014 might be damaged. == Current data == Queries on current data are such well-adopted and in widespread use that I will not break backward compatibility. Essentially only two features have been added: Sparse queries are now faster. An example is http://overpass-turbo.eu/s/3FS That query would have taken hours with such a large bounding box in the past. Now it should run in less than 15 seconds. There is still room for improvement: It doesn't work yet that fast for regular expressions, but this is postponed to the next version. And features like ways and relations can expose their geometry directly on the feature. This is triggered by adding the word geom to the out statement. This currently works only for XML output. For JSON output, the final format is discussed on https://github.com/drolbr/Overpass-API/issues/93 They should mimic GeoJSON as close as it makes sense. Once again, the feature is very useful, but doesn't affect the database debugging. Thus it is postponed. == Augmented diffs == The Augmented Diffs are redesigned to be always generated on the fly. This is because the Augmented Diffs have been piling up on the server to almost a terabyte of data and we are running out of space. A second advantage is that their generating does no longer block applying the diffs to the current database. You can access them via e.g. http://overpass-api.de/api_0750/augmented_diff?id=905039 (by changing the id to the number you actually need). There are also advantages to the users: Augmented diffs now carry a number that can be straightforward computed from the desired date. Number 1 starts at 2012-09-12T06:56:00Z, the effective license change date. And the date interval of Augmented Diff n is expressed in seconds since the epoch always (n - 1347432900 ) / 60 until one minute later. You can get the current Augmented Diff with the call http://overpass-api.de/api_0750/augmented_diff_status which is deduced from http://overpass-api.de/api_0750/timestamp giving the date of the current database state. A second advantage is that filtering now makes sense. The full Overpass QL language can be used for filtering on Augmented Diffs, and it almost always makes the request faster. For that purpose take the request from http://overpass-api.de/api_0750/augmented_diff?id=905039debug=yes and adapt it to your needs. A third advantage is that you can requests arbitrary timeframes, not only minutes. However, the price to pay is that these changes are not completely backwards compatible; I'll take the reservation that the format was experimental. The info elements are no longer available, in favour of the geometry features of ways and relations. This is the format more widely adopted, and offering the info variant would have taken too much time to implement again. == Attic data == This is a completely new kind of feature, and complements the Historic dumps of OSM data. I'll prefer the notion attic data like in version control conventions to avoid confusion with mapping of historical features. You can run a query against the database state as it were at an arbitrary date in the past. Just put [date:2014-06-02T20:00:00Z]; in the front of your query. In a similar way, you can get what has changed in the results by adding [diff:2014-06-02T20:00:00Z]; or [diff:2014-06-02T20:00:00Z,2014-06-02T20:00:00Z]; in front. The first form takes as second time implicitly the current time. If you need in addition deletion dates, you can use [adiff:2014-06-02T20:00:00Z]; or [adiff:2014-06-02T20:00:00Z,2014-06-02T20:00:00Z]; In this case, you will get meta
Re: [OSM-talk] overpass query to get last used editor
Hi, Is it possible to also get the editor used by the last user? No, it is not possible to get the last used piece of software to change a given object. This information is present, if at all, in the changesets comments. Changesets comments don't go into the ordinary diffs. Thus Overpass API never even sees these comments. I would basically be happy to also provide changesets comments in the long run, but at the moment I don't know from what kind of feed (could also be XML diffs) I can get them from. Please note that even then it is optional for an editing software to put information about itself into the changeset. Cheers, Roland ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Worldwide non-surveyed tag edits
Dear John, I'm glad that you got into discussion. The OpenStreetMap community has some consensus that look ouright nonsense from a computer scientist or programmiers usual point of view. So it is helpful to explain every now and then what is common sense, checking whether those decisions are still valid. Consistent data is useful and typos and mistakes are common place. Unifying these so they are machine readable so they are useful is, in fact, useful. Just some examples: We have streets with housenumbers 3, 5, 9, 7, 11, 13 Is it an obvious mistake? It's on purpose, because the housenumbers sometimes are in that order on the ground. We have in Germany cities with a street named Cäcilienstraße and others with a street named Cecilienstraße (both with exactly the same pronounciation, and both variants of the same surname). The literal translation of connecting way into German is Verbindungsweg. This is also the offical name of a living street in Siegburg, Germany. By contrast, for good reason not connected in the database are these roads: http://blog.openstreetmap.de/blog/2013/05/wochennotiz-nr-147/ There was an automatic bot changing road names ending in ...strasse to ...straße (means ... street in German, second is the standard spelling). This did fail both in Switzerland (where ...strasse is the authorized spelling) and on the name Gleistrasse, which means railway track right of way and only contains conincidentially the substring strasse. There are probably more examples. They don't leave much space for obvious corrections that are without doubt justified. That's why the rule exists that mechanical edits are accepted unless somebody complains: If nobody complains then the edit was a posteriori a correction of the obvious. We have no a priori criterion for obvious correction. The rules for mechanical edit are frankly ridiculous. Have you read them. Our most valuable resource is not data but people who curate their share of data. Changing data in a way that might be considered harmful or is unintentionally outright wrong may shy away those who keep the data current. The sometimes rude feedback was identified as a probable cause for OpenStreetMap having few contributing women. So correcting those obvious errors requires communication with the mappers (male, female, or else) who have made these errors, in a way that always at first encourages them to carry on mapping (hopefully with less mistakes). On the other hand, a mechanical change of data can be performed as easy during postprocessing than in the database. This is known in programming in don't store an information when it is easier to recompute it. You may earn real fame if you have a good filtering ruleset that flatirons all suspect data. If you publish this as a postprocessing script, it is useful. If you apply that to flatiron the database, in 99% justified cases and 1% on otherwise on purpose crafted data, then you will earn shame instead, because that same script could be perceived as doing vandalism. It's potentially feasible to postprocess data. It's hard to collect data. So please don't make collecting data harder. Please make rather postprocessing data easier. Best regards, Roland ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Worldwide non-surveyed tag edits
Couldn't this be even worse than applying those changes directly in the database? The postprocessing refers to the final data consumer, not the map on osm.org. The map on osm.org is specifically designed for giving mappers feedback. Therefore, it has no such postprocessing and will never have. Best regards, Roland ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Worldwide non-surveyed tag edits
Lets pick 1 simple established tag and value combination. highway = residential Please go to taginfo https://taginfo.openstreetmap.org/ choose highway there, browse through the values and tell me which of the values you would like to change. Best regards, Roland ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Overpass API v0.7.50 almost done
Hi Martijn, thank you for the feedback. Also, do the attic queries already work fully when I use Overpass Turbo? No, that's the reason for almost. With much more intensive testing than before I've found a couple of bugs in corner cases, and I'm still fixing them. When I run [adiff:2014-06-12T20:00:00Z]; way(11027880); out; I do get an old and new section but they don't contain the deletion date. Please use out meta; instead. In the laguage design, out means to not provide meta data but only geometry, memberships and tags. Roland ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
[OSM-talk] Attic data on Overpass API
Dear all, the attic feature of Overpass API should now work properly. An example how to use this feature is http://www.openstreetmap.org/user/ikonor/diary/23329 A big thank you to Ikonor at this point. In detail, the database has been rebuilt. I tried to do as much checks as possible, and this has shown further bugs, which in turn led me to fix this in the code and then re-build the database again. A thank you to Stephan and Markus who managed to obtain a temporary powerful server; this ultimately enabled to do the last database rebuild within less than a week. I will give details about the kind of bugs that kept me busy: To keep the database as small as possible (currently more than 400 GB are painful, compare this to 25 GB of a PBF planet file), we store attic data as delta to the next newer version. Unchanged details like not changed tags aren't stored at all. For example, the tags of node version=3 lat=50 lon=10 timestamp=2011-01-01 tag k=name v=something/ tag k=highway v=bus_stop/ tag k=bus v=yes/ tag k=FIXME v=check_name/ /node node version=4 lat=50 lon=10 timestamp=2012-01-01 tag k=name v=something else/ tag k=highway v=bus_stop/ tag k=bus v=yes/ tag k=shelter v=yes/ /node are stored as current: name = something else current: highway = bus_stop current: bus = yes current: shelter = yes attic: before 2012: name = something attic: before 2012: FIXME = check_name attic: before 2012: shelter = void This allows us to save space for the repeated tags highway and bus. The main traces of the bug were checksum disparities in five of the 6000 checked first augmented diffs, spanning roughly 12 September to 16 September 2012. In detail, the diffs generated from the database state as of 1st October 2012 (thus, representing deltas to the then-current state of 1st October 2012) were not consistent with the diffs generated for the same minutes based on deltas to the database state as of June 2014. A detailed re-generation of the augmented diffs of both database states has shown that an extra tag was present on the node 1700083447 in its attic state of September 2012 computed from June 2014 in comparison to the same node at the same attic state computed from the database as of October 2012. Do you have guessed what has gone wrong? Me not so far, so I had to understand the subtle details. There are millions of nodes carrying tags, so what is so special about this one? It turned out that the node has moved forth and back over a significant distance, see versions 11, 14, and 15. While the movement itself is not so large (about 2 km), it happened to change its quadtile index to a new value and then back to the old value. And in version 13, after the move forth, the tag is_capital=country has been added and not changed when the node moved back. I managed in the delta computation to set a marker on the quadtile index of the older position that the tag is present, but I forgot to set a marker on the new position that the tag isn't present on earlier versions on this quadtile index. Now: Why didn't this pop up earlier? Haven't there been tests? There have been tests, but obviously not enough, and you always only know afterwards which tests have been missing. The whole problem doesn't appear to a node when the node never existed at this place before (only 3 in a million of nodes ever get back to a quadtile index where they have been before). And it doesn't appear either when the tag had any value on this older node versions (so the total number of affected nodes is less than 100), because there is then a marker for this older version and this specific key. Both cases have been tested individually, but not both together. In total: I've worked to get the number of bugs down, and I'm confident to call the database state now consistent, but there might be other arcane bugs that affect only few objects in specific versions. So please be bold to report suspect query results to me, but be also bold in using the new attic database. Cheers, Roland ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Proposed mechanical edit to convert alt_name tags
Hello Andrew, First of all, thank you for discussing the issue. Let's have a look about the numbers and distribution: For objects that carry a ; within their alt_name tag, we find nodes: almost 3000 ways: almost 3000 relations: almost 400 (see http://overpass-turbo.eu/s/4ZM and similar) and in each category they are distributed all over the world. By contrast, objects that have a key starting with the string alt_name_ exist nodes: about 40 outside Western Africa ways: about 50 rels: currently 6 (see http://overpass-turbo.eu/s/4ZL and similar) and a large number of nodes inside Western Africa. This underpins the two theses: 1. The solution with ; is far more established than the solution with prefix alt_name_. 2. It is altogether a sometimes but not often used feature. The first point means that enforcing an automated edit will embarass quite a lot of people or at least require a lot of personal talk. Altogether, it is almost for sure a no-go. If it is really worth to you to spend some hundred hours on it, then the next step would be to contact all mappers that have been last editor of the object, the editors that have introduced the tag and all editors in between. They can tell you which tools may rely on this syntax. The second point means that you have to expect a problem that often harms wiki proposals: It is more likely that people having time to discuss general topics will answer than people that have knowledge about the tag. Hence, whether the change will or will not cause outrage is quite unrelated with the response on this list (or any other communication channel, or even a combination of common communication channels). If you don't believe this then please verify that I never have touched an object in question, but I do discuss in this thread. Thus, a far more community-friendly solution than an at least controversal mechanical edit would be to just use the alt_name_ prefix. It may annoy other people, but it is then their uphill-battle to convince you to change your toolchain. As breaking compatibility is deprecated for a good reason, I would ask you to explain then why you have chosen this approach. This could be easily explained if their are objects that cannot be modeled with the semicolon approach, or if you have tools that won't work with the semicolon approach, preferrably for more profound reasons than that it is just not implemented. Altogether I appreciate that you have seeked communication and I would like to ask to you to get familiar that multiple approaches to a problem may exist in parallel in OpenStreetMap. Cheers, Roland ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Overpass turbo Has it been amended?
Ok, so you recommend to use QL instead of XML. Is there some feature that are in QL and not in XML ? The XML will be maintained for how long ? and quickosm should switch from XML to QL in the short term ? Currently, all features are available in both XML and QL variants, but the most recent additions have only been announced and documented for the QL language. I don't think that the XML language will be dumped anytime soon. Maybe Roland can tell us more about his long-term plans about deprecating the XML language. Thank you for bringing up this question. Acouple of people have asked me in personal communication, but I haven't made so far a public statement to bring clarity. Both languages have unlimited support. The software is designed in such a way such that the XML syntax makes no extra effort. In fact, the names of the statements are the names of the internal classes, so the XML support is tightly integrated. QL makes a little bit more effort with a dedicated parser, but that language has a couple of advantages: It is more briefly, which simplfies to write and publish code of it. Secondly, most programmers are nowadays familiar with C-style syntax, so there is less learning effort. The probably most striking aspect might be a cultural. Forced updates are amongst the things in the software world I hate most, so I would not deprecate things unless I'm absolutely forced to do so. The program code by design doesn't require deprecation, so it is unlikely that I would deprecate one of the languages soon. Another issue is documentation. I'm lagging behind a lot in this regard. And to catch up more quickly with incoming minor fixes and extensions, I would prefer to write the documentation for QL only in the future and publish for the XML syntax just a translation help. Best regards, Roland ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
[OSM-talk] overpass-api.de: Emergency rollback
Dear all, the Overpass API instance on overpass-api.de will receive in a few hours a data rollback to 22nd Oct 2014. This means a shutdown for two to three hours. Then it will catch up from 22nd October to recent data. The other instances on - http://overpass.osm.rambler.ru/cgi/ - http://api.openstreetmap.fr/oapi/ aren't affected. They will continue to deliver current data. Recent attic data will not be available for some days. I'm sorry for the inconvenience. However, attic data before Oct 22nd should be consistently available also during rollback. Details about what most likely happened: On saturday morning I've made a software update to version 0.7.51. As there was no change in the database format, the change went smoothly and all indicators looked fine. However, I've wrongly configured the dispatcher for areas to also care on meta data. Once the areas dispatcher triggered the first update of areas, .i.e. a few hours later, it corrupted the meta data, in particular the *.idx files. The processes have run with dispatcher --osm-base --db-dir=/opt/ssd/v0.7.50/ --attic dispatcher --areas --db-dir=/opt/ssd/v0.7.50/ --attic They should have run with dispatcher --osm-base --db-dir=/opt/ssd/v0.7.50/ --attic dispatcher --areas --db-dir=/opt/ssd/v0.7.50/ At this point I would like to thank the people that have complained. This gives me the impression that a fast and prospectous resuce attempt is better than a lengthy investigation without meta data. Given the wrong parameter was the cause, future software versions will be protected about these kinds of wrong parameters to the possible extent. I'm sorry for the service disruption. Best regards, Roland ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
