Re: [OSM-dev] Automatic generation of short_name / alt_name where missing and obvious
Is there some existing code/library/tool for generating obvious short_name / alt_name from other tagged data? Specifically for shortened names, this list of common abbreviations could help: https://wiki.osm.org/Name_finder:Abbreviations Thank you for the pointer. At least for German (and for French as far as I can judge) please take it with a grain of salt: - The by far fast common practices to shorten names are to resort to KFZ-Kennzeichen (district codes), i.e. "K-Süd" is the short_name for "Köln-Süd" - In 95% of Germany (everywhere except Berlin), the abbreviaton of "Bahnhof" is "Bf" - the suffix "straße" resp. "Straße" is shortened to "str." resp. "Str.", everything else is uncommon - French street names are shortened by entirely removing the generic parts, i.e. "Place Victor Hugo" has short_name "Victor Hugo" I would not be surprised if the other languages have regional or personal biases, too. Cheers, Roland ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: A functioning Public Transport plugin
Hi all, thank you for the feedback. Furthermore, this plugin is closed-source (as far as I know), so it cannot be "fixed". The plugin is open source, see https://github.com/openstreetmap/josm-plugins/tree/master/public_transport The problem is that maintaining the plugin is a lot of work. I abandoned the development long ago because public transport v2 would have meant too much work, because the scheme has inherent flaws. Any such flaw does fall on the developer multiple times, for implementation, for developing test cases for all the undefined corner cases, for a UI that explains what the software actually does. By contrast, updating to a single different set of tags for stop poles is not a substantial problem. This would mean you could immediately add all the stops in the click of a button, and sort broken relations in a click of a button. Since writing this plugin, the relation editor has superseded most of the way sorting features. Thus, it no longer makes sense to duplicate the sorting capabilities in a distinct plugin. I would nowadays add buttons to the relation editor rather than a separate relation editor. There is also an unfinished routing algorithm in the plugin. I never had found a reasonable UI to exhibit that to the end user. Best regards, Roland
Re: JOSM development in the last year
>> - 54% Windows, 37% Linux, 9% MacOS (Linux numbers are going up...) I'm surprised by the Windows/Linux ratio. I always noticed a quite stable ratio around 70% Windows, 20% Linux, 10% Mac (+/- 5% margin). Isn't just that more Linux people used JOSM during the holidays than Windows people? Such an increase looks strange otherwise. It might be Oracle's license conditions. In the corporate environment, using the Oracle JRE requires a commercial license. Installing the JRE of OpenJDK is possible and JOSM runs with it, but counter-intuitive in multiple steps. I myself have written instructions for my collegues how to install the OpenJDK. Given that Windows is in particular popular amongst companies and users with limited tech competence, that new threshold might have scared off users. Best regards, Roland
[OSM-dev] Overpass v0.7.55.7 fixes vulnerability
Hello everybody, a new update of Overpass API is available. As this fixes a security issue, I strongly encourage you to install the fix right now. The release is as usual available via https://dev.overpass-api.de/releases/ resp. https://dev.overpass-api.de/releases/osm-3s_v0.7.55.7.tar.gz The public servers have already been updated. The issue is XSS, i.e. you can place arbitrary HTML such that it appears to originate from the Overpass server by sending a crafted request to the server. No personal data has been leaked because Overpass servers do not process any. No attack in the wild is known so far. Details will follow in a couple of days. I would like to thank the people that have reported the vulnerability. Best regards, Roland ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Talk-GB] OSM augmented reality project - affordable hosting recommendations or Overpass?
Hi,
As an alternative, I was wondering how acceptable it would be to use the
Overpass API to obtain the data? Downloaded data would be cached on the
device so for a given area, data would only need to be downloaded once.
I'm fine with such a usage. The fine print is about other issues:
- Overpass API does support GeoJSON indirectly, but GeoJSON does not
support EPSG:3857, see
https://tools.ietf.org/html/rfc7946#section-4
To get GeoJSON I suggest
[out:json];
way(south,west,north,east)[highway];
convert link ::=::,::geom=geom();
out geom;
where (south,west,north,east) is the bounding box.
As an act of courtesy I suggest to set the "Accept-Encoding: deflate,
gzip" header and use
[out:json];
way(south,west,north,east)[highway];
if (count(ways) < 2)
{
convert link ::=::,::geom=geom();
out geom;
}
else
{
make error what="Too many ways in this bounding box";
out;
}
This compresses the data and bails out if there are more than 2 ways
in the bounding box, corresponding to between 1 MB and 2 MB of data.
Overpass would happily deliver about 1 GB per user and day, but the
users may have data plans with rather 1 GB per month.
Thanks,
Roland
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev
Re: josm Keyboard input stops working
Hi all, Look at https://josm.openstreetmap.de/ticket/13160 For me its dead simple to reproduce ... Still exists in current josm with openjdk 8 and 11. I am on Linux but others have reported the Problem on Windows. I had the issue both on Windows and Linux (Windows 7 64Bit, Ubuntu 14.04, and Ubuntu 16.04). It did never materialize in any reproducible way, sometimes after barely a minute, sometimes not on session over multiple hours. At the moment, everything is fine (on Windows 7 and Ubuntu 16.04). This includes the setting that you have described. Given that the JOSM core developers might face the same situation, I suggest to collect as much information as possible, in particular the Java environment, the active plugins, all their version and so on. Could you check whether the problem persists even on a JOSM without any plugins? Best regards, Roland
Re: [OSM-dev] How to find the way with the most relation memberships
Hi, https://overpass-turbo.eu/s/BpE shows that this has happended already 2015 and in changeset 33711981. It is notable that - the user made otherwise many useful contributions - there are two version fo that relation within the changeset, only 6 minutes apart This makes it most probable that the editor (an old Potlatch version) had a bug, and that the bug most likely had been fixed long ago. The other affected relation 5328067 (replace twice in the query) is from the same changeset and has the same number of duplicate members, which also is well in line with an editor bug. Best regards, Roland ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] GDPR implementation on planet.osm.org
Hi,
brief and frank: The suggested way that users of Overpass API have to
sign up as OSM users would cause a downtime of some months and a
development backlog of more than a year, or kill the project entirely.
Because this sounds harsh, I will explain that further down.
The key point is: please do not bind information intended for data
processors to OSM user accounts.
The only alternatives I can see would be:
1. Stop distributing who-did-what-when information
[...] it would create a privileged class inside OSM
[...] 2. Take the view that distributing the data is what we do and tough
luck, you've signed up to it.
As Simon has pointed out there is another alternative. And I have
understood so far the OSMF that way wanted to follow that way:
as has been outlined before, 3rd parties using OSM data with user data
will be acting as independent data controllers and will not be
processing data on behalf of the OSMF (which would require a DPA and all
the associated complications). They will have to make their own
determinations on how to deal with the situation. We will provide some
support to such entities to help them fulfil their legal obligations
(for example a list of deleted users), but that's it.
In particular, data processors do a much better job if they do not deal
with OSM accounts at all, avoiding having and triggering extra
who-viewed-what data.
Most important, privacy relevance varies heavily with context. Hence I
will and should inform users about different risks than the OSMF, and
HDYC may again decide to stress other aspects. A central ToU cannot do
that. Also, for that reason the GDPR is a law and not a suggestion for a
contract, and the OSMF decided to handle it as such.
To give an analogy, think of blades. It is forbidden by law to injure or
kill someone, and blades of any kind do pose a risk. Kitchen knives can
be used to stubb someone, but nobody every got stubbed with a kitchen
blender. By contrast, user may harm themselves when using a kitchen
blender. For that reason, you should be informed about the blades in the
kitchen blender's manual, but no knife salesman in the world would
require you to sign a contract not to stubb someone else with the knife.
Conversely, giving too detailed information what approaches of stubbing
are physologically risky and which are harmless could be abused as
how-to-stub instructions.
Taking GDPR serious means every data processor must decide which use
cases they make simple, which use cases they make hard, and tailor the
documentation according to that. For example, for that reason Overpass
API has no feature to track all actions of a single user. I have
proposed a declaration tailored to Overpass API on the FOSSGIS list (the
FOSSGIS is sponsoring the server operations), and I would prefer to go
forward with that one. A central ToU does not help, hence having it
ticked or not is of no interest to the data processor.
Then there is the problem that regardless whether you expect that OSM
users will read or just tick the box, you have downsides:
- If you expect that users do read the ToU then we will scare away users
that just signed up to fix a POI and find themselves scrolling through
pages of legalese on a mobile phone
- If you do not expect that users read the ToU then the bad guys in
particular won't do, and no judge ever would count that as an
appropriate technical protection of data
In addition, this is stealing users' attention from more important
matter. Our contributor terms have quite some content and that so for a
reason.
On the technical side, things are even worse. The elephant in the room
is OAuth. OAuth is built on in particular the assumptions that
- the consumer ("the website") acts stateful
- sessions are relatively long-lived, i.e. some seconds to some hours
- the identity provider has the cross-origin assets
All three are not true for Overpass API which means that I have to work
around OAuth or significantly mess with it.
For example, implementing to have sessions on Overpass API will require
to develop a full-fledged security system to deal with the hundres of
potential modes of attacks on session based systms. Even if that works,
the median runtime for a request on Overpass API is well below a second,
and just the roundtrip times for the OAuth threesome communication sum
up to more. We have not even started to talk about the plethora of error
messages that need to be formulated, explained, and implemented.
On top of that, the OAuth idea means that each and every sequence of
user data access will trigger an event on the central OSM OAuth server.
This is quite Orwellian. Even if you do not store that information, your
friendly agency of choice will do so on the line that connects the server.
Additionally, if you monitor "independend processors" so closely, it is
questionable whether they are not seen as disguised contractors by a judge.
I can live with the
[OSM-dev] Changes on planet.osm.org
Hello everybody, given that the GDPR is going into effect tomorrow and there have been plans announced to restrict the minute diffs: Whet is the state of this? Is it sufficient to send HTTP basic auth (via HTTPS) with a dedicated user from tomorrow on to continue consuming the diffs (with metadata)? Or are there specific requirements, like using OAuth? Or is there simply no change tomorrow, and the restriction will be announced separately? What I would like to avoid is to poison the client side database with fake user data. Best regards, Roland ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Working with OSM data with less or no metadata
Hi, - timestamps however cannot only potentially be used in lieu of changeset ids to group contributions, the information itself is problematic because it allows to profile contributions over time Timestamps are necessary to correctly figure out which nodes have belonged to a certain version of a way, and similarly for ways and nodes belonging to relations. More generally: - What is planned with regard to minute diffs? Stripping extra information will inevitably break tools like Achavi - Tools will need substantial time (I would estimate 3-6 months for Overpass API) to adapt in a meaningful way. What is the schedule of the LWG to take decisions? - How about simply asking the users for consent? We could then -- make a clear-cut last complete history dump before the date -- start with a planet dump without history before that date afterwards that then accumulates history only from users that have given consent Personally, I would prefer a solution as easy as dropping usernames and uids but retaining changeset ids, timestamps and the geometry/tag data. That way we display goodwill, but do not cripple the tools that have proven useful or crucial to run the project. Please note that in the context of an API without user interface, it is a substantial challenge in itself to have any form of (OAuth or so) authentification. Cheers, Roland ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] overpass query question
Hi Jason,
In addition to mmd's answer, you can let the intersection take place in
the element queries:
query = """[timeout:25];
(
area[admin_level=8][boundary=administrative][name="{0}"] INTERSECTS
area[admin_level=4][boundary=administrative][name="Massachusetts"]
)->.searchArea;
(
way["landuse"="conservation"](area.searchArea);
relation["landuse"="conservation"](area.searchArea);
);
(._;>;);
out body
can be realised as
[timeout:60];
area[admin_level=4][boundary=administrative][name="Massachusetts"]->.state;
area[admin_level=8][boundary=administrative][name="{0}"]->.muni;
(
way["landuse"="conservation"](area.state)(area.muni);
relation["landuse"="conservation"](area.state)(area.muni);
);
(._;>;);
out body;
An example with Worcester: https://overpass-turbo.eu/s/uaC
Best regards,
Roland
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Querying for non-native characters in name field
I want to be able to do an overpass query for Iceland where name= field contains non-Icelandic characters. These could be for example Chinese, Cyrillic or even other European characters (such as âà for example). I'm guessing it could be difficult for the latin characters but hopeful it would be easier for non-latin alphabets. Is there a magic formula for achieving this? I suggest, as a refinement of Ilya's query, this one: http://overpass-turbo.eu/s/lCk As it may help for other languages, I explain how I got to this: 1. Start with area["name:en"="Iceland"]; node(area)[name]; out count; This is basically an all-nodes-in-Iceland-with a name. The important part is the "out count". This assures that you are not flooded with results. For the same reason it is enough to start with nodes: We do not want a final result now. But we want to create a senstive search term. For this reason, we will even get down to just a subset of all nodes in a second. 2. Clamp down to area["name:en"="Iceland"]; node(area)[name~"[^a-zA-Z]"]; out count; These are all nodes that contain at least one character different from a latin letter. These are still many. Therefore: 3. Get examples with area["name:en"="Iceland"]; node(area)[name~"[^a-zA-Z]"]; out 100; This prints some random 100 results (in fact: the 100 matches with lowest node id). Now we can look at the name fields and get an idea what we would like to exclude in addition. 4. Start to narrow down with area["name:en"="Iceland"]; node(area)[name~"[^a-zA-Z0-9 ]"]; out 100; Spaces and digits are OK even before we start to accept all the special characters from Icelandic. This process is now repeated until the sample contains no more false positives. Finally, we expand this to all three types of OSM elements, in the expectation that not much false positives appear. Cheers, Roland ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Overpass-API 0.7.52 : HTTP 429 error even after kill_my_queries
Hi all, It seems that GET kill_my_queries request won't prevent oAPI to give HTTP 429 error. thank you for reporting the issue. I've made an extra API call to improve the situation. To faciliate documentation, I've written the details at the overpass developers mailing list: http://listes.openstreetmap.fr/wws/arc/overpass/2016-06/msg7.html Best regards, Roland ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Minute replication hiccup
Dear all, something strange has happened to http://planet.osm.org/replication/minute/001/788/263.osc.gz The overpass-api.de (and probably other servers) have received a version with 3322 bytes on 2016-02-12 02:25 UTC, but the current version is much bigger. Subsequent diffs will miss OSM nodes and ways if you try to work based on the older version. To get out of the problem, I have stopped the minute update process right now. I'll replay the clone from http://dev.overpass-api.de/clone and re-apply the updates again, with the newly downloaded http://planet.osm.org/replication/minute/001/788/263.osc.gz Best regards, Roland ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Output relations from Overpass API geo analysis
Hi all, Example : 2 nodes inside 1 way (total : 6 nodes). Output : A relation without any OSM id with 3 members : the way (role=enclosing) and 2 nodes (role=inside). I'm sorry, there is no such feature at the moment. I'll put it on the list of requested features. Best regards, Roland ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Acceptable use of Overpass API?
Hi Nick, However I don't want users to have to download a huge OSM or GraphHopper file covering the whole of England and Wales: I would prefer users to download small tiles of OSM data (say 10km x 10km) of their local area, which could then be cached on their device. The conversion to GraphHopper format could be done on the device. It is perfectly well within the usage policy. The quota, by the way, doesn't apply here because it is per IP address, not per appplication. Such a tile is (gzip compressed) usually between 100 KB and 1 MB each. Thus, a user trying to download 1 tiles would anyway exceed his mobile plan for the month. furthermore I could route requests through a proxy on my own server which ensures that no more than 5000 (to be safe) requests to Overpass per day are made. While I appreciate that offer, the direct access is not a problem and an advantage in terms of quotas per IP. What I would rather want to ask you is to include a user agent. I addition, both your users and I would be grateful if you can send the header Accept-Encoding: gzip, deflate That will save bandwidth in particular on the mobile plans (and also speed up handling on the server side). Thanks, Roland ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] New overpass development mailing list
Hi, is there any English interface to subscribe or read the archive of this list? Yes, look at the lower left corner and select English (or even German). Or this there any other reason why you did not let the list being created at lists.openstreetmap.org? Please see https://github.com/drolbr/Overpass-API/issues/198 Best regards, Roland ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Overpass ql problem using around/radius
Hi, thank you for the feedback. This is a perfectly valid enhancement of the software. Could you please open an issue at https://github.com/drolbr/Overpass-API With a bounding box, this works: [bbox:51.2911,-2.4750,51.4435,-2.2219]; ( node [leisure=park]; way [leisure=park]; relation [leisure=park]; ); out body; ; out skel qt; But if If substitute the first line for around: [around:15000,51.38138, -2.35968]; it throws an error: *Error*: line 1: parse error: ']' expected - ',' found. Is this expected behaviour? Yes. There is no global around feature so far. This is for a couple of reasons. The most important is that around is currently painfully slow, in particular a lot slower than bbox. I would like to improve the performance first to make this hypothetical feature run within an acceptable time for the user. Best regards, Roland ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Mapnik-Glitch Schwarzwald
Hello, Although the rendering stack in general works excellent, there is a strange glitch near Stuttgart in Germany: http://yevaud.openstreetmap.org/11/1076/704.png contains the string Schwarzwald (which doesn't belong there) but there is no element with a tag of any key with value Schwarzwald anywhere near, c.f. http://overpass-turbo.eu/s/64L The phenomenon has sustained a re-rendering, with rendering times from Sat Nov 15 19:03:44 2014 and Sun Nov 16 01:45:21 2014 Can somebody please explain where this Schwarzwald comes from? Or give a hint where to look next to understand the glitch? Best regards, Roland ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] JOSM plugin public transport update (Google Summer of Code program)
Hello Markus, I'm sorry for the delayed answer. I'm currently preparing my talks on the FOSSGIS conference (German version of SotM) and this took much more time than expected. I thought about algorithm guessing connection type based on: - common tags on ways that contain selected nodes This is really smart. It would avoid most of the time bothering the user. - what type was previously selected (i.e. if a user had selected a type X in the past, then this selection will be assumed as more likely the next time) Also a good idea although I would see this rather as choosing which item to highlight. Some types would be predefined. There would be possibility to edit existing and add more. A connection type would have: - name - icon (not sure, requires testing whether it would be helpful) - set of data about tags that would identify ways that can be included - tags that must appear - tags that indicate this route type - tags that should not appear and indicate that it is not this type - tags that must not appear Additionally, features with certain tags could and should block routing even if they just cross the path without sharing a nnode. Think of a gate (barrier=gate or a military no-go area). I like this idea, but is it big enough to qualify as entire project? Maybe yes, especially after considering that things often are harder than expected and take far more time than it seemed at start. Yes, it is big enough. It is more more meritful to arrive at a useful piece of code first and then do one or more exciting features on top than a monolithic goal that could (and then, by Murphy's law, will) fail completely. In this project I see following parts - creation of ticket on JOSM bugtracker (I am not expecting this, but in case that JOSM developers reject the entire idea I would instead create plugin with this functionality). - collecting use cases (two nodes + what ways would be expected to be found by algorithm) - setting up development environment, with ability to build JOSM - interface (new position in tool menu, icon, fetching data about what is selected, error messages) - algorithm that will find suitable connection (probably A* with cost functions) - Deal with various tagging variants. - Deal with features intersecting without nodes when they should block routing. - Make a proper handling of implicit way splitting (may require user action, but the less the better. Users wouldn't accept to click through hard-to- understand and possibly multiple modal windows). - Make a proper Undo (This one being particular important) - Maybe cope with runtime and/or space constraints Subsequently I would add handling of connection types - fixing bugs reported by user, encountered in released version - modification of algorithm (multiple A* searches? Maybe something smarter and faster would be needed.) - interface (selecting connection type) - loading user-defined connection types in reasonable and editable format - definitions of some obvious types - Making a user interface to handle user-defined connection types, might be a quite large sub-editor). - Fitting that into the various JOSM storage ideas (File format, storage in the central JOSM wiki repository, store in local user preferences). - Probably i18n for the interface parts. - Find sensitive testing case and defend the whole thing on various community mailing lists (I will assist you, but it is inevitable to meet the real needs of various local comunities, and inevitable to make the software really useful). At this point it should be already useful for many tasks and faster than configuring filters to select linear feature consisting of multiple ways. It would depend on how JOSM developers would see it, but it could be at this stage merged into JOSM trunk (in case of backup plan I would release a plugin). I think it would be helpful to first have it as a plugin and see it in action. This allows to proof it has proper encapsulation for the other code parts. That has proven in the past very useful for code maintenance and is therefore common for new and significant improvements). Is there anything that I missed here? Is my plan realistic (GSoC coding starts on 19 May and finishes on 11 August)? Yes, I'm confident that this will work. I'll have a closer look at your proposal in the train tomorrow and then comment as soon as I found my Wifi on the conference venue :) Cheers, Roland ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Problem with XAPI and ref based search
hello everybody, I start by asking XAPI to give me the France admin_level 2 object like so: http://www.overpass-api.de/api/xapi?relation[boundary=administrative][admin_ level=2][name=France] this returns me the object, which contains among other things: member type=relation ref=79981 role=outer/member member type=relation ref=2202120 role=outer/member member type=relation ref=2627308 role=outer/member member type=relation ref=2177258 role=outer/member Please try recurse down: http://www.overpass- api.de/api/interpreter?data=(relation[boundary=administrative][admin_level=2] [name=France];;);out+meta; The XAPI syntax is restricted to very simple use cases, and even the question for all members of a certain relation cannot be formulated. So the solution is to use the more comprehensive Overpass QL language. The line relation[boundary=administrative][admin_level=2][name=France]; searches for the relation (the same way as before) and ; adds all members, members of members and so on of the last result. This gives the full pyramidal boundary construction. If this is too much data (these are more than 110 MB), you can resolve on step with http://www.overpass- api.de/api/interpreter?data=(relation[boundary=administrative][admin_level=2] [name=France];rel(r););out+meta; Here, rel(r) resolves only one level of relation members. Best regards, Roland ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Overpass API around query with lat/lon not possible in QL?
Note that the lat/lon values are missing, so this query does not work. Is this a bug in the converter or does around in QL really not allow to specify a location? No, this was a bug in the converter. I have just fixed it, and the code will become live with the soon upcoming version 0.7.4 during this week. Best regards, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Expiring Tiled OSM Data
I'm looking at the augmented diffs stuff but the documentation [0] appears to be stale. Thank you for pointing me at this. The documentation was totally out-of-date. I've rewritten it to reflect the current and stable state of affairs. Can you describe what the waymember and indirectmember attributes are describing in the XML? They're not mentioned on the wiki. Thre flag waymember=yes is added to the action tag of a node if the node is a member of a way that appears in the diff. Similarly, the flag indirectmember=yes is added to the action tag of a node if the node is a member of a way that is in turn member of a relation that appears in the diff. In both cases: If the node itself has changed somehow, all ways and relations it belongs to are listed. If the node is unchanged but present because it is member of a changed way or relation, the respective way or relation is present. The only downside are relations that have relation members. These relation memberships are assumed not to contribute to the geometry and therefore do not appear if only their members have changed. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Expiring Tiled OSM Data
Indeed. Although there's probably a much clever approach than the one I'm suggesting. Basically, the Augmented Diffs comprise exactly that information. They carry for each changed object the geometry information, either directly or by referencing the also in the file contained nodes. From that it should not be too hard to mark affected tiles as dirty. I'd suggest the id-sorted variant with info=yes. It is bigger, but the nodes are required to get the geometry for relations right. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] The Overpass Augmented Changesets API
Hi all, http://www.webmonkey.com/2013/04/watch-openstreetmap-improve-in-real-time/ Cool! I could get addicted to sit down and watch every time again. A great thank you to Ian. So: questions. Thank you for the hint. I have added at least some documentation at http://wiki.openstreetmap.org/wiki/Overpass_API/Augmented_Diffs#The_augmented_diff_API_call that answers the questions below: What does `info=yes` and `info=no` mean? Boundary relations tend to bloat the Augmented Diffs: whenever a change somewhere on the boundary happens, even a minor one, the nodes and ways of the entire boundary are transmitted. This makes in turn the file one or two orders of magnitude bigger, without high profile information gain. The biggest one I have observed so far had more than 50 MB. With info=no, you can boil down file sizes but keep the geometry information for ways and nodes always intact. Does the BBOX argument work, and what argument order does it accept? The OpenLayers order (lon,lat,lon,lat). Are there any other querystring arguments that might be useful - for instance, a limit argument would be wonderful, since right now the API returns 'all items', which is around 3.5MB gzipped per user per minute. No, not yet. But I'm open to ideas although it may take some time until I can implement them. In the long run, I would like to offer full history information on Overpass API, and the then-developed Diff format would be the format for the diff between the same query at two arbitrary moments of time. But this may take half a year or longer until I arrive there. This is the reason why I'm keen in getting the format and API as useful as possible. For the limit argument: What should be the order to determine what to cut? Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [josm-dev] function to find nodes next to a way
Hi, I would like to be able to find nodes with certain tags next to a way, within a few meters (configurable). Please have a look at the public transport plugin, in class RoutePatternAction, at lines 1806-1819. This does exactly the desired. I agree it would bencessary to refactor the code. I'll be happy to answer questions for it. Cheers, Roland ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [OSM-dev] Is there any API to get data from OSM as geoJson?
Hello Stefan, That's what I conclude from [1] i.e. osm-script output=json ... /osm-script The format option JSON meets the format described here: http://wiki.openstreetmap.org/wiki/Xappy.js This has exactly the same semantics like OSM XML and only different syntax. However, JSON makes some JavaScript applications a lot easier than XML, so I followed the suggestion to add it. What makes me hesitate about GeoJSON is essentially that there is no clear rule whether an elements becomes a linestring or a polygon. Solving this will require both a test for validness (e.g. self intersections, but also some other corner cases. Most are listed on http://wiki.openstreetmap.org/wiki/Multipolygon#Valid_Multipolygon_conditions ) as well as a semantic interpretation of tags. The most likely mid-term solution is to convert ways always to linestrings, but the area type to polygons. This has the advantage that both the code to validate polygons and the choice of tags is done by code that does this anyway for area creation. I will code this proper together with a area output implementation conforming to Jochen's outline http://blog.jochentopf.com/2012-11-26-an-area-datatype-for-osm.html so it will take some time until it gets available. I expect it is complete some weeks before the FOSSGIS conference, together with a couple of other larger changes. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Reverts from the woodpeck_repair account
Hi, The second problem is the automated editing. Perhaps now as OSM becomes more and more popular it is time to start looking at some more general solutions to these kind of problems with data and bots. The solution is simple and straightforward: A database design must be able to cope with the the edits that are uploaded. If you don't consider all edits valuable, you are free to drop data in the target database. By contrast, any additional decision logic in the main API does really clutter OSM because the main database as sigle point of failure gets more prone to errors. More important, it may shy away mappers if they found that their particular situation on the ground cannot be mapped properly. SomeoneElse got his work damaged by other mappers, for no important reason. A less self-confident mapper may have been lost at that point. That's the reason why we have freedom of expression in tagging and mostly in producing changesets. Data consumers still have all freedoms to process the data to any rules they may find suitable, but no harm is done to the community or the core database. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Fwd: OSM item tracking
Hi, I think you will need several tools, because the posed questions aren't solved all by the same tool. My use case is: a school district has a list of 15 schools. They have all the updated information. They keep this information in a website to show to people, an they want to share it in ISM and keep it updated in OSM. How can they be notified of a change? This requires somebody who regularly looks after the reports about potential changes. A tool that can watch for changes is for example http://www.openstreetmap.fr/~zorglub/watch/ Ypu can be bold and select all schools in a quite big bounding box around your district - these will still be less than one change per day on average. How would they make a change on their website and then sync that data back over to OSM? The best way to make a change in OSM is really to use an editor and do the change. All automatic solutions have problems. The Permanent ID tool for example, is deliberately designed only to link to objects in OSM, not to feed data there. Best regards, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [josm-dev] route handling (was: Find bus stops along a way)
Hi, I have been looking at that and I'll have to look again as it seemed quite complicated :-) Yes, it is. Unfortunately it needs a lot of refactoring. It is even not the most efficient approach, but I simply went short of time. I did see that you only use highway=bus_stop for bus routes, but I'm slowly but surely in the process of converting all of them to public_transport=platform. That's a good idea. I think at current state of affairs it should take into account both highway=bus_stop and public_transport=platform. Feel free to update the code. I have created a Python script which does quality control on numbered node cycle and walking routes, which I'd like to extend to PT now. As a first step I'd like to do what your plugin does; detect stops and propose to add them to the route. Later on, I'd like to actually create a route relation based on stops that need to be served. (Which will, of course, need to be checked manually for correctness before it is uploaded) There is even code prepared for this in the plugin. A call to new PublicTransportAStar(Node start, Node end).shortestPath() will return a data structure that contains the shortest path found by an A*-algorithm, written as partial ways (because it may use ways only partial). What is missing here is an appropriate filter (don't route over stairs or other non-streets), which would go where the comment // Further tests whether the way is eligible. is. And the projection from off-road bus stops to starting nodes. And then to make sense of the result. In particular, the plugin has no code to split ways automatically, and JOSM won't mark partly ways AFAIK. Once again: feel free to work with the code. Given my OSM agenda, I won't work on the public_transport plugin code before the end of the year, but I will give any help to make it useful (or recycle the useful parts in a new plugin). Cheers, Roland ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] Find bus stops along a way
Hi, I want to find all bus stops along a side of the road. There is a function doing this in the plugin public_transport. Feel free to copy it. Cheers, Roland ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [OSM-dev] Downloading data via the Export tab
What would it take for Overpass API export to be listed in the export tab? Is it about service reliability? No, Overpass API is reliable and stable since some years, and the service is funded until mid-2015 by FOSSGIS. The EWG has recently discussed the matter. http://www.osmfoundation.org/wiki/Working_Group_Minutes/EWG_2012-10-08 The common sense would more towards a all-in-one change of the design of the export tab, and it is not yet decided what to include in what way. The Geofabrik extracts are also a very good candidate to be somehow integrated, and maybe other services. Best regards, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OWL + OSM Activity Server
I think it could be based on OWL however I'm a bit put off by the C++ part to be honest - I don't think the XML parsing and database operations part really needs to be in C++ since it is certainly not a bottleneck. I will try to speak with Matt and find out if he would accept changing the C++ part into Osmosis plugin (which is something he mentioned to me I think on IRC but said there were no hooks in Osmosis). If you are C++ averse, the good news is that you may resolve the real bottleneck independent of the programming language. I've written down some notes about the Overpass API implementation with similar scalability problems: http://wiki.openstreetmap.org/wiki/Overpass_API/Technical_details However, XML parsing (and gzip compression) are heavily CPU intensive. Thus, it might be a good idea to retain these in C++ or getting done in carefully chosen external libraries. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] What is the main API? (was Re: OWL + OSM Activity Server)
Not sure I buy into this dichotomy. Application level integration != user facing integration. Or in other words: it's thinkable to have features on openstreetmap.org that appear seamlessly integrated to the user but are powered from a separate tier - separate application or separate application on separate server. No? I'm sorry for the caused confusion. I was talking about the backend. In general, almost all clearly stated wishes are for new backend services. What I wanted to clarify: for scalability, realiability and maintainabilty it is much more desirable to have a specialized database, specialized API and one or more enthusiastic maintainers of that for each purose than to have a big multi-purpose main database. This applies even to services that look like they could be easily done on-the-fly by the main database, but in fact would create significant load, in particular changeset queries by coordinate. The user experience is a completely different story. A good user interface can and should integrate all APIs that are useful for its purpose. Yet again, the strength of OSM is that different user interfaces for the same API can exist. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] What is the main API? (was Re: OWL + OSM Activity Server)
This specific functionality seems like something we'd want directly supported in the railsport, no? Or is it so expensive that it has to be tiered out into its separate application? It is expensive. To do bboxing properly you have to look at each element in the changeset whether it is inside or outside the provided bounding box. This has roughly the same effort than downloading all elements in that bounding box. This export feature is so expensive that it is limited to small bounding boxes, delegating people to third party services for larger bounding boxes. On the other hand, you can implement it trivially on a third party service: just take the Augmented Diffs (which have full geometry), split it into small bounding box buckets (a tool for filtering bounding boxes, process_augmented_diffs, is at the Overpass API, another is in the osmconvert toolbox) and deliver the relevant stream to the user. I think the core problem is that there are two different concepts of main API around: There is the lean approach, or technical point of view, geared towards scalability: Obviously, we need a central instance for managing the id space, and that place can also seralize all edits to the database. This has good chances to scale, but means that anything beyond immediate editing support has to go off the main API sooner or later when it turns out to be the roadblocker at that time of scalability. And there is some kind of approval point of view, or marketing point of view, seeing tools with a .osm.org domain in some sense approved/superior/preferred or whatever. This comes from our usual experience with brands or large companies that differentiate the inside from the outside. It doesn't match neither the common sense nor the factual situation around the project, think of JOSM (which is not on osm.org) or the Geofabrik and CloudMade extracts and several others, but it is very straightforward. I tend to prefer the first point of view, but I accept that the second point of view is fairly often unchallenged told. From the first point of view, it is pretty clear that a history stream belongs to a third party tool. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] how to make way and relation history less expensive
Currently if you try to look at the history of a bigger way or relation (and/or one with many versions) it is most probable that you run into a timeout. A solution to this would be The solution to an API overload is always: use an appropriate third party tool. In this case for example: https://github.com/MaZderMind/osm-history-splitter If you want to make a good service to the community then please make the splitted files publicly accessible on some server. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Notes - OSM improvements BoF at SOTM PDX
As to other sites, the problems are several fold - how do we decide to bless one particular site over another? Please be honest. You, I, and everbody else also set preferences on at best educated guesses, often more trust in what or whom we know or don't know. We use OpenLayers and not Leaflet on the main page, we don't have routing on the main page although several services are realiable. We have certain tile sources and others not. Some services have an address redirect off .osm.org, others not, for no particular reason. 1. The simplest way would be a public voting. Ask - whether somebody does volunteer to do the administration even longterm - let the community give comments on the project and site and person 2. A more elaborate way would be to design a benchmark for the task and test how the service in question performs. Then publish and discuss the test results. 3. An even more elabortae way would to have a test installation, announce the service as beta, observe how usage develops and give a report after some months. I encounter 1. easily possible, and 2. with some enthusiasm. I don't think 3. is necessary, but it may be worth considering. In any case, the produced material and documents suffice for a diligent decision. not to mention the fact that we might end up driving more traffic to such sites than they could handle. Could you give figures what amount of traffic you expect? 1 GB per day, 1 GB per hour, 1 GB per minute, or even more? Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Notes - OSM improvements BoF at SOTM PDX
This would mean folding http://planet.osm.org/ into a future data tab, no? How do the planet osm maintainers see this? http://planet.openstreetmap.org is the interface from the main DB to all kind of data consuming services, including the renderers, Nominatim, Overpass API and several others. It has roughly the same importance than the main API and it is the place where minute and other replication streams are distributed in an efficient way along with the Planet.osm. As this is the data hub with appropriate user interface for the technical folks, I suggest rather not to bend it into something of minor importance like an Export tab on a website, even if that one easily visible to newbies. On the other hand, for somebody who is not yet familiar with the OSM data format (XML, PBF or both), the planet.osm.org is unlikely to be useful, so both are services with quite distinct audiences. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Notes - OSM improvements BoF at SOTM PDX
Hello everyone - ## Getting data out of osm.org - Export tab on osm.org is one of the most popular locations, but needs a lot of improvement. E.g. it does not explain how the downloaded data can be used. How can export be more actionable? - if an export fails due to its size, it's not clear where to go to get a larger dump. We could let various services let plug-in into the export dialogue and completely remove the raw data export (the rest works on some random tests). Implement some registration process on the server such that the UI makes a callback on the chosen service with the selected bounding box. Overpass API (and jXAPI if there is a running instance) can deliver much bigger areas and just takeover the given bouding box. - How can third party services like geofabrik's shapefile exports (or jXAPI) be tied in better? See above. Such services could also do conversions (e.g. to Garmin data, another map style, open it in JOSM or whatever) and thus offer more ideas what to do with the exported data. Best regards, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM Wishlist
Imagine there is list of schools in a city. This list is maintained by a school district office with updated website, phone numbers, and office opening times. They maintain this list through a simple PHP form on their website that allows them to make updates to each listing (node). If any node on their list is changed by anyone but them or the people listed as able to edit that form on their own website (with their OSM Ids) then they are notified and can correct or confirm acceptance of the change. They do not need to be the last editors of the node to know that the information contained therein is approved by them. Nobody can block somebody else from editing something. If you mean approve in this direction, it would pretty never happen. However, the school district can fork the Planet into an own database and move newer edits as act of approvement to its database. This is a typical use case of the idea of federated databases. No work has been done so far, but the point * Tools to move, merge, and diff objects between different OSM databases is surely a good point for the tools list. The federated databases will also solve most issues around imports, including patching an updated import without damaging exisiting data. The second approach is to get a concise report of all updates that happened. It is then up to the stuff of the school district to rollback or accept the edits. Concering tools to watch updates, a lot of things have happened very recently. However, none of these tools work perfectly for the school district case yet, but this will significatly improve in the next weeks. Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] (no subject)
Hi Toby, The way was added to the relation in version 51 at 2010-06-20T19:29:34Z and then the way was deleted one second later at 2010-06-20T19:29:35Z. In theory this should not be possible. However this edit was done in Potlatch 1 in live edit mode and according to RichardF on IRC, there were some race conditions between Potlatch 1 and the API that could be to blame. It seems like this is the most likely explanation for this data. After running a query oh my local database I think there are a total of 208 relations which contain ways that are deleted. I may go through and clean them up after I get back from SOTM-US. Hopefully since P1 is declining in usage, there won't be any more created. Thank you very much for both informations. This explains all questions, how it happened, and whether it may happen again. I would be very grateful if you could clean these relations up. Best regards, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] (no subject)
Dear all, could somebody please explain why http://www.openstreetmap.org/api/0.6/relation/970776 contains member type=way ref=62318915 role=/ but http://www.openstreetmap.org/api/0.6/way/62318915/history looks deleted? Best regards, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Overpass API: Augmented Diffs
Dear diff users, The good news is that the Augmented Diffs now also resolve relations, i.e. they contain for every mentioned relation all their members. However, I also fund some bugs during the implementation such that I anyway had to restart the Augmented Diff generation an hour ago. Now the Augmented Diffs should be stable for a longer period since there are no known bugs and open feature requests. If you have yet another feature request for Augmented Diffs, feel free to add it to http://wiki.openstreetmap.org/Overpass_API/Augmented_Diffs Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Minute Diffs
Dear all, the final step of the ODbL transistion has apparently started. The minute diffs are suspended. For that reason, of course, also the minute updates of Overpass API have been suspended. Overpass API will reflect the last CC-BY-SA data state of 2012-09-12 07:00 UTC until about Saturday. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] XML planets (was Re: New proposed directory layout for planet.openstreetmap.org)
If overall duration is an issue, is it worth patching a planet using hourly/minutely diffs before importing the planet itself? Ideas are welcome, but I'm quite sure this won't help. The primary issue is developing time, not overall duration for this step. For the patching itself: Reading 25 GB from disk and then writing it back alone will take about 800 seconds, which equals 13 minutes, for a minute diff. Thus, it will take for the assumed 2-16 days delay between a month and 8 month, even if Osmosis takes no computation time at all. Of course, you can use daily diffs instead for complete days and hour/minute diffs for the rest, but so you can do with Overpass API. It is just not worth the hassle. Ths is what the whole story is about: There are projects whose main concern is to process a planet file. In such a project, a lot of developing effort and increased system requirements are justified a 25% speedup. But there are also projects where a Planet file just needs to be read, in the simplest possible way, because it has little impact on the project's performance. For those, having a fancy compression which needs some extra software to decompress it that may or may not work on the target platform, is rather a nuisance than a developers welcome sign. Cheers, Roland___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] XML planets (was Re: New proposed directory layout for planet.openstreetmap.org)
Hi, Overpass is an amazing resource, but I can't believe it relies on a XML dump of the database being released every two weeks? How does that work? The planet file is necessary for the first startup. Afterwards, it can work forever solely on minute diffs. And new instances can be cloned from the exisiting instance over a public interface, without a planet file. This initial startup has usually the following timing: 2 hours to download planet.osm.bz2 12-24 hours to import planet.osm.bz2 into the database. This contains various optimizations that are fine tuned on the properties of the XML planet structure and maybe gets worse with a differently organised file format. 4-30 hours to catch up by applying the minute diffs. This depends on how many days the planet file is old (always at least two days, one for the planet file itself, a second because we have done the above import procedure). Altogether, there is not much point in just improving the download time, because it has a diminishing share of the total time needed. It would be by far more important to get the import step again fast. I doubt that converting the PBF into a XML is done in the saved half an hour on the download, so the simplest solution to convert after download is surely a slowdown. Yes, about 25 GB. Every two weeks. Ok, we come closer to the problem. I agree that probably very few people need old planet files. I deem the following useful to have - the last CC-BY-SA Planet - a CC-BY-SA full history planet up to that date - an ODbL full history planet from time to time (lets say four times a year or so) - the latest two planet files These are altogether less than 300 GB compressed XML, so I assume this manageable unless the server administrators tell otherwise. Then I would rather offer to host the above mentioned files than to drop them. Best, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Augmented Diffs - Compatibility?
1. Tagging Why did you not use solely the existing XML tags create, modify and delete? Because they don't properly describe what's happening. The core idea of the augmented diffs is to include some unchanged but related elements. This doesn't happen in diffs. Thus, the new category keep is used. On the other hand, modify doesn't apply to augmented diffs, because it brings the new version only. Thus, the augmented diffs use a pair of delete/insert for each modified element, with delete containing the old version. Of course you can argue that either delete should also be renamed or insert should be called create. This was accidental. Altogether, please note: the augmented diffs aren't an extension of the normal diffs and even less a replacement for normal diff. They are two distinct diff formats, for distinct purposes. 2. Sequence A lot of software depends on getting _sorted_ input data, usually ascending by object type, then by ID. Why did you break this order? The order by type is maintained. And all the tools I've seen so far don't need the second order criterion by id. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Reliability (was Re: New proposed directory layout for planet.openstreetmap.org)
Dear Martijn, There are much more tools around reading OSM files, in particular the XML format, than just Osmosis. And even more important: It is easy to write a piece of software that reads XML, and that is _because_ XML is human readable. So you really shy off potential developers. It may be 20% or 80% of all potential developers; both are numbers to get frightened. So I strongly oppose to remove any established format, in particular the OSM XML. I'm fine with the proposed directory structure. On the other hand, what do you gain by not having XML planet files? 25 GB of disk space? Agreed, but most of what you would want to do with grep is possible with other tools like osmosis, osmconvert and osmfilter that work much faster on pbf and o5m files. I think you miss the point. The argumentation Don't continue an established toolchain when a fancy new one exists is exactly what killed the Gnome project. Look for Linus Torwalds' reply in https://plus.google.com/115250422803614415116/posts/hMT5kW8LKJk The money quote is: One of the core kernel rules has always been that we never ever break any external interfaces. Transferred to our situation this means: we shall carry on the XML format forever because there are already a plenty of tools that rely on the XML format and they are worth protection, and because this is a clear signal to developers that we are a reliable partner. To make it even clearer: being cut off the XML planet means that Overpass API will starve for some month until I have implemented the quite complex file format PBF, and with it some hundred frustrated users, just to mention one tool. Do you seriously want to loose a huge part of the OSM community to save 25 GB of disk space? Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Overpass API v0.6.99
Dear all, a new version of Overpass API, v.0.6.99, is now available. It includes the augmented diffs in their current form and some other improvements. Please see http://wiki.openstreetmap.org/wiki/Overpass_API/versions The next version is almost for sure version 0.7 because there are no version numbers left In fact this version already freezes the features of version 0.7. The only thing that will change is maybe the format of the Augmented Diffs. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Augmented Diffs
Dear all, One complete different thing that would also been very interesting to have in even more augmented diffs is the previous version of an object. This would allow to do analysis on edits without the need of a database... and also apply diff backwards. Of course, the side effect will be larger files, but is that a real problem ? This is not a real problem. In fact, the diffs already contain the previous version of all changed objects. Last question (for the moment)... have you looked at .o5c file format to produce near pbf compact diffs ? Not really. I spent a weekend in January to understand PBF and how to generate it but I didn't understand the documentation and put it aside. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] nominatim-like wildcard search
For exempel Bre* would return Bremen, Brenner etc. I will limit my queries by bbox, You can use Overpass API: E.g. http://overpass-api.de/api/interpreter?data=(rel[name~^Bre](50.6,7.0,50.8,7.3);way[name~^Bre](50.6,7.0,50.8,7.3);node[name~^Bre](50.6,7.0,50.8,7.3););out; gives all elements that have a name tag starting with Bre. The result is not sorted, but this should be no problem on the client side, given that these queries don't produce too much data. The bounding box order is (south,west,north,east). The term ^Bre is the regular expression matching all strings beginning with Bre. Of course, you can also search without a bounding box, but in that case you should give further restrictions: http://overpass-api.de/api/interpreter?data=(rel[name~^Bre][place=city];way[name~^Bre][place=city];node[name~^Bre][place=city];);out; If you are interested, feel free to ask for more details. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Augmented Diffs
A few questions... When the change on a node or way is just a tag change and does not involve geometry change, are the diff still providing all linked nodes/ways ? Yes, they do. Thank you for pointing this out. I think it would be resonable to not include a way when the only change is a tag change of the connected node. However, the opposite is may or may not apply: Imagine somebody runs a database of all italian restaurants in Paris. He stores for each way that is the outline of an italian restaurant the center coordinates. Now somebody reclassifies a restaurant from whatever to italian, or corrects a typo like cuisine=italien to cuisine=italian. Then the database maintainer cannot derive the coordinates of the way because the underlying nodes are not contained in the augmented diffs. Of course you could argument that the same applies to relations, and that the database maintainer could ask for all nodes once he knows that a relevant restaurant appeared. So a general question to all: should we keep off the nodes of ways that have not changed their geometry? Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Augmented Diffs
The docs you list are a bit confusing, so I went to the XML itself, and that has some issues, mainly name/document related. 1. The osm tag name is already used for a format, so I suggest not using it. Similarly osmChange or osc files are already maybe you should call this something else? Thank you for pointing me at this. I think of osmAugmented or osmAugmentedChange and subsequently a file name extension .osa. 2. The sections are a bit confusing in that the order is very important. That makes parsing a bit more difficult than it might need to be if you just renamed sections? Eg keep means two different things in two different contexts, but by using the same name, it means your parser must keep a lot of state. What do you mean exactly? The purpose of the elements in keep is always the same: this is data that is unchanged but related to changed data. Of course there are different reasons why an element arrives in keep. However, this will make things difficult: an element could be present for more than one reason, for example a node because it is referred by more than one element. What do you think of the hierarchy for nodes: - coords changed - tags changed - part of a changed geometry for ways: - members changed - geometry of members changed - tags changed and for relations the same, maybe with a separate set of events for nested relations, in the sense that a element is listed only once at the event of highest precedence. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Augmented Diffs
Dear all, Overpass API now offers Augmented Minute Diffs. The idea goes back to a talk of Matt Amos at SOTM 2010. These diffs allow e.g. to keep a geographically limited database extract up to date. The details are documented in the wiki http://wiki.openstreetmap.org/wiki/Overpass_API/Augmented_Diffs However, these diffs address developer, not end users, because there is no software yet that can process these diffs. Therefore, I would be happy for comments and suggestions on this matter that arise when writing such a software. If there are good reasons to change the format, I'll try to do it only once, at the same time like the database reimport after the license change. Happy updating, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Augmented Diffs
Hi Paweł, This looks really good and may be useful for a project that I'm working on. Quick question - have you considered adding relation members data to these diffs? Yes, I have considered it. The reason not to do this so far are the really huge relations. Examples for these are national borders (relations 51477, 111 and several other), trans European routes (e.g. relation 2148361) and potentially similar relations. If we do the simplest approach to handle relations exactly like ways, we get 3.5 MB of minute diff in all minutes when a single node in these relations is moved (and much more if several relations are involved). Also, this would really produce heavy load on the server. So what I need towards relations is an approach that is simple to understand, reasonable concerning the leverage of processed data, and that fulfills the need of potential consumers. Things I have considered but not deemed easy enough to understand are to make it dependand on a certain tag in the relation, on the number of the relations' elements or not do deliver nodes when delivering unchanged ways because they are members of changed relations. It is all unsatisfactory, so good ideas are welcome. Best regards, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Server-side data validation
Hi Pawel, Hi Peter, Thanks for the response. So for now I'm trying to discuss this at a more abstract level - that the contract would be we can't have X in the database but how it is implemented (at changeset close maybe?) - I cannot say (yet) as I am no expert in OSM. For now more important is whether this kind of thinking even makes sense for you. I would not make much sense. There are a lot of ideas what could be done on the API, please see http://wiki.openstreetmap.org/wiki/API_v0.7 The more import thing is that OpenStreetMap is about a community, not about data. Business logic in the API will sooner or later include controversial topics and this will end with losing valuable contributers. For example: if you want the API to reject nodes at the same location, you may demotivate indoor mappers who model structural identical building levels with different shops per level. Or even simpler: every change in the API breaks temporarily or permanently tools. And the mapper who finds that his or her tool doesn't work any more is lost forever. On the other hand, duplicate or wrong data is not really a problem. There are a lot of applications (renderers, routers etc.) that do succesfully manage the existing data. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql failing with low cache size
Hi, Reading in file: ./data/latest.xml StartElement: Unknown element name: note Unknown node type 3 EndElement: Unknown element name: note StartElement: Unknown element name: meta EndElement: Unknown element name: meta Segmentation fault (core dumped) Can you sanitize your XML by running it through Osmosis (--read-xml latest.xml --write-xml foo.xml) and then try osm2psql again on the resulting file? Just to be safe that there's nothing strange in there that breaks it. The note element contains a license remark, the meta element a timestamp. Both are in the beginning of the file. You could simply remove them with a text editor. It is a little bit surprising to me because the osm2pgsql version I tried had happily ignored the extra elements. The rationale behind these two elements (and a third element error appearing only in case of an error) is to provide useful information in additional tags. This allows any tool to safely just ignore tags which it doesn't know. The license remark is a matter of good style - nobody should say he didn't know about the license of the excerpt. And the timestamp can enable editors like JOSM with the mirrored_download plugin to decide whether a given set of data is outdated or not. Cheers, Roland -- Empfehlen Sie GMX DSL Ihren Freunden und Bekannten und wir belohnen Sie mit bis zu 50,- Euro! https://freundschaftswerbung.gmx.de ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Extended diffs
I had a brief conversation with Roland Olbricht about this who suggested that maybe his Overpass API could be fashioned into producing such diffs but I don't think anybody had really implemented anything. That the good thing about the dev, talk and talk-de mailing lists: I put this on priority nice to have after the FOSSGIS conference, but given the fact that it is actively requested here, I'll include it in the next version, to be released during June. I think it would be best done on the next Karlsruhe Hack Weekend in June, but at the moment it looks like I can't participate due to another appointment that weekend. Cheers, Roland -- NEU: FreePhone 3-fach-Flat mit kostenlosem Smartphone! Jetzt informieren: http://mobile.1und1.de/?ac=OM.PW.PW003K20328T7073a ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Overpass API questions
Ignoring the relations: ( way(35.40,-120.80,35.50,-120.00)[key!=value]; node(w); ); out meta; Now, the problem is how do I add standalone nodes (points of interest, etc.), since I only have ways? You can just add another line with a bounding box for the nodes: ( way(35.40,-120.80,35.50,-120.00)[key!=value]; node(w); node(35.40,-120.80,35.50,-120.00); ); out meta; The query within the parentheses are connected by or. Thus, every additional request is just added to the result. In the same way ( way(35.40,-120.80,35.50,-120.00)[key!=value]; node(w); node(35.40,-120.80,35.50,-120.00); rel(35.40,-120.80,35.50,-120.00); node(r)-.x; way(r); node(w); ); out meta; will add even the relations with the ways and nodes that they refer to. Another problem appears to be that [key!=value] filters out any way that does not have a key tag at all, in addition to those that have key=value, which is not what I need. Yes, this has been the behaviour expected by design. The rationale behind this is to filter for oneway but not oneway=no. *The point of this is to be able to eliminate ways and their nodes that account for up to 90% of the data being downloaded (i.e. inflating the data by 1000%) when they are unnecessary for a particular editing session. However, this is a convincing argument. For this reason, and because the negation has not found widespread usage so far, I will change the semantics of != in the very next version 0.6.98. Version 0.6.98 is scheduled to be released on Monday, 30 April. This means, from 30 April on, you can get with the above query what you intended to get. 2. The example given at the top almost implements the normal API map call. However, the API map call returns the entire extent of ways that intersect the bbox, not just the nodes that fall within the bbox. Yes, that's right. Both variants are used. It appears to depend on the client what behaviour is expected. Nonetheless, the above mentioned call does include the nodes for the entire way, even if the particular node is outside the bounding box. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Overpass-API error codes
Am Freitag, 30. März 2012 11:15:26 schrieb Peter Wendorff: Hi. In a software I'm going to use the Overpass-API to fetch OSM-Data, and I think about handling error codes. As far as I see, Overpass prints errors in a remark-Tag in human readable text, but nowwhere as machine-readable text, while most error codes I guess could be mapped on HTTP-response states. Well, I'm not sure whether HTTP-response codes are here the best way to encode error states. The by far most important error situations are 1. A timeout on the server The client shouldn't use the result, as it may be incomplete. It can retry with a longer timeout. 2. Connection refused This would happen in case of overload, does happen rarely at the moment, but it should be addressed by a client. The client should retry after a timespan of 1 to 5 seconds again. 3. A malformed request It would be unusual if this happens when the request is generated programmatically. If a human user has entered the query, the client should pass the HTML document to the human user. The more simpler approach would be to process the result if it is OSM XML and to pass it to the end user if it is HTML. Specifically the first one cannot be handled by a HTTP status code. The status code is always the first thing to be written. Thus, to send a different HTTP status code in case of a timeout, the servers needs to held back the payload until the request has been completed. That impedes a couple of use cases where the client could take advantage of early arriving responses. Thus, a response started with HTTP 200 may still fail later with a timeout. Thus, in the end a client (think of a JavaScript framework would need to handle both some kind of errors coming from HTTP status codes and timeouts which cannot appear anyway else than as tags in the already started document. Is it planned (or a good idea that's planned now ;) ) to add a machine readable error state somehow? this could be done e.g. by adding a error-code attribute to the remark-attribute. What do you think about a switch to enforce XML as output format and error what=timeout/ error what=overload/ error what=bad_request/ as error indication? This would allow to ignore all remark/ tags in case you want to reliably respond to fatal errors and to ignore the rather diagnostic information. If somebody needs error details, he or she can read the messages in the remark- tags or resend the query to get a HTML explaining the error. Best regards, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Overpass Question
Hi lets go through the questions one by one. - which query-language is suited for what? I suggest using the Overpass QL syntax. This and the XML language both offer the same semantics, but Overpass QL is more concise. Overpass QL has been created because the XML language looks cumbersome to most people. - what's this .-x or .-_ about? Overpass has an imperative execution model. In particular, the staements are executed one after another, and each statement is terminated by a semicolon. Each statement puts its results into a container in its memory named by default _. If a statement needs input, it reads its input from _. For example, the print statement reads from _. Sometimes it is useful to use more than one container during a more complex query. In this case, the output can be redirected to another container, e.g. with name x. This is what the above syntax controls. - will a print-command print all results of all unions or only the last one? It prints the content of the container _ at execution time. This comes very close to the last one. - are two queries in a union get ANDed or ORed? They are ORed. I tried to fiddle an overpass-query together that would give me: - all relations tagged with type=boundary or type=multipolygon - their way-members - nodes used by thode way-members The query [timeout:86400]; ( rel[type=boundary]; rel[type=multipolygon]; ); ( ._; way(r); node(w); ); out; does that. but I was unable to get this result. I'd love to see a walktrgough or a tutorial that explains the building blocks and how they interact in a chronological order without adding stuff that isn't explained. Lets walk through the example: [timeout:86400]; is necessary in this case because we expect a really large result. The 86400 is an amount of time in seconds and means that we expect a runtime up to a whole day. The default value is 180 seconds, which fits well to the usual timeouts of browsers or other HTTP clients. rel[type=boundary]; collects all relations from the database that have a tag with key type and value boundary. The result is stored in the memory of the query server, in the container _, because this is the default behaviour. If you want to see what has happened so far, you can print the content of the container at this point: http://overpass-api.de/api/interpreter?data=rel[type=boundary];out; Similarly, rel[type=multipolygon]; collects all relations from the database that have a tag with key type and value multipolygon. Check the results with http://overpass-api.de/api/interpreter?data=rel[type=multipolygon];out; This is already quite a lot of data. Now, the union statement comes into effect. It takes a copy of each output and produces as result the union of each output. ( rel[type=boundary]; rel[type=multipolygon]; ); This means that here, first the output of rel[type=boundary]; is collected, then the output of rel[type=multipolygon]; is collected. The union of both is stored at the end of the statement into the container _ and replaced the content of container _ after the last substatement, rel[type=multipolygon]. To see the conent of the container _ at this point, run http://overpass-api.de/api/interpreter?data=(rel[type=boundary];rel[type=multipolygon];);out; On a semantic level, we now have in the container _ all relations that are of type boundary or of type multipolygon. We now get to the second union block: ( ._; way(r); node(w); ); The first substatement, ._, is only useful in a union block: It has as output in the container _ the input from container _. While this doesn't change container _, it lets union copy the current content of container _ to its own output. Thus, we have now all relations of the interesting type in both the container _ and the union's internal container. The next substatement, way(r); reads its input from the container _ and writes its output again to the container _, replacing the input data. Thus, on a semantic level, way(r); puts all ways that are members of a relation of interesting type into the container _. Because it is a substatement of union, this unoin block adds this output to its internal storage. The next statement, node(w); again reads its input from the container _ and writes its output to the container _. It finds all nodes that are members of the ways in its input. Because container _ contains at this point of time all ways that are members of a relation of interesting type into the container _, we now have exactly the nodes that we want in the container _. And because we are still in the union block, the internal union storage now contains the relations (from ._), the ways (from way(r);), and the nodes (from node(w);) that we want. At the end of the union statement, in the source code at );, the statement puts this into the cotainer _. In a final step, we only need to print this, adding an out; statement. If you want meta information, you may
Re: [OSM-dev] Overpass API areas
Thank you for asking. I particular, listing the IDs of all elements helped a lot to find the bug. I'm sorry that Overpass API doesn't work correctly. I have been using http://overpass.osm.rambler.ru/query_form.html for testing the queries Please try http://overpass-api.de/query_form.html instead at the moment. The area creation on overpass.osm.rambler.ru has got out of sync (it is some days old for now) and will be back on place on Tuesday after I have installed a software update there. I'm pretty sure it has missed the latest edits on relation 1875866. As I encountered the area feature as infrequently used so far, I handled it with lower priority than the core OSM and meta data. For that reason it is always good to ask, because areas get higher priority now. After trouble with the last software update, I update overpass.osm.rambler.ru always with some days delay, because in case of serious faults I could keep the old software there instead avoiding a total blackout. This means the inconvenience that the area bug fix arrives on overpass.osm.rambler.ru with some days delay. The relations themselves look correct so far. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] New version of API
Hi there, what about new version of API (0.7)? Since 2009 demands on OSM changed drastically so new API is really needed... I can not really find any activity in this arey is going on... http://wiki.openstreetmap.org/wiki/API_v0.7 There is a lot of stuff. But I frankly say that it would be insane to do any API change before the license change, because all OSM tool writers are more or less busy at the moment with the license change. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Overpass API: new big server, IPv6
Hello everybody, there are two new amazing news around Overpass API: I'm happy to announce a much more powerful (8-core, 64 GB RAM) machine, hosted voluntarily by Rambler. It offers the full interface of currently Overpass API 0.6.94, including meta data. Please have a try at http://overpass.osm.rambler.ru/query_form.html More information is on the wiki http://wiki.openstreetmap.org/wiki/Overpass_API http://wiki.openstreetmap.org/wiki/OSM_Servers_in_Rambler Second, the server at overpass-api.de now supports IPv6. Because I'm a complete newbie on IPv6 and I don't have IPv6 on my personal internet access, I've set up a second address http://ipv6.overpass-api.de/query_form.html pointing to the same server overpass-api.de/query_form.html . If it doesn't work, please contact me at my personal mail address. Happy access, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Bug Fixed (was: Re: Overpass API delivers randomly invalid data)
Hello everybody, I expect that it would take several days to determine the cause. I'll publish any results here and on http://wiki.openstreetmap.org/wiki/OSM3S/status Ok, now Overpass API should again deliver all meta data correctly. A now fixed bug in the transaction management interacted strangely with some residuals from a dirty restart at 23rd October. Thank you to everybody for the patience and in particular to Ruediger for reporting this and finding a way to reproduce the bug. Best regards, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Minute Diffs broken?
Hello everybody, It looks like the minute replicate diffs have stopped at 1071883. http://planet.openstreetmap.org/minute-replicate/001/057/ Could someone please check? Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Overpass API delivers randomly invalid data
Hi, I wrote a small test script to give you more help. The following results are reproducible: [..] 'way id=39990889' looks wrong at query Thank you, that helps a lot. I've also found way #4732211 that shows no history information at all. Altogether, about 250 ways of 140 million ways and probably also a small fraction of all nodes are affected, without any obvious pattern. The ways show the same behaviour on a independent database instance. I expect that it would take several days to determine the cause. I'll publish any results here and on http://wiki.openstreetmap.org/wiki/OSM3S/status Best regards, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Overpass API delivers randomly invalid data
Hi, We are querying data for a bounding box and randomly entries are delivered as way id=30134048 or node id=31881194 lat=47.5137295 lon=11.3250178/ The entries are cut before the version attribute. If we query the entries with the id all attributes are delivered. I've tried to reproduce this. But all changesets I have receieved have complete meta data. So could you please give more detailed information, in particular - At what time as exactly as possible have you tried to download? There have been an outage on Sunday including a backlog for meta data until Monday evening, and I would need to figure out whether it is related to this or not. - Which query exactly failed? - Does is fail sometimes or reproducably? - Are in a damaged response no meta data at all or is it partly missing? Best regards, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OverpassAPI down?
Now that it is back up, I am not getting any metadata on nodes that I have created since then, but the [@meta] predicate works OK on older nodes. Ok, this should have been solved now. The update routine has got a misconfiguration. I have also added a server status page on http://wiki.openstreetmap.org/wiki/OSM3S/status . I hope, it is easier to track the status there than distributed on several mails. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OverpassAPI down?
Hello, I'm not sure whether I'm addressing the right people. Please point me into the right direction if not. [...] Who is maintaining this API? Would be nice if this could be fixed. It's me who is maintaining the API. Thank you for reporting the problem. The problem should be fixed now. The server did a hard reset for an unknown reason (it is hosted at a service provider) and did not come up properly again. Another sudden restart would cause a similar failure at the moment, but this is the first hard restart after more than a year, so I won't expect any other sudden restarts. Best regards, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Error in overpass api
i've found an error in the result of overpass queries. E.g. relation id=1645727 version=2 timestamp=2011-07-05T19:58:33Z changeset=8642384 uid=244422 user=Sonny76 member type=way ref=119868575 role=different/ the role value contains '' and '' which must be 'lt' and 'gt' Thank you for reporting this. I've just fixed it on the instance on www.overpass-api.de. For those who want to fix their instance: change line 297 of src/overpass_api/statements/print.cc to \ role=\escape_xml(roles[skel.members[i].role])\/\n; Otherwise, the fix will be included in the next regular update. Best regards, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM XML declaration, JOSM, Osmosis et al.
Is Overpass so geared for tools that don't care about uid/date/version/visible etc? Actually, yes. Note that these tools include map rendering, routing, location based search and probably every other tool that consumes the data. The data model is: the state of the Planet database (or an excerpt) at a fixed point in time, which makes perfectly sense. Even keeping a database up to date would be possible without meta data: It suffices to know the timestamp of the patches as a whole, not of every individual element, despite the behaviour of possibly picky tools. The old XAPI had mainly been suffering from a permanent shortage of hardware ressources. You now get a switch to download data three times faster if you omit data that you discard anyway later - if you render a map, do routing, or a lot of other useful things. This would even lower the burden on the still hardware-constrained Overpass API server. A lot of users encounter that useful but some can't make sense of the error messages somewhere later in the toolchain. I can't and I won't rewrite every tool, but I can happly write in the header whatever the tools expect. I simply asking for a consensus for the things to write in the header. Discussing the usefulness of the current history model is a different question. As I got never an answer to an earlier post on that topic http://lists.openstreetmap.org/pipermail/talk/2011-September/060027.html (it's [osm-talk], I'm sorry), I assume that history is only a minor concern to most users. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] OSM XML declaration, JOSM, Osmosis et al.
Hello everybody, A question to all maintainers of OSM XML processing software: What combinations of root tag attributes do you expect for XML with and without meta data? If I deliver from Overpass API to JOSM the XML data with meta data and with a plain osm tag, JOSM crashes with a NullPointerException on the first version attribute. If I deliver, as suggested from a JOSM developer, with a osm version=0.6 generator=Overpass API root tag, JOSM and Osmosis both complain about the absence of version attributes on the individual OSM elements for data without meta data. Having OSM data with or without meta data is really a useful feature. The same data with meta data is three times bigger than without, in a context, where data size matters, and the meta data is usually only needed when one wants to re-upload the data to the main database. So what attributes would you, as a toolmaker, expect on the root tag for OSM data with or without meta data? Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM XML declaration, JOSM, Osmosis et al.
Is Overpass so geared for tools that don't care about uid/date/version/visible etc? Actually, yes. Note that these tools include map rendering, routing, location based search and probably every other tool that consumes the data. The data model is: the state of the Planet database (or an excerpt) at a fixed point in time, which makes perfectly sense. Even keeping a database up to date would be possible without meta data: It suffices to know the timestamp of the patches as a whole, not of every individual element, despite the behaviour of possibly picky tools. The old XAPI had mainly been suffering from a permanent shortage of hardware resources. You now get a switch to download data three times faster if you omit data that you discard anyway later - if you render a map, do routing, or a lot of other useful things. This would even lower the burden on the still hardware-constrained Overpass API server. A lot of users encounter that useful but some can't make sense of the error messages somewhere later in the tool chain. I can't and I won't rewrite every tool, but I can happily write in the header whatever the tools expect. I simply asking for a consensus for the things to write in the header. Discussing the usefulness of the current history model is a different question. As I got never an answer to an earlier post on that topic, http://lists.openstreetmap.org/pipermail/talk/2011-September/060027.html I assume that history is only a minor concern to most users. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Minute-Diff
Hello, does anybody know what has happened to http://planet.openstreetmap.org/minute-replicate/ ? It apparently stopped half an hour ago. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Fastest way to extract a bounding box
Hello Benjamin, I'm looking for the fastest way to extract a bounding box from osm-data. The intention is to get a section (a square of about 5km * 5km) around a user's current position in a few seconds. [...] I extracted such a bounding-box from germany.osm.pbf with osmosis in about 110 seconds.[...] in about 53 seconds. You may want to try OSM3S as a server. I got the 5km x 5km around my home in about 2 seconds from the planet size database. Please have a look at http://wiki.openstreetmap.org/wiki/OSM3S/install Feel free to ask what you don't understand. As the server developer and maintainer I'm happy to use feedback to improve the manual and/or user interface. Best regards, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] XAPI and using a home server
The only thing is that it ends with: runtime error: open64: 2 ./db/area_tags_local.bin File_Blocks:1 and does not provide the closing /osm-derived tag. Do you have any suggestions on what might cause that? Yes, it's a bug in the software. Thank you for reporting it. First, for a feasible workaround please place empty files with names area_tags_local.bin area_tags_local.idx in the database directory. The probably fastest way is pushd YOUR_DATABASE_DIR touch area_tags_local.bin touch area_tags_local.idx popd The cause is a little bit intricate. OSM3S should allow for the usage of derived data structures, and as a showcase the handling of areas is included. This feature is not mature enough to offer it as part of the regular interface, but areas if present, can be printed with the print statement. Unfortunately, the print statement throws an error if a corresponding file of the database doesn't exist. While useful in general to find errors, its unhelpful for this optional data structure, which may be absent for good reason. However, the workaround above should do it for the moment. I'll fix the behaviour, by making areas truly optional also in the print statement, with the next bugfix version. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] XAPI and using a home server
Roland, I have compiled osm3s and imported the latest planet file. Thank you for this. I'm sorry that there are still holes in the instructions. I think this was successful - no errors anyway and took about 24 hours as you suggested and ended with [...] max_written_role_id 4458 R 1298757941 r 1298757941 This looks a bit abrupt, but I tried a small osm file and got a similar result, so I think this is a good sign? Yes, it is. It looks fine so far. Well, the next version will have a more informative final message. How do I use it? And although it lets me type things at the terminal (like your example queries), it does not respond - is it really listening on standard input? Yes, it does. It waits until the input stream is finished. So, please finish the input stream by Ctrl+D when you type at the console (works in general on all UNIX consoles). A more convenient way (especially when invoking it in a tool chain) might be to pass a file with the content to it. For example echo 'query type=nodebbox-query n=51.0 s=50.9 w=6.9 e=7.0/has- kv k=amenity v=pub//queryprint/' | bin/osm3s_query --no-mime --db- dir=YOUR_DB_DIR Your instructions on the wiki say put a query on the standard input, but when I start osm3s_query it prints out: encoding remark: No input found from GET method. Trying to retrieve input by POST method. You can safely ignore the message if you call osm3s_query on the command line. It's a leftover from the purpose of using it via the CGI interface of a web browser. Sorry if this is obvious - I haven't tried to look at your code to see what it is doing! Questions are always welcome. Even if it were obvious (it is not), it is still a good hint for me how to improve the instructions. The other question is whether anyone has written a php script (or similar) to make this emulate xapi queries to use as a direct replacement for xapi? AFAIK no yet. I think it would be feasible for /map?bbox=.. (gets 'bbox-query n= s= w= e=/print/') /node[..] (with or without bbox) (like the first query in the instructions, maybe without bbox) /way[..] (gets 'query type=wayhas-kv k= v=//queryprint/' or maybe 'query type=wayhas-kv k=//queryprint/' /relation[..] (gets 'query type=relationhas-kv k= v=//queryprint/') /* (all three without bbox) (gets: union query type=nodehas-kv k= v=//query query type=wayhas-kv k= v=//query query type=relationhas-kv k= v=//query /union print/ ) but the other queries may get more difficult. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Something wrong with API
Hello, I'm unable since about an hour to open a new changeset, i.e. the request http://api.openstreetmap.org/api/0.6/changeset/create hangs with no response. I've initially tried to upload by JOSM with my username roland.olbri...@gmx.de, but the server just hangs on any response, with or without a user name, from different IP adresses and with different tools (JOSM, wget, curl and firefox). I don't get any error message, just no response. Could anybody please explain what's going wrong? Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [josm-dev] Another 66 relations bite the dust
[1] http://trac.openstreetmap.org/ticket/2652 You can get an approximate solution: http://78.46.81.38/api/rels-extended?id=34631 This points to the OSM3S server and returns the desired relation-ids for the given relation id. It's compressed XML at the moment, but I can change that to are more appropriate format if the format is the problem. The second drawback is that the server is some hours behind the main database, but most of the relations mentioned here are days old or even older and would have been found by this API call. But still this would be a great step from deleting relations accidentally. Incidentally, the server is down this evening from 22h15 to 12h00 tomorrow because the server provider (Hetzner) is moving inside Germany. Cheers, Roland ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [OSM-dev] Opening hours
The rough plan is that my site become essentially a validation layer for producing the opening_hours tag and a service for getting the current state of a element. I'd use the OSM oauth for authentication. [..] Let me know if you're interested in helping or have any advice. I've had a good read of this wiki page and discussion, it seems very sensible to me. http://wiki.openstreetmap.org/wiki/Opening_hours The syntax for opening hours still has its difficulties: try to express that a service is open from October to March only at the weekend and all days during the summer. Or a service that is open on the evenings _before_ public holidays. It's just not possible. On the other hand, a line like Mo-Fr 09:00-12:00; Tu 14:00-17:00 may read - on Tuesdays also in the afternoon or - on Tuesdays only in the afternoon. While the former is far more common, the latter is what the current syntax defines. Even worse, what about Mo-We 09:00-12:00; Tu-Fr 14:00-17:00 or Mo-Sa 09:00-12:00; We-Fr 14:00-17:00? I'd rather suggest a syntax like Month Day_of_Week Modificator From-To[,From-To]* ;]* where - Month is in (Jan, .., Dec) or a range or empty - Day_of_Week is in (Mo, .., Su) or a range or empty - Modificator is one of WD (working day), SD (school day), SH (school holiday), PH (public holiday), PH- (day before PH) or empty - From, To are times HH:MM and intersections between days (e.g. Mo-Fr ..; Tu ..) are considered as syntax error. This would be much more concise yet were more versatile. And 90% of all existing opening hour data remain valid. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Using relations to save data
Am Dienstag, 4. Mai 2010 22:12:02 schrieb Andreas Kalsch: I am still reading some old mailing list posts ... What about a relation with type=data, which is a relation that can include tags and other relations recusively? It is a really good idea to store definitions of tags directly into the database, but relations leave some drawbacks open: - A description might easily surpass 255 characters. Or you might want to use markup (e.g. for a link or for emphasizing things). - Often tags apply only to a part of the world. Or, even worse, do have different meanings in different part of the worlds. Think of different maxspeed restrictions on motorways in different parts of the world. I'd encourage you to start using those relations now but we should have a more versatile solution with the next API. See http://wiki.openstreetmap.org/wiki/API_v0.7#Classes [1]http://www.remote.org/frederik/tmp/towards-a-new-data-model-for-osm.pdf Looks interesting. But I thin it is largely outdated because it does not consider relations. And, of course, a couple of new problems have arisen in the meantime. How about starting an updated version of that paper somewhere in the wiki or elsewhere? Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Query-Formats
Hello, I'm currently thinking about the interface on how to query this resource and i'd like to be able to ask queries like all highway=trunk and highway=primary with a name what is not possible with the current XAPI-Interface. This would look like osm-script timeout=180 union query type=way has-kv k=highway v=trunk/ has-kv k=name/ /query query type=way has-kv k=highway v=primary/ has-kv k=name/ /query /union print mode=body/ /osm-script in OSM3S. Bboxes for ways are possible but I haven't implemented them yet - things that happen if a lonely guy develops a full blown multi-purpose server. This specific query is quite slow, because it returns data from across the entire planet. what do you think about supplying a raw SQL-Where via URL? query.php?(tags ? 'name') AND (tags ? 'highway') AND ((tags-'highway' = 'primary') OR (tags-'highway' = secondary')) I have thought about that when designing the interface, but I think it will be close to impossible to avoid - extremely long running a resource intensive queries - malicious queries (simple example ...'; drop table *') because SQL is simply not designed for to be offered over the web. The second observation was that at least MySQL was way too slow to be useful - MySQL delivered about 10'000 nodes per minute, while OSM3S delivers roughly 1'000'000 to 2'000'000 nodes per minute (for a query with is spatially local, like downloading all data from a city or county). From that starting point it looks unlikely that another general purpose DB engine would have a satisfactory performance. Thus, I have preferred a query language with good predictability properties. See http://78.46.81.38/#chapter.concepts The server is productive and used for some services with the current language, but comment and also a deep refactoring of the language are welcome. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[josm-dev] Where to put documentation for a plugin?
Hello everybody, where shall I put the documentation for my JOSM plugin? I've written a html file containing all the information about the public transport plugin into the svn, but it is delivered as text/plain: http://svn.openstreetmap.org/applications/editors/josm/plugins/public_transport/resources/public_transport.html I suspect that this is intentional. But where is the designated place for the documentation? Where should the more info in the plugin download list point to? Cheers, Roland ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
[josm-dev] Java 1.5 versus 1.6 for plugins
Hello, I've just developed a plugin (Public Transport) for JOSM which should be as portable as possible. Now I've received a request of a Mac OS user using Java 1.5 that the plugin doesn't work. I've set property name=ant.build.javac.target value=1.5/ in http://svn.openstreetmap.org/applications/editors/josm/plugins/public_transport/build.xml and I don't use any special features of Java 1.6. So has anybody an idea why the plugin doesn't work with Java 1.5? Cheers, Roland ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [OSM-dev] Tag-key with ? in planet but not in API
in the planet file there are some tags containing a ? in the key, eg. in line 725579517 node id=187690242 lat=49.2199004 lon=12.6700505 timestamp=2009-08-24T13:30:09Z version=5 changeset=2243654 user=klausis uid=85761 tag k=amenity v=parking / tag k=name?? v=Parkdeck? / tag k=poi v=vehicle.parking.garage / /node These aren't question marks. These are white space. A hexdump shows 6e 61 6d 65 09 0d for the k-value, meaning name, then a tab, followed by a CR. The second question mark is just a CR. The characters are legal in both XML and the OSM format and do appear in the planet as well as in http://www.openstreetmap.org/browse/node/187690242 But with the planet, your viewer seems to convert them for whatever reason to question marks. In case of the API, the whitespace characters in the HTML source code are ignored by your browser. This is intended by the specification of HTML. Anyway, you can edit away the characters with Potlatch or just upload a corrected version with any other application (e.g. JOSM) via the API. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Best way to select from a region
This one is 224495 - nine-five at the end instead of five-nine - though. 224459 has both a name and admin_level, it's returned by: [...] Thank you for pointing me to that. In fact it missing relation revealed a larger bug in the system: no relation beyond 187000 is considered as a potential area. I'm at the moment trying to fix it. Ok, the problem should now be fixed. Please try it again. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] (no subject)
Hello everybody, way 15581339 contains node 448856251 but this node is deleted. Both elements date from 27th July, so they are edited (and the node also created) after the change to API 0.6. How can this happen? Is this intended? Cheers, Roland -- GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT! Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Best way to select from a region
That's an interesting number. Which definition of correct do you use? The mathematically most general: take all way members of the relation as borders. This forms an area if and only if every node if references by an odd number of way segments. The system takes into account any relation that has an admin_level-key or is of type multipolygon. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Best way to select from a region
Off topic, but what might be a reason for a boundary relation (such as 224459) not appearing in any coords-query, for locations inside and outside it alike, and returning no data for an area-query, but also no error? I don't suppose node number limit? relation id=224495 member type=way ref=39907923 role=outer/ member type=way ref=39907925 role=inner/ tag k=type v=multipolygon/ /relation It doesn't have a name. The possibly applying rule osm-script name=Area::Create_from_multipolygon version=2 query type=relation has-kv k=type v=multipolygon/ has-kv k=name/ !-- * -- /query [...] /osm-script requires a name (see the line with the *). The reason is that an area without a name would not be very informative. But if you want those relations also appear, I'll change the rule. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Best way to select from a region
http://wiki.openstreetmap.org/wiki/OSM_Server_Side_Script that sounds cool. Did not know it before. It's not advertised a lot ;) What are the limits? Frederik mentioned a query against a polygon might be slow. Which area could be returned? Only a city? Bavaria? France? The query against a polygon is in general not very expensive. Retrieving a single polygon takes a single hard disk hit, and the computational effort can be neglected against that. The limiting factor is the RAM size of the server (4 GB). To avoid congestions, I've set a weighted limit of 10'000'000 elements so far (this takes about 500 MB of RAM to process the data, so four queries and the update process could safely run in parallel) - think of roughly 5'000'000 nodes. The amount of RAM is needed to sort the nodes, ways and relations in ascending order of their ids. Bavaria would be a corner case and is worth a try ... It doesn't work, it has already over 10 million nodes. Or maybe only query the nodes near the border against the expensive polygon? Whatever system you use, it will probably automatically use less a expensive query deep inside a polygon. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Best way to select from a region
In 95% of cases you will find that the borders in the database are unsuitable because they do not form a proper closed polygon. Plus, PostGIS definitely sucks at point-in-polygon performance, especially if your polygon has 30k points. There are currently at least 51090 correct boundary polygons (as of 2009-09-01 03h00 UTC) in the database. I doubt that these are only 5% of all boundaries (would the be more than 1'000'000) in the OSM database - there aren't even a million relations at all. Have you considered just downloading the list? If you don't want to have all the trouble with running a GIS database, you may directly use XAPI or OSM3S to obtain specific data in XML format from a region, see http://wiki.openstreetmap.org/wiki/XAPI http://wiki.openstreetmap.org/wiki/OSM_Server_Side_Script Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Problem retrieving wiki pages
Of course, but the server (or more likely the proxy) is still mis-behaving: if the client does not send an 'Accept-Encoding' header, the server must return plain text, not gzipped or deflated text. No. Have a look at the HTTP 1.1 specification http://tools.ietf.org/html/rfc2616#section-14.3 If no Accept-Encoding field is present in a request, the server MAY assume that the client will accept any content coding. If you think of a highly frequented server like the wiki, it's a good decision to compress the data whenever possible. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Spatial queries in Mysql
Does anyone have experience with Mysql and 2d-range-queries? I am looking for the optimal way of: a) selecting all nodes in a given bounding-box and b) selecting all ways intersecting a given bounding-box. I am not bound to any existing schema. What works best? I've started OSM3S with mySQL (and myIsam tables), but it had very poor performance for spatial queries with all the nodes from the planet. For mySQL, I used an extra index containing the latitude rounded to an integer, then the longitude. Thus, a spatial query like the one below was aligned or nearly aligned with the index. This had the same performance than a quadtile index with latitude and longitude interleaved as binary numbers. The legacy system to which I compare uses a quadtile index. I don't have exact test results, but for retrieving a bbox like 51.0lat52.0, 7.0lon8.0 the legacy system takes 2 seconds while mySQL took more than two hours. Similar results have appeared for the ways table. The tests were performed on a Intel T9500 with 4 GB RAM, operated by Hardy Heron 32-bit. Monitoring the process showed that mySQL was busy with the disk all the time. So must probably, mySQL organises the data on the disk in a way that is inappropriate for retrieving large amounts of data. Unfortunately, there is no documentation how the data is arranged on the disk and no switch how to let it be organised in a more useful way. So if you are intending to do spatial queries on a planet file sized amount of data, you should probably pass to a more performant system. PostgreSQL does not document its disk storage strategy either, so the same problem might or might not appear. Somebody has suggested the Cern database Root, but I have not tried this one. The source code of the mentioned legacy system is also available. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] broken utf8 in minute changeset 200907140650
1. Go to http://www.systemeD.net/stuff/keycode.html 2. Type some non-ASCII characters into the top box (letters with accents, the sort of thing you might want to enter as an OSM tag value) 3. For each one, tell me what it returns in the next two boxes Entering äöüÄÖÜß yields - in the first box äöüÄÖÜß - in the second box c3 a4 c3 b6 c3 bc c3 201e c3 2013 c3 153 c3 178 - in the third box C3 83 C2 A4 C3 83 C2 B6 C3 83 C2 BC C3 83 E2 80 9E C3 83 E2 80 93 C3 83 C5 93 C3 83 C5 B8 Doing the same via copypaste yields - in the first box äöüÄÖÜß - in the second box e4 f6 fc c4 d6 dc df - in the third box C3 A4 C3 B6 C3 BC C3 84 C3 96 C3 9C C3 9F Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] how many api calls/time for an app allowed?
Hello, i am playing with the idea of providing real time error checking on osm data. usually one would use the XAPI. but since it is unreliable and obviously misses data and it seems no one really cares (no ansers to questions about that matters on the mailing list - at least not satisfying ones) one would use the API instead. Well, one of the design goals of OSM3S was to provide an almost real time error checking. Pleas have a look at http://78.46.81.38/#section.debug_area to see how to find bugs in the borders of areas with OSM3S. This is just a showcase. The big idea to project is after is to divide between the code to query the database (currently C++) and the code that describes the error checks themselves (a scripting language specifially crafted to support advanced OSM data queries and error checking rules). Thus the former can be ugly but performant. The latter should be human readable and so safe that it can be directly submitted from users through the internet (as opposed to arbitrary SQL queries). Due to the complexity of the computations (anyway some hours for the entire planet), I'm aiming to be perpetually 4 to 6 hours behind the main API. The concept can be developed to be only 60 to 90 minutes behind the main API, but this needs a lot more effort. The project is still in an early stage, e.g. the documentation of the OSM3S API is not yet complete. I would be glad to share efforts. This includes developing the scripting language towards the features it really needs, as well as improvements to the database source code (by the way: what is an appropriate way to publish the source?). I don't insist on C++ or the scripting language syntax or semantics; I have chosen it because the tools for it are at hands. I would also like to discuss the entire concept as it is a test case for a larger project. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] (no subject)
Hello everybody, Wolfgang has written and why (an area is a closed way with some tag attached simply doesn't cut it). Or, in case that was still not concrete enough: in Austria alone, there are currently on the order of 750 geometries that are perfectly valid in osm but not digestible by quite a few GIS-enabled databases Well, just take it from a mathematical point of view. One can define any area by providing its border. A point is inside if and only if it is separated by the south pole by an odd number of borders. I you need a mathematical proof, I'll put one on the wiki. So there is no need to bother the mappers with any outer, inner tag, order on relation members or whatever. The only thing needed would be a tag to reverse the area to make areas possible that cover the south pole. The last thing I would think about is to bully the community with another bot that changes what a poorly designed application fails to read. There are at the moment more than 44,000 valid areas in the planet data (multipoygons and administrative units). There are a lot of tools that can digest this data. The interpretation of this data would need neither a relevant amout of code nor computation time. Feel free to ask for sample code. So if a certain database or tool fails on mathematical facts, I would have doubts about that database resp. tool, not about OSM. Cheers, Roland -- GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT! Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Way 33187566
Hello everybody, a query by the api for way 33187566 http://www.openstreetmap.org/api/0.6/way/33187566 returns only one node. On the contrary, a query for its history http://www.openstreetmap.org/api/0.6/way/33187566/history returns quite a long list of nodes. So does the latest planet file. But all the nodes seem to have gone. Can somebody please explain what has happened? Which is the definitive source for that way? Something similar seem to have happened to the ways 28792317 33187577 Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] country boundary polygons
I am looking for a simple answer to the question 'what country is
lat+long in?',
Has there been any progress
made with this?
Yes, I'm working on a server side scripting host for OSM data. This includes a
reverse gazetteer as a sample application. The gazetteer works fine, but the
more down-to-earth-things like updating the database without a complete data
reload aren't yet working stable. So I think, the system will be available
about mid-May.
echo select osm_id,name,admin_level from planet_osm_polygon where way
transform('SRID=4030;POINT(6.0333 45.7666)',900913) and
_ST_Contains(way,transform('SRID=4030;POINT(6.0333 45.7666)',900913)); |
psql gis
If you have an instance of PostGIS running anyway, this is a good solution.
and for this I need the country borders as polygons.
You may want to have a look at
http://wmaz.math.uni-wuppertal.de/olbricht/osm/
I abandoned to simply produce the country polygons that way because, as Sly
has mentioned, the country borders still are of rather poor quality. There
are roughly 17 countries out of 200 for which the boundaries will constitute
complete polygons. That's the reason why we should have a more advanced
system for error detection (the script server will contribute to this).
By contrast, the local borders from imported data (Kreisgrenzen in Germany
and similar import in France and Italy) are quite complete and consistent.
Cheers,
Roland
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] auto-generating landuse=residential polygon around highway=residential -streets
Hello, issue: Where they are missing, I would like to generated landuse=residential automatically in these places by evaluating the strongly connected graphs of residential-streets. Where such polygons have been created by mappers they will be used instead but there are too few of them to rely on them to be present. question: Dies anyone have a good idea for an algorithm that could do this? What is the particular situation, i.e. what accuracy is necessary? A good starting point might be a scanline approach as for the 12-nm-brim, which is a matter of minutes for the entire planet but not very precise: - We assume that every point which is less than a given distance (e.g. 100 m) from a residental road is residental and draw a line around that area. - Imagine the planet (or a smaller area) covered with equidistant scanlines along the latitudes (the distance between two scanlines e.g. 0.0001 degrees of latitude ~= 11,1 m). Now you add for every segment of every residental way all the intervals of points on each scanline that are less than 100m from any point of that way (simplified: take into account only the endpoints of the segment and the intersections with the scanlines - the inaccuracies won't matter compared to the scanline gaps or the projection correction). After the tool has taken the union of all those intervals, it spans a way from interval border to interval border - that's the desired polygon around the residental area. - the algorithm is n log n in the number n of intersection points, which in turn are linear in the number of scanlines or the total length of all residental roads. Regards, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
