[OSM-dev] Questions on Nominatim and administrative boundaries + osm2pgsql
Dear fellow OSM hackers, We are working on improving MapOSMatic, especially for international support. We are using http://nominatim.openstreetmap.org to query OSM in a human-friendly way, which, indeed, is a wonderful tool! Now we would like to use its results to query our own copy of the DB (created with osm2pgsql). We found a way to do this, but we are not quite sure it's the right way (tm) to do it... For example, we query Nominatim for Paris. One of the results is an administrative boundary limit (the one with osm_id 7444). For that entry, we would like to query our OSM DB to determine its administrative level. If we do: select * from planet_osm_line where osm_id=7444; then we don't get anything. However, when we do: select * from planet_osm_line where osm_id=-7444; then we get exactly what we need. Is this the right way to do it? Can we assume that, when nominatim returns an entry with class=boundary type=administrative and osm_type=relation, then we can safely query the planet_osm_line table with the _opposite_ of osm_id? Now, we already know that it does not always work like this... For example, if we try this with the entry for Paris, Kentucky, United States of America (osm_id 130722), then it simply does not work at all. In fact, it doesn't seem to work at all with none of the other administrative boundary query results that nominatim returns us for Paris. Should we instead query the planet_osm_rels table? Can we assume this table is always present in the database? If yes, how do we parse its fields to get the IDs to the other tables? Are there any documentation on the DB tables produced by osm2pgsql? Thanks a lot in advance! Regards, david -- for MapOSMatic dev team ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Questions on Nominatim and administrative boundaries + osm2pgsql
2009/12/20 David MENTRE dmen...@linux-france.org: If we do: select * from planet_osm_line where osm_id=7444; then we don't get anything. However, when we do: select * from planet_osm_line where osm_id=-7444; then we get exactly what we need. I think ways are positive and relations are negative, that's just what I've observed but I could be wrong. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Maposmatic-dev] Questions on Nominatim and administrative boundaries + osm2pgsql
David MENTRE wrote: Is this the right way to do it? Can we assume that, when nominatim returns an entry with class=boundary type=administrative and osm_type=relation, then we can safely query the planet_osm_line table with the _opposite_ of osm_id? A _way_ will be in the tables with a positive osm_id, where a _relation_ will always have a negative osm_id. This is due to non-unique numbering between OSM object classes. Nominatim tells you* whether you got back a relation or a way, and you can then search for the negative or positive osm_id in planet_osm_polygon, for relations or closed ways, respectively. * At least, it does so in http://nominatim.openstreetmap.org/details.php?place_id=49949948 but I don't know what the bare API result looks like. -- Lennard ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Questions on Nominatim and administrative boundaries + osm2pgsql
On Sun, 2009-12-20 at 11:55 +0100, David MENTRE wrote: Dear fellow OSM hackers, We are working on improving MapOSMatic, especially for international support. We are using http://nominatim.openstreetmap.org to query OSM in a human-friendly way, which, indeed, is a wonderful tool! Now we would like to use its results to query our own copy of the DB (created with osm2pgsql). We found a way to do this, but we are not quite sure it's the right way (tm) to do it... For example, we query Nominatim for Paris. One of the results is an administrative boundary limit (the one with osm_id 7444). For that entry, we would like to query our OSM DB to determine its administrative level. If we do: select * from planet_osm_line where osm_id=7444; then we don't get anything. However, when we do: select * from planet_osm_line where osm_id=-7444; then we get exactly what we need. Is this the right way to do it? Can we assume that, when nominatim returns an entry with class=boundary type=administrative and osm_type=relation, then we can safely query the planet_osm_line table with the _opposite_ of osm_id? This is correct. When osm2pgsql creates entries from relations in the line (or roads) table then it uses the negative of the relation ID in the osm_id column. Now, we already know that it does not always work like this... For example, if we try this with the entry for Paris, Kentucky, United States of America (osm_id 130722), then it simply does not work at all. In fact, it doesn't seem to work at all with none of the other administrative boundary query results that nominatim returns us for Paris. The boundary 130722 must be a complete closed ring, this means it ends up in the polygon table instead. Again it has a negative ID because it was generated from a relation: gis= select osm_id,name,boundary,admin_level from planet_osm_polygon where osm_id in (130722,-130722); osm_id | name |boundary| admin_level -+---++- -130722 | Paris | administrative | 8 I did consider putting a copy of these polygons into the line (or roads) tables but in the end I decided against it. The ways which make up the 7444 relation must have an error in them somewhere which prevents osm2pgsql from forming a polygon. Should we instead query the planet_osm_rels table? Can we assume this table is always present in the database? If yes, how do we parse its fields to get the IDs to the other tables? The _rels table is present if you use the --slim mode. The fields are a little trickier to parse, you may get some inspiration from the queries used by osm2pgsql otherwise you'll need to read up on how the postgresql intarray feature works. Are there any documentation on the DB tables produced by osm2pgsql? Not that I am aware of. One thing that you might like to know is that the backend database for nominatim is also generated using osm2pgsql, using the gazetteer mode. http://wiki.openstreetmap.org/wiki/Nominatim http://svn.openstreetmap.org/applications/utils/export/osm2pgsql/gazetteer/README.txt Thanks a lot in advance! Regards, david -- for MapOSMatic dev team ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Questions on Nominatim and administrative boundaries + osm2pgsql
On Sun, 2009-12-20 at 11:27 +, Jon Burgess wrote: On Sun, 2009-12-20 at 11:55 +0100, David MENTRE wrote: Dear fellow OSM hackers, We are working on improving MapOSMatic, especially for international support. We are using http://nominatim.openstreetmap.org to query OSM in a human-friendly way, which, indeed, is a wonderful tool! Now we would like to use its results to query our own copy of the DB (created with osm2pgsql). We found a way to do this, but we are not quite sure it's the right way (tm) to do it... For example, we query Nominatim for Paris. One of the results is an administrative boundary limit (the one with osm_id 7444). For that entry, we would like to query our OSM DB to determine its administrative level. If we do: select * from planet_osm_line where osm_id=7444; then we don't get anything. However, when we do: select * from planet_osm_line where osm_id=-7444; then we get exactly what we need. Is this the right way to do it? Can we assume that, when nominatim returns an entry with class=boundary type=administrative and osm_type=relation, then we can safely query the planet_osm_line table with the _opposite_ of osm_id? ... The ways which make up the 7444 relation must have an error in them somewhere which prevents osm2pgsql from forming a polygon. On closer inspection, the Paris boundary does form a complete closed polygon and is stored in the _polygon table as well: gis= select osm_id,name,boundary,admin_level from planet_osm_polygon where osm_id in (7444,-7444); osm_id | name |boundary| admin_level +---++- -7444 | Paris | administrative | 8 All boundaries which form a closed ring should have entries in the polygon table. Jon ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Use API to get bbox of way/relation ids?
Thanks... seems a little wasteful to have to get all the individual nodes and tags. Is this the only way to get the bbox of a given way without one's own planet db? - L On 20 Dec 2009, at 02:37, John Smith wrote: 2009/12/20 Laurence Penney l...@lorp.org: What's the best method of getting a bbox for a given way or relation id, using the API? Is it cached anywhere? If you know the node/way/relation ID you don't need to specify the bbox.. http://www.openstreetmap.org/api/0.6/node/nodeid http://www.openstreetmap.org/api/0.6/way/wayid/full http://www.openstreetmap.org/api/0.6/relation/relationid/full ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Use API to get bbox of way/relation ids?
On Sun, Dec 20, 2009 at 3:43 PM, Laurence Penney l...@lorp.org wrote: Thanks... seems a little wasteful to have to get all the individual nodes and tags. Is this the only way to get the bbox of a given way without one's own planet db? How does http://www.openstreetmap.org/browse/relation/# do it? Considering how long it took to load http://www.openstreetmap.org/browse/relation/62899, I'm guessing it downloads the entire relation and uses javascript to calculate the bbox and draw the relation. Pretty inefficient, but looking at the database schema (http://wiki.openstreetmap.org/wiki/Database_schema), that information doesn't even seem to be cached in the db, so it's something that has to be recalculated every single time by someone - just a matter of whether to waste bandwidth or CPU. I'd say if this is something you're going to be doing a lot of, it's probably best to download the whole planet (or the part of it you're working on). For the sake of not overloading the API as much as for making things easier on your end. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] duplicated entities with osmosis diff-update
Thank you all! now it works! I switched to --rri tasks. This is what I did: * get latest osmosis version 0.31.2 dated december 20th (I had an older 0.31.2 version before, but who knows...) * initialize the working directory with osmosis --rrii workingDirectory=/blabla/0 * changed the URL to http://planet.openstreetmap.org/hour-replicate/ and maxInterval = 0 in workingDirectory/configuration.txt * fetched the appropriate state.txt file from said URL and don't forget to save it as state.txt (_NOT_ 450.state.txt!) in the working directory * called osmosis --rri workingDirectory=/blabla/0 --simplify-change --rx /blabla/0.osm --ac --bb left=8 top=49 right=17 bottom=46 idTrackerType=BitSet completeWays=yes completeRelations=yes --wx /blabla/0.osm.new now happy with my shiny new dump file :-)) Thank you, best regards Harald On Sun, Dec 20, 2009 at 9:47 AM, Frederik Ramm frede...@remote.org mailto:frede...@remote.org wrote: Be aware that --simplify-change is meant to be used with the replicating diffs, and --rci is used to process the old-style (non-replicating) diff files. Oh, I overlooked that, sorry. But then I wonder why Harald has the problem he has - maybe his Osmosis version contains the slightly erroneous patch that I once applied and that was later reverted, where objects are considered not the same if they differ in version number? Hopefully that's the problem, otherwise I'm confused as well. Harald, you might want to investigate the newer --rri task. I really need to update the wiki with up to date task details ... it's getting badly out of date. Brett ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Use API to get bbox of way/relation ids?
2009/12/21 Laurence Penney l...@lorp.org: Thanks... seems a little wasteful to have to get all the individual nodes and tags. Is this the only way to get the bbox of a given way without one's own planet db? If you put /full at the end of the URL it will get all the ways and/or nodes needed. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Use API to get bbox of way/relation ids?
2009/12/21 John Smith deltafoxtrot...@gmail.com: 2009/12/21 Laurence Penney l...@lorp.org: Thanks... seems a little wasteful to have to get all the individual nodes and tags. Is this the only way to get the bbox of a given way without one's own planet db? If you put /full at the end of the URL it will get all the ways and/or nodes needed. Which may be well over 100 MB of xml data for a couple of relations I know of. Cheers ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Use API to get bbox of way/relation ids?
I'd say if this is something you're going to be doing a lot of, it's probably best to download the whole planet (or the part of it you're working on). For the sake of not overloading the API as much as for making things easier on your end. Or you may suggest a patch for the api's source. Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Removing Minutely and Hourly Changesets
On Sun, Dec 20, 2009 at 10:44 AM, Brett Henderson br...@bretth.com wrote: On Sun, Dec 20, 2009 at 6:44 AM, Florian Lohoff f...@rfc822.org wrote: Ich am still using the normal hourly diffs as of lately the replicated diff stuff has not been documented and i didnt want to read java code to find out ... Okay, that's fair enough. I'll update the wiki as soon as I can. I can keep the old ones running for a while if necessary, I don't believe they're causing any major problems, I'm just keen to start cleaning things up to simplify maintenance. The wiki has been updated now. Or at least the --read-replication-interval task has been. http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage#--read-replication-interval_.28--rri.29 Note that you'll need to run the --read-replication-interval-init task first to create a working directory, edit the config file if necessary, and download an appropriate state.txt from the server (some basic instructions are on the wiki). Usage is fairly similar to the old --read-change-interval task. If you get stuck or find the doco lacking let me know. Brett ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Removing Minutely and Hourly Changesets
On Sat, Dec 19, 2009 at 3:06 PM, Brett Henderson br...@bretth.com wrote: Are there any objections to me disabling the old minute and hourly changesets and deleting their files from the planet server? http://planet.openstreetmap.org/minute/ http://planet.openstreetmap.org/minute-slow/ http://planet.openstreetmap.org/hourly/ They are replaced by the (hopefully) reliable transaction based replication changesets. http://planet.openstreetmap.org/minute-replicate/ http://planet.openstreetmap.org/hour-replicate/ I haven't heard any complaints about the replication changesets so I'm assuming they're working well. I'll leave the daily changesets running for now because they are running with a much longer delay and shouldn't miss data. I'm not aware of anybody still using the minute diffs. I'll disable them early in the new year. I won't delete the files straight away so they can be re-enabled if somebody suddenly screams. I hope to disable the hour diffs around the same time but will wait until Florian has moved off them first. Brett ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Changeset files (was Removing Minutely and Hourly Changesets)
Hi Brett, Do you also maintain the changesets files (e.g. changesets-091216.osm.bz2)? Now that the full history dump is out I can confirm that there are a relatively large number of corrupted key/values in that file. For example, see http://www.openstreetmap.org/browse/changeset/2498325 and http://www.openstreetmap.org/browse/changeset/966232 . The dumper is converting control characters into question marks. Anthony ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Changeset files (was Removing Minutely and Hourly Changesets)
Hi Anthony, No, that's not me. I only look after the osmosis changesets. Hmm, the word changeset has become overloaded ... by osmosis changesets I mean the diff/delta files. The planet and changesets-xx.osm.bz2 files are created using the planet dumper tool. I believe Jon and Grant look after those. Brett On Mon, Dec 21, 2009 at 4:09 PM, Anthony o...@inbox.org wrote: Hi Brett, Do you also maintain the changesets files (e.g. changesets-091216.osm.bz2)? Now that the full history dump is out I can confirm that there are a relatively large number of corrupted key/values in that file. For example, see http://www.openstreetmap.org/browse/changeset/2498325 and http://www.openstreetmap.org/browse/changeset/966232 . The dumper is converting control characters into question marks. Anthony ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Changeset files (was Removing Minutely and Hourly Changesets)
Cool. If anyone familiar with the planet dumper tool is listening...
In http://svn.openstreetmap.org/applications/utils/planet.osm/C/output_osm.c
} else if ((*in = 0) (*in 32)) {
escape_tmp[len] = '?';
len++;
should be something like
} else if ((*in 0) (*in 32)) {
len+=sprintf(escape_tmp[len], #%d;, *in);
Something like as in I haven't even checked if that compiles :).
Of course, another thing to consider is that 1024 bytes isn't enough for the
truly pathological cases. I think you need like 1531 or something to handle
that. Fixing this might be enough to properly process the current db,
though.
Any chance of adding num_changes?
On Mon, Dec 21, 2009 at 12:54 AM, Brett Henderson br...@bretth.com wrote:
Hi Anthony,
No, that's not me. I only look after the osmosis changesets. Hmm, the
word changeset has become overloaded ... by osmosis changesets I mean the
diff/delta files.
The planet and changesets-xx.osm.bz2 files are created using the planet
dumper tool. I believe Jon and Grant look after those.
Brett
On Mon, Dec 21, 2009 at 4:09 PM, Anthony o...@inbox.org wrote:
Hi Brett,
Do you also maintain the changesets files (e.g.
changesets-091216.osm.bz2)? Now that the full history dump is out I can
confirm that there are a relatively large number of corrupted key/values in
that file. For example, see
http://www.openstreetmap.org/browse/changeset/2498325 and
http://www.openstreetmap.org/browse/changeset/966232 . The dumper is
converting control characters into question marks.
Anthony
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
[josm-dev] Picture KML support ?
Would it be possible to extend JOSM to be able to load pictures from KML ? (via AgPifoJ plugin or in the core). Like the Potlatch flash editor can - it loads KML in the following format: Document Placemark id=Photo0 description ![CDATA[a href=http://photos.openstreetphoto.org/tijs/dataset/testset/test1/DSC00161.JPG; img src=http://thumbs.openstreetphoto.org/tijs/dataset/testset/test1/DSC00161.JPG; alt=//a]] /description name/name Icon hrefhttp://thumbs.openstreetphoto.org/tijs/dataset/testset/test1/DSC00161.JPG/href /Icon Point coordinates5.37255650003,51.7343925/coordinates /Point /Placemark Placemark id=Photo1 [...] and then shows image placeholders at specified coordinates, and when you click on them, shows you the thumbnail and/or full picture (from URL encoded in KML file)... Such a photo KML files are produced by various tools, for example: http://wiki.openstreetmap.org/wiki/OpenStreetView http://wiki.openstreetmap.org/wiki/OpenStreetPhoto IMO, that would be a great addition. Currently, I've written a small perl script [1] which downloads and geotags all the pictures specified in KML, so those can then be loaded in AgPifoJ plugin; but that is rather suboptimal, as one usually only looks on small subset of pictures. [1] http://linux.voyager.hr/osm/openstreetview_dl.txt -- Opinions above are GNU-copylefted. ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
