[OSM-dev] Import NASA SRTM3 data into Postgres (SOC update)
I think I finished my script that imports NASA SRTM3 data into Postgres. http://sprovoost.nl/2008/05/31/import-nasa-srtm3-data-into-postgres/ Kind regards, Sjors ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] Datamodel relation/member constraints
Thomas Wood schreef: >> type=node ref=1 role=from >> type=node ref=2 role=at >> type=node ref=1 role=at >> type=node ref=3 role=to > If by ref you mean node id, then yes, the second 1 is not allowed. > The API used to throw an unhandled exception back at you (500) if you > did it because of an index constraint on the database. > This has recently been fixed to actually return a nice error and stop > corrupting the database. (Although it may not have been deployed yet) Then I would suggest a change for 0.6 to allow a relation as a path :) Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] Datamodel relation/member constraints
On Fri, May 30, 2008 at 9:21 PM, Stefan de Konink <[EMAIL PROTECTED]> wrote: > Second question: > > Are there currently any constraints on relations members based on: > > 1) Sequence > > and/or > > 2) 'Reuse' > > > So would it be allowed to do: > > type=node ref=1 role=from > type=node ref=2 role=at > type=node ref=1 role=at > type=node ref=3 role=to > > Or is the second 1 not allowed? > > Next to this; can a user explicitly trust the sequence 1,2,1,3; or is > this currently not guaranteed? > > > Stefan > If by ref you mean node id, then yes, the second 1 is not allowed. The API used to throw an unhandled exception back at you (500) if you did it because of an index constraint on the database. This has recently been fixed to actually return a nice error and stop corrupting the database. (Although it may not have been deployed yet) -- Regards, Thomas Wood (Edgemaster) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
[OSM-dev] Datamodel relation/member constraints
Second question: Are there currently any constraints on relations members based on: 1) Sequence and/or 2) 'Reuse' So would it be allowed to do: type=node ref=1 role=from type=node ref=2 role=at type=node ref=1 role=at type=node ref=3 role=to Or is the second 1 not allowed? Next to this; can a user explicitly trust the sequence 1,2,1,3; or is this currently not guaranteed? Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] Datamodel tag/key constraint
Stefan de Konink wrote: > Reading the DTD it seems to be unbound to have multiple tagn having the > same key. Is this allowed? (JOSM doesn't allow it, but is it a constraint?) Currently the API allows duplicate tags, but we don't believe any of the editors do. We plan to remove support for them in API 0.6 as things stand at the moment. Tom -- Tom Hughes ([EMAIL PROTECTED]) http://www.compton.nu/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] Datamodel tag/key constraint
Stefan de Konink wrote: >Sent: 30 May 2008 9:04 PM >To: Openstreetmap Dev list >Subject: [OSM-dev] Datamodel tag/key constraint > >Hi, > > >Reading the DTD it seems to be unbound to have multiple tagn having the >same key. Is this allowed? (JOSM doesn't allow it, but is it a constraint?) > As far as I recall its never been a constraint at the database end, only the editors didn't support it. Cheers Andy > > >Stefan > > > >___ >dev mailing list >dev@openstreetmap.org >http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev > >No virus found in this incoming message. >Checked by AVG. >Version: 8.0.100 / Virus Database: 269.24.4/1473 - Release Date: 29/05/2008 >7:53 PM ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
[OSM-dev] Datamodel tag/key constraint
Hi, Reading the DTD it seems to be unbound to have multiple tagn having the same key. Is this allowed? (JOSM doesn't allow it, but is it a constraint?) Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] C++ implementation of the API
Alex Wilson schreef: > Should have an example C++ api serving from an apache module online soon. I guess it is a good thing that there is development going on for this API thingie :) If you have a demo too, please make a wiki. My demo can be found here: http://wiki.openstreetmap.org/index.php/Cherokee_Handler_OSM Will be working today to get the other GET requests working. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] C++ implementation of the API
Am Freitag, den 30.05.2008, 13:37 +0200 schrieb Steven te Brinke: > SELECT * > FROM current_war_tags > WHERE id IN ( > SELECT * > FROM current_ways > WHERE tile IN (...) >AND latitude BETWEEN ... AND ... >AND longitude BETWEEN ... AND ... > ) > ORDER BY id; > > However, I do not know how the db handles subqueries. As Tom H. said, badly. In a little more Detail: All subqueries with IN are rewritten to EXISTS an executed for each row in the main table. Subqueries with EXIST work as expected. Subqeries in the FROM clause (that can not be correlated) are executed only once. SELECT wt.* FROM current_war_tags wt JOIN ( SELECT * FROM current_ways WHERE tile IN (...) AND latitude BETWEEN ... AND ... AND longitude BETWEEN ... AND ... ) w ON w.id=wt.id ORDER BY id; should do what you want. Non-correlated subselects returning values should also work well. A short general remark on the attitude towards the database: Love her or leave her (also she may not be perfect). Sincerely, Joachim ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
[OSM-dev] Tiger CFCC code conversion
I previous post pointed me to the page on the wiki: http://wiki.openstreetmap.org/index.php/TIGER_to_OSM_Attribute_Map which has most of the Tiger CFCC code conversions. But it is missing the Classification D, E, F, and H. Is there info for these classifications? Specifically, I am interested in D and H. Looking at the OSM style sheet, it looks like D is broken into 5 different styles (using multiple attributes): leisure, amenity, places, stations, and maybe power_towers. It appears H would be in 2 styles (again, multiple attributes). Any places I can look? - Brian ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] C++ implementation of the API
In message <[EMAIL PROTECTED]> Steven te Brinke <[EMAIL PROTECTED]> wrote: > The second query uses a join only to select the correct values, not > because you really want to join the tables. I think that you can achieve > the same by a query like this: > > SELECT * > FROM current_war_tags > WHERE id IN ( > SELECT * > FROM current_ways > WHERE tile IN (...) >AND latitude BETWEEN ... AND ... >AND longitude BETWEEN ... AND ... > ) > ORDER BY id; > > However, I do not know how the db handles subqueries. Thus, I am not > sure if the performance of this query is better. MySQL is rubbish at that kind of subquery - it will table scan the current_ways table and check each one with a lookup in the external. That's why I wrote it as a join ;-) My selection criteria were nonsense of course, as this is ways not nodes... Tom -- Tom Hughes ([EMAIL PROTECTED]) http://www.compton.nu/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] Algorithm help
> Consider a way from 0°, -189.999° to 0°, 180°. that is, in our current > datamodel, a way that's roughly 40,000 km long. No. In the datamodel, it's just a way between two nodes. The nodes are defined in lat, lon, effectively in spherical co-ordinates, hence them being angles. (ignoring not-quite-spherical deformities for now) If a tool says the length is 40,000km, the tool is incorrect. But the datamodel doesn't specify the length, just a sequence of nodes. Tom (assuming you mean -179.99, not -189.99. That would be different) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] C++ implementation of the API
Hi Steven, Thanks for the suggestion. I think it would work nicely on a more sophisticated db, but in my limited experience - MySQL is pretty dumb about running subqueries and you can end up with O(n2) run time. So it's better to stay with dumb joins unless OSM moves to something a little more heavyweight! Alex 2008/5/30 Steven te Brinke <[EMAIL PROTECTED]>: > The second query uses a join only to select the correct values, not > because you really want to join the tables. I think that you can achieve > the same by a query like this: > > SELECT * > FROM current_war_tags > WHERE id IN ( > SELECT * >FROM current_ways >WHERE tile IN (...) > AND latitude BETWEEN ... AND ... > AND longitude BETWEEN ... AND ... > ) > ORDER BY id; > > However, I do not know how the db handles subqueries. Thus, I am not > sure if the performance of this query is better. But it seems to be a > more logical way than using a join. (I am not very sure if using > subqueries works at all, because I know some dbs have problems with that.) > It might be worth trying. > > Steven > > > Tom Hughes schreef: > > You will get some duplication, yes. How much that matters will be > > down to profiling as you say. > > > > The other option is to run two queries in parallel: > > > > SELECT * > > FROM current_ways > > WHERE tile IN (...) > > AND latitude BETWEEN ... AND ... > > AND longitude BETWEEN ... AND ... > > ORDER BY id; > > > > and: > > > > SELECT cwt.* > > FROM current_way_tags cwt > > INNER JOIN current_ways cw ON cwt.id = cw.id > > WHERE cw.tile IN (...) > > AND cw.latitude BETWEEN ... AND ... > > AND cw.longitude BETWEEN ... AND ... > > ORDER BY cwt.id > > > > then read both result sets at the same time and match them up. > > > > Tom > > > > > ___ > dev mailing list > dev@openstreetmap.org > http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev > ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] Algorithm help
Tom Evans schrieb: Our data-model does not support earth as a sphere. Uh? The data model supports it fine as far as I can see. Yes, virtually all the tools don't, because it isn't a problem so far (and is harder...). But they could be improved in the future and the data model would be fine. Or am I missing something? Consider a way from 0°, -189.999° to 0°, 180°. that is, in our current datamodel, a way that's roughly 40,000 km long. -- Dirk-Lüder "Deelkar" Kreie Bremen - 53.0952°N 8.8652°E signature.asc Description: OpenPGP digital signature ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] C++ implementation of the API
The second query uses a join only to select the correct values, not because you really want to join the tables. I think that you can achieve the same by a query like this: SELECT * FROM current_war_tags WHERE id IN ( SELECT * FROM current_ways WHERE tile IN (...) AND latitude BETWEEN ... AND ... AND longitude BETWEEN ... AND ... ) ORDER BY id; However, I do not know how the db handles subqueries. Thus, I am not sure if the performance of this query is better. But it seems to be a more logical way than using a join. (I am not very sure if using subqueries works at all, because I know some dbs have problems with that.) It might be worth trying. Steven Tom Hughes schreef: > You will get some duplication, yes. How much that matters will be > down to profiling as you say. > > The other option is to run two queries in parallel: > > SELECT * > FROM current_ways > WHERE tile IN (...) > AND latitude BETWEEN ... AND ... > AND longitude BETWEEN ... AND ... > ORDER BY id; > > and: > > SELECT cwt.* > FROM current_way_tags cwt > INNER JOIN current_ways cw ON cwt.id = cw.id > WHERE cw.tile IN (...) > AND cw.latitude BETWEEN ... AND ... > AND cw.longitude BETWEEN ... AND ... > ORDER BY cwt.id > > then read both result sets at the same time and match them up. > > Tom > ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] C++ implementation of the API
Thanks Tom. I'll have a play with the various alternatives - and keep the code flexible enough to allow them to be swapped without much fuss. Alex 2008/5/30 Tom Hughes <[EMAIL PROTECTED]>: > In message <[EMAIL PROTECTED]> > Alex Wilson <[EMAIL PROTECTED]> wrote: > > > Ah. good point - a sensible third way. Apologies for missing it, I'm a > > relative SQL novice. However, if you fetch the tags along with the ways > > using a join, given that there are potentially many tags per way - > couldn't > > you end up with considerable duplication of the way data in the query > > result? I suppose one would have to profile both the join method and my > > local storage method to see which had better memory and time performance. > > You will get some duplication, yes. How much that matters will be > down to profiling as you say. > > The other option is to run two queries in parallel: > > SELECT * > FROM current_ways > WHERE tile IN (...) >AND latitude BETWEEN ... AND ... >AND longitude BETWEEN ... AND ... > ORDER BY id; > > and: > > SELECT cwt.* > FROM current_way_tags cwt > INNER JOIN current_ways cw ON cwt.id = cw.id > WHERE cw.tile IN (...) >AND cw.latitude BETWEEN ... AND ... >AND cw.longitude BETWEEN ... AND ... > ORDER BY cwt.id > > then read both result sets at the same time and match them up. > > Tom > > -- > Tom Hughes ([EMAIL PROTECTED]) > http://www.compton.nu/ > ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] C++ implementation of the API
In message <[EMAIL PROTECTED]> Alex Wilson <[EMAIL PROTECTED]> wrote: > Ah. good point - a sensible third way. Apologies for missing it, I'm a > relative SQL novice. However, if you fetch the tags along with the ways > using a join, given that there are potentially many tags per way - couldn't > you end up with considerable duplication of the way data in the query > result? I suppose one would have to profile both the join method and my > local storage method to see which had better memory and time performance. You will get some duplication, yes. How much that matters will be down to profiling as you say. The other option is to run two queries in parallel: SELECT * FROM current_ways WHERE tile IN (...) AND latitude BETWEEN ... AND ... AND longitude BETWEEN ... AND ... ORDER BY id; and: SELECT cwt.* FROM current_way_tags cwt INNER JOIN current_ways cw ON cwt.id = cw.id WHERE cw.tile IN (...) AND cw.latitude BETWEEN ... AND ... AND cw.longitude BETWEEN ... AND ... ORDER BY cwt.id then read both result sets at the same time and match them up. Tom -- Tom Hughes ([EMAIL PROTECTED]) http://www.compton.nu/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] C++ implementation of the API
Ah. good point - a sensible third way. Apologies for missing it, I'm a relative SQL novice. However, if you fetch the tags along with the ways using a join, given that there are potentially many tags per way - couldn't you end up with considerable duplication of the way data in the query result? I suppose one would have to profile both the join method and my local storage method to see which had better memory and time performance. Alex Can't you use a join to fetch the tags along with the ways and then > stream the ways along with their tags? > > Tom > > -- > Tom Hughes ([EMAIL PROTECTED]) > http://www.compton.nu/ > > ___ > dev mailing list > dev@openstreetmap.org > http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev > ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] C++ implementation of the API
In message <[EMAIL PROTECTED]> Alex Wilson <[EMAIL PROTECTED]> wrote: > I'm taking the printf method below - except streaming out using the apache > module interface rather than stdout. I'm also streaming as much stuff from > the db as possible (nodes, ways, . However I believe for the map query > there's a trade-off between local process memory usage and minimising db > queries: For instance when outputting the ways, one can either have > previously built up a local dictionary of way ids to way tag key-value > pairs, or one can query the db each time for the tags for each way. I've > chosen the former - because in C/C++ I think the overhead of storing the > strings locally is fairly minimal. Can't you use a join to fetch the tags along with the ways and then stream the ways along with their tags? Tom -- Tom Hughes ([EMAIL PROTECTED]) http://www.compton.nu/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] C++ implementation of the API
I'm taking the printf method below - except streaming out using the apache module interface rather than stdout. I'm also streaming as much stuff from the db as possible (nodes, ways, . However I believe for the map query there's a trade-off between local process memory usage and minimising db queries: For instance when outputting the ways, one can either have previously built up a local dictionary of way ids to way tag key-value pairs, or one can query the db each time for the tags for each way. I've chosen the former - because in C/C++ I think the overhead of storing the strings locally is fairly minimal. Should have an example C++ api serving from an apache module online soon. Alex 2008/5/30 Raphaël Jacquot <[EMAIL PROTECTED]>: > Joachim Zobel wrote: > > Am Montag, den 26.05.2008, 15:07 +0100 schrieb Tom Hughes: > >> The reason is that we have to allow about 600Mb or so for each call > >> and that quickly mounts up as you try and add extra simultaneous > >> accesses. > > > > If _that_ amount of memory is needed this probably means the XML is > > build in memory. This could be done the SAX way instead. > > > considering how simple the xml is, it could probably be done the printf > way, that sure would make it less memory abusive. > one thing that it may be doing, is getting *all results* in one go from > the database server, instead of one result at a time, resulting in > having the entire set of data in the web server's machine memory at some > point. > requesting one row at a time *will* result in instant memory consumption > cutoff and won't have any effet on the database server, which is much > better at handling this sort of thing (it can handle the same row in ram > for various requests at the same time) > > > Sincerely, > > Joachim > > raphael > (who now works on seismic data sets...) > > > ___ > dev mailing list > dev@openstreetmap.org > http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev > ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
