Re: [OSM-dev] OSM formats optimised for client-side vector rendering?
yes, for the record, in Cartagen the limitation was that JavaScript cannot read XML, so my fork of the Rails port generates JSON instead ( https://github.com/jywarren/openstreetmap-website). But there is a parsing step where the node, way, and relation IDs are used to reference them all together once they are imported as JavaScript objects. This is expensive but only performed upon import. This also allows for knockout of duplicate areas of import and such, and for reconnecting relations across vector tile boundaries. On import Cartagen also calculates way area. And if you try reassembling way/node/relation references on the server side on the fly, your system will slow to a crawl. So there are lots of advantages to logically separating raw data from efficiently renderable data. Jeff On Thu, Feb 17, 2011 at 7:24 AM, Igor Brejc igor.br...@gmail.com wrote: Hi, There are two different things here: - A format for efficiently transmitting storing OSM data in the memory - A data model for efficient rendering. These are not the same, they serve different purposes and usually if you want memory efficiency, you'll loose out on the rendering speed side. In Maperitive I keep these things separate, and the data structures for rendering are quite different from the OSM data model. Handling multipolygons, for example, requires transforming the OSM data model into something more useful for rendering (i.e. polygons with holes). And there are other kinds of relations which are too expensive to reconstruct each time you want to update the screen map. Another example: when Maperitive renders residential roads (or any other kind of way, for that matter), it first constructs a graph of these roads to be able to better place labels, shields etc. This is done only once and then reused while the map is alive (or until the rendering stylesheet changes). On the other hand, loosing node references would make it almost impossible to construct such a graph. So I think you still need the base OSM model somewhere - keeping just x,y coords is not enough (well, for Maperitive anyway). Igor On Thu, Feb 17, 2011 at 11:20 AM, Nick Whitelegg nick.whitel...@solent.ac.uk wrote: Hi, Much of the stuff I'm working on or have been recently involves loading OSM data from file or the web, and rendering it client side (e.g. WebGL OSM viewer and an augmented-reality app for walkers which I'm just starting work on now). Because of this, my server (OSM PostGIS-based) provides OSM data not in the standard .osm format, but in a rendering-optimised format with ways consisting of a series of x,y coords rather than node references, e.g.: http://www.free-map.org.uk/ws/trsvr.php?x=4079y=2740z=13 However I'm wondering if there is any consensus on a standard OSM data format optimised for vector rendering. There seems to be the OSM Mobile Binary Protocol though by the looks of things, that's only used in one application: the Symbian app whereami. So does everyone just use their own or are there any formats used by multiple apps? Thanks, Nick ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] tms translator
Hello all - At OpenMapsCaucasus/JumpStart and in other places I've had trouble reconciling the standard OSGeo implementation (origin at bottom-left) of TMS which gdal2tiles.py generates, and the OSM convention (origin at top-left). There may be a simpler way to do this, but for on-the-fly translation between the two implementations, we've created this tool: http://tms.mapspot.ge/ You enter one or the other, and it generates links for both formats. It asks you to register (name, description etc. all optional) and generates a relatively short URL for each. Please feel free to use it, fork the source: http://github.com/jywarren/tms-filter or tell us how to improve it. This has, for example, allowed JOSM viewing of gdal2tiles.py derived tiles with minimal fuss. Jeff ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Cartagen Knitter TMS, GeoTiff, can be used with JOSM
yes, google maps is a reference design - the code accepts any TMS. I'm going to add a few public domain options (USGS National Map is good within the US, for example) and an arbitrary TMS/WMS input. Though i do wonder if there's ownership precedent for derived orthorectification... makes sense, but how would, for example, Yahoo feel about it given that they've allowed tracing. Of course you can also orthorectify against existing OSM data, which is the default. Kind of chicken/egg, but it's basically allowing you to use a few GPX traces or what have you as a basis for rectifying a much larger raster dataset. Some people put big red X's on the ground before taking aerial imagery, noting their coordinates for this purpose. Paul, did you mean a place to store the aerial imagery itself within OSM? If so, you might check out OpenAerialMap, which it seems folks are getting ready to re-attempt. Jeff On Mon, Aug 9, 2010 at 7:41 PM, andrzej zaborowski balr...@gmail.comwrote: On 4 August 2010 15:14, Jeffrey Warren war...@mit.edu wrote: Therefore you can now use aerial images from balloons or kites (or whatever! slingshots!) to stitch a map, then export it for printing or import it into many other geospatial tools. All in the browser, no plugins. That last one, TMS, means you can now use the Slippy Map plugin for JOSM to trace your aerial imagery and submit it to OpenStreetMap. Something that would be useful for OSM would be a way to add new background layers from a WMS or TMS url, maybe there is a way to do that that I missed. I only see Google Maps layer using which will make your georeferenced imagery incompatible with OSM. Another nice feature would be adding more control points on the edges of a picture or even in the middle (this would probably require splitting the pic canvas in four squares for each new gcp). Cheers ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Cartagen Knitter TMS, GeoTiff, can be used with JOSM
Hi all - I added a range of new publication options to the Cartagen Knitter software (http://cartagen.org/maps) yesterday, amongst them: - Output as high-resolution JPG and GeoTIFF - Output as TMS tile service for use in OpenLayers, etc. Therefore you can now use aerial images from balloons or kites (or whatever! slingshots!) to stitch a map, then export it for printing or import it into many other geospatial tools. All in the browser, no plugins. That last one, TMS, means you can now use the Slippy Map plugin for JOSM to trace your aerial imagery and submit it to OpenStreetMap. See TMS output of a 5.5 km long map made in Georgia with Jumpstart Internationa/OpenMapsCaucasus here: http://cartagen.org/tms/mestia/openlayers.html That was made in 3 days with a balloon and a canon camera. See the flight paths here: (http://www.flickr.com/photos/jeffreywarren/4785997308/) A couple pictures here: http://www.flickr.com/photos/jeffreywarren/4859434543/, http://www.flickr.com/photos/jeffreywarren/4860184506/ But most importantly, the new Export button is in the upper-right corner of the screen. Beware, exporting a map can take quite a while! *I'd appreciate it if you didn't export the Mestia map because it takes over an hour and slams my server.* Also please be nice in general as its just been put up and bugs may persist. Best, Jeff ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] OSM diagram usage
Hi all - i'm writing about OSM and was wondering if I could use the diagram at: http://wiki.openstreetmap.org/wiki/File:OSM_Components.png It's not marked as CC or anything. Would whoever owns it mind releasing it or giving permission? Thanks much! Jeff ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Edit in JOSM link
It'd be very cool to have JOSM register a custom URL scheme as in: http://burnttoys.blogspot.com/2008/07/adding-url-scheme-to-qt-application.html however this would only work if you had JOSM installed; and if you didn't it might leave the user with a broken link. But this would be cool: josm://openstreetmap.org/edit?lat=xlon=x Jeff On Mon, Mar 8, 2010 at 11:05 PM, John Smith deltafoxtrot...@gmail.comwrote: On 9 March 2010 13:55, Bernhard zwischenbrugger b...@datenkueche.com wrote: To open a map view in potlatch I use something like: http://www.openstreetmap.org/edit?lat=11.561150369579765lon=104.91780691575758zoom=17 But what's to link to open the map view JOSM? The remote control plugin must be loaded in JOSM then you can call this URL: http://localhost:8111/load_and_zoom?left=151.2019right=151.2219top=-33.8379bottom=-33.8579select=relation384683 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] I converted the rails port to Git
I just did a svn/googlecode to git/github import for Cartagen: github.com/jywarren/cartagen/ and ran this perl script to import issues. It worked pretty well, could be adapted to trac: http://gist.github.com/101002 jeff On Sun, Feb 21, 2010 at 8:35 PM, Chris Browet c...@semperpax.com wrote: For what it's worth, you have my full support regarding moving to git. I was considering making a merkaartor clone to gitorious myself, SVN seems so limited when you start working with git or bazaar from multiple branches. As for myself, I like bazaar because the learning curve seems easier coming from svn, and lauchpad is a nice repository with great support for translations. Perhaps a little bit too ubuntu-esque, though. Beware that Trac+Git might not be hassle-free, apparently. Maybe switch to redmine? IMHO, redmine is to trac what git is to svn... - Chris - ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Noobie Question - Custom Map
You could take a look at wiki.cartagen.org or perhaps CloudMade's Style Editor. Jeff On Fri, Dec 4, 2009 at 7:30 AM, Ben Marchbanks b...@magazooms.com wrote: Are there any opportunities to control the style of vectors on the maps ? For example I export a large map as PDF but the street vectors are so small until you zoom in. Would like to have a map render with street vector options that allow: 1. Styling options for vectors a.stroke width / color for streets 2. Option to choose street levels to include/exclude on the map a. Interstate b. Secondary ( state roads) c. unpaved , neighborhood streets What a great resource ! Looking forward to any contribution I can make to the effort. -- *Ben Marchbanks* www.magazooms.com [image: Signature] Email: b...@magazooms.com Phone: (864) 284.9918 www.alQemy.com www.iCampAmerica.com ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev mzLogo.gif___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Quantifying
Hi all - I tried to do a rough analysis (for an article I'm writing) of the composition of an API response by character count. If anyone knows more about this or has opinions or ideas for better (but lightweight) means of quantifying this, i'd love to hear them. I’ve been wondering how much of the data transferred by the OpenStreetMap API is actual geometric data as opposed to timestamp and author data. I ran some rough numbers on a typical API response (in JSON, not XML, though these are relative measurements, so it shouldn’t matter too much). The file I examined is here: dr5ru0.jsonhttp://unterbahn.com/wp-content/uploads/2009/09/dr5ru0.json but my count is by number of characters and I did include formatting. Graph and blog post here: it'd be nice to have comments and answers posted here, so they're archived: http://unterbahn.com/2009/10/composition-of-a-openstreetmap-api-responses/ Thanks! Jeff ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Osmosis and Postgresql
No, the data is not corrupt, nor are there duplicate entries. Quite a few current_way_nodes are not getting deleted when they cease to be 'current', that is, once they're only history and no longer part of the current planet. I asked earlier about what mechanism actually deletes current_way_nodes if marked for deletion (i think the tag is 'visible=false' but i might be mistaken) but nobody seemed to know - it seems the best route is to go through and delete those orphans, but I couldn't write a single SQL query to do so efficiently and before writing a script to do it, I realized that the rails port doesn't really care if they're there, or if the constraint is created. Jeff On Tue, Jul 7, 2009 at 3:24 PM, Shaun McDonald sh...@shaunmcdonald.me.ukwrote: On 7 Jul 2009, at 18:47, Joachim Zobel wrote: Am Mittwoch, den 01.07.2009, 09:47 +1000 schrieb Brett Henderson: Jeffrey Warren wrote: And along those lines, should we remove the foreign key constraints from the schema, Brett, if they're not necessary and they cause Osmosis imports to fail? I don't mind really. I only make them available as a convenience so people don't need to run the rails migrations themselves. I don't want to put much effort into maintaining them so I just leave them as is. Is this something you'd be interested in looking after? I am somewhat confused, maybe I missed something. Either you can recreate the constraint if the import is done or the data is _corrupt_. If you are loading a larger volume of data you probably want to drop _all_ constraints and indexes before that, recreate some in the middle and the rest afterwards. Could it be that your constraint failures are due to a node/way/ relation containing two keys that have different capitalisation? Shaun ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Osmosis and Postgresql
Hmm, I'm not sure I'm equipped to do that. On Tue, Jun 30, 2009 at 7:47 PM, Brett Henderson br...@bretth.com wrote: I don't mind really. I only make them available as a convenience so people don't need to run the rails migrations themselves. I don't want to put much effort into maintaining them so I just leave them as is. Is this something you'd be interested in looking after? ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Osmosis and Postgresql
I'd like to, but how does the schema you want differ from the schemas Brett Henderson hosts? And along those lines, should we remove the foreign key constraints from the schema, Brett, if they're not necessary and they cause Osmosis imports to fail? Jeff On Tue, Jun 23, 2009 at 4:48 PM, Graham Jones grahamjones...@googlemail.com wrote: Jeffrey, I can't help with your error, I'm afraid, but if you have an up to date description of the database schema I would be happy to add it ot the summary page I am writing. I think that http://wiki.openstreetmap.org/wiki/Database/Model is supposed to be a description of the main API DB schema, but it is labelled as out of date, so maybe this would be a good place for your description to be stored? Regards Graham. 2009/6/22 Jeffrey Warren war...@mit.edu Hi, I'm compiling a very long description of a Postgres/Rails port schema import, though I'm writing directly to the db. I'd be happy to share/post my notes on the process as I managed to get around a number of undocumented problems. I'm still not done right now (i've been pinging the list here occasionally) and my latest problem is that I'm getting several errors like the following: ERROR: insert or update on table current_way_nodes violates foreign key constraint current_way_nodes_node_id_fkey DETAIL: Key (node_id)=(17704640) is not present in table current_nodes. ** Error ** ERROR: insert or update on table current_way_nodes violates foreign key constraint current_way_nodes_node_id_fkey SQL state: 23503 Detail: Key (node_id)=(17704640) is not present in table current_nodes. These are current_way_nodes which were not deleted from the current_ table when their matching current_nodes were deleted. I'm now trying to delete them manually since this only occurs where nodes are visible=false, but it's confusing when this causes the foreign key constraint creation to fail: ALTER TABLE ONLY current_way_nodes ADD CONSTRAINT current_way_nodes_node_id_fkey FOREIGN KEY (node_id) REFERENCES current_nodes(id); Has anyone else encountered this? Is there a better way to resolve it? I tried following how the rails port actually deletes from current_nodes, but was unable to find it; is this done outside rails code? Can someone point to a line number where this is done so I can try to find out why the dependent current_way_node record wasn't deleted also? Hopefully this will eliminate this problem for future submitted changesets. Jeff history of a node with leftover current_way_node == ?xml version=1.0 encoding=UTF-8? osm version=0.6 generator=OpenStreetMap server node id=17704640 lat=51.5465346 lon=-2.4116768 changeset=127793 user=southglos uid=3937 visible=true timestamp=2006-10-06T11:08:17Z version=1 tag k=created_by v=JOSM/ /node node id=17704640 lat=51.5465346 lon=-2.4116768 changeset=601639 user=Strange but untrue uid=57932 visible=true timestamp=2009-04-16T22:28:59Z version=2/ node id=17704640 lat=51.5465346 lon=-2.4116768 changeset=601639 user=Strange but untrue uid=57932 visible=true timestamp=2009-04-16T22:31:23Z version=3/ node id=17704640 lat=51.5465346 lon=-2.4116768 changeset=601639 user=Strange but untrue uid=57932 visible=true timestamp=2009-04-16T22:32:07Z version=4/ node id=17704640 lat=51.5465346 lon=-2.4116768 changeset=601639 user=Strange but untrue uid=57932 visible=true timestamp=2009-04-16T22:32:19Z version=5/ node id=17704640 lat=51.5465346 lon=-2.4116768 changeset=601639 user=Strange but untrue uid=57932 visible=false timestamp=2009-04-16T22:32:21Z version=6/ /osm On Mon, Jun 22, 2009 at 4:09 PM, 80n 80n...@gmail.com wrote: On Mon, Jun 22, 2009 at 2:29 AM, Brett Henderson br...@bretth.comwrote: Graham Jones wrote: Brett, I'll have a look at --write-pgsql-dump and add some words about that too. Then I'll have a go at MySQL... The other thing I would like to add is a write-up on choosing a database to use, because it is not obvious to me which would be the best, but I suspect this has been looked at a lot before now. I think the choices are: I'm not aware of a page summarising this, so it sounds very useful. Lots of people ask about it. * PostgreSQL/PostGIS - Can handle big datasets, and has geographic extensions. What I do not know is whether these extensions are fast or not, or if for simple things like selecting for a bounding box it would be quicker to just select on lat/lon directly. Mapnik uses this database, but the schema is optimised for rendering. There's actually 3 PostgreSQL schemas which causes much confusion: 1. The API database. This is the database behind the main OSM ruby-based API. This is supported by the osmosis apidb tasks. The only db capable of holding full history. 2. The Mapnik database. Used by the mapnik renderer. Loaded using osm2pgsql
Re: [OSM-dev] Osmosis and Postgresql
Agreed, it's quite confusing. The page you reference seems hard to maintain since it's basically an image. Perhaps a sub-page for each schema would be nice? At least mentioning the simple-schema, the rails-port schema (the one in the migrations) and the one brett hosts, those three would be quite nice to list with some notes on each and a link to download them. The simple schema comes with Osmosis and excludes history, i think. I have notes on getting the migrations to run so if you stub out the pages I'll put what notes i have in them. Jeff On Tue, Jun 30, 2009 at 1:00 PM, Graham Jones grahamjones...@googlemail.com wrote: I don't think I want a different one, I was just struggling to understand what the differences are between all these different databases that people use so I can choose one, and thought it would be worth documenting it once I had worked it out! Graham 2009/6/30 Jeffrey Warren war...@mit.edu I'd like to, but how does the schema you want differ from the schemas Brett Henderson hosts? And along those lines, should we remove the foreign key constraints from the schema, Brett, if they're not necessary and they cause Osmosis imports to fail? Jeff On Tue, Jun 23, 2009 at 4:48 PM, Graham Jones grahamjones...@googlemail.com wrote: Jeffrey, I can't help with your error, I'm afraid, but if you have an up to date description of the database schema I would be happy to add it ot the summary page I am writing. I think that http://wiki.openstreetmap.org/wiki/Database/Model is supposed to be a description of the main API DB schema, but it is labelled as out of date, so maybe this would be a good place for your description to be stored? Regards Graham. 2009/6/22 Jeffrey Warren war...@mit.edu Hi, I'm compiling a very long description of a Postgres/Rails port schema import, though I'm writing directly to the db. I'd be happy to share/post my notes on the process as I managed to get around a number of undocumented problems. I'm still not done right now (i've been pinging the list here occasionally) and my latest problem is that I'm getting several errors like the following: ERROR: insert or update on table current_way_nodes violates foreign key constraint current_way_nodes_node_id_fkey DETAIL: Key (node_id)=(17704640) is not present in table current_nodes. ** Error ** ERROR: insert or update on table current_way_nodes violates foreign key constraint current_way_nodes_node_id_fkey SQL state: 23503 Detail: Key (node_id)=(17704640) is not present in table current_nodes. These are current_way_nodes which were not deleted from the current_ table when their matching current_nodes were deleted. I'm now trying to delete them manually since this only occurs where nodes are visible=false, but it's confusing when this causes the foreign key constraint creation to fail: ALTER TABLE ONLY current_way_nodes ADD CONSTRAINT current_way_nodes_node_id_fkey FOREIGN KEY (node_id) REFERENCES current_nodes(id); Has anyone else encountered this? Is there a better way to resolve it? I tried following how the rails port actually deletes from current_nodes, but was unable to find it; is this done outside rails code? Can someone point to a line number where this is done so I can try to find out why the dependent current_way_node record wasn't deleted also? Hopefully this will eliminate this problem for future submitted changesets. Jeff history of a node with leftover current_way_node == ?xml version=1.0 encoding=UTF-8? osm version=0.6 generator=OpenStreetMap server node id=17704640 lat=51.5465346 lon=-2.4116768 changeset=127793 user=southglos uid=3937 visible=true timestamp=2006-10-06T11:08:17Z version=1 tag k=created_by v=JOSM/ /node node id=17704640 lat=51.5465346 lon=-2.4116768 changeset=601639 user=Strange but untrue uid=57932 visible=true timestamp=2009-04-16T22:28:59Z version=2/ node id=17704640 lat=51.5465346 lon=-2.4116768 changeset=601639 user=Strange but untrue uid=57932 visible=true timestamp=2009-04-16T22:31:23Z version=3/ node id=17704640 lat=51.5465346 lon=-2.4116768 changeset=601639 user=Strange but untrue uid=57932 visible=true timestamp=2009-04-16T22:32:07Z version=4/ node id=17704640 lat=51.5465346 lon=-2.4116768 changeset=601639 user=Strange but untrue uid=57932 visible=true timestamp=2009-04-16T22:32:19Z version=5/ node id=17704640 lat=51.5465346 lon=-2.4116768 changeset=601639 user=Strange but untrue uid=57932 visible=false timestamp=2009-04-16T22:32:21Z version=6/ /osm On Mon, Jun 22, 2009 at 4:09 PM, 80n 80n...@gmail.com wrote: On Mon, Jun 22, 2009 at 2:29 AM, Brett Henderson br...@bretth.comwrote: Graham Jones wrote: Brett, I'll have a look at --write-pgsql-dump and add some words about that too. Then I'll have a go at MySQL... The other thing I would like to add is a write-up on choosing a database to use
Re: [OSM-dev] Osmosis and Postgresql
I dropped it as well; there are many instances of data in the planet dump which do not meet the constraint; do you know what part of the codebase does in fact rely upon the constraints? Jeff On Tue, Jun 23, 2009 at 9:09 AM, Brett Henderson br...@bretth.com wrote: Jeffrey Warren wrote: Hi, I'm compiling a very long description of a Postgres/Rails port schema import, though I'm writing directly to the db. I'd be happy to share/post my notes on the process as I managed to get around a number of undocumented problems. I'm still not done right now (i've been pinging the list here occasionally) and my latest problem is that I'm getting several errors like the following: ERROR: insert or update on table current_way_nodes violates foreign key constraint current_way_nodes_node_id_fkey DETAIL: Key (node_id)=(17704640) is not present in table current_nodes. ** Error ** ERROR: insert or update on table current_way_nodes violates foreign key constraint current_way_nodes_node_id_fkey SQL state: 23503 Detail: Key (node_id)=(17704640) is not present in table current_nodes. These are current_way_nodes which were not deleted from the current_ table when their matching current_nodes were deleted. I'm now trying to delete them manually since this only occurs where nodes are visible=false, but it's confusing when this causes the foreign key constraint creation to fail: ALTER TABLE ONLY current_way_nodes ADD CONSTRAINT current_way_nodes_node_id_fkey FOREIGN KEY (node_id) REFERENCES current_nodes(id); Has anyone else encountered this? Is there a better way to resolve it? When testing local imports I drop that constraint from the database. I'm not aware of any better ways of solving the problem. Brett ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Osmosis error, duplicate
I've been importing with: motel:osmosis-0.31 eco$ bin/osmosis --read-xml-0.6 file=/Volumes/Planet/planet.osm --write-apidb-0.6 populateCurrentTables=yes host=localhost database= user= password= validateSchemaVersion=no and it's been running the following query for about 24 hours: SELECT Count(*) AS changesetCount FROM changesets WHERE id=$1 I don't remember this occurring in previous import attempts. Has it stalled, and should I cancel and try again? (As mentioned above I'd like to skip changesets if they're going to add a lot of time to the import. Also I couldn't find documentation for skipping changesets; maybe I should just remove that table and it'll figure it out?) I've been using the first half of the apidb06-latest schema, following directions here: http://gis.hsr.ch/wiki/HowTo_OpenStreetMap#Howto_mit_PostgreSQL Jeff On Tue, Jun 2, 2009 at 4:36 PM, Joachim Zobel jzo...@heute-morgen.dewrote: Am Dienstag, den 02.06.2009, 11:52 -0700 schrieb Jeffrey Warren: I was importing a planet dump to pgsql with the --write-apidb-0.6 and got the error: ERROR: insert or update on table current_way_nodes violates foreign key constraint current_way_nodes_node_id_fkey Detail: Key (node_id)=(395051992) is not present in table current_nodes. Why would that cause osmosis to fail? Not shure. Osmosis should work with foreign keys, so this may be a bug. I would however recommend loading the data into a database without any indexes or foreign keys and then recreate them. See http://gis.hsr.ch/wiki/HowTo_OpenStreetMap#Howto_mit_PostgreSQL This is much faster for that size. The reason is that INSERTing otherwise writes the data _and_ the indexes. This causes the heads to move back and forth (unless you have seperate disks for the data and all its indexes), which means a major slowdown. I seem to be left with only 42448 current_ways, and no current_relations or current_relation_tags. Can I restart osmosis without re-running the entire import (5 days)? Sorry, probably you can't. Sincerely, Joachim ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Osmosis error, duplicate
Thanks, I was tracing the original foreign key constraint error (reproduced in full below) and this line caught my eye: Detail: Key (node_id)=(395051992) is not present in table current_nodes. current_nodes has no node_id column! The constraint is created in the latest pgsql schema, here: (apidb06-pgsql-latest.sql:2132) -- -- TOC entry 2018 (class 2606 OID 35690) -- Dependencies: 1579 1961 1589 -- Name: current_way_nodes_node_id_fkey; Type: FK CONSTRAINT; Schema: public; Owner: osm -- ALTER TABLE ONLY current_way_nodes ADD CONSTRAINT current_way_nodes_node_id_fkey FOREIGN KEY (node_id) REFERENCES current_nodes(id); I changed line 2139 to: ADD CONSTRAINT current_way_nodes_node_id_fkey FOREIGN KEY (id) REFERENCES current_nodes(id); I'm going to run it now to see what happens. Best, Jeff ### motel:bin eco$ cd ../motel:osmosis-0.31 eco$ bin/osmosis --read-xml-0.6 file=/Volumes/Planet/planet.osm --write-apidb-0.6 populateCurrentTables=yes host=localhost database=osm-planet user=eco password=poopies validateSchemaVersion=no May 26, 2009 8:59:51 PM org.openstreetmap.osmosis.core.Osmosis run INFO: Osmosis Version 0.31 May 26, 2009 8:59:51 PM org.openstreetmap.osmosis.core.Osmosis run INFO: Preparing pipeline. May 26, 2009 8:59:51 PM org.openstreetmap.osmosis.core.Osmosis run INFO: Launching pipeline execution. May 26, 2009 8:59:51 PM org.openstreetmap.osmosis.core.Osmosis run INFO: Pipeline executing, waiting for completion. Jun 1, 2009 6:50:20 PM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion SEVERE: Thread for task 1-read-xml-0.6 failed org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to load current way nodes. at org.openstreetmap.osmosis.core.apidb.v0_6.ApidbWriter.complete(ApidbWriter.java:980) at org.openstreetmap.osmosis.core.xml.v0_6.XmlReader.run(XmlReader.java:110) at java.lang.Thread.run(Thread.java:637) Caused by: org.postgresql.util.PSQLException: ERROR: insert or update on table current_way_nodes violates foreign key constraint current_way_nodes_node_id_fkey Detail: Key (node_id)=(395051992) is not present in table current_nodes. at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:1592) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1327) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:192) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:451) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:350) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:343) at org.openstreetmap.osmosis.core.apidb.v0_6.ApidbWriter.complete(ApidbWriter.java:977) ... 2 more Jun 1, 2009 6:50:20 PM org.openstreetmap.osmosis.core.Osmosis main SEVERE: Execution aborted. org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks failed. at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146) at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:85) at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:30) On Fri, Jun 5, 2009 at 1:29 PM, Joachim Zobel jz-2...@heute-morgen.dewrote: Am Freitag, den 05.06.2009, 12:40 -0400 schrieb Jeffrey Warren: and it's been running the following query for about 24 hours: SELECT Count(*) AS changesetCount FROM changesets WHERE id=$1 Take a look on Brets pgsql-latest.sql script (after the COPYs) and add the index on id: ALTER TABLE ONLY changesets ADD CONSTRAINT pk_changesets PRIMARY KEY (id); Sincerely, Joachim ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Osmosis error, duplicate
OK, that makes sense. But then what is the error: Detail: Key (node_id)=(395051992) is not present in table current_nodes. from? That's what crashed my import. I'm trying to find where Osmosis requests a node_id from current_nodes; or am I misreading the error? Thanks, Jeff On Fri, Jun 5, 2009 at 3:39 PM, Joachim Zobel jz-2...@heute-morgen.dewrote: Am Freitag, den 05.06.2009, 14:58 -0400 schrieb Jeffrey Warren: Detail: Key (node_id)=(395051992) is not present in table current_nodes. The field that is present is actually named node_id and that is the one that is mentioned. ALTER TABLE ONLY current_way_nodes ADD CONSTRAINT current_way_nodes_node_id_fkey FOREIGN KEY (node_id) REFERENCES current_nodes(id); This is correct, the table current_way_nodes does have a nodes_id column which points to current_nodes.id. Sincerely, Joachim ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] gweb.bretth.com
http://gweb.bretth.com/apidb06-pgsql-latest.sql is returning: The bandwidth or page view limit for this site has been exceeded and the page cannot be viewed at this time. Once the site is below the limit, it will once again begin serving as normal. Does anyone have a copy of the latest schema they could forward me? Thanks, Jeff ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Osmosis error, duplicate
I was importing a planet dump to pgsql with the --write-apidb-0.6 and got the error: ERROR: insert or update on table current_way_nodes violates foreign key constraint current_way_nodes_node_id_fkey Detail: Key (node_id)=(395051992) is not present in table current_nodes. Why would that cause osmosis to fail? I seem to be left with only 42448 current_ways, and no current_relations or current_relation_tags. Can I restart osmosis without re-running the entire import (5 days)? Thanks for any help - Jeff ==Full error:= motel:bin eco$ cd ../motel:osmosis-0.31 eco$ bin/osmosis --read-xml-0.6 file=/Volumes/Planet/planet.osm --write-apidb-0.6 populateCurrentTables=yes host=localhost database=osm-planet user=eco password=poopies validateSchemaVersion=no May 26, 2009 8:59:51 PM org.openstreetmap.osmosis.core.Osmosis run INFO: Osmosis Version 0.31 May 26, 2009 8:59:51 PM org.openstreetmap.osmosis.core.Osmosis run INFO: Preparing pipeline. May 26, 2009 8:59:51 PM org.openstreetmap.osmosis.core.Osmosis run INFO: Launching pipeline execution. May 26, 2009 8:59:51 PM org.openstreetmap.osmosis.core.Osmosis run INFO: Pipeline executing, waiting for completion. Jun 1, 2009 6:50:20 PM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion SEVERE: Thread for task 1-read-xml-0.6 failed org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to load current way nodes. at org.openstreetmap.osmosis.core.apidb.v0_6.ApidbWriter.complete(ApidbWriter.java:980) at org.openstreetmap.osmosis.core.xml.v0_6.XmlReader.run(XmlReader.java:110) at java.lang.Thread.run(Thread.java:637) Caused by: org.postgresql.util.PSQLException: ERROR: insert or update on table current_way_nodes violates foreign key constraint current_way_nodes_node_id_fkey Detail: Key (node_id)=(395051992) is not present in table current_nodes. at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:1592) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1327) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:192) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:451) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:350) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:343) at org.openstreetmap.osmosis.core.apidb.v0_6.ApidbWriter.complete(ApidbWriter.java:977) ... 2 more Jun 1, 2009 6:50:20 PM org.openstreetmap.osmosis.core.Osmosis main SEVERE: Execution aborted. org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks failed. at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146) at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:85) at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:30) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] type four_byte_unsigned does not exist
Ok well i have the rails port running (and serving JSON instead of XML, hooray!), but I'd like to try getting the tests passing too; where's a place to document that process thoroughly? the wiki? Jeff On Wed, May 27, 2009 at 2:16 AM, Andy Allan gravityst...@gmail.com wrote: On Wed, May 27, 2009 at 12:19 AM, Jeffrey Warren war...@mit.edu wrote: It's just a big slice of Rome, so i doubt the data is the problem. I got a lot of errors, most were because i couldn't get libpgosm.so to compile for OSX Leopard, (maptile_for_point() does not exist, etc), but there were a lot more too. Given that mine is not really a standard install (lacking libpgosm) i don't know if you'd be interested in seeing my errors... if you are, tell me and I'll post them. Jeff If the tests have errors that generally means something is broken in the setup. Maybe db permissions or some such. I check the tests before I even get to booting up WeBrick for the first time. Cheers, Andy ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] type four_byte_unsigned does not exist
I noticed that during db:migrate it doesn't pick up the /lib/migrate.rb type definitions... so I just changed it to :integer, :limit = 8, which is how four_byte_unsigned is defined in migrate.rb. I'm also setting up the rails port and am having trouble so if you want to compare notes I'd love to. Best, Jeff On Tue, May 26, 2009 at 1:31 PM, Dan Karran d...@karran.net wrote: I'm trying to set up a copy of the rails site on my machine, using Postgres, but I'm running into a problem on the initial db:migrate ... can anyone point me to how to fix this? = rake db:migrate --trace (in /Users/dankarran/Sites/openstreetmap) ** Invoke db:migrate (first_time) ** Invoke environment (first_time) ** Execute environment ** Execute db:migrate == TileTracepoints: migrating -- add_column(gps_points, tile, :four_byte_unsigned) rake aborted! An error has occurred, this and all later migrations canceled: PGError: ERROR: type four_byte_unsigned does not exist LINE 1: ALTER TABLE gps_points ADD COLUMN tile four_byte_unsigne... ^ : ALTER TABLE gps_points ADD COLUMN tile four_byte_unsigned = Thanks, Dan -- Dan Karran d...@karran.net www.dankarran.com ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] type four_byte_unsigned does not exist
which migration is that? I need pgsql so I'm sticking to my guns, but i think i'm doing all right now... i briefly had a problem with btree_gist but after struggling to make that, i retried rake db:migrate and i successfully completed the migrations. if you have booted the application in another shell before running rake db:migrate, you get a much more verbose set of errors in that first script/server shell, which really helped me figure out what was failing. what's failing specifically in the migration in question? BTW, Tom's fix is a much better idea than mine, esp. since it'll be making its way into the trunk code whereas mine's just a hack. Jeff On Tue, May 26, 2009 at 2:08 PM, Dan Karran d...@karran.net wrote: Hi Jeff, 2009/5/26 Jeffrey Warren war...@mit.edu: I noticed that during db:migrate it doesn't pick up the /lib/migrate.rb type definitions... so I just changed it to :integer, :limit = 8, which is how four_byte_unsigned is defined in migrate.rb. I'm also setting up the rails port and am having trouble so if you want to compare notes I'd love to. Thanks, though it doesn't seem to get much further than that once I've made the change, stopping after trying to remove the index points_idx. Would you be able to send over a diff of any necessary changes you've made? Is it any more successful with MySQL? :) Cheers, Dan Best, Jeff On Tue, May 26, 2009 at 1:31 PM, Dan Karran d...@karran.net wrote: I'm trying to set up a copy of the rails site on my machine, using Postgres, but I'm running into a problem on the initial db:migrate ... can anyone point me to how to fix this? = rake db:migrate --trace (in /Users/dankarran/Sites/openstreetmap) ** Invoke db:migrate (first_time) ** Invoke environment (first_time) ** Execute environment ** Execute db:migrate == TileTracepoints: migrating -- add_column(gps_points, tile, :four_byte_unsigned) rake aborted! An error has occurred, this and all later migrations canceled: PGError: ERROR: type four_byte_unsigned does not exist LINE 1: ALTER TABLE gps_points ADD COLUMN tile four_byte_unsigne... ^ : ALTER TABLE gps_points ADD COLUMN tile four_byte_unsigned = Thanks, Dan -- Dan Karran d...@karran.net www.dankarran.com ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev -- Dan Karran d...@karran.net www.dankarran.com ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] type four_byte_unsigned does not exist
OK I managed to get it booted, but I encountered this error in rails:
NoMethodError in ApiController#map
You have a nil object when you didn't expect it!
The error occurred while evaluating nil.visible?
RAILS_ROOT: /Users/eco/Sites/openstreetmap
Application Trace | Framework Trace | Full Trace
app/models/relation.rb:128:in `to_xml_node'
app/models/relation.rb:119:in `to_xml_node'
app/controllers/api_controller.rb:185:in `map'
app/controllers/api_controller.rb:184:in `each'
app/controllers/api_controller.rb:184:in `map'
Request
Parameters:
{bbox=12.4502,
41.861,
12.5341,
41.9227}
I fixed it by modifying this line in relation.rb, to check if member.member
is nil before evaluating .visible?
This may be a very special case and doesn't need to be committed... but I'm
not that familiar with the codebase so just tell me if I should submit this
to Trac or whatever.
# /app/models/relation.rb:128
# if member.member.visible?
if member.member member.member.visible?
Jeff
On Tue, May 26, 2009 at 3:33 PM, Thomas Wood grand.edgemas...@gmail.comwrote:
2009/5/26 Tom Hughes t...@compton.nu:
Dan Karran wrote:
I'm trying to set up a copy of the rails site on my machine, using
Postgres, but I'm running into a problem on the initial db:migrate ...
can anyone point me to how to fix this?
It's a bug, related to a change of some sort in rails 2.2 (it used to
work as is in 2.1). Add a require of lib/migrate to the top of that
migration (as some of the others already have) and it will work.
The fix is already on the i18n branch but not on the head right now.
Tom
I've merged the fix across to head.
--
Regards,
Thomas Wood
(Edgemaster)
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] type four_byte_unsigned does not exist
It's just a big slice of Rome, so i doubt the data is the problem. I got a
lot of errors, most were because i couldn't get libpgosm.so to compile for
OSX Leopard, (maptile_for_point() does not exist, etc), but there were a lot
more too. Given that mine is not really a standard install (lacking
libpgosm) i don't know if you'd be interested in seeing my errors... if you
are, tell me and I'll post them.
Jeff
On Tue, May 26, 2009 at 3:49 PM, Shaun McDonald
sh...@shaunmcdonald.me.ukwrote:
Do all the tests run? That will show if there are any problems with your
setup.
Is there anything in the data that would cause this problem?
Shaun
On 26 May 2009, at 23:40, Jeffrey Warren wrote:
OK I managed to get it booted, but I encountered this error in rails:
NoMethodError in ApiController#map
You have a nil object when you didn't expect it!
The error occurred while evaluating nil.visible?
RAILS_ROOT: /Users/eco/Sites/openstreetmap
Application Trace | Framework Trace | Full Trace
app/models/relation.rb:128:in `to_xml_node'
app/models/relation.rb:119:in `to_xml_node'
app/controllers/api_controller.rb:185:in `map'
app/controllers/api_controller.rb:184:in `each'
app/controllers/api_controller.rb:184:in `map'
Request
Parameters:
{bbox=12.4502,
41.861,
12.5341,
41.9227}
I fixed it by modifying this line in relation.rb, to check if member.member
is nil before evaluating .visible?
This may be a very special case and doesn't need to be committed... but I'm
not that familiar with the codebase so just tell me if I should submit this
to Trac or whatever.
# /app/models/relation.rb:128
# if member.member.visible?
if member.member member.member.visible?
Jeff
On Tue, May 26, 2009 at 3:33 PM, Thomas Wood
grand.edgemas...@gmail.comwrote:
2009/5/26 Tom Hughes t...@compton.nu:
Dan Karran wrote:
I'm trying to set up a copy of the rails site on my machine, using
Postgres, but I'm running into a problem on the initial db:migrate ...
can anyone point me to how to fix this?
It's a bug, related to a change of some sort in rails 2.2 (it used to
work as is in 2.1). Add a require of lib/migrate to the top of that
migration (as some of the others already have) and it will work.
The fix is already on the i18n branch but not on the head right now.
Tom
I've merged the fix across to head.
--
Regards,
Thomas Wood
(Edgemaster)
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] JSON/GeoJSON output format for 0.6 api
I'm adding GeoJSON and also structured OSM JSON output formats to a rails port 0.6 API I'm running; are these changes I could potentially commit, so such formats could be offered by OSM proper, or are they too specific a need? I'll be posting the changes online regardless. The osm JSON will be an exact JSON version of the current OSM XML, while the GeoJSON will pack Nodes into their parent Ways and Relations, as I believe that more accurately follows the GeoJSON spec. (correct me if I'm wrong) Best, Jeff ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] API 0.6 returns 403 forbidden
Yes, thank you. I had looked for an API policy and had not found one... I'm looking for bulk data but in fact live bulk data... working on a live javascript editor/renderer. I have a planet dump but had avoided working on it since the idea is to use data as it comes in. Is there a solution for that that's sustainable? I'm not familiar with the diffs and changesets as I've been focused on client-side work in the past week, but if there's an efficient way to sync my planet dump with the live dataset I'd love to do that. My understanding is that the planet dumps are done weekly, and downloading changesets continuously seems not much better than hitting the API... I suppose this may simply be outside the scope of OSM's available interfaces, no? Jeff On Thu, May 14, 2009 at 4:46 AM, Andy Robinson (blackadder-lists) ajrli...@googlemail.com wrote: one that includes the API as far as I can see. We should either add ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] API 0.6 returns 403 forbidden
Tels - I'd like to reiterate my offer of a hosted machine with lots of bandwidth and space for a planet dump for our applications. But i also agree that another solution might be better long-term. For the short term, are you interested in working with me to get a dump running and serving... JSON? Jeff On Thu, May 14, 2009 at 12:35 PM, Tels nospam-ab...@bloodgate.com wrote: Moin, On Thursday 14 May 2009 09:31:56 Tom Hughes wrote: Jeffrey Warren wrote: Hi, I suddenly started getting 403 - Forbidden from my any machine. I also get a 403 when i simply go to www.openstreetmap.org http://www.openstreetmap.org in a browser (just discovered this...). Have I been blacklisted or is this an outage? I suspect you're the person I blocked last night because you were apparently trying to scrape map data from the API with a large number of presumably automated map calls. If you are the person I blocked then in the last four days you've made a total of 8186 map calls, an average of about 1.5 a minute. If you want bulk data, please use the planet dump not the API to get it. I presume my own application (http://bloodgate.com/wiki/map) will then be blocked, too. Using the plant dump + diffs sounds easy, but is in fact impossible for me. My online server does not have the capacity to store the entire world, and I do not have the computing power and time to convert it up-front, either. (Not to mention do upload the converted data to my server from my home DSL line :( (There is also the question what happens if importing an hourly diff takes regulary longer than 60 minutes. So far the only data I have on that is someone mentioning that it takes between 40 and 70 minutes with a fast harddisk...) Also, to download the whole dump, update it with minute diffs and pregenerate a huge database is vastly more complicated and uses a lot more processing power than just to fetch (and cache for X days) the few areas someone is actually looking at. So, how will OSM handle this situation in the future? Will there be a fast API server (plus a few shadow, copy servers) than can deliver tile data in (semi) real-time, or will each and every person that wants to create a view of the data have to go to the hassle of dealing with the entire download dump, update it and convert it scheme? The latter seems like it will rule out quite a few applications just because it places a really high burden on each and every project to duplicate the same work. And that is without the small detail that next year the dump might be easily twice as large :) All the best, Tels -- Signed on Thu May 14 18:26:38 2009 with key 0x93B84C15. View my photo gallery: http://bloodgate.com/photos PGP key on http://bloodgate.com/tels.asc or per email. . . . my work, which I've done for a long time, was not pursued in order to gain the praise I now enjoy, but chiefly from a craving after knowledge, which I notice resides in me more than in most other men. And therewithal, whenever I found out anything remarkable, I have thought it my duty to put down my discovery on paper, so that all ingenious people might be informed thereof. -- Antony van Leeuwenhoek. Letter of June 12, 1716 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] API 0.6 returns 403 forbidden
Hi, I suddenly started getting 403 - Forbidden from my any machine. I also get a 403 when i simply go to www.openstreetmap.org in a browser (just discovered this...). Have I been blacklisted or is this an outage? Concerned, Jeff ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] API 0.6 returns 403 forbidden
This is only true from my server, correction... typo on the other machine. Late night. But my server definitely gets 403 for any openstreetmap.orgresource... On Wed, May 13, 2009 at 10:20 PM, Jeffrey Warren war...@mit.edu wrote: Hi, I suddenly started getting 403 - Forbidden from my any machine. I also get a 403 when i simply go to www.openstreetmap.org in a browser (just discovered this...). Have I been blacklisted or is this an outage? Concerned, Jeff ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] API 0.6 returns 403 forbidden
Yeah, my local machine gets 200 fine... i guess i was more asking if there's some kind of ban policy or whether i should be checking my own network... thanks, jeff On Wed, May 13, 2009 at 10:36 PM, Michael Bemmerl osm-t...@mx-server.dewrote: Jeffrey Warren schrieb: This is only true from my server, correction... typo on the other machine. Late night. But my server definitely gets 403 for any openstreetmap.org http://openstreetmap.org resource... Getting HTTP/1.1 200 OK here, must be something local. Bye, Michi ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Cartagen - client-side vector based map renderer, dynamic maps
* The proxy receives XML from the api or xapi server. Currently it
requests the full dataset.
* Then it removes unnec. tags (like note, fixme, attribution and a whole
bunch of others that are not needed for rendering). Some of them are
very minor, but 1 nodes with attribution=veryvery long string
here can make up like 40% of all the data, and just clog the line and
browser :)
Yes, I'm thinking of trying to cache locally but still request changesets if
the ?live=true tag is set... caching locally is great for more static data
but for the live viewer, I'm trying to not use caching, but increase
efficiency in the requests.
* The data is then pruned into (currently 3) levels and stored in a
cache:
* level 0 - full
* level 1 - no POI, no paths, streams, tracks etc. used for zoom 11
* level 2 - no tertiary roads etc. used for zoom 10 and below
* The client is served the level it currently requested as JSON.gz.
Great, this is what I'm working on too. I'm thinking a ruleset about what
features are relevant for what zoom levels could be something to work
together on? I was also thinking of correlating tags with a certain zoom
level. But maybe each tag should be associated with a range of zoom levels,
like way: { zoom_outer: 3, zoom_inner: 1 }. Thoughts?
* There are three servers in the list (api.openstreetmap,
xapi.informationfreeway and tagwatch) and a lot of them do not complete
the request (internal error, not implemented etc. etc.). It can take a
lot of retries to finally get the data.
* Even when you get the data, it takes seconds (10..40 seconds
is normal) to minutes - upwards to 360 seconds just to serve one
request.
So currently all received data is stored in the cache for 7 days to
avoid the very very long loading times.
Ideas of fetching the full dataset and pre-computing the cache simple
don't work because I don't have a big enough machine and no big enough
online account to store the resulting JSON :(
Also, somehow processing 150 Gbyte XML into JSON will prove to be a
challange :)
So I'm having the same problems with the APIs. The standard 0.6 api has been
pretty good but of course it serves XML, not JSON. The xapi is not very
responsive to me, it seems. I thought parsing XML in JS would be molasses,
so if you're interested, we should put up our own XAPI or custom api off the
planet.osm file, and send JSON?
I have an quad-core Intel Mac Pro with 1.5 TB and a bunch of RAM we can
dedicate to this effort, with plenty of bandwidth. And perhaps when Stefan's
work is published, we could run it as well, since it seems to be a great
solution to requesting fewer nodes for large ways... but for now do you
think you could use an XAPI? i think all my requests fit into that api.
Alternatively, Stefan points out that the dbslayer patch for the Cherokee
server allows direct JSON requests to a database. So some very thin db
wrapper might serve us for now? This isn't my area of expertise, so if you
have better ideas on how to generate JSON direct from the db, like GeoServer
or something, and still have tag-based requests, i'm all ears.
Yes, but reducing the polygons is also a lot of work :) I haven't
started on this yet, because on zoom 12 or higher you need to render
almost anything, anyways. Plus, you would then need to cache thepartial
data somehow (computing it is expensive in JS..)
Seems like Stefan's work may address this, no? Or if we did cache it, seems
like we'd calculate it on the server side.
d) oh, and localStorage. I've partially implemented that but haven't
had much testing... other work... ugh. So caching on a few levels,
basically.
I fail to see what localstorage actually gains, as the delivered JSON is
put into the browser cache, anyway and the rest is cached in memory.
Could you maybe explain what your idea was?
Yes, localStorage persists across sessions so you could build up a permanent
local cache and have more control (in JS) over requesting it and
timestamping when you cached it, not to mention applying only changesets and
not complete cache flushes. This has some advantages over the browser cache,
although that does of course persist across sessions too.
* There is a talk I proposed for State of the Map and I don't want to
spoil everything before :)
yes, me too! so if you want to discuss off-list that's fine.
Of course, semi-dynamic rules like color them according to feature X by
formula Y are still useful and fun, and avoid the problems above.
(Like: use maxspeed as the color index ranging from red over green to
yellow :).
Yes, this is an exciting area to me, for example the color by authorship
stylesheet i posted before:
http://map.cartagen.org/find?id=parisgss=http://unterbahn.com/cartagen/authors.gss
or this one i threw together yesterday, based on the tags of measured width
instead of on a width rule:
http://map.cartagen.org?gss=http://unterbahn.com/cartagen/width.gss
A more fully-rendered screenshot
Re: [OSM-dev] Cartagen - client-side vector based map renderer, dynamic maps
Great, this is a good discussion. I've put up a wiki page with some of the
things we've covered, with pros/cons. I hope we can continue to talk about
our approaches and as we optimize for different problems post some of it
back up here:
http://code.google.com/p/cartagen/wiki/FeatureTradeoff
I put in what I could gather about Temap, but feel free to update and add
more pros and cons... this is just my thought process so far. We might also
add a status column so we can annotate what we learn from each approach.
Best,
Jeff
On Fri, May 8, 2009 at 3:00 PM, Tels nospam-ab...@bloodgate.com wrote:
Moin,
On Friday 08 May 2009 20:04:48 you wrote:
* The proxy receives XML from the api or xapi server. Currently it
requests the full dataset.
* Then it removes unnec. tags (like note, fixme, attribution and a
whole bunch of others that are not needed for rendering). Some of
them are very minor, but 1 nodes with attribution=veryvery
long string here can make up like 40% of all the data, and just
clog the line and browser :)
Yes, I'm thinking of trying to cache locally but still request
changesets if the ?live=true tag is set... caching locally is great
for more static data but for the live viewer, I'm trying to not use
caching, but increase efficiency in the requests.
I fear loading data live from the API server is just not feasible,
unless you:
* only load diffs (minute-diffs?) and update your already cached at the
proxy data with that. OTOH I read that importing a one-hour diff into a
postgres database can take 40..70 minutes, e.g. depending on load you
might not even manage to update your DB with the diffs fast enough...
* invent an API server that is about 1000 times faster :)
* do never zoom out from level 18, anything below will request so much
data that you can't get it live :)
Currently I consider live-view not an achiveable goal, I am happy if I
can render data that is about 1 day or so old.
* The data is then pruned into (currently 3) levels and stored in a
cache:
* level 0 - full
* level 1 - no POI, no paths, streams, tracks etc. used for zoom
11 * level 2 - no tertiary roads etc. used for zoom 10 and below *
The client is served the level it currently requested as JSON.gz.
Great, this is what I'm working on too. I'm thinking a ruleset about
what features are relevant for what zoom levels could be something to
work together on? I was also thinking of correlating tags with a
certain zoom level. But maybe each tag should be associated with a
range of zoom levels, like way: { zoom_outer: 3, zoom_inner: 1 }.
Thoughts?
My rules do have a minimum zoom level, smaller than that and they are
not rendered. The levels are inspired by the osmarenderer and mapnik
outputs, but I moved a few of them down so you can render really high
resolution maps.
However, the pruning at the proxy is something else and not connected to
that. For instance, somebody might not want to see tertiary roads on
level 13, but others want. So I make sure that I only prune out data
that is never be able too seen on that level. E.g. a conservative
pruning.
Also, about 90% of the data-pruning is about removing unwanted data
(like note=blah :) and not about the smaller zoom levels because
currently it is simple not feasible to render below 10 and even for
zoom 10 you need a really really beefy machine and a long wait time
* There are three servers in the list (api.openstreetmap,
xapi.informationfreeway and tagwatch) and a lot of them do not
complete the request (internal error, not implemented etc. etc.).
It can take a lot of retries to finally get the data.
* Even when you get the data, it takes seconds (10..40 seconds
is normal) to minutes - upwards to 360 seconds just to serve one
request.
So currently all received data is stored in the cache for 7 days to
avoid the very very long loading times.
Ideas of fetching the full dataset and pre-computing the cache
simple don't work because I don't have a big enough machine and no
big enough online account to store the resulting JSON :(
Also, somehow processing 150 Gbyte XML into JSON will prove to be a
challange :)
So I'm having the same problems with the APIs. The standard 0.6 api
has been pretty good but of course it serves XML, not JSON. The xapi
is not very responsive to me, it seems.
Neither for me, but the API server is very slow, too. It seems it can't
manage to send me more than 17Kbyte/s (but maybe it is bandwidth
limited?).
I thought parsing XML in JS
would be molasses,
When I tried it, it used ungodly amounts of memory (because the data
structure is not usefull for rendering and it contains so much cruft),
and I also never managed to extract the actual node data for rendering
from it...
so if you're interested, we should put up our own
XAPI or custom api off the planet.osm file, and send JSON?
Yeah,
Re: [OSM-dev] Cartagen - client-side vector based map renderer, dynamic maps
Hi, Tels - It's not been optimized yet, so loading is a little slow, but I'm optimistic that it will scale. Based on my experience, I can tell you right away it won't scale :) Not to discourage you, but: * the amount of data is really huge. Throwing a few dozend megabyte XML or even JSON at the browser will bring it to its knees. Not to mention the data you need to render even a small city. * re-rendering everything takes a long time. You want to avoid that :) I was actually talking about server-side load time. I'm running it off the 0.6 API, so it packs up XML, sends it to my server, i unpack, re-encode to JSON, send to the browser, render. Obviously that's SUPER inefficient, so I'm looking forward to cutting a lot of that out in the next week or so. Actually, rendering in the browser's been pretty good - for example this page loaded with no noticeable slowdown, and I haven't even begun optimizing: http://www.flickr.com/photos/jeffreywarren/3476532351/ But you're right, it's a challenge. I'm impressed that you rendered a whole city like Berlin - do you have some code online so I can see, or a screenshot? I bet it looks great... What I'm looking at now is: a) rendering only some tags per zoom-level, so no rendering footpaths and buildings as you zoom out... but that's dependent on the xapi, which I haven't been able to fetch from reliably (help anyone?) b) cutting the API out of the loop and running direct from a planet.osm, but then you can't use it to view live edits, like you can here: http://vimeo.com/4435969 c) trying to serve partial polygons... I'd like to try plotting only every 3rd or 10th node... do the polygons collapse? Can i cull nodes in a more intelligent way? Someone on this list or geowanking pointed to a company that can serve lower-res polys over an API. I'm sure folks have worked on this in tile systems, so if you know anything about it and are willing to share, I'm all ears. This becomes really relevant as you zoom out... don't want to render every node for the coast of Argentina, for example. d) oh, and localStorage. I've partially implemented that but haven't had much testing... other work... ugh. So caching on a few levels, basically. What strategies have you employed, if you're willing to share? Also agreed that GSS is not technically spectacular - the driving motivation is that it is legible to those new to mapping, being CSS-like. So really an adoption decision, though the JavaScript-ability of it is a nice bonus - dynamic rules are fun. Anyways, I'm excited to hear you've been working on this kind of stuff too. I'm happy to collaborate or just share information, the whole codebase is at http://code.google.com/p/cartagen/. Best, Jeff My app has already quite a few optimizations, and it still chokes at big cities like Berlin or London. However, I am confident that things can be improved :) (Browser limitations non-withstanding. Single-threaded dead-slow JS and incomplete Canvas spec without dashe I hate thee... :( Regarding the rule sets and CSS: I've already considered adding a different rule-set (just to show that it can be done). However, from a technical viewpoint, that is not that spectacular. As long as the renderer is flexible enough to handle the wanted features, it doesn't really matter in what format the rules are (CSS, GSS, JSON, XML, you name it) or where they come from (hard-coded, web, URI, user input), as long as you can load, parse and convert them, it can display them. In my eyes the much bigger impact is that you no longer need different sets of tiles or tile providers - just the data and the rules to display it and the map can morph in real-time from mapnik to cyclemap to whatever-you-want. And one more button click and the user can save it locally. That's at least my vision I work towards :) All the best, Tels ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] xapi status?
Hmm, i can load yours, but I'm getting a lot of stalling in general, also after about 5.8 Kb. I'll try later in the day, maybe it's as you say, just transient dropouts. On Sat, May 2, 2009 at 12:19 PM, 80n 80n...@gmail.com wrote: Actually your request seems to be working fine now. The stall was probably a transient network dropout. On Sat, May 2, 2009 at 5:14 PM, 80n 80n...@gmail.com wrote: Jeff This url works for me: http://www.informationfreeway.org/api/0.6/*[elephant=*]http://www.informationfreeway.org/api/0.6/*%5Belephant=*%5D I just tried yours and it appears to have stalled after downloading 5.2Kb. Not sure why yet, I'll investigate futher. 80n On Sat, May 2, 2009 at 3:50 PM, Jeffrey Warren war...@mit.edu wrote: Hmm, I'm now getting a blank response after consistently 26.4 seconds. Is that a timeout? I'm only asking for buildings=yes in an area where I know there are very few... i request all ways for this bbox from the regular 0.6 api all the time. I tried: http://osmxapi.hypercube.telascience.org/api/0.6/*[building=yes][bbox=-71.089,42.359,-71.087,42.361]http://osmxapi.hypercube.telascience.org/api/0.6/*%5Bbuilding=yes%5D%5Bbbox=-71.089,42.359,-71.087,42.361%5D http://www.informationfreeway.org/api/0.6/*[building=yes][bbox=-71.089,42.359,-71.087,42.361]http://www.informationfreeway.org/api/0.6/*%5Bbuilding=yes%5D%5Bbbox=-71.089,42.359,-71.087,42.361%5D Is there a request you can confirm working, that you can give me a URI for so I can troubleshoot using that as an example? Thanks a lot - Jeff On Fri, May 1, 2009 at 2:41 AM, 80n 80n...@gmail.com wrote: This characteristically happens if all the daemons are busy servicing other people's requests. The server you are accessing is not particularly fast. The other xapi servers are faster and can handle more capacity. Most people use the xapi instance on the telascience server: http://osmxapi.hypercube.telascience.org/api/0.5/http://xapi.openstreetmap.org/api/0.5/*%5Bamenity=hotel%5D%5Bbbox=-71.089,42.359,-71.087,42.361%5D There's also a 0.6 instance running on that server at: http://osmxapi.hypercube.telascience.org/api/0.6/http://xapi.openstreetmap.org/api/0.5/*%5Bamenity=hotel%5D%5Bbbox=-71.089,42.359,-71.087,42.361%5D I'll update the platform status page as it's not quite accurate at the moment. 80n The 0.5 service is serving data as at 17th April. On Fri, May 1, 2009 at 6:27 AM, Jeffrey Warren war...@mit.edu wrote: Hello all - I'm trying to ping the xapi.openstreetmap.org service for: http://xapi.openstreetmap.org/api/0.5/*[amenity=hotel][bbox=-71.089,42.359,-71.087,42.361]http://xapi.openstreetmap.org/api/0.5/*%5Bamenity=hotel%5D%5Bbbox=-71.089,42.359,-71.087,42.361%5D or perhaps less ambitious: http://xapi.openstreetmap.org/api/0.5/*[building=yes][bbox=-71.089,42.359,-71.087,42.361]http://xapi.openstreetmap.org/api/0.5/*%5Bbuilding=yes%5D%5Bbbox=-71.089,42.359,-71.087,42.361%5D I'm not getting a response - rather, i'm getting a blank page. Am I making a grievous error or is the xapi server not running pending the 0.6 upgrade? Platform status says Currently serving data as at 0.5 cut-off. 0.6 service will start shortly. Thanks in advance - Jeff ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] xapi status?
Hello all - I'm trying to ping the xapi.openstreetmap.org service for: http://xapi.openstreetmap.org/api/0.5/*[amenity=hotel][bbox=-71.089,42.359,-71.087,42.361] or perhaps less ambitious: http://xapi.openstreetmap.org/api/0.5/*[building=yes][bbox=-71.089,42.359,-71.087,42.361] I'm not getting a response - rather, i'm getting a blank page. Am I making a grievous error or is the xapi server not running pending the 0.6 upgrade? Platform status says Currently serving data as at 0.5 cut-off. 0.6 service will start shortly. Thanks in advance - Jeff ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
