Re: [OSM-dev] Seeking advice for OSM based routing application
On Sun, Mar 17, 2013 at 4:52 PM, Matt Hoover the.one.ele...@gmail.com wrote: I recently developed a fairly simple open-source application for creating/editing/analyzing GPX files. It relies heavily on OSM tiles and JMapViewer (a JOSM component). If interested, the beta version is hosted at http://www.gpxcreator.com. Currently the only way to create a GPX file is by pointing and clicking on the map repeatedly, for each point in the route. My ultimate goal though is to have start-to-end routing, and I'd like it to work for backcountry hiking/biking/etc trails. OSM seems to be the best map system I've found yet as far as having a lot of these trails included in its maps (in USA west coast at least, where I've looked so far). I haven't worked much with actual OSM data yet (only tiles), but I understand the main way to download the data is in an XML format. In trying to accomplish my goal of adding routing to my application, I'm faced with what I see as a few limited options, and none of them seem ideal. I'd appreciate insight/advice from people more experienced both with OSM data and routing implementations. Here are the options as I see them: 1) Host OSM data myself and run a routing algorithm on the server. This would be ideal in theory, but in reality I doubt I can afford to do something so big. 2) Have the routing algorithm run on client machine, and request the data it needs from OSM with each routing attempt. This seems bad for a number of reasons. Even to do a relatively short route ( 100 miles), it seems like a pretty big chunk of data would need to be downloaded to the client (like 50MB at least in some quick tests I did). This is not ideal for the client, nor is it nice to the OSM server. Plus all that XML would need to be parsed on the client and re-structured into some format the routing algorithm could use, which would be time consuming for the client. 3) Use one of the existing servers which has its own data, routing implementation, and public API. This sounds like a very promising idea at first. However, I'm running into some weird issues, and I've tried every single option listed in OSM wiki. I first tried YOURS. See the linked image here, an attempted route near [38.198108, -119.883907]. The routing algorithm refuses to jump from the road to the trail, instead choosing to do a long roundabout trip. I tried several other online routers, and they all made the same choice, so I naturally assumed maybe there was a problem with the OSM data at this point. But I examined it up close with JOSM and the ways seem connected properly (though I could be mistaken since I haven't used JOSM much). To make matters stranger, I finally found a single service (out of all the many I tried), which is able to route correctly across that junction! This service is BRouter, which the linked image here shows performing as desired. However, BRouter does not have a public API, so I would seem to be out of luck for all options here. 4) Client side routing logic using OSM server-side data? The routing algorithm would make many single element requests to the server, as it runs, downloading individual nodes/ways needed for the routing, rather than all data in vicinity as in option 2 above. I don't even think this is a legitimate idea, since I would guess that there would be way too many messages sent back and forth between client and server, and communication cost would be far too expensive (though overall amount of data downloaded would be far less than option 2 above). But the idea crossed my mind so I figured I'd add it to the list. Maybe I'm wrong here though and the communication cost is affordable? Are the any other options I've missed? Are any of my considerations of these 4 options wrong? Which is the best option for my situation, in your opinion? Thanks in advance for any advice anybody can offer. I really appreciate it. Well options 2 and 4 are absolutely not feasible, assuming you are talking about using the osm.org map query API. The OSM API is for editing the map only, not for clients to query for use in some 3rd party service. Of course some small non-editing related calls get through but if you start querying a lot of data you will get blocked pretty quickly. For non edit related querying there are other services like jxapi or overpass. But even then, this does not sound like the best idea. The specific example you linked to does seem very odd. It does look like the MapQuest Open routing service behaves correctly thought: http://mapq.st/1067Po9 Looking on a couple of other routing services, the OSRM demo is set for cars only so it won't even try to go on the highway=path. No clue where the cloudmade router's head is. It will obviously use highway=path but for some reason it is refusing to transition to it right there. I can't figure out why either. Toby ___ dev mailing
Re: [OSM-dev] Coastline changes Antarctica
On Tue, Mar 12, 2013 at 10:09:30AM +0100, Jochen Topf wrote: On Tue, Mar 12, 2013 at 01:53:30AM -0700, Paul Norman wrote: You should not import the -180/180 part until a plan for tile.osm.org has been worked out. If this isn't solved before we do the import we just add those bogus coastline ways back in as a temporary measure. Christoph did the import on the weekend. The bogus coastline is not in there. If somebody still needs it, he can add it back in. These maps already have all the current data and mostly current tiles: http://www.openstreetmap.de/karte.html http://polar.openstreetmap.de/antarctica Current coastlines are available from: http://openstreetmapdata.com/ You probably want for z10+: http://data.openstreetmapdata.com/land-polygons-split-3857.zip (thats the replacement for processed_p) and for z0-9: http://data.openstreetmapdata.com/simplified-land-polygons-complete-3857.zip (thats the replacement for shoreline_300) If you want to do this yourself, they are generated with: osmcoastline -o coastlines-split-3857.db --no-index --srs=3857 --max-points=1000 --bbox-overlap=50 --output-lines --output-polygons=both coastlines-current.osm.pbf ogr2ogr -f ESRI Shapefile land-polygons-split-3857 coastlines-split-3857.db land_polygons and: osmcoastline -o coastlines-complete-3857.db --no-index --srs=3857 --max-points=0 --bbox-overlap=0 coastlines-current.osm.pbf spatialite -batch -bail -echo coastlines-complete-3857.db simplify.sql ogr2ogr -f ESRI Shapefile simplified-land-polygons-complete-3857 coastlines-complete-3857.db simplified_land_polygons Instead of the simplified polygons you can also use the complete polygons, but they are 10 times as large and in the low zoom levels you will not see much difference. The split polygons will probably not work for small zoom levels because you will see the seams between polygons. This could be solved by using larger overlaps or by using the gamma parameter when rendering, but it is probably easiest just to use the new simplified polygons because you don't have to change anything in your Mapnik config. We'll probably need some experimentation to see which configuration is best in the long run. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[josm-dev] JOSM Server
Hello, additional to Pauls text some background information: We had server troubles regarding the database performance for about 2 years now. Mainly these issues have been caused by the increasing activity of you all which resulted in growing database and activity. SqLite is a very simple database and got to its limits. The JOSM-Trac was too successful (one of the reasons why there are so many spam attempts :-) To fix the database issues I wanted to change to postgresql instead, which is more suitable for the current requirements. As the change of the database is a big issue and the current server had no IPV6 connectivity I simply asked Hetzner for sponsoring a new server not really expecting a positive answer. But they said yes immediatelly, so database upgrade also got server move. I promised them some unobtrusive advertising for their sponsoring, but there is no contract enforcing this. BTW: Why Hetzner - I use them private and in company a lot and they are pretty good. So I got a complete new server instead of only a new database. The hosting with FOSSGIS was fine, nothing to blame about this (the IPV6 issue is due to the fact GFZ has no IPV6 yet). Thought an own server is always a bit better. So instead of only doing the database update we also have a new server now (still database migration was much more work than the server move - we have a pretty good guide for server migration right now, we did it 2 times in the past :-). Regarding new server, database: There needs to be a little more finetuning, so the setup does not exceed the available ressources - The defaults of web server and database are very ressource hungry for no good reason. Please report any issues - I expect some more troubles in the next days, but hopefully we have some time until we exceed limits next time. So lots of thanks again to FOSSGIS who provided server space for us for such a long time. As there is enough demand for computer power in the OSM community I believe the old server soon will get used for other fancy purposes. Ciao -- http://www.dstoecker.eu/ (PGP key available) ___ josm-dev mailing list josm-...@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [OSM-dev] mod_tile in other projections?
you can change the mapnik configuration file for the projection. mod_tile haven't nothing with projection or ogc protocol. mapproxy is better if you need WMS WMTS TMS protocol to distribuite your tiled map in more projection. On Thu, Mar 7, 2013 at 4:36 PM, Jason Lee jaslee...@gmail.com wrote: Hi all, It might seem an odd question but I'll give it a shot - would mod_tile/renderd support the dynamic generation of map tiles in a map projection other than web mercator? If not, then would it be possible/how difficult would it be to modify the code to support it? The only other thing I can think of is using MapProxy with Mapnik backend but I'm not sure the scalabilty/performance compared to mod_tile. Thanks and regards, Jason ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Generating local apidb diff files questions
Hi - Any advice on generating a local set of APIdb diffs to support minutely editing feedback for a local rails instance? (www.openhistoricalmap.org) I believe I've found what I need to get started at: http://wiki.openstreetmap.org/wiki/Osmosis/Replication#Server-side_Replication and http://wiki.openstreetmap.org/wiki/Planet.osm/diffs But... before I went off and started stubbing my toes, I wanted to check if there was anywhere else I should be looking. Any - _any_ - advice or questions (like, Why are you doing this?!?!?) are most welcome. Thanks, Jeff -- Jeff Meyer Global World History Atlas www.gwhat.org j...@gwhat.org 206-676-2347 OpenStreetMap: Mapping with a Human Touch http://www.openstreetmap.org/user/jeffmeyer osm: Open Historical Map (OHM)http://wiki.openstreetmap.org/wiki/Open_Historical_Map / my OSM user page http://www.openstreetmap.org/user/jeffmeyer t: @GWHAThistory https://twitter.com/GWHAThistory f: GWHAThistory https://www.facebook.com/GWHAThistory ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [josm-dev] JOSM Server and Database
On Mon, 18 Mar 2013, Vincent Privat wrote: Is this big and ugly new sponsor logo really needed as it ? I find very disturbing to display such a large commercial logo without any discussion first ? Large one? I placed a rather small logo on the start page. The idea was to show the sponsor, but very decent. Ciao -- http://www.dstoecker.eu/ (PGP key available) ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] JOSM Server and Database
Hi! 2013/3/18 Dirk Stöcker openstreet...@dstoecker.de: On Mon, 18 Mar 2013, Vincent Privat wrote: Is this big and ugly new sponsor logo really needed as it ? I find very disturbing to display such a large commercial logo without any discussion first ? Large one? I placed a rather small logo on the start page. The idea was to show the sponsor, but very decent. Ah - now I found the logo! Took me some time to spot it. If they sponsor the server why should this be hidden? Everyone who's willing to pay the bills to get rid of this big and ugly new sponsor logo may now stand up. Anyone? Hello? @Dirk: thanks for the update! Best regards, Martin ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] JOSM Server and Database
On Mon, 18 Mar 2013, Martin Vonwald wrote: Large one? I placed a rather small logo on the start page. The idea was to show the sponsor, but very decent. Ah - now I found the logo! Took me some time to spot it. If they sponsor the server why should this be hidden? Everyone who's willing to pay the bills to get rid of this big and ugly new sponsor logo may now stand up. Anyone? Hello? They don't request a large and ugly logo. They are pretty open. Thought I promised them a decent solution. And I find the current one rather non-disturbing. Maybe there are browser configurations where it looks ugly? Ciao -- http://www.dstoecker.eu/ (PGP key available) ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] JOSM Server and Database
On 3/18/13 8:23 AM, Martin Vonwald wrote: Hi! 2013/3/18 Dirk Stöcker openstreet...@dstoecker.de: On Mon, 18 Mar 2013, Vincent Privat wrote: Is this big and ugly new sponsor logo really needed as it ? I find very disturbing to display such a large commercial logo without any discussion first ? Large one? I placed a rather small logo on the start page. The idea was to show the sponsor, but very decent. Ah - now I found the logo! Took me some time to spot it. If they sponsor the server why should this be hidden? Everyone who's willing to pay the bills to get rid of this big and ugly new sponsor logo may now stand up. Anyone? Hello? that's a really restrained logo. keep it. richard signature.asc Description: OpenPGP digital signature ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
[josm-dev] new server
Hi all, you may be wondering why there is a new sponsor logo on the front page of the JOSM website and why there has been some downtime lately. Let me give you some updates. In the last few days, we moved to a new server which is kindly sponsored by the web hoster Hetzner [1]. The offer stands for one year, but may be continued after that time. We've placed an inconspicuous reference at the bottom of each wiki page and a more visible logo on the the front page. I think it's fair, the regular costs would be 19,90 EUR / month. While he was at it, Dirk switched the database backend for trac, our bug-tracking- and wiki-software (from sqlite to postgresql). So hopefully, we will no longer see the notorious database locked messages. Please report if you still get this or a similar error. Big thanks to the FOSSGIS society for the continuous server hosting so far. We will still need your support, but hopefully, some resources are freed for other projects. Btw., all the credit goes to Dirk, he found the sponsor and migrated the sever and database. [1] http://www.hetzner.de/en/hosting/produkte_vserver/vq19 Paul ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] JOSM Server and Database
On 03/18/2013 12:38 AM, Vincent Privat wrote: Is this big and ugly new sponsor logo really needed as it ? I find very disturbing to display such a large commercial logo without any discussion first ? I've slightly adapted the size and position: from http://josm.openstreetmap.de/wiki/WikiStart?version=191 tohttp://josm.openstreetmap.de/wiki/WikiStart?version=192 I'm sorry, we haven't involved you earlier, see my other mail for some background info. Paul ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] JOSM Server
On Mon, Mar 18, 2013 at 11:14 AM, Dirk Stöcker openstreet...@dstoecker.dewrote: Hello, additional to Pauls text some background information: We had server troubles regarding the database performance for about 2 years now. Mainly these issues have been caused by the increasing activity of you all which resulted in growing database and activity. SqLite is a very simple database and got to its limits. The JOSM-Trac was too successful (one of the reasons why there are so many spam attempts :-) Why does JOSM rely on a database-driven wiki at all? It's great that you use trac and the wiki for project management, but the stuff that JOSM loads is all relatively static, right? The imagery, presets, styles, and front page documents all change infrequently and could be served as static files somewhere so you don't require database actions every time someone starts JOSM. ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] JOSM Server
On Mon, 18 Mar 2013, Ian Dees wrote: Why does JOSM rely on a database-driven wiki at all? It's great that you use trac and the wiki for project management, but the stuff that JOSM loads is all relatively static, right? The imagery, presets, styles, and front page documents all change infrequently and could be served as static files Some of these are pregenerated files (thought delivered dynamically, like maps, presets, styles, plugins, ...). Others are dynamic and usually for good reason. Others are plain static (like e.g. the downloads, icons, ...). There is little we can gain by optimizing this further. What creates a lot of load are necessary database accesses and sometimes large results. Reducing them would reduce the service of JOSM. As long as we get the required server power there is no need to restrict service. somewhere so you don't require database actions every time someone starts JOSM. Actually the amount of connects by JOSM itself can be ignored compared to the web spiders, SPAM bots, hacker attempts and all the other things accessing a webpage nowadays. And the wiki, ticket and svn stuff is pretty dynamic. If only real users would access the webpage we would have less trouble. Blocking stuff for spiders on the other hand is also no good idea, as web search engines like Google are the main entrypoint to the pages (f.e. we have pretty good clicktrough rates with Google.) To have good performance for the users there must be a lot performance left to serve all the others. I don't have a problem with the fact that the server has to do lots of work. It shows that JOSM is nowhere near beeing a dead project. :) Ciao -- http://www.dstoecker.eu/ (PGP key available) ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] JOSM Server
Hi all, On Mon, 2013-03-18 at 17:14 +0100, Dirk Stöcker wrote: So lots of thanks again to First of all big thanks to the JOSM server admins (Dirk and Paul, I guess) for administrating, optimising, keeping alive the server. There's hardly a bigger obstacle than a not working project management for distributed free software development. In this light, I'm happy with the recent server migration (thanks to Hetzner, at this point). In relation to annoying ads and useless other images, the logo is hardly visible. :-) Also thanks to all (other) JOSM contributors and user for the cool recent features, great ideas, translations, the not-forgotten documentation etc. I think we are a great team! Cheers, Simon (simon04) ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] JOSM Server
On Mon, Mar 18, 2013 at 12:39 PM, Dirk Stöcker openstreet...@dstoecker.dewrote: On Mon, 18 Mar 2013, Simon Legner wrote: What about re-generating this file on modification and shipping as fast as possible (using Apache directly, Nginx or Varsnish). Maybe, having a subdomain for static content might be worth a try (getting rid of cookies, sessions etc.). Actually while I wrote that text I thought that maybe /maps actually is a point for improvement and could be delivered static :-) Thought when I change that, I think about splitting that large page into smaller subpages, as editing the page also got a demanding task today. OTOH that makes stuff more complicated - will have to think about a useful solution. Would you consider requesting the imagery.xml from a separate server? I've slowly been working on a project to store the imagery presets across all editors [0] so that someone who wanted to add their imagery layer could do it in one spot and have it show up in all editors that use the same source files. For now, that project contains a script that spits out tons of JSON documents. The goal is to store those JSON documents in GitHub and write scripts that write out files in formats specific to the various editors. These files could then be grabbed by the editors themselves. I'm considering doing the same sort of thing for presets, but obviously that would require more coordination. [0] https://github.com/iandees/editor-imagery-index ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] JOSM Server
On Mon, 18 Mar 2013, Ian Dees wrote: Would you consider requesting the imagery.xml from a separate server? I've slowly been working on a project to store the imagery presets across all editors [0] so that someone who wanted to add their imagery layer could do it in one spot and have it show up in all editors that use the same source files. For now, that project contains a script that spits out tons of JSON documents. The goal is to store those JSON documents in GitHub and write scripts that write out files in formats specific to the various editors. These files could then be grabbed by the editors themselves. I'm considering doing the same sort of thing for presets, but obviously that would require more coordination. That is already possible. Additional XML-Sources can be supplied in advanced prefs. You could provide an preferences loader-file including the link, which can be installed in josm very simple. I have my own private XML as well, which I install in all my josm instances. For the default maps sources externals will not be allowed. That would disable the security checks. Ciao -- http://www.dstoecker.eu/ (PGP key available) ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] JOSM Server
On Mon, Mar 18, 2013 at 1:20 PM, Dirk Stöcker openstreet...@dstoecker.dewrote: For the default maps sources externals will not be allowed. That would disable the security checks. What security checks are you talking about? ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] JOSM Server
On 18.03.2013 18:31, Simon Legner wrote: On Mon, 2013-03-18 at 17:46 +0100, Dirk Stöcker wrote: Thanks for your work. Actually the amount of connects by JOSM itself can be ignored compared to the web spiders, SPAM bots, hacker attempts and all the other things accessing a webpage nowadays. And the wiki, ticket and svn stuff is pretty dynamic. If only real users would access the webpage we would have less trouble. Blocking stuff for spiders on the other hand is also no good idea, as web search engines like Google are the main entrypoint to the pages (f.e. we have pretty good clicktrough rates with Google.) To have good performance for the users there must be a lot performance left to serve all the others. Some service like search do not need to be crawled What about re-generating this file on modification and shipping as fast as possible (using Apache directly, Nginx or Varsnish). Maybe, having a subdomain for static content might be worth a try (getting rid of cookies, sessions etc.). Thought trac is using caching itself already. Good setup would be nginx + varnish and then trac. nginx could offer http and https and looping internal on varnish with http (maybe socks). Definitely, a bit more complicated than using apache and only theory. Just my two cents Colliar ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] JOSM Server and Database
Ok thanks, it's much better right now, integrated on the right side of the front page :) I recognize I may have have over-reacted, sorry for that. Thank you Dirk for finding us a new server :) 2013/3/18 Paul Hartmann phaau...@googlemail.com On 03/18/2013 12:38 AM, Vincent Privat wrote: Is this big and ugly new sponsor logo really needed as it ? I find very disturbing to display such a large commercial logo without any discussion first ? I've slightly adapted the size and position: from http://josm.openstreetmap.de/**wiki/WikiStart?version=191http://josm.openstreetmap.de/wiki/WikiStart?version=191 to http://josm.openstreetmap.de/**wiki/WikiStart?version=192http://josm.openstreetmap.de/wiki/WikiStart?version=192 I'm sorry, we haven't involved you earlier, see my other mail for some background info. Paul __**_ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.**org/listinfo/josm-devhttp://lists.openstreetmap.org/listinfo/josm-dev ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
