Re: [OSM-dev] How to earn undying fame
On Tue, Jun 22, 2010 at 9:40 AM, Tom Hughes t...@compton.nu wrote: On 22/06/10 08:27, Lambertus wrote: So what about my proposal to use the OSM dev server for the purpose of determining the server specs? http://wiki.openstreetmap.org/wiki/Servers/errol Sure, if somebody wants to run some tests there then that's fine. Can I have an account on it please ? ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] How to earn undying fame
On 2010-06-22 00:40, Tom Hughes wrote: On 21/06/10 19:37, Nic Roets wrote: On Mon, Jun 21, 2010 at 7:09 PM, Tom Hughest...@compton.nu wrote: All we need now is a volunteer to run the server in the way that Brian does for the geocoding server. I'll do it. We can start with a small (2GB RAM) server. No we can't - we have already established that it needs more than that. Not that I have a spare server of that class to give you anyway. The point of this conversation is to work out what we need to procure if we are to do this, and to make sure we have people to run it. Tom So what about my proposal to use the OSM dev server for the purpose of determining the server specs? http://wiki.openstreetmap.org/wiki/Servers/errol ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] How to earn undying fame
On 22/06/10 08:27, Lambertus wrote: So what about my proposal to use the OSM dev server for the purpose of determining the server specs? http://wiki.openstreetmap.org/wiki/Servers/errol Sure, if somebody wants to run some tests there then that's fine. Tom -- Tom Hughes (t...@compton.nu) http://compton.nu/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] How to earn undying fame
Tom Hughes t...@compton.nu writes: On 21/06/10 17:19, Kai Krueger wrote: My guess would be that Nic meant: Fork (i.e. copy) the git rails_port repository. Add the desired functionality and then once done try and get it merged back into trunc. So the usual development way to get new functionality into the main site. In case someone would try and do it, it would probably be a good idea to make the frontend code as backend agnostic as possible. In that case it may even be possible for people to choose which routing engine they want to use. Be it gosmore, some other yet to be written opensource routing engine, open route service, or even a commercial provider like cloudmade's routing. Well if somebody can provide a backend that returns a GPX or KML or something then I'm sure somebody will hack up the rails port to be able to display those routes. Nominatim is the model here - a separate server with an API the rails code can call to get what it needs. There is an OSM wiki page that lists many online routers (presumably meaning they already have a web interface), several of which are free software: http://wiki.openstreetmap.org/wiki/Routing/online_routers They offer a wide variety of features (according to the comparison table) and nearly all of them offer results as GPX format. Before I finish I should declare an interest here: I wrote one of the free software routers listed on that wiki page. -- Andrew. -- Andrew M. Bishop a...@gedanken.demon.co.uk ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] How to earn undying fame
The result of the uservoice survey[1] is in: The community wants routing on our website more than anything else. And most of the work has already been done. An opensource routing engine that can do the job already exists [2]. The routing database can be updated weekly. The hardware requirements are quite reasonable (64-bit processor with tens of GB of RAM). If a new server is needed, I'm sure we'll get the money in fairly quickly. All that remains is for someone with a little bit of programming experience to volunteer. Make a copy of the osm.org website* and add routing buttons. Then write some code that converts the routing engine output to a few formats. Open Layers, GPX, human readable text should be enough. [1] http://osm.uservoice.com/ [2] http://wiki.openstreetmap.org/index.php/Gosmore *: Copy only the html and the javascript and exclude /edit, /history, /api, /user etc. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] How to earn undying fame
On 21/06/10 16:00, Nic Roets wrote: The result of the uservoice survey[1] is in: The community wants routing on our website more than anything else. Like we didn't know that already. And most of the work has already been done. An opensource routing engine that can do the job already exists [2]. The routing database can be updated weekly. The hardware requirements are quite reasonable (64-bit processor with tens of GB of RAM). If a new server is needed, I'm sure we'll get the money in fairly quickly. I'm not sure tens of Gb of RAM is that reasonable, though it does rather depend how many tens you mean. It's also rather important to know how fast that memory requirement is likely to grow. Equally I would like to have some idea of how many routes/second you believe this engine to be capable of on that hardware. Time to run a weekly update on that hardware is something else we would want to know, along with some idea of the likely rate of growth of that over time. Basically we want to scale the hardware so that it not only works now, but in a year's time as well. We also want it not to collapse under the load when hundreds of people decide to query it simultaneously because we've just been on some TV programme. All that remains is for someone with a little bit of programming experience to volunteer. Make a copy of the osm.org website* and add routing buttons. Then write some code that converts the routing engine output to a few formats. Open Layers, GPX, human readable text should be enough. No. Whatever solution is adopted should absolutely not involve making a copy of the web site. It should be an addon to the web site, not a copy of it. Tom -- Tom Hughes (t...@compton.nu) http://compton.nu/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] How to earn undying fame
On 21 June 2010 16:00, Nic Roets nro...@gmail.com wrote: The result of the uservoice survey[1] is in: The community wants routing on our website more than anything else. And most of the work has already been done. An opensource routing engine that can do the job already exists [2]. The routing database can be updated weekly. The hardware requirements are quite reasonable (64-bit processor with tens of GB of RAM). If a new server is needed, I'm sure we'll get the money in fairly quickly. All that remains is for someone with a little bit of programming experience to volunteer. Make a copy of the osm.org website* and add routing buttons. Then write some code that converts the routing engine output to a few formats. Open Layers, GPX, human readable text should be enough. Without wanting to pour cold water on the idea, I would like to know what the load is a server that you have specified, and how many requests per second can you do? The traffic of the site means that the load will be very big and we might need more than one server in the end. In addition, you also have to modify Gosmore to remove dependencies like GTK as for a server they are not useful. The app will need to be modified to work on a server condition by adding FCGI abilities for example, which is not necessarily trivial. Updating the database weekly might not be able to meet people expectation like the instant rendering that we have. It is more than just a tab to add. Emilie Laffray ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] How to earn undying fame
On 01/-10/-28163 08:59 PM, Tom Hughes wrote: On 21/06/10 16:00, Nic Roets wrote: All that remains is for someone with a little bit of programming experience to volunteer. Make a copy of the osm.org website* and add routing buttons. Then write some code that converts the routing engine output to a few formats. Open Layers, GPX, human readable text should be enough. No. Whatever solution is adopted should absolutely not involve making a copy of the web site. It should be an addon to the web site, not a copy of it. My guess would be that Nic meant: Fork (i.e. copy) the git rails_port repository. Add the desired functionality and then once done try and get it merged back into trunc. So the usual development way to get new functionality into the main site. In case someone would try and do it, it would probably be a good idea to make the frontend code as backend agnostic as possible. In that case it may even be possible for people to choose which routing engine they want to use. Be it gosmore, some other yet to be written opensource routing engine, open route service, or even a commercial provider like cloudmade's routing. Then the decision of which routing engine scales well enough and supports sufficient richness in OSM routing tags to warrent being run on OSMF hardware becomes a somewhat separate issue from the coding of the front end and might at least allow the ball to get rolling. Kai Tom ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] How to earn undying fame
In your letter dated Mon, 21 Jun 2010 16:21:33 +0100 you wrote: In addition, you also have to modify Gosmore to remove dependencies like GTK as for a server they are not useful. The app will need to be modified to = work on a server condition by adding FCGI abilities for example, which is n= ot necessarily trivial. For the people who don't know this already: take a look at http://yournavigation.org/ It is created by Lambertus IJsselstein and does worldwide navigation based on gosmore. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] How to earn undying fame
Hopefully I can help a bit with some numbers, I hope Nic doesn't mind. These numbers originate from yournavigation.org which is a full opensource routing website using (an older version) Gosmore as routing backend. The server that runs the website is a VM with two Xeon 2GHz cores and 2GB RAM. This very limited amount of RAM makes it a bit difficult to do representative benchmarks (harddisk I/O is killing), so please keep that in mind before reading on... Under optimal conditions where the required bit of the route database is already loaded into RAM, Gosmore can calculate most routes in a few milliseconds to about 100 ms with peaks of half a second using a single core. A single core should therefore be able to sustain 10 requests per second, but I've seen 20 requests per second too. I think that Gosmore will scale nicely in handling concurrent requests when given more cores, provided enough RAM is available. I'm doing Gosmore database updates on a C2D P8600 (2.4 GHz) laptop with 4GB RAM is about 8 to 10 hours, so weekly updates won't be a problem. Database sizes: October 2009: 7 GB America's, 4.9 GB Europe, Asia, Africa, Oceania May 2010: America's estimated: 10 GB, 9.9 GB Europe, Asia, Africa, Oceania The database not only contains the road network but also almost all other items in the OSM database (administrative, landuse, poi, houses etc). These could all be left out, but I don't have the figures. My own reservation towards this plan is that (older versions at least) Gosmore doesn't handle very long routes well but, again, this might be a result of the low amount of ram in my server. And I don't know if newer Gosmore versions handle long routes better...Nic? Would it be a good idea to setup a test on the OSM dev server? This server already has a lot of RAM and should behave much closer to a real world server. I can provide a request trace log from yournavigation.org that covers e.g. a whole week (which would be up to 20.000 requests) and run it through a script against Gosmore. That should provide a real-world test and gives you an idea of what to expect. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] How to earn undying fame
On 21/06/10 17:19, Kai Krueger wrote: My guess would be that Nic meant: Fork (i.e. copy) the git rails_port repository. Add the desired functionality and then once done try and get it merged back into trunc. So the usual development way to get new functionality into the main site. In case someone would try and do it, it would probably be a good idea to make the frontend code as backend agnostic as possible. In that case it may even be possible for people to choose which routing engine they want to use. Be it gosmore, some other yet to be written opensource routing engine, open route service, or even a commercial provider like cloudmade's routing. Well if somebody can provide a backend that returns a GPX or KML or something then I'm sure somebody will hack up the rails port to be able to display those routes. Nominatim is the model here - a separate server with an API the rails code can call to get what it needs. Tom -- Tom Hughes (t...@compton.nu) http://compton.nu/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] How to earn undying fame
On 2010-06-21 17:21, Emilie Laffray wrote: Without wanting to pour cold water on the idea, I would like to know what the load is a server that you have specified, and how many requests per second can you do? The traffic of the site means that the load will be very big and we might need more than one server in the end. I'm currently pondering on a routing cluster (somewhat academic interest on my part). In addition, you also have to modify Gosmore to remove dependencies like GTK as for a server they are not useful. Gosmore, when used headless, doesn't need GTK. Libxml2 and g++ will do nicely. The app will need to be modified to work on a server condition by adding FCGI abilities for example, which is not necessarily trivial. You mean running Gosmore as a service and interface with it through e.g. pipes? That could mean some speed boost, but I'm using PHP to interface with Gosmore and spawn a new process for each request without serious impact, see: http://trac.openstreetmap.org/browser/applications/routing/yours/trunk/www/api/dev/gosmore.php Spawning Gosmore doesn't take that much time. Updating the database weekly might not be able to meet people expectation like the instant rendering that we have. It is more than just a tab to add. Yeah well, our gazetteer isn't updated instantly either, not even weekly. Also, I would be seriously impressed if someone could come up with a routing engine that can have it's database updated in situ and still be able to service the whole world with a performance that doesn't drive everyone away instantly. Please tell me if you know of such an opensource routing engine. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] How to earn undying fame
On 21/06/10 17:42, Lambertus wrote: Updating the database weekly might not be able to meet people expectation like the instant rendering that we have. It is more than just a tab to add. Yeah well, our gazetteer isn't updated instantly either, not even weekly. Also, I would be seriously impressed if someone could come up with a routing engine that can have it's database updated in situ and still be able to service the whole world with a performance that doesn't drive everyone away instantly. Please tell me if you know of such an opensource routing engine. It was being updated fairly instantly, and hopefully will be again soon. There was just a bit of a glitch of late... That said, weekly updates is certainly better than nothing. Tom -- Tom Hughes (t...@compton.nu) http://compton.nu/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] How to earn undying fame
On 2010-06-21 18:41, Tom Hughes wrote: Well if somebody can provide a backend that returns a GPX or KML or something then I'm sure somebody will hack up the rails port to be able to display those routes. I know it's PHP and I'm deeply sorry for that, but this code exists: http://trac.openstreetmap.org/browser/applications/routing/yours/trunk/www/api/dev/gosmore.php Get the result back in KML, GPX or geoJSON formats on request :-) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] How to earn undying fame
On 2010-06-21 18:48, Lambertus wrote: On 2010-06-21 18:41, Tom Hughes wrote: Well if somebody can provide a backend that returns a GPX or KML or something then I'm sure somebody will hack up the rails port to be able to display those routes. I know it's PHP and I'm deeply sorry for that, but this code exists: http://trac.openstreetmap.org/browser/applications/routing/yours/trunk/www/api/dev/gosmore.php Get the result back in KML, GPX or geoJSON formats on request :-) Ooh, and I forgot to say that some documentation is here: http://wiki.openstreetmap.org/wiki/YOURS#API_documentation ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] How to earn undying fame
On Mon, Jun 21, 2010 at 5:17 PM, Tom Hughes t...@compton.nu wrote: On 21/06/10 16:00, Nic Roets wrote: The result of the uservoice survey[1] is in: The community wants routing on our website more than anything else. Like we didn't know that already. Tom, you're in London and is an OSM insider. Even Steve Coast indicated that he finds uservoice a useful tool. I feel very out of the loop here in South Africa. I don't know that the supply and demand for routing server software is and that's the main reason why I have spent very little time on the routing back-end during the last 20 months. And most of the work has already been done. An opensource routing engine that can do the job already exists [2]. The routing database can be updated weekly. The hardware requirements are quite reasonable (64-bit processor with tens of GB of RAM). If a new server is needed, I'm sure we'll get the money in fairly quickly. I'm not sure tens of Gb of RAM is that reasonable, though it does rather depend how many tens you mean. It's also rather important to know how fast that memory requirement is likely to grow. Equally I would like to have some idea of how many routes/second you believe this engine to be capable of on that hardware. It depends on how long each route is and how dense the road network is in that area. So I must admit that I can't tell you because I don't have access to a such a machine. The amount of time and RAM used is proportional to the square of the distance between the 'from' and the 'two' marker. So the load can easily be managed by placing an upper limit on it. So when we see an increase in the load (e.g. TV show), a small reduction in that upper limit will solve the problem. No. Whatever solution is adopted should absolutely not involve making a copy of the web site. It should be an addon to the web site, not a copy of it. Like Kai said: A demonstration copy that will be merged with the main site. Updating the database weekly might not be able to meet people expectation like the instant rendering that we have. AFAIK Google Earth takes a month to update their community layer. So weekly updates will be a good start. In case someone would try and do it, it would probably be a good idea to make the frontend code as backend agnostic as possible. Well if somebody can provide a backend that returns a GPX or KML or something then I'm sure somebody will hack up the rails port to be able to display those routes. You can look at the current cgi spec. It's richer than plain GPX, but still quite simple. Using sed or awk to change it to GPX is quite straight forward. (Thanks Lambertus for answering the other questions). ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] How to earn undying fame
Nic Roets nro...@gmail.com writes: The result of the uservoice survey[1] is in: The community wants routing on our website more than anything else. [...] The routing database can be updated weekly. It might make for a strange user experience if a different version of the data is used for routing and rendering. Also, the value of the routing service for debugging purposes would increase significantly if the routing database would follow the minutely updates. Matthias ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] How to earn undying fame
On 21/06/10 18:02, Nic Roets wrote: The amount of time and RAM used is proportional to the square of the distance between the 'from' and the 'two' marker. So the load can easily be managed by placing an upper limit on it. So when we see an increase in the load (e.g. TV show), a small reduction in that upper limit will solve the problem. Things which require manual intervention are not ideal though. We don't have a staff of full time administrators in a position to monitor and adjust things like that. Well if somebody can provide a backend that returns a GPX or KML or something then I'm sure somebody will hack up the rails port to be able to display those routes. You can look at the current cgi spec. It's richer than plain GPX, but still quite simple. Using sed or awk to change it to GPX is quite straight forward. Cool. Not that we'll be using sed or awk of course, but that doesn't matter so long as there is something. All we need now is a volunteer to run the server in the way that Brian does for the geocoding server. I would also add that I'm aware of at least one other candidate for the job of providing a routing solution so I'm not sure exactly where we will go from here but it is certainly something I would expect to see being addressed soon. Tom -- Tom Hughes (t...@compton.nu) http://compton.nu/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] How to earn undying fame
On Mon, Jun 21, 2010 at 7:09 PM, Tom Hughes t...@compton.nu wrote: All we need now is a volunteer to run the server in the way that Brian does for the geocoding server. I'll do it. We can start with a small (2GB RAM) server. My ssh key is (also attached): ssh-rsa B3NzaC1yc2EBIwAAAQEArghsg+FgXJgbT+vViUArdtiQy+gUv+fCPQlXHZ5qxwGdOagPy2bQYig9JFxpcmxBLZDKAnQsXFTiE1vLQQZ7rorPBJyKQ1IYB4tKBCMizg1GUmh4EGHEeoksG9ViN9xguHA1SDoxpMfX8kjS0K7Q9lMEFqru+ZxgkHg0zegL+eoHXJphhYfsm5kyZoKp3H2/4fkjOd+6zxUNGXUE9evvVbw+8MAy156OhGnkbfmaJeVuYS6zO5Pkr3w/jK1dwn447+t6iiw1Cd4845eTzaR9eOl8J8O2MhJZUth4tup1Pg//poYvcrVZSa6LoIgwyvaH+hsOpUEag3PdWQG6Kozq8Q== n...@nick id_rsa.pub Description: Binary data ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] How to earn undying fame
On Mon, Jun 21, 2010 at 1:37 PM, Nic Roets nro...@gmail.com wrote: On Mon, Jun 21, 2010 at 7:09 PM, Tom Hughes t...@compton.nu wrote: All we need now is a volunteer to run the server in the way that Brian does for the geocoding server. I'll do it. We can start with a small (2GB RAM) server. My ssh key is (also attached): ssh-rsa B3NzaC1yc2EBIwAAAQEArghsg+FgXJgbT+vViUArdtiQy+gUv+fCPQlXHZ5qxwGdOagPy2bQYig9JFxpcmxBLZDKAnQsXFTiE1vLQQZ7rorPBJyKQ1IYB4tKBCMizg1GUmh4EGHEeoksG9ViN9xguHA1SDoxpMfX8kjS0K7Q9lMEFqru+ZxgkHg0zegL+eoHXJphhYfsm5kyZoKp3H2/4fkjOd+6zxUNGXUE9evvVbw+8MAy156OhGnkbfmaJeVuYS6zO5Pkr3w/jK1dwn447+t6iiw1Cd4845eTzaR9eOl8J8O2MhJZUth4tup1Pg//poYvcrVZSa6LoIgwyvaH+hsOpUEag3PdWQG6Kozq8Q== n...@nick I just so happen to have a 2GB prgmr instance. Nic, please try logging in to nro...@openstreetmap.xen.prgmr.com using the above key. E-mail directly if you have problems. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] How to earn undying fame
On 21/06/10 19:37, Nic Roets wrote: On Mon, Jun 21, 2010 at 7:09 PM, Tom Hughest...@compton.nu wrote: All we need now is a volunteer to run the server in the way that Brian does for the geocoding server. I'll do it. We can start with a small (2GB RAM) server. No we can't - we have already established that it needs more than that. Not that I have a spare server of that class to give you anyway. The point of this conversation is to work out what we need to procure if we are to do this, and to make sure we have people to run it. Tom -- Tom Hughes (t...@compton.nu) http://compton.nu/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
