Re: [OSM-dev] Lean and mean Tile- and XML-API-Server
Matt Amos wrote: premature optimisation is the root of all evil ;-) A premature optimisation would be starting with integers, C, and more. Now even I implemented a relatively 'old' API (0.5) and started with doubles ;) Last night I introduced a comparison table with the 2^32/360 stuff. To see there is 10ms improvement per bbox request ;) Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Lean and mean Tile- and XML-API-Server
On Fri, Nov 21, 2008 at 11:45 AM, Stefan de Konink [EMAIL PROTECTED] wrote: Matt Amos wrote: premature optimisation is the root of all evil ;-) A premature optimisation would be starting with integers, C, and more. premature optimisation would be writing something in C(++) when it isn't the bottleneck. the database is currently the bottleneck and i'm pretty sure they already wrote that in C :-) Now even I implemented a relatively 'old' API (0.5) and started with doubles ;) Last night I introduced a comparison table with the 2^32/360 stuff. To see there is 10ms improvement per bbox request ;) what was %age gain? i mean, if its a 10ms improvement on a 500ms call then it might not be such a big deal. if its a 10ms gain on an 11ms call that would be fantastic. don't forget that the rails port also handles user management, friends, messages, diaries, gps points, etc... it would be better to say you have implemented part of 'old' API 0.5. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Lean and mean Tile- and XML-API-Server
Matt Amos wrote: On Fri, Nov 21, 2008 at 11:45 AM, Stefan de Konink [EMAIL PROTECTED] wrote: Matt Amos wrote: premature optimisation is the root of all evil ;-) A premature optimisation would be starting with integers, C, and more. premature optimisation would be writing something in C(++) when it isn't the bottleneck. I hear a different thing when I browse on this list about ruby memory usage :) the database is currently the bottleneck and i'm pretty sure they already wrote that in C :-) The database that is used in production isn't the most efficient one too ;) So that is also optimised ;) Now even I implemented a relatively 'old' API (0.5) and started with doubles ;) Last night I introduced a comparison table with the 2^32/360 stuff. To see there is 10ms improvement per bbox request ;) what was %age gain? i mean, if its a 10ms improvement on a 500ms call then it might not be such a big deal. if its a 10ms gain on an 11ms call that would be fantastic. Double: 50~100ms Integer: 40~80ms But the calltrace revealed in both search situation more optimisations where possible. don't forget that the rails port also handles user management, friends, messages, diaries, gps points, etc... it would be better to say you have implemented part of 'old' API 0.5. I have implemented API0.5 + XAPI. User management and GPX was on the todo. If I implement user management I'll implement an OpenStreetProxy, for now I don't even keep changes (for reasons I already mentioned). Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Lean and mean Tile- and XML-API-Server
On Fri, Nov 21, 2008 at 2:06 PM, Stefan de Konink [EMAIL PROTECTED] wrote: Matt Amos wrote: On Fri, Nov 21, 2008 at 11:45 AM, Stefan de Konink [EMAIL PROTECTED] wrote: Matt Amos wrote: premature optimisation is the root of all evil ;-) A premature optimisation would be starting with integers, C, and more. premature optimisation would be writing something in C(++) when it isn't the bottleneck. I hear a different thing when I browse on this list about ruby memory usage :) ah well, thats the price you pay for being able to easily write code without having to manage memory manually (or stick shared_ptr all over the place). there is a bug in there too, i think, which results in libxml not fully free()ing all the memory it is using. the database is currently the bottleneck and i'm pretty sure they already wrote that in C :-) The database that is used in production isn't the most efficient one too ;) So that is also optimised ;) how did you optimise it? (other than converting ways to relations)? Double: 50~100ms Integer: 40~80ms But the calltrace revealed in both search situation more optimisations where possible. 20% is pretty significant. obviously thats an optimisation worth having. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Lean and mean Tile- and XML-API-Server
Matt Amos wrote: ah well, thats the price you pay for being able to easily write code without having to manage memory manually (or stick shared_ptr all over the place). there is a bug in there too, i think, which results in libxml not fully free()ing all the memory it is using. Come on you can give better arguments than that :) C/C++ is not a prototype language. You will need to have your functional diagrams ready before start 'scripting'. So I don't mind pointers, and in this case my webserver's api takes care of the string management, so I am a happy coder :) the database is currently the bottleneck and i'm pretty sure they already wrote that in C :-) The database that is used in production isn't the most efficient one too ;) So that is also optimised ;) how did you optimise it? (other than converting ways to relations)? Not using MySQL, but MonetDB. It uses column based storage. Double: 50~100ms Integer: 40~80ms But the calltrace revealed in both search situation more optimisations where possible. 20% is pretty significant. obviously thats an optimisation worth having. Jup, but extra mathematical overhead in query generation that should not be forgotten. Every output has to be atoi - double back. And in the case of storing doubles/floats the input can directly be passed to the user. I need to figure out if the overhead of data translation is not bigger than querying speed. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Lean and mean Tile- and XML-API-Server
On Fri, Nov 21, 2008 at 2:22 PM, Stefan de Konink [EMAIL PROTECTED] wrote: Matt Amos wrote: ah well, thats the price you pay for being able to easily write code without having to manage memory manually (or stick shared_ptr all over the place). there is a bug in there too, i think, which results in libxml not fully free()ing all the memory it is using. Come on you can give better arguments than that :) but i don't need to :-) C/C++ is not a prototype language. do you mean http://en.wikipedia.org/wiki/Prototype-based_programming ? You will need to have your functional diagrams ready before start 'scripting'. So I don't mind pointers, and in this case my webserver's api takes care of the string management, so I am a happy coder :) i don't mind pointers either, but i know a lot of people who do. C++ is a powerful and complex language, which takes a long time to learn to use properly. in an open-source project, this means either potential contributors are excluded, or that they submit poorly-written code. of course, its possible for any developer to make a mistake, but at least in ruby it is less likely to cause a colossal failure. how did you optimise it? (other than converting ways to relations)? Not using MySQL, but MonetDB. It uses column based storage. interesting. the web site describes it as an in-memory database, but i assume it can store to disk as well. so why is everyone still using mysql / postgres / oracle? Jup, but extra mathematical overhead in query generation that should not be forgotten. Every output has to be atoi - double back. And in the case of storing doubles/floats the input can directly be passed to the user. I need to figure out if the overhead of data translation is not bigger than querying speed. to be fair, the latency between here and the states is about 150ms, so you may not need to optimise further. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Lean and mean Tile- and XML-API-Server
Matt Amos wrote: C/C++ is not a prototype language. do you mean http://en.wikipedia.org/wiki/Prototype-based_programming ? No I mean a language where you start scripting and others have no clue what you are doing until it finished and it works. Other words; Proof-of-Concept. Low level languages are not good for this, too much distractions with memory management etc., but once you know exactly what you want. Don't waste your time on a language that does 90% for you sadly has no compiler to native machine code. how did you optimise it? (other than converting ways to relations)? Not using MySQL, but MonetDB. It uses column based storage. interesting. the web site describes it as an in-memory database, but i assume it can store to disk as well. so why is everyone still using mysql / postgres / oracle? I know some pretty big Dutch banking/insurance organisations are using an ancient version of it, that was tailor made and tested specifically on their applications. There are some datawarehouses that use it. Plus this database implements SQL92 by the letter, no funny extensions. So we can call it; nobody knows, no body is using it. Jup, but extra mathematical overhead in query generation that should not be forgotten. Every output has to be atoi - double back. And in the case of storing doubles/floats the input can directly be passed to the user. I need to figure out if the overhead of data translation is not bigger than querying speed. to be fair, the latency between here and the states is about 150ms, so you may not need to optimise further. You miss the point of concurrent usage every performance improvement will allow more users to be served simultaneously. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Lean and mean Tile- and XML-API-Server
2008/11/21 Stefan de Konink [EMAIL PROTECTED]: Matt Amos wrote: C/C++ is not a prototype language. do you mean http://en.wikipedia.org/wiki/Prototype-based_programming ? No I mean a language where you start scripting and others have no clue what you are doing until it finished and it works. Other words; Proof-of-Concept. Low level languages are not good for this, too much distractions with memory management etc., but once you know exactly what you want. Don't waste your time on a language that does 90% for you sadly has no compiler to native machine code. Meh, proper languages are often over rated. Machines are cheap, developers are expensive, and code lives far longer than most people hope. Maintaining well written code is easier than maintaining crap code, and many of your prototype languages promote well written code. Especially when you talk about ruby on rails. The entire point is an easy to maintain code base, with MVC architecture, database schema versioning, and all the crud and boiler plate taken out of your way. Point a rails developer at a well written rails app and they can figure out what's what in very little time. C/C++ has it's place, and I'm not claiming it's not faster (or even better) for this application. But that doesn't make RoR a prototyping system. Or as a mirror for your last statement: don't waste your time on a language that compiles 90% quicker when it doesn't matter. Which leaves the important question: does it matter for OSM? (and is a full rewrite the best way of fixing it if it does) Dave ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Lean and mean Tile- and XML-API-Server
Dave Stubbs wrote: Meh, proper languages are often over rated. Machines are cheap, developers are expensive, and code lives far longer than most people hope. Who is paying for new machines in OSM? And who is getting paid for writing code? Which leaves the important question: does it matter for OSM? (and is a full rewrite the best way of fixing it if it does) As long it is slow, and someone is stupid enough to try and fix it, it matters :) Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Lean and mean Tile- and XML-API-Server
On Fri, Nov 21, 2008 at 3:42 PM, Dave Stubbs [EMAIL PROTECTED] wrote: 2008/11/21 Stefan de Konink [EMAIL PROTECTED]: Don't waste your time on a language that does 90% for you sadly has no compiler to native machine code. i'd have said the same thing about javascript a couple of years ago. Meh, proper languages are often over rated. Machines are cheap, developers are expensive, and code lives far longer than most people hope. to put it another way: computers are getting better and faster, but human brains have been the same for hundreds of thosands of years. Maintaining well written code is easier than maintaining crap code, and many of your prototype languages promote well written code. except for perl, of course ;-) Which leaves the important question: does it matter for OSM? (and is a full rewrite the best way of fixing it if it does) we'll see soon whether api 0.6 will make things better. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Lean and mean Tile- and XML-API-Server
Matt Amos wrote: On Fri, Nov 21, 2008 at 3:42 PM, Dave Stubbs [EMAIL PROTECTED] wrote: 2008/11/21 Stefan de Konink [EMAIL PROTECTED]: Don't waste your time on a language that does 90% for you sadly has no compiler to native machine code. i'd have said the same thing about javascript a couple of years ago. Yes this is *exactly* my point. Python/Ruby are great high level languages. But until there is any real progress in getting them to work at native speeds (or at least in the python case, better speeds are possible) it is not an alternative for HP applications, while for HA it can be a choice because of the design. Meh, proper languages are often over rated. Machines are cheap, developers are expensive, and code lives far longer than most people hope. to put it another way: computers are getting better and faster, but human brains have been the same for hundreds of thosands of years. Faster is overrated; hence we are looking here at I/O and scheduling bottlenecks. Not even a Pentium X is going to get more speed out of a sata harddisk. Which leaves the important question: does it matter for OSM? (and is a full rewrite the best way of fixing it if it does) we'll see soon whether api 0.6 will make things better. Very true :) Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Lean and mean Tile- and XML-API-Server
2008/11/21 Stefan de Konink [EMAIL PROTECTED]: Dave Stubbs wrote: Meh, proper languages are often over rated. Machines are cheap, developers are expensive, and code lives far longer than most people hope. Who is paying for new machines in OSM? And who is getting paid for writing code? 1) The OSMF through donations and membership, other random donations etc. 2) Nobody (not by OSM(F) at any rate), so that means we're limited by how much time people are willing to provide for free (either companies or individuals). And besides which it was a general statement in response to your general statement about languages. Any applicability to OSM in particular is a fairly complex question which is, unsurprisingly, going to result in compromises. Which leaves the important question: does it matter for OSM? (and is a full rewrite the best way of fixing it if it does) As long it is slow, and someone is stupid enough to try and fix it, it matters :) Sure, except that the code, once written, doesn't just sit there. People have to maintain this stuff. It's not just about, can it be written?, it's can it be written and maintained?. So I think a better reply (well, maybe just more complete) is that as long as it is slow, and someone is stupid enough to try and fix it, and they can convince the guys who are going to end up responsible for deploying and maintaining it, it matters. Sometimes speed becomes important and it makes sense to replace certain parts of the system, ie: the GPX importer which has already been down this path. Dave ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Lean and mean Tile- and XML-API-Server
Dave Stubbs wrote: 1) The OSMF through donations and membership, other random donations etc. 2) Nobody (not by OSM(F) at any rate), so that means we're limited by how much time people are willing to provide for free (either companies or individuals). And besides which it was a general statement in response to your general statement about languages. Any applicability to OSM in particular is a fairly complex question which is, unsurprisingly, going to result in compromises. Many corporate boys always go to battle with the stupid argument you made. Your general statement only points out laziness of developers, doing things in shorter time resulting (usually) in suboptimal solutions. Since we are a group of people that not get paid and therefore need to produce better work in order to use our facilities more optimal it is to defend to always try to improve the work, because the userbase grows and the amount of hardware is limited. Which leaves the important question: does it matter for OSM? (and is a full rewrite the best way of fixing it if it does) As long it is slow, and someone is stupid enough to try and fix it, it matters :) Sure, except that the code, once written, doesn't just sit there. People have to maintain this stuff. It's not just about, can it be written?, it's can it be written and maintained?. So I think a better reply (well, maybe just more complete) is that as long as it is slow, and someone is stupid enough to try and fix it, and they can convince the guys who are going to end up responsible for deploying and maintaining it, it matters. The amount of people here that feel responsible for each others code is very low. If you look at the tools available for processing OSM data it is also easy to see that some are superseded a long time ago and others get bitrot by updated APIs. In my perspective; if everything is unittested and the limitations are known, they don't break if they are untouched. To use another well known sales boy quote: no one is irreplaceable, there will thus always be someone that is capable of reading the documentation, and changing the code. Sometimes speed becomes important and it makes sense to replace certain parts of the system, ie: the GPX importer which has already been down this path. Exactly, and so can we do with the API/XAPI if we want to :) Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Lean and mean Tile- and XML-API-Server
On Fri, Nov 21, 2008 at 4:37 PM, Stefan de Konink [EMAIL PROTECTED] wrote: Matt Amos wrote: On Fri, Nov 21, 2008 at 3:42 PM, Dave Stubbs [EMAIL PROTECTED] wrote: 2008/11/21 Stefan de Konink [EMAIL PROTECTED]: Don't waste your time on a language that does 90% for you sadly has no compiler to native machine code. i'd have said the same thing about javascript a couple of years ago. Yes this is *exactly* my point. Python/Ruby are great high level languages. But until there is any real progress in getting them to work at native speeds (or at least in the python case, better speeds are possible) it is not an alternative for HP applications, while for HA it can be a choice because of the design. people are making progress. http://antoniocangiano.com/2008/05/31/maglev-rocks/ http://blog.fallingsnow.net/2008/09/05/rubinius-status/ for high performance it may be worth porting some bits to lower-level languages (e.g: the map call). but some people at the api 0.6 hack weekend had some ideas for improving performance while staying within RoR. Faster is overrated; hence we are looking here at I/O and scheduling bottlenecks. Not even a Pentium X is going to get more speed out of a sata harddisk. install a raid array? ;-) at what point do you decide that your code is fast enough? cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Lean and mean Tile- and XML-API-Server
Matt Amos wrote: Exactly, and so can we do with the API/XAPI if we want to :) There's only one API though, so you need to do one awesome job of convincing the right people :-) and ensure that no functionality is lost (e.g: user diaries, comments, friends, etc...). i don't use these features, but there are people who like them. Interesting that you mention them, but where can all data be downloaded that is obviously there under CC-SA-BY? The same counts for the full history of OSM. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Lean and mean Tile- and XML-API-Server
On Fri, Nov 21, 2008 at 7:53 PM, Stefan de Konink [EMAIL PROTECTED] wrote: Matt Amos wrote: and ensure that no functionality is lost (e.g: user diaries, comments, friends, etc...). i don't use these features, but there are people who like them. Interesting that you mention them, but where can all data be downloaded that is obviously there under CC-SA-BY? The same counts for the full history of OSM. obviously there are privacy issues with providing users' data. but the source code of the current API is public, so you could create test data pretty easily. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Lean and mean Tile- and XML-API-Server
Matt Amos wrote: On Fri, Nov 21, 2008 at 7:53 PM, Stefan de Konink [EMAIL PROTECTED] wrote: Matt Amos wrote: and ensure that no functionality is lost (e.g: user diaries, comments, friends, etc...). i don't use these features, but there are people who like them. Interesting that you mention them, but where can all data be downloaded that is obviously there under CC-SA-BY? The same counts for the full history of OSM. obviously there are privacy issues with providing users' data. but the source code of the current API is public, so you could create test data pretty easily. Are the user prefs also in an api? Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Lean and mean Tile- and XML-API-Server
Shaun McDonald wrote: On 21 Nov 2008, at 21:07, Stefan de Konink wrote: Matt Amos wrote: Are the user prefs also in an api? Yes user preferences are in the API. http://wiki.openstreetmap.org/wiki/Api Could you add it to Other API's? Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Lean and mean Tile- and XML-API-Server
On Sat, Nov 22, 2008 at 12:46 AM, Stefan de Konink [EMAIL PROTECTED] wrote: Shaun McDonald wrote: On 21 Nov 2008, at 21:07, Stefan de Konink wrote: Are the user prefs also in an api? Yes user preferences are in the API. http://wiki.openstreetmap.org/wiki/Api Could you add it to Other API's? It's in API 0.5: http://wiki.openstreetmap.org/wiki/OSM_Protocol_Version_0.5#Preferences -- /emj ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Lean and mean Tile- and XML-API-Server
Stefan Keller wrote: Would'nt it be worthwhile to rewrite everything in a single well known language (like Ruby or Java)? The reason for the rewrite to C(++) of common used tools within OSM (api server/editor) is purely performance and not a prototype language like Python, Ruby. Or a language that is cross platform but is memory hungry as Java. Next to this. I don't know if you actually try to setup an API server yourself ;) But I think you will be shocked how much time the 'official' builds will take you ;) Yes I am biased, but also capable to setup a new server within the download time of the planetfile ;) Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Lean and mean Tile- and XML-API-Server
On Fri, Nov 21, 2008 at 1:07 AM, Stefan de Konink [EMAIL PROTECTED] wrote: Stefan Keller wrote: Would'nt it be worthwhile to rewrite everything in a single well known language (like Ruby or Java)? iirc, the only thing in a weird language is osmxapi which, as you point out, is written in MUMPS. the server software is mostly in ruby. osmosis, the OSM swiss-army knife is in java. various little libraries and tools are in C/C++. ROMA is in perl. all of these are well-known, if heterogeneous. The reason for the rewrite to C(++) of common used tools within OSM (api server/editor) is purely performance and not a prototype language like Python, Ruby. Or a language that is cross platform but is memory hungry as Java. premature optimisation is the root of all evil ;-) Next to this. I don't know if you actually try to setup an API server yourself ;) But I think you will be shocked how much time the 'official' builds will take you ;) Yes I am biased, but also capable to setup a new server within the download time of the planetfile ;) the instructions here http://wiki.openstreetmap.org/wiki/The_Rails_Port shouldn't take more than 20 minutes to set up on a linux box and, at a minimum, requires no compilation whatsoever. the planet to mysql import takes some extra time, but that must be true of your C++ port as well. there are optional ruby and mysql libraries to improve performance, which do need to be compiled. however, they are not necessary and you can easily run a personal API without them. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Lean and mean Tile- and XML-API-Server
I'd like to organize - or implement if needed - an own server which offers the 'usual' Tile- and XML-API (read only, ev. with diff upload) services for OSM data. I've so far found OnDemandTileServer, The_Rails_Port, Mapnik and other tools. These are all Ok, but I'm looking for something lean and mean, i.e. based on only postgres, mapserver and a (one!) modern language (Ruby, Java, etc.) and on as few dependencies as possible. Any suggestions? -- Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Lean and mean Tile- and XML-API-Server
Stefan Keller wrote: Any suggestions? http://repo.or.cz/w/handlerosm.git We are building a VM around it. Some OGR patches still need to be modified, but then we have a live Mapserver running on our database. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
