Hi, > Am I missing something here...? People are complaining about how bogged > down and slow the current service is, so its being re-written in java? > Is there any language slower or more resource intensive than java? If > the service isnt designed to be portable (it only runs on one system > currently, in the world), then who cares about java, why isnt it written > in optimized C or some other similarly lowish level language, rather > than java? You miss a whole lot in that picture. First of all openstreetmap has a really nice growth rate when it comes to mappers but we still lack developers, especially ones that create "infrastructure" and not nice webapps for the general public. So if one of the not-so-many developers actually finds the time and starts to improve a service that is in dire need of developer love who can dictate which tools he is to use?
The current (or some would say old) XAPI is written in mumps with a GT.M key-value database. This language has quite a bit of history and it (as well as the XAPI implementation in it) have been able to keep up with openstreetmaps growth for quite a long time and have provided an invaluable data source. A while back the tiles@home clients switched away from XAPI to the newly created TRAPI read only mirrors to reduce load from simple "map" calls on the XAPI in the hope that this would free enough resources for the more complicated queries to the XAPI. Never the less it is no longer able to keep up with the increasing demand of "I need all organic=yes nodes on london and I need it now" requests. Part of the problem is a whole bunch of stupid data scraper which just use huge map calls (that the main API disallows) to XAPI instead of using TRAPI or going straight to the planet dump or extracts there of. Now given that XAPI is written in a rather exotic language other developers can not just go in and try to improve the current state of affairs and 80n is busy doing other things and I don't blame him. I know very well how much time can be gobbled up by developing for openstreetmap. This is about the situation we face end of last year / start of this year. Several developers including myself started to say "if we can't improve the current XAPI we might have to rewrite it". At first we said that as a joke because we very well know how large such a task is. It needs tons of manpower to get to the point where the existing XAPI is, you need a fairly large box to handle all of the planet and you might have moments where you have to reload the database from scratch and can't develop for a whole week. At this point iandees stepped up to the task and I'm immensely glad that he did. iandees now chose java as the language of choice, a language that I personally don't like and that I questioned when he first proposed it. However he gave a couple of reasons that really convinced me: * "It has to be a language that I know." Now how could I argue with that? I had been thinking about a rewrite quite some time but didn't find the time and motivation. I know how badly I would have reacted if somebody told be "you can't use C" so who am I to tell iandees "you can't use java". You want a C version? Go write it. * "It has to be a language that is spoken by a large number of developers." Now mumps was the favorite tool of 80n and it worked very well from him judging from the nice service he wrote in it. Be there aren't all that many mumps developers around, at least not in the openstreetmap ecosphere. I personally don't like java too well but at least I can read (and write if need be) it and I know a whole bunch of developers who use java. So if iandees doesn't have time for openstreetmap in 5 years or gets eaten by a grue then jXAPI wont die but others can take over and keep it running. * The third point, and this is the point that really sold be, was: "If I don't do it completely wrong the real bottleneck wont be parsing the result but the speed at which I can retrieve the data from hard disk. Consequently I wont write that part myself but I will use postgis, a well-known, well-optimized piece of software that is written in C by a whole lot of people who can throw way more manpower at it then I ever could." I have to applaud him for that idea and I complete agree with his analysis that java performs sufficiently well to act as the thin glue layer between HTTP queries and the postgis backend. So far we have no real performance numbers for the new jXAPI much less a direct comparison to XAPI-mumps on the same hardware. If you want to benchmark that: Go ahead! If you ask politely we can even hand you a (anonymised!) query log to the XAPI to use as a basis for your benchmark. If you think you can write a faster alternative then postgis in C or another low-level language: Go ahead! We wont hold our breath though. Patrick "Petschge" Kilian _______________________________________________ talk mailing list [email protected] http://lists.openstreetmap.org/listinfo/talk
