On 10/19/2012 03:06 PM, Roland Olbricht wrote:
If you are C++ averse, the good news is that you may resolve the real bottleneck independent of the programming language.
I'm OK with C++ when it is justified. I personally am more productive in Java/Ruby/Python than C++.
I've written down some notes about the Overpass API implementation with similar scalability problems: http://wiki.openstreetmap.org/wiki/Overpass_API/Technical_details
Thanks, this is great stuff, I would definitely be interested in learning from people more experienced in processing OSM data.
I just simply don't have server resources to work with anything larger than a Europe dump which already is crawling on the hardware I've got available.
However, XML parsing (and gzip compression) are heavily CPU intensive. Thus, it might be a good idea to retain these in C++ or getting done in carefully chosen external libraries.
It would be interesting to see what is the overhead of Ruby when parsing with libxml-ruby (or Nokogiri which is another thin layer on top of libxml) compared to C/C++.
Maybe another way to go would be to use PBF which is supposed to be much faster for machine reading.
Paweł _______________________________________________ dev mailing list [email protected] http://lists.openstreetmap.org/listinfo/dev
