I'd be happy to look at the code, but XML parsing is no the slow part: writing to the format that Mongo expects (BSON) is.
On Mon, Jul 5, 2010 at 1:43 PM, Andreas Kalsch <[email protected]> wrote: > What about rewriting this stuff in C? I have written a MySQL importer in C > some time ago, so what about reusing its XML parsing part? > > If you are interested, I'll put it on Github. > > Andi > > Am 03.07.10 22:43, schrieb Ian Dees: > > On Sat, Jul 3, 2010 at 1:17 PM, Nolan Darilek <[email protected]>wrote: > >> On 07/03/2010 01:09 PM, Nolan Darilek wrote: >> >>> On 07/02/2010 01:52 PM, Serge Wroclawski wrote: >>> >>>> Similarly, Ian Dees and I have written a server using MongoDB, which >>>> also provides functionality such as auto-sharding and built in >>>> map/reduce. >>>> >>>> >>> Is this work available anywhere? How did you find performance to be, and >>> to what uses did you put it? >>> >>> I've done some experiments creating a LibOSM MongoDB backend and found >>> its performance fairly bad, but I don't have the most optimal server for it, >>> and probably didn't use MongoDB to its limits. If you experienced good >>> performance for real-time operations then I'd be very interested in seeing >>> how you managed it so I might adopt the techniques and see if I have any >>> better luck. >>> >>> It seemed to me that a dump of the entire planet would require a >>> substantial server to serve up, so I abandoned the work, but would very much >>> like to revive it if it's at all workable. >>> >>> > The code is here: http://github.com/iandees/mongosm > > It took several days to import a planet file. The majority of the CPU > time was spent serializing/deserializing BSON in Python and the Mongo server > had very little CPU time so if I use a language with a faster BSON > implementation it might be faster. > > Serge was working on a way to import diff files to maintain mintutely > updates. > > I was working on an HTTP API interface. > > > _______________________________________________ > dev mailing list > [email protected]http://lists.openstreetmap.org/listinfo/dev > > >
_______________________________________________ dev mailing list [email protected] http://lists.openstreetmap.org/listinfo/dev
