Hi Stefan, Stefan de Konink wrote: > How do you handle the LargeFile allocation? Because I tried mmaping on > some obscure 32 userspace valgrind 64 kerner lately, and that was prone > to failure.
It's a (very crude) "windowing" system - a smaller chunk of the file is read in, scanned, and scooted down. If a single XML node were larger than the window, it would fail. But...even in 32 bits you can make a very large window - I think we are unlikely to have a single tag be > 1 GB, for example. :-) In the long term I want to switch from my hacky crude lousy XML parser to expat...expat runs through XML at about half the speed mine does, but then it is a real correct robust XML parser whereas I am simply grepping blindly for strings (and wouldn't handle legal small changes in char spacing well). Run time was about 18 hours last time I ran it...the biggest limit is the number of open file descriptors - only 1024 on my Mac. Currently I do multiple passes over the input file to avoid open/closing output files...the locality of output was really poor when I examined the problem. The program uses gzip on both input/output to let someone work entirely in terms of compressed XML. It could easily be converted to bz2. > Welcome in the fantastic world of OSM :D I thought had fixed all those > instances two days ago :) Cool you found another one. At least - it only finds one and dies right now. :-) > The idea with that situation is for ways: > - Download the way; and just reupload it, then it will be solved in the > next run (because the way download hides invisible nodes) Okay - so simply omitting the node would be a reasonable thing to do. Thanks!! cheers ben -- Scenery Home Page: http://scenery.x-plane.com/ Scenery blog: http://xplanescenery.blogspot.com/ Plugin SDK: http://www.xsquawkbox.net/xpsdk/ X-Plane Wiki: http://wiki.x-plane.com/ Scenery mailing list: x-plane-scen...@yahoogroups.com Developer mailing list: x-plane-...@yahoogroups.com _______________________________________________ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
