Hello WanMil, the attached patch reverts the changes that calculated the MaxNodeId and nodeCount values. The default is now to assume a highest node id of 2^31 and resize when needed. This still needs much fewer memory compared to r181 because it allocates exactly two of these large arrays, while r181 typically allocated 8. So, on my machine splitting germany.osm.pbf took 938 secs with r181 (GC very busy because of -Xmx1600m ) and only 638 secs with patchv4 (optimize-mem=false). With optimize-mem=true, the program is typically ~ 10% slower.
I've also increased the allowed number of areas per pass, because this is now limited only by available memory and by the number of elements in the dictionary. If the latter reaches 32767 the program will crash, but I doubt that this will happen with real OSM data. Ciao, Gerd > Date: Sat, 12 Nov 2011 12:41:11 +0100 > From: [email protected] > To: [email protected] > Subject: Re: [mkgmap-dev] [PATCH]splitter memory usage > > Am 11.11.2011 14:58, schrieb Gerd Petermann: > > > Date: Wed, 9 Nov 2011 19:24:37 +0100 > > > From: [email protected] > > > To: [email protected] > > > Subject: Re: [mkgmap-dev] [PATCH]splitter memory usage > > > > > > Thanks Gerd! > > > > > > I didn't yet review the patch but I have commited (r184) it so it's > > > available at http://www.mkgmap.org.uk/splitter/ > > > > > > I am looking forward that your patches will improve splitter for several > > > use cases. > > > > > > WanMil > > > > Hello WanMil, > > > > please apply the attached patch on the memory_optimization branch. It > > contains: > > a) code simplification > > b) correction regarding the possibilty that a way id is higher than the > > highest node id > > c) small performance tweaks > > > > ciao, > > Gerd > > > > Hi Gerd, > > commited it to r187. > > Have fun! > WanMil > > _______________________________________________ > mkgmap-dev mailing list > [email protected] > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
memory_patch_v4.patch
Description: Binary data
_______________________________________________ mkgmap-dev mailing list [email protected] http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
