On Mon, 2011-02-07 at 20:00 +0100, Frederik Ramm wrote: > Hi, > > when you run osm2pgsql in slim mode with a large cache, then could > it not release the cache memory once it reaches the "Sorting data and > creating indexes" stage? It is this phase where postgresql tends to > consume a lot of memory for index creation, and the specific settings on > the machine I'm running it on at the moment have led to all of > osm2pgsql's 10 GB being swapped out - which, if the memory is really > unused, isn't a big deal but I wonder if the memory could not have been > freed anyway?
I think what you are describing is an issue I am aware of and even have a patch which I use myself (attached). The memory allocations are being freed correctly but the overall 'sbrk' memory can not be released due to fragmentation. With the attached patch the block allocator makes larger malloc requests which glibc fulfils via an mmap() based scheme which avoids this fragmentation (see MMAP_THRESHOLD in the malloc man page). The reason that I have not applied this to the main tree is that it triggers more out-of-memory errors for people that are importing small extracts. A better long term solution would be some way of automatically detecting whether the data being imported seems to be dense (i.e. a whole planet) vs sparse (a country extract) and tweaking some of the internal parameters to optimize for these two situations. Jon
Index: middle-pgsql.c =================================================================== --- middle-pgsql.c (revision 25083) +++ middle-pgsql.c (working copy) @@ -178,7 +178,7 @@ int used; }; -#define BLOCK_SHIFT 10 +#define BLOCK_SHIFT 16 #define PER_BLOCK (1 << BLOCK_SHIFT) #define NUM_BLOCKS (1 << (32 - BLOCK_SHIFT)) Index: middle-ram.c =================================================================== --- middle-ram.c (revision 25083) +++ middle-ram.c (working copy) @@ -64,7 +64,7 @@ * */ -#define BLOCK_SHIFT 14 +#define BLOCK_SHIFT 16 #define PER_BLOCK (1 << BLOCK_SHIFT) #define NUM_BLOCKS (1 << (32 - BLOCK_SHIFT))
_______________________________________________ dev mailing list [email protected] http://lists.openstreetmap.org/listinfo/dev
