I'm doing an a pgsnapshot --wpd on a machine with a lot of RAM, and ran
across a garbage collector memory error.
When running, I had the options -Xmx60g -Xms8g and the osmosis command
line --rb --lp --wpd enableLinestringBuilder=yes
nodeLocationStoreType=yes. About 1.8b nodes in, I ran into a
java.lang.OutOfMemoryError: GC overhead limit exceeded. This is
apparently to stop applications from spending too much time garbage
collecting because the heap is too small. At 60GB, I hope the heap is
not too small.
I've tried making a few changes and am running again
* -server (doh, should of had this before)
* -Xms24g (We know it's going to need lots of heap)
* -XX:+UseParallelGC (Apparently higher throughput, and latency or
pauses isn't an issue on a bulk ETL job)
* -XX:-UseGCOverheadLimit (Disables the overhead limit check)
My thought is that there may be some point in the process where it does
a *lot* of garbage collection at once, and this is triggering the limit
check, but periodically doing a lot of GC isn't a huge problem on an ETL
task which runs overnight, the total time spend on GC is more important.
Does anyone have any suggestions for improvement?
Something else that's come up is that the node store for --wpd is very
inefficient. osm2pgsql shows that you can store all the nodes in the
planet in ~20GB with its dense/sparse cache, or in ~23GB as a simple
flat array (--flat-nodes). Osmosis needed about twice that the last time
I checked.
_______________________________________________
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev
