I'm beginning to implement a tile expiration solution and have run into
some issues with the new tile expiry expansion.

I'm afraid I do not have precise timings, but I'm seeing what appears
to be at least an order of magnitude performance penalty, probably due
to memory exhaustion.

Test machine is Debian stretch 4CPUs, 26GB RAM, SSD Array. Initial osm
import was performed with osm2pgsql 0.92 and finished in under 48 hours.

Under 0.92 I was running multiple render chains while importing
changesets. Tests w/ 0.96 have been run against an otherwise idle

Database & flatnodes file were restored to initial state between each
round of testing.

WEEKLY changeset using 0.92:
time $OSM2PGSQL --append -s -C 3000 -G --hstore -d gis -H $PGHOST -U \
$PGUSER -r xml changes.osc \
--flat-nodes /database/postgresql/OSM-FLATNODES --slim \
--number-processes 4 --style openstreetmap-carto.style \
--tag-transform-script openstreetmap-carto.lua -e19 -o \

With -e19, I was able to import a weekly changeset in roughly 24 hours.

Using git repository (5/27? pull):
time $OSM2PGSQL --append -s -C 3000 -G --hstore -d gis -H $PGHOST -U
$PGUSER -r xml changes.osc \
--flat-nodes /database/postgresql/OSM-FLATNODES --slim \
--number-processes 4 --style openstreetmap-carto.style \
--tag-transform-script openstreetmap-carto.lua -e10-16 -o \

Consistently crashed w/ a bad alloc(). I didn't note any unusual output
in the compile. Crashes even with a 24 hour changeset.

DAILY changeset using Debian backport to stretch (0.96):
As above, using -e10-16. 

22 hours spent processing a 24 hour changeset and still importing new
relations. It's 35GB into swap, with osm2pgsql claiming 89% of the
memory usage.


I realize there is a large difference between zoom level 19 and 10-16,
but I assume it should take significantly less RAM/CPU for 10-16.

Please feel free to point out any stupidity I've generated here and/or
recommend a better way to generate a list of dirty tiles at lower zoom
levels based on the 0.92 output.

To be fair, I am not sure this is even related to tile expiration. I
have not tried 0.96 updates without tile expiration as a baseline.


dev mailing list

Reply via email to