Re: [OSM-dev] osm2pgsql tile expiry performance
On Tue, 4 Jun 2019 14:28:38 +0200 Michael Reichert wrote: > Just to ensure that all important test conditions are known. > > Do you mean the weekly changesets dump from > https://planet.openstreetmap.org/planet/changesets-latest.osm.bz2? Or > how did you create changes.osc or where did you download it from? > changesets-latest.osm.bz contains changeset metadata only. changes.osc was created via osmosis. 0.92 was running w/ a max interval of 7 days. 0.96 was running w/ a max interval of 1 day. For reference, this is the command used to create the changeset: osmosis --read-replication-interval workingDirectory=$WORKDIR_OSM --simplify-change --write-xml-change changes.osc.gz Replication url from osmosis configuration.txt: https://planet.openstreetmap.org/replication/day/ > The tests were done with Europe only which is roughly 2/3 of the whole > planet. I did not realize Europe accounted for such a large portion of the OSM data! I did notice that your test reported a 10GB memory usage for the import. I did not pay attention to the memory usage for 0.92 imports, but tile expiration seems to be requiring significant memory as of 0.96. My own test hit 22GB before I ^C'd. 21GB more than required w/o expiration. > I will have a deeper look into this but my test machine is currently > busy doing other work. My own dev box is very busy for the next couple of weeks, but I can spin something up for testing purposes if needed. j ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql tile expiry performance
Hi, Am 29.05.19 um 21:40 schrieb j: > WEEKLY changeset using 0.92: > -=-=-=- > time $OSM2PGSQL --append -s -C 3000 -G --hstore -d gis -H $PGHOST -U \ > $PGUSER -r xml changes.osc \ > --flat-nodes /database/postgresql/OSM-FLATNODES --slim \ > --number-processes 4 --style openstreetmap-carto.style \ > --tag-transform-script openstreetmap-carto.lua -e19 -o \ > $WORKDIR_OSM/expired-tiles Just to ensure that all important test conditions are known. Do you mean the weekly changesets dump from https://planet.openstreetmap.org/planet/changesets-latest.osm.bz2? Or how did you create changes.osc or where did you download it from? changesets-latest.osm.bz contains changeset metadata only. Btw, "-H $PGHOST" forces connection via TCP instead of Unix sockets. If you can use Unix sockets, -H 127.0.0.1 is not required. Best regards Michael -- Michael Reichert www.geofabrik.de Geofabrik GmbHHandelsregister: HRB Mannheim 703657 Amalienstr. 44Geschaeftsfuehrung: C. Karch, F. Ramm 76133 Karlsruhe Tel: 0721-1803560-3 reich...@geofabrik.de Fax: 0721-1803560-9 signature.asc Description: OpenPGP digital signature ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql tile expiry performance
Hi, Am 30.05.19 um 00:09 schrieb j: > I note that initial testing of this code used an extract of Europe: > https://github.com/openstreetmap/osm2pgsql/pull/747 > > I did not note it earlier, but I am working with the planet. I doubt > these issues would present themselves on a modestly sized machine > working with Europe only. The tests were done with Europe only which is roughly 2/3 of the whole planet. The main difference to your test is that my test applied the diff of a single day from download.geofabrik.de. Diffs from download.geofabrik.de are a pure diff of the old and the new file. I will have a deeper look into this but my test machine is currently busy doing other work. Best regards Michael -- Michael Reichert www.geofabrik.de Geofabrik GmbHHandelsregister: HRB Mannheim 703657 Amalienstr. 44Geschaeftsfuehrung: C. Karch, F. Ramm 76133 Karlsruhe Tel: 0721-1803560-3 reich...@geofabrik.de Fax: 0721-1803560-9 signature.asc Description: OpenPGP digital signature ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql tile expiry performance
j wrote: > I'm afraid I do not have precise timings, but I'm seeing what appears > to be at least an order of magnitude performance penalty, probably due > to memory exhaustion. I can not confirm this. I have running the new tile Expiry code at least since December 2017. As I am about to setup two machines for this purpose right now I reworked the scritps a bit. They are available here: https://github.com/giggls/tileserver-scripts Regards Sven -- Um Kontrolle Ihres Kontos wiederzugewinnen, klicken Sie bitte auf das Verbindungsgebrüll. (aus einer Ebay fishing Mail) /me is giggls@ircnet, http://sven.gegg.us/ on the Web ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
