[OSM-dev] Slow osmosis import

2019-10-04 Thread merspieler
I'm currently trying to import the planet.osm.pbf file with osmosis. While it's quite fast with nodes (took about 7h) it massively lows down when it comes to the ways. INFO: Processing Node 6814667967, 307145.5708858228 objects/second. Oct 04, 2019 9:13:37 AM org.openstreetmap.osmosis.core.progres

Re: [OSM-dev] Slow osmosis import

2019-10-04 Thread Frederik Ramm
Hi, first question: are you absolutely sure you need an Osmosis import - does your use case not work with an osm2pgsql import? Best Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ dev mailing list dev@openstre

Re: [OSM-dev] Slow osmosis import

2019-10-04 Thread Yves
Same as Frederic, but also proposing Imposm, also quite fast. A brief hardware description would allow to exclude some bottlenecks. Yves ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Slow osmosis import

2019-10-04 Thread merspieler
I've wanted to use osm2pgsql but the schema is a different one. The software [1] I'm going to use the db with only supports the osmosis one. As for the hardware: 2x Xeon E5 8 cores/16 threads 96GB ram 5x 4TB HDD in a RAIDZ2 I've done some benchmarking of the raid and osmosis doesn't even reach 5%

Re: [OSM-dev] Slow osmosis import

2019-10-04 Thread Yves
No, Imposm as it's own schema. I never used Osmosis to import a complete planet file, but I would find reasonable to start with a small extract like stated in osm2city documentation. Yves___ dev mailing list dev@openstreetmap.org https://lists.openstreet

Re: [OSM-dev] Slow osmosis import

2019-10-04 Thread merspieler
I've imported small extracts in the past but I've never actually monitored the performance of these as they were done in reasonable time. I'll try a smaller area again... Yves: > No, Imposm as it's own schema. > I never used Osmosis to import a complete planet file, but I would find > reasonable

Re: [OSM-dev] Slow osmosis import

2019-10-04 Thread merspieler
The same problem applies to a 3.4 GB .pbf file. The nodes were done quickly but as soon as it started processing the ways, it got super slow. merspieler: > I've imported small extracts in the past but I've never actually > monitored the performance of these as they were done in reasonable time. >