On Thu 17th Sep 2015 at 10:51 +0100 Badita Florin <baditaflorin at gmail.com> wrote:
>Now, to import the planet it will take around 6-20 days with osmosis >Other tools behave in the same way ... >What i think it is missing, is a monthly postgis dump of the planet. Or weekly, whatever >To have loaded in postgis the whole planet, and once a week/day/hour/minute ( what works best ) to apply just the changefiles >The dump to be as concise as possible, having just the bare necessities, without the index, etc >In this way, people that are interested in doing some statistics out of OSM can >do it in a simple way, without spending 1-2 weeks only downloading the 29 >osm.pbf file, waiting 6 days for osmosis to load the file, if it will not crush, , >having 1500 GB to do that, just to revert back to 200-700 GB >I cannot say how much it will take the database, because i was never able to >load the entire planet, all the time i had to revert to filtering, clipping, calculate >for each separate continent, amenity or type of POI, and then combine it all >back. >A dump of a planet file would make the osm data more accessible to scientist, .>researchers and the general osm community >I would be able to try and find a server where the planet file would lay , with the >osmosis script that would update the script and the psql script that will generate >once per month a dump file for the whole planet, if somebody can help me . >create such a setup, if possible >Have a good day, >Florin Badita Does it really take this long to import a planet file these days? I would be interested to know if anyone else has imported one recently. I may import one myself at some point, but if it takes 6 days I probably won't bother. > > On a side note, this is for a piece of work looking at using big data tools to look at using OSM data. More info is available here https://lists.openstreetmap.org/pipermail/dev/2015-January/028227.html. Just as a comparative benchmark I was able to import Europe (roughly half the size of the whole DB) into a suitable spatial form in around 4 hours using 8 nodes with a combined 104GB of RAM (OK, I know that's a lot but this is Big Data!). Indexing takes a little longer and I'm not sure the querying is as quick as PostGIS but it certainly won't take you days to get up and running. I'll write up some more about this in due course. Stephen
_______________________________________________ talk mailing list [email protected] https://lists.openstreetmap.org/listinfo/talk
