Note: This is a copy of a former post that did not make it on the mailinglist.
Hi, On http://wiki.openstreetmap.org/wiki/Nominatim/Installation it says index creation "will take a very long time - up to 10 days for a full planet on a small machine.". This is probably information added at 10th September 2010 and in the meantime, the plantefile has grown a bit, but: Currently, I am indexing a planet file (planet-110126.osm.pbf) for use with Nominatim and indexing has now nearly run for 800hs (that's more than a month). It’s a 8 core machine > cat /proc/cpuinfo ... model name : Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz stepping : 5 cpu MHz : 2668.000 cache size : 8192 KB ... The 8 cores are completely busy all the time (predominantly usr, so CPU is the bottleneck here) and there is plenty of free RAM. There are huge reads (from 1,5 - 62 MB/s with an average of ~16MB/s) and from time to time small writes (20 - 8000 KB with an average of ~100 KB, but most of the time there are no writes. Here is an example: > dstat ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system-- usr sys idl wai hiq siq| read writ| recv send| in out | int csw 65 0 31 3 0 0|6108k 939k| 0 0 | 14k 5372B|1510 4578 87 0 11 2 0 0| 22M 0 | 60B 842B| 0 0 |2206 7190 94 0 5 0 0 0| 33M 0 | 106B 400B| 0 0 |2398 7069 93 1 5 1 0 0| 32M 0 | 60B 346B| 0 0 |2229 20k 64 0 31 5 0 0| 11M 56k| 60B 346B| 0 0 |2235 6518 92 0 7 2 0 0| 744k 0 | 60B 362B| 0 0 |1923 3812 87 0 11 2 0 0|3244k 0 | 60B 346B| 0 0 |2085 3997 86 0 12 2 0 0|2236k 0 | 60B 346B| 0 0 |2056 3983 81 1 16 2 0 0| 21M 0 | 60B 346B| 0 0 |2244 5375 83 0 15 2 0 0| 31M 0 | 60B 346B| 0 0 |2454 7021 79 1 19 2 0 0| 32M 0 | 60B 346B| 0 0 |2478 11k 83 1 14 1 0 0| 49M 0 | 60B 346B| 0 0 |2607 13k 76 1 21 2 0 0| 45M 0 | 60B 346B| 0 0 |2487 15k 77 1 20 2 0 0| 30M 0 | 60B 346B| 0 0 |2356 20k 84 1 14 1 0 0| 28M 0 | 60B 346B| 0 0 |2332 11k 81 1 17 1 0 0| 37M 0 | 60B 346B| 0 0 |2314 11k 88 0 10 1 0 0| 27M 0 | 60B 346B| 0 0 |2304 5928 95 1 4 1 0 0| 31M 0 | 60B 346B| 0 0 |2362 5464 86 0 12 1 0 0| 34M 0 | 60B 346B| 0 0 |2346 6887 70 0 26 4 0 0| 968k 32k| 60B 346B| 0 0 |1957 4238 67 0 29 4 0 0|7892k 0 | 60B 362B| 0 0 |2177 10k 58 1 36 5 0 0|4836k 0 | 60B 346B| 0 0 |2317 12k 34 0 57 9 0 0|8752k 256k| 60B 346B| 0 0 |2225 9076 82 1 16 1 0 0| 39M 0 | 60B 362B| 0 0 |2366 14k 83 1 14 2 0 0| 42M 0 | 60B 346B| 0 0 |2487 14k 96 0 3 0 0 0| 25M 0 | 60B 346B| 0 0 |2216 4317 91 0 7 2 0 0| 12M 56k| 60B 346B| 0 0 |2003 3668 94 1 4 1 0 0| 17M 0 | 60B 362B| 0 0 |2048 7270 98 0 2 0 0 0| 12M 0 | 60B 346B| 0 0 |1974 3316 95 0 4 0 0 0| 30M 0 | 60B 346B| 0 0 |2104 11k 92 1 7 1 0 0| 18M 0 | 60B 346B| 0 0 |2107 5712 88 1 10 1 0 0| 25M 0 | 60B 346B| 0 0 |2275 8254 91 0 8 1 0 0| 13M 40k| 60B 346B| 0 0 |2090 6409 96 0 3 0 0 0| 28M 0 | 60B 362B| 0 0 |2342 5899 97 1 1 0 0 0| 28M 0 | 60B 346B| 0 0 |2337 3895 92 1 6 1 0 0| 25M 0 | 106B 400B| 0 0 |2200 9122 84 1 14 2 0 0| 29M 0 | 60B 346B| 0 0 |2129 12k 94 1 4 1 0 0| 29M 0 | 60B 346B| 0 0 |2363 10k 96 1 3 0 0 0| 20M 0 | 60B 346B| 0 0 |2117 11k ... I think, it should run faster and wonder whether this is somehow related to Postgres 9.0 and the different GIN creation strategy, cf. http://gis.638310.n2.nabble.com/Patch-to-osm2pgsql-for-faster-updates-tp5952996p5952996.html So I am interested to hear, what experiences others made in this step and additionally, whether there is a way to estimate the "ttf" (time to finish) of that process (maybe index sizes?). Thanks fatzopilot -- View this message in context: http://gis.638310.n2.nabble.com/Superslow-nominatim-index-creation-with-Postgres-9-0-tp6180717p6210768.html Sent from the Developer Discussion mailing list archive at Nabble.com. _______________________________________________ dev mailing list [email protected] http://lists.openstreetmap.org/listinfo/dev
