On 29 April 2013 23:34, Sarah Hoffmann <[email protected]> wrote: > On Mon, Apr 29, 2013 at 10:20:55PM +0100, Simon Nuttall wrote: >> On 29 April 2013 08:30, Sarah Hoffmann <[email protected]> wrote: >> > Hi Simon, >> > >> > On Mon, Apr 29, 2013 at 03:39:37AM +0100, Simon Nuttall wrote: >> >> Using our installation script mentioned at: >> >> http://lists.openstreetmap.org/pipermail/geocoding/2013-April/000779.html >> >> >> >> I started a Europe wide import on a 4 CPU processor machine with 16GB RAM. >> >> >> >> That was 13 days ago, on 16 April, and by 21 April the log file said: >> >> >> >> GRANT SELECT ON place_classtype_natural_islet TO "www-data";drop index >> >> idx_placex_classtype;# Done special phrases Sun Apr 21 20:29:08 BST >> >> 2013 >> >> # Nominatim website created Sun Apr 21 20:29:09 BST 2013 >> > >> > It looks like your initial import has finished. After this line >> > your script seems to start the update process directly, about here: >> > https://github.com/cyclestreets/nominatim-install/blob/master/run.sh#L237 >> > >> > This line actually runs the update process in an endless loop, so your >> > script will never finish and is happily keeping the database up to date >> > now. But the database should already be ready to be used. >> >> Yeah I did try using that for a while: >> >> http://nominatim.cyclestreets.net/ >> >> but because the script was still running and was I/O bound the >> performance was slow. So I decided to stop and wait until the script >> finished. > > It's mostly osm2pgsql that causes responsiveless to decrese. It helps a > lot to keep the updates small. Basically, I wouldn't recommend to import > updates of more than an hour at a time, even when the machine is offline and > just catching up. The osm instance is currently set to half hour chunks > which seems to work well even when it is under heavy load. > > You seem to have set your system to arbitrarily large chunks. Not sure > how that happend. You'll find the setting in settings/configuration.txt > under 'maxInterval'.
Here is that file: # The URL of the directory containing change files. baseUrl=http://planet.openstreetmap.org/replication/minute # Defines the maximum time interval in seconds to download in a single invocation. # Setting to 0 disables this feature. maxInterval = 3600 > >> >> Processing: Node(307016k 2.2k/s) Way(39382k 0.09k/s) Relation(118300 >> >> 14.95/s) >> > >> > Now that is odd. Normally, an update step just imports a few k of data >> > at most. Here you seem to reimport almost the entire planet again. It looks >> > like the state file was not set up correctly and you have been applying >> > updates of the last half year. >> > >> > It's hard to say how long the update step above will continue. It is >> > probably >> > almost done. So you might just let it run for a bit more but be aware that >> > the osm2pgsql processing you see above is followed by indexing which will >> > again take quite a bit of time. >> > >> > It might work to just cancel the process and let the DB roll back. If you >> > do that, check afterwards if all rows in the placex table that have >> > indexed_status = 0. If that is the case, you should be savely back to the >> > state of your initial import. Then you can try to figure out what the >> > problem with the state file is (settings/state.txt), get the correct one >> > and restart your updates. >> >> I've let it carry on running, and you're right it seems like its doing >> the whole import again. Currently I'm seeing lines like: >> >> Done 1118475 in 44519 @ 25.123543 per second - Rank 26 ETA >> (seconds): 152234.937500 >> Done 1118482 in 44519 @ 25.123699 per second - Rank 26 ETA >> (seconds): 152233.718750 >> Done 1118489 in 44520 @ 25.123293 per second - Rank 26 ETA >> (seconds): 152235.890625 >> >> The settings/state.txt file reads: >> >> #Mon Apr 22 07:40:54 BST 2013 >> sequenceNumber=247388 >> timestamp=2013-03-03T20\:59\:02Z > > As far as I know, osmosis should have updated that by now which would mean > that you have imported around 4 months worth of updates and are still only > in March. Hard to say what went wrong without knowing what exactly was in > the state file before the update started. You'll probably be faster with a > reimport at this point. OK, well I could cancel it as our new machine will come available in a couple of days. > >> > NB: geofabrik has now dayly diffs for excerpts which will work fine with >> > Nominatim. Maybe that is of interest for you. You need to get the latest >> > Nominatim version from github, though, and you probably need to reimport >> > because the DB schema has changed a little bit. >> >> A new machine has become available to us and I'll try the installing >> the latest version on that. We have the luxury of choosing anywhere >> between 16 GB and 64 GB RAM for the Nominatim VM although I'd like to >> choose at most 32GB to allow more memory for our other VMs on that >> host. >> >> Does it make a lot of difference? Would 32GB be enough to process the >> whole planet or should I just stick to Europe? > > 32GB should be ok for a planet. The more limiting factor is I/O. > If you could add an SSD to the machine that would give you a huge > performance boost, especially when running minutely updates in the background. Noted, thanks. _______________________________________________ Geocoding mailing list [email protected] http://lists.openstreetmap.org/listinfo/geocoding
