Re: [OSM-dev] Nominatim Index bloat? Try pgindexrebuild, a production friendly index debloater

2016-08-19 Thread Paul Norman

On 8/19/2016 4:30 AM, Rory McCann wrote:

Although I'm using this often, I'm always open to suggestions about
braindead things I might be doing. ☺ Suggestions welcome!


It's good to see commands for a concurrent reindex scripted.

For rendering tables I'd recommend pausing updates, creating a new 
ordered table, building indexes on the new table, then putting the new 
table in place. The reclustering adds performance beyond what a reindex 
does. 
http://paulnorman.ca/blog/2016/06/improving-speed-with-reclustering/ has 
example SQL.


For slim tables, I'd probably just pause updates and REINDEX.

http://reorg.github.io/pg_repack/ would be interesting on the slim 
tables but can't operate on the rendering tables.




___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


[OSM-dev] Nominatim Index bloat? Try pgindexrebuild, a production friendly index debloater

2016-08-19 Thread Rory McCann
Hello all,

If you use/update Nominatim (or any PostgreSQL) server for a long time,
one problem you may come across is index bloat. Your database will grow
in size on your disk, your indexes getting larger on disk. It's no fun
when your database fills up the disk and won't start anymore.

There are several solutions to this, but I wanted a way to fix index
bloat while still being able to use the database as a production
nominatim (or tile) database, so I wrote pgindexrebuild:

https://github.com/rory/pgindexrebuild

It uses PostgreSQL's CREATE INDEX CONCURRENTLY to create a new index and
replace the old bloated one afterwards. The CONCURRENTLY ensure that you
can still read and write from the table while it is creating a new
index. It has some nice things like lock files, bloat thresholds, and
logging, to allow you to put it in cron and forget about it. "Set up and
forget" is a design goal.

Although I'm using this often, I'm always open to suggestions about
braindead things I might be doing. ☺ Suggestions welcome!

This is inspired by [pgtoolkit](https://github.com/grayhemp/pgtoolkit)
which does the same thing for table bloat.

-- 



signature.asc
Description: OpenPGP digital signature
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev