On 07/25/2011 02:46 PM, Tim Alder wrote: > It's possible to read the reason why expiring was broken in: > https://jira.toolserver.org/browse/TS-971 (march of this year) > > The reason why we don't restart expiring was performance problems, we > had a lot of queries that running longer than 20 minutes for > low-zoomlevels renderings (time-out) and the Queue seems not stable. (We > were too slow.)
I don't know what the parameters for the expiry process were, but the low zoom levels shouldn't be automatically expired. Z12 or Z11 seem like a good place to cut off the automatic expiry. This way, the long running low-zoom tiles don't matter. These can simply periodically be added via tirex-batch to the background rendering queue. Does it matter if the rendering queue is not stable? It will eventually asymptote. At the very latest when all tiles are in the queue. If a long queue itself causes performance problems (which I don't think it does) then one could set a maximum queue size (like renderd does) after which requests get dropped. This way more frequently visited tiles have a hire chance of making it into the queue, thus adding a certain amount of "priority queue" towards the most frequently visited tiles. It will still result in a faster rerendering schedule than no expiry at all. > > Now (with the clustering) it seems better and the Queue is going done > today. Tiles on z=9 needs very long (>600s). But it seems acceptable. It does seem to be getting through the queue in a reasonably rapid pace, which perhaps indicates that the expiry won't be to bad. With the tile storage now on a separate set of disks, the actual touching of meta-tiles for expiry will also no longer interfere with db access for the rendering process. I presume you clustered on the geometry index? (I am asking as it is not entirely clear from https://jira.toolserver.org/browse/TS-1122). However, there still seems to be a factor of 5 - 10 times more disk operations per rendered tile (ignoring non tirex related db ops) than on osm's tile server and I suspect figuring out why is key to solving the performance issues. > --- > We have now indexes for geometry,hstore and osm-id. If somebody needs > index for name or ref -> please give me a signal. > > It seems that we can restart the db updating tomorrow. If it is not too much trouble, it might be good to update to the latest osm2pgsql. I added a potential optimization a week or so ago, that may help performance in the case of slow disks and low node cache hit ratio (which the update process often is). May I also suggest to not do a minutely update. If the tiles are only expired once every 8 month or so, it makes little sense to keep the db minutely up to date. Hourly, or perhaps even only daily, would probably help performance both with the expiry process, the rendering, as well as the db update process. Kai > ---- > So or so I will try to optimize the rendering strategy from zoom-level > 9-15. see talk-de: > http://lists.openstreetmap.org/pipermail/talk-de/2011-July/087862.html > > Greetings Tim alias Kolossos > > > > > Am 25.07.2011 17:20, schrieb Kay Drangmeister: >> Hi River, >> >> Am 25.07.2011, 16:54 Uhr, schrieb River Tarnell<[email protected]>: >> >>> Is there a particular reason why expiry was disabled? The load seems >>> fine at the moment, even though quite a few tiles are being rendered. >> >> I don't know either who disabled it or why it has been disabled. >> Probably it had been deactivated in favour of another process... >> >> Kay >> >> _______________________________________________ >> Maps-l mailing list >> [email protected] >> https://lists.wikimedia.org/mailman/listinfo/maps-l >> > > > _______________________________________________ > Maps-l mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/maps-l _______________________________________________ Maps-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/maps-l
