Re: [HACKERS] Autovacuum fails to keep visibility map up-to-date in mostly-insert-only-tables

2014-10-29 Thread Robert Haas
On Mon, Oct 27, 2014 at 5:51 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Jeff Janes wrote: It is only a page read if you have to read the page. It would seem optimal to have bgwriter adventitiously set hint bits and vm bits, because that is the last point at which the page can be

Re: [HACKERS] Autovacuum fails to keep visibility map up-to-date in mostly-insert-only-tables

2014-10-28 Thread Alvaro Herrera
Jeff Janes wrote: It is only a page read if you have to read the page. It would seem optimal to have bgwriter adventitiously set hint bits and vm bits, because that is the last point at which the page can be changed without risking that it be written out twice. At that point, it has been

Re: [HACKERS] Autovacuum fails to keep visibility map up-to-date in mostly-insert-only-tables

2014-10-21 Thread Jeff Janes
On Mon, Oct 20, 2014 at 5:46 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-10-20 17:43:26 -0700, Josh Berkus wrote: On 10/20/2014 05:39 PM, Jim Nasby wrote: Or maybe vacuum isn't the right way to handle some of these scenarios. It's become the catch-all for all of this stuff,

Re: [HACKERS] Autovacuum fails to keep visibility map up-to-date in mostly-insert-only-tables

2014-10-21 Thread Jim Nasby
On 10/21/14, 4:36 PM, Jeff Janes wrote: On Mon, Oct 20, 2014 at 5:46 PM, Andres Freund and...@2ndquadrant.com mailto:and...@2ndquadrant.com wrote: On 2014-10-20 17:43:26 -0700, Josh Berkus wrote: On 10/20/2014 05:39 PM, Jim Nasby wrote: Or maybe vacuum isn't the right way to

Re: [HACKERS] Autovacuum fails to keep visibility map up-to-date in mostly-insert-only-tables

2014-10-20 Thread Andres Freund
On 2014-10-19 20:43:29 -0500, Jim Nasby wrote: On 10/19/14, 11:41 AM, Andres Freund wrote: On 2014-10-18 21:36:48 -0500, Jim Nasby wrote: The weird part is that if it's not doing a freeze it will just punt on a page if it can't get the cleanup lock. I don't think that's particularly wierd.

Re: [HACKERS] Autovacuum fails to keep visibility map up-to-date in mostly-insert-only-tables

2014-10-20 Thread Jim Nasby
On 10/20/14, 3:11 PM, Andres Freund wrote: On 2014-10-19 20:43:29 -0500, Jim Nasby wrote: On 10/19/14, 11:41 AM, Andres Freund wrote: On 2014-10-18 21:36:48 -0500, Jim Nasby wrote: The weird part is that if it's not doing a freeze it will just punt on a page if it can't get the cleanup lock.

Re: [HACKERS] Autovacuum fails to keep visibility map up-to-date in mostly-insert-only-tables

2014-10-20 Thread Josh Berkus
On 10/20/2014 05:39 PM, Jim Nasby wrote: Or maybe vacuum isn't the right way to handle some of these scenarios. It's become the catch-all for all of this stuff, but maybe that doesn't make sense anymore. Certainly when it comes to dealing with inserts there's no reason we *have* to do anything

Re: [HACKERS] Autovacuum fails to keep visibility map up-to-date in mostly-insert-only-tables

2014-10-20 Thread Andres Freund
On 2014-10-20 17:43:26 -0700, Josh Berkus wrote: On 10/20/2014 05:39 PM, Jim Nasby wrote: Or maybe vacuum isn't the right way to handle some of these scenarios. It's become the catch-all for all of this stuff, but maybe that doesn't make sense anymore. Certainly when it comes to dealing

Re: [HACKERS] Autovacuum fails to keep visibility map up-to-date in mostly-insert-only-tables

2014-10-19 Thread Andres Freund
On 2014-10-18 21:36:48 -0500, Jim Nasby wrote: On 10/9/14, 4:19 PM, Andres Freund wrote: On 2014-10-09 18:16:46 -0300, Alvaro Herrera wrote: Andres Freund wrote: On 2014-10-09 18:03:00 -0300, Alvaro Herrera wrote: Bruce Momjian wrote: I agree this is a serious problem. We have

Re: [HACKERS] Autovacuum fails to keep visibility map up-to-date in mostly-insert-only-tables

2014-10-19 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: There's rub here though. We unconditionally do: /* Do post-vacuum cleanup and statistics update for each index */ for (i = 0; i nindexes; i++) lazy_cleanup_index(Irel[i], indstats[i], vacrelstats); and that's not

Re: [HACKERS] Autovacuum fails to keep visibility map up-to-date in mostly-insert-only-tables

2014-10-19 Thread Andres Freund
On 2014-10-19 12:50:30 -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: There's rub here though. We unconditionally do: /* Do post-vacuum cleanup and statistics update for each index */ for (i = 0; i nindexes; i++) lazy_cleanup_index(Irel[i],

Re: [HACKERS] Autovacuum fails to keep visibility map up-to-date in mostly-insert-only-tables

2014-10-19 Thread Jim Nasby
On 10/19/14, 11:41 AM, Andres Freund wrote: On 2014-10-18 21:36:48 -0500, Jim Nasby wrote: On 10/9/14, 4:19 PM, Andres Freund wrote: On 2014-10-09 18:16:46 -0300, Alvaro Herrera wrote: Andres Freund wrote: On 2014-10-09 18:03:00 -0300, Alvaro Herrera wrote: Bruce Momjian wrote: I agree

Re: [HACKERS] Autovacuum fails to keep visibility map up-to-date in mostly-insert-only-tables

2014-10-18 Thread Jim Nasby
On 10/9/14, 4:19 PM, Andres Freund wrote: On 2014-10-09 18:16:46 -0300, Alvaro Herrera wrote: Andres Freund wrote: On 2014-10-09 18:03:00 -0300, Alvaro Herrera wrote: Bruce Momjian wrote: I agree this is a serious problem. We have discussed various options, but have not decided

[HACKERS] Autovacuum fails to keep visibility map up-to-date in mostly-insert-only-tables

2014-10-09 Thread Alexey Bashtanov
Hello! Autovacuum daemon performs vacuum when the number of rows updated/deleted (n_dead_tuples) reaches some threshold. Similarly it performs analyze when the number of rows changed in any way (incl. inserted). When a table is mostly insert-only, its visibility map is not updated as vacuum

Re: [HACKERS] Autovacuum fails to keep visibility map up-to-date in mostly-insert-only-tables

2014-10-09 Thread Bruce Momjian
On Thu, Oct 9, 2014 at 02:34:17PM +0400, Alexey Bashtanov wrote: Hello! Autovacuum daemon performs vacuum when the number of rows updated/deleted (n_dead_tuples) reaches some threshold. Similarly it performs analyze when the number of rows changed in any way (incl. inserted). When a table

Re: [HACKERS] Autovacuum fails to keep visibility map up-to-date in mostly-insert-only-tables

2014-10-09 Thread Alvaro Herrera
Bruce Momjian wrote: I agree this is a serious problem. We have discussed various options, but have not decided on anything. The TODO list has: https://wiki.postgresql.org/wiki/Todo Improve setting of visibility map bits for read-only and insert-only workloads

Re: [HACKERS] Autovacuum fails to keep visibility map up-to-date in mostly-insert-only-tables

2014-10-09 Thread Andres Freund
On 2014-10-09 18:03:00 -0300, Alvaro Herrera wrote: Bruce Momjian wrote: I agree this is a serious problem. We have discussed various options, but have not decided on anything. The TODO list has: https://wiki.postgresql.org/wiki/Todo Improve setting of visibility map

Re: [HACKERS] Autovacuum fails to keep visibility map up-to-date in mostly-insert-only-tables

2014-10-09 Thread Kevin Grittner
Alvaro Herrera alvhe...@2ndquadrant.com wrote: Bruce Momjian wrote: I agree this is a serious problem. We have discussed various options, but have not decided on anything. The TODO list has: https://wiki.postgresql.org/wiki/Todo Improve setting of visibility map bits for

Re: [HACKERS] Autovacuum fails to keep visibility map up-to-date in mostly-insert-only-tables

2014-10-09 Thread Alvaro Herrera
Andres Freund wrote: On 2014-10-09 18:03:00 -0300, Alvaro Herrera wrote: Bruce Momjian wrote: I agree this is a serious problem. We have discussed various options, but have not decided on anything. The TODO list has: https://wiki.postgresql.org/wiki/Todo Improve

Re: [HACKERS] Autovacuum fails to keep visibility map up-to-date in mostly-insert-only-tables

2014-10-09 Thread Alvaro Herrera
Kevin Grittner wrote: Wouldn't we get substantially the same thing just by counting tuple inserts toward the autovacuum vacuum threshold? I mean, it unless the table is due for wraparound prevention autovacuum, it will only visit pages that don't have the all-visible bit set, right? And

Re: [HACKERS] Autovacuum fails to keep visibility map up-to-date in mostly-insert-only-tables

2014-10-09 Thread Andres Freund
On 2014-10-09 18:16:46 -0300, Alvaro Herrera wrote: Andres Freund wrote: On 2014-10-09 18:03:00 -0300, Alvaro Herrera wrote: Bruce Momjian wrote: I agree this is a serious problem. We have discussed various options, but have not decided on anything. The TODO list has:

Re: [HACKERS] Autovacuum fails to keep visibility map up-to-date in mostly-insert-only-tables

2014-10-09 Thread Jim Nasby
On 10/9/14, 4:03 PM, Alvaro Herrera wrote: Bruce Momjian wrote: I agree this is a serious problem. We have discussed various options, but have not decided on anything. The TODO list has: https://wiki.postgresql.org/wiki/Todo Improve setting of visibility map bits for