Re: [HACKERS] Freeze avoidance of very large table.

2015-11-14 Thread Amit Kapila
On Sat, Nov 14, 2015 at 1:19 AM, Andres Freund wrote: > On 2015-10-31 11:02:12 +0530, Amit Kapila wrote: > > On Thu, Oct 8, 2015 at 11:05 PM, Simon Riggs wrote: > > > > > > On 1 October 2015 at 23:30, Josh Berkus wrote: > > >> > > >> On 10/01/2015 07:43 AM, Robert Haas wrote: > > >> > On Thu, Oc

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-14 Thread Amit Kapila
On Sat, Nov 14, 2015 at 1:12 AM, Bruce Momjian wrote: > > On Tue, Nov 3, 2015 at 09:03:49AM +0530, Amit Kapila wrote: > > I think in that case we can call it as page info map or page state map, but > > I find retaining visibility map name in this case or for future (if we want to > > add another

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-13 Thread Andres Freund
On 2015-10-31 11:02:12 +0530, Amit Kapila wrote: > On Thu, Oct 8, 2015 at 11:05 PM, Simon Riggs wrote: > > > > On 1 October 2015 at 23:30, Josh Berkus wrote: > >> > >> On 10/01/2015 07:43 AM, Robert Haas wrote: > >> > On Thu, Oct 1, 2015 at 9:44 AM, Fujii Masao > wrote: > >> >> I wonder how much

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-13 Thread Bruce Momjian
On Tue, Nov 3, 2015 at 09:03:49AM +0530, Amit Kapila wrote: > I think in that case we can call it as page info map or page state map, but > I find retaining visibility map name in this case or for future (if we want to > add another bit) as confusing.  In-fact if you find "visibility and freeze >

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-12 Thread Amit Kapila
On Fri, Nov 13, 2015 at 4:48 AM, Masahiko Sawada wrote: > > > Thank you for reviewing the patch. > > I changed the patch so that the visibility map become the page info > map, in source code and documentation. > One thing to notice is that this almost doubles the patch size which might makes it s

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-05 Thread Masahiko Sawada
On Wed, Nov 4, 2015 at 12:19 PM, Amit Kapila wrote: > On Wed, Nov 4, 2015 at 4:45 AM, Masahiko Sawada > wrote: >> >> On Tue, Nov 3, 2015 at 12:33 PM, Amit Kapila >> wrote: >> > On Tue, Nov 3, 2015 at 5:04 AM, Robert Haas >> > wrote: >> >> >> >> On Sat, Oct 31, 2015 at 1:32 AM, Amit Kapila >> >

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-05 Thread Kyotaro HORIGUCHI
Hello, I had a look on v21 patch. Though I haven't looked the whole of the patch, I'd like to show you some comments only for visibilitymap.c and a part of the documentation. 1. Patch application patch command complains about offsets for heapam.c at current master. 2. visitibilymap_test(

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-03 Thread Amit Kapila
On Wed, Nov 4, 2015 at 4:45 AM, Masahiko Sawada wrote: > > On Tue, Nov 3, 2015 at 12:33 PM, Amit Kapila wrote: > > On Tue, Nov 3, 2015 at 5:04 AM, Robert Haas wrote: > >> > >> On Sat, Oct 31, 2015 at 1:32 AM, Amit Kapila > >> wrote: > >> > > >> > What is your main worry about changing the name

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-03 Thread Masahiko Sawada
On Tue, Nov 3, 2015 at 12:33 PM, Amit Kapila wrote: > On Tue, Nov 3, 2015 at 5:04 AM, Robert Haas wrote: >> >> On Sat, Oct 31, 2015 at 1:32 AM, Amit Kapila >> wrote: >> > >> > What is your main worry about changing the name of this map, is it >> > about more code churn or is it about that we mig

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-03 Thread Robert Haas
On Mon, Nov 2, 2015 at 10:33 PM, Amit Kapila wrote: > On Tue, Nov 3, 2015 at 5:04 AM, Robert Haas wrote: >> >> On Sat, Oct 31, 2015 at 1:32 AM, Amit Kapila >> wrote: >> > >> > What is your main worry about changing the name of this map, is it >> > about more code churn or is it about that we mig

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-02 Thread Amit Kapila
On Tue, Nov 3, 2015 at 5:04 AM, Robert Haas wrote: > > On Sat, Oct 31, 2015 at 1:32 AM, Amit Kapila wrote: > > > > What is your main worry about changing the name of this map, is it > > about more code churn or is it about that we might introduce new issues > > or is it about that people are alre

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-02 Thread Robert Haas
On Sat, Oct 31, 2015 at 1:32 AM, Amit Kapila wrote: > On Thu, Oct 8, 2015 at 11:05 PM, Simon Riggs wrote: >> >> On 1 October 2015 at 23:30, Josh Berkus wrote: >>> >>> On 10/01/2015 07:43 AM, Robert Haas wrote: >>> > On Thu, Oct 1, 2015 at 9:44 AM, Fujii Masao >>> > wrote: >>> >> I wonder how mu

Re: [HACKERS] Freeze avoidance of very large table.

2015-10-31 Thread Amit Kapila
On Fri, Oct 30, 2015 at 6:03 AM, Masahiko Sawada wrote: > > > > v20 patch has a bug in result of regression test. > Attached updated v21 patch. > Couple of more review comments: -- 1. @@ -615,6 +617,8 @@ typedef struct PgStat_StatTabEntry PgS

Re: [HACKERS] Freeze avoidance of very large table.

2015-10-30 Thread Amit Kapila
On Thu, Oct 8, 2015 at 11:05 PM, Simon Riggs wrote: > > On 1 October 2015 at 23:30, Josh Berkus wrote: >> >> On 10/01/2015 07:43 AM, Robert Haas wrote: >> > On Thu, Oct 1, 2015 at 9:44 AM, Fujii Masao wrote: >> >> I wonder how much it's worth renaming only the file extension while >> >> there ar

Re: [HACKERS] Freeze avoidance of very large table.

2015-10-29 Thread Masahiko Sawada
On Fri, Oct 30, 2015 at 1:26 AM, Masahiko Sawada wrote: > On Wed, Oct 28, 2015 at 12:58 PM, Amit Kapila wrote: >> On Sat, Oct 24, 2015 at 2:24 PM, Masahiko Sawada >> wrote: >>> >>> On Sat, Oct 24, 2015 at 10:59 AM, Amit Kapila >>> wrote: >>> > >>> > I think we can display information about rela

Re: [HACKERS] Freeze avoidance of very large table.

2015-10-29 Thread Masahiko Sawada
On Wed, Oct 28, 2015 at 12:58 PM, Amit Kapila wrote: > On Sat, Oct 24, 2015 at 2:24 PM, Masahiko Sawada > wrote: >> >> On Sat, Oct 24, 2015 at 10:59 AM, Amit Kapila >> wrote: >> > >> > I think we can display information about relallfrozen it in >> > pg_stat_*_tables >> > as suggested by you. It

Re: [HACKERS] Freeze avoidance of very large table.

2015-10-27 Thread Amit Kapila
On Sat, Oct 24, 2015 at 2:24 PM, Masahiko Sawada wrote: > > On Sat, Oct 24, 2015 at 10:59 AM, Amit Kapila wrote: > > > > I think we can display information about relallfrozen it in pg_stat_*_tables > > as suggested by you. It doesn't make much sense to keep it in pg_class > > unless we have some

Re: [HACKERS] Freeze avoidance of very large table.

2015-10-24 Thread Masahiko Sawada
On Sat, Oct 24, 2015 at 10:59 AM, Amit Kapila wrote: > On Mon, Oct 5, 2015 at 9:53 PM, Masahiko Sawada > wrote: >> >> On Mon, Oct 5, 2015 at 11:03 PM, Fujii Masao >> wrote: >> > On Fri, Oct 2, 2015 at 8:14 PM, Masahiko Sawada >> > wrote: >> >>> +#define Anum_pg_class_relallfrozen12 >> >

Re: [HACKERS] Freeze avoidance of very large table.

2015-10-23 Thread Amit Kapila
On Mon, Oct 5, 2015 at 9:53 PM, Masahiko Sawada wrote: > > On Mon, Oct 5, 2015 at 11:03 PM, Fujii Masao wrote: > > On Fri, Oct 2, 2015 at 8:14 PM, Masahiko Sawada wrote: > >>> +#define Anum_pg_class_relallfrozen12 > >>> Why is pg_class.relallfrozen necessary? ISTM that there is no user o

Re: [HACKERS] Freeze avoidance of very large table.

2015-10-22 Thread Masahiko Sawada
On Thu, Oct 22, 2015 at 4:11 PM, Torsten Zühlsdorff wrote: > On 21.10.2015 02:05, Masahiko Sawada wrote: >> >> On Sat, Oct 10, 2015 at 4:20 AM, Robert Haas >> wrote: >>> >>> On Thu, Oct 8, 2015 at 1:52 PM, Andres Freund wrote: I don't see the problem? I mean catversion will reliably te

Re: [HACKERS] Freeze avoidance of very large table.

2015-10-22 Thread Torsten Zühlsdorff
On 21.10.2015 02:05, Masahiko Sawada wrote: On Sat, Oct 10, 2015 at 4:20 AM, Robert Haas wrote: On Thu, Oct 8, 2015 at 1:52 PM, Andres Freund wrote: I don't see the problem? I mean catversion will reliably tell you which format the vm is in? Totally agreed. We could additionally use the

Re: [HACKERS] Freeze avoidance of very large table.

2015-10-21 Thread Alvaro Herrera
Jim Nasby wrote: > On 10/21/15 8:11 AM, Andres Freund wrote: > >Meh. Adding complexity definitely needs to be weighed against the > >benefits. As pointed out e.g. by all the multixact issues you mentioned > >upthread. In this case your argument for changing the name doesn't seem > >to hold much wat

Re: [HACKERS] Freeze avoidance of very large table.

2015-10-21 Thread Jim Nasby
On 10/21/15 8:11 AM, Andres Freund wrote: Meh. Adding complexity definitely needs to be weighed against the benefits. As pointed out e.g. by all the multixact issues you mentioned upthread. In this case your argument for changing the name doesn't seem to hold much water. ISTM VISIBILITY_MAP_FRO

Re: [HACKERS] Freeze avoidance of very large table.

2015-10-21 Thread Andres Freund
On 2015-10-20 20:35:31 -0400, Simon Riggs wrote: > On 9 October 2015 at 15:20, Robert Haas wrote: > > > On Thu, Oct 8, 2015 at 1:52 PM, Andres Freund wrote: > > > I don't see the problem? I mean catversion will reliably tell you which > > format the vm is in? > > > > Totally agreed. > > > > Thi

Re: [HACKERS] Freeze avoidance of very large table.

2015-10-20 Thread Simon Riggs
On 9 October 2015 at 15:20, Robert Haas wrote: > On Thu, Oct 8, 2015 at 1:52 PM, Andres Freund wrote: > > I don't see the problem? I mean catversion will reliably tell you which > format the vm is in? > > Totally agreed. > This isn't an agreement competition, its a cool look at what might cause

Re: [HACKERS] Freeze avoidance of very large table.

2015-10-20 Thread Masahiko Sawada
On Sat, Oct 10, 2015 at 4:20 AM, Robert Haas wrote: > On Thu, Oct 8, 2015 at 1:52 PM, Andres Freund wrote: >> I don't see the problem? I mean catversion will reliably tell you which >> format the vm is in? > > Totally agreed. > >> We could additionally use the opportunity to as a metapage, but t

Re: [HACKERS] Freeze avoidance of very large table.

2015-10-09 Thread Robert Haas
On Thu, Oct 8, 2015 at 1:52 PM, Andres Freund wrote: > I don't see the problem? I mean catversion will reliably tell you which > format the vm is in? Totally agreed. > We could additionally use the opportunity to as a metapage, but that seems > like an independent thing. I agree with that, to

Re: [HACKERS] Freeze avoidance of very large table.

2015-10-08 Thread Andres Freund
On October 8, 2015 7:35:24 PM GMT+02:00, Simon Riggs wrote: >The problem is we won't be able to tell the two formats apart, since >they >both are just lots of bits. So we won't be able to tell if the file is >old >format or new format, which could lead to loss of information that >relates >to vi

Re: [HACKERS] Freeze avoidance of very large table.

2015-10-08 Thread Simon Riggs
On 1 October 2015 at 23:30, Josh Berkus wrote: > On 10/01/2015 07:43 AM, Robert Haas wrote: > > On Thu, Oct 1, 2015 at 9:44 AM, Fujii Masao > wrote: > >> I wonder how much it's worth renaming only the file extension while > >> there are many places where "visibility map" and "vm" are used, > >>

Re: [HACKERS] Freeze avoidance of very large table.

2015-10-08 Thread Masahiko Sawada
On Thu, Oct 8, 2015 at 7:03 PM, Fujii Masao wrote: > On Mon, Oct 5, 2015 at 7:31 PM, Masahiko Sawada wrote: >> On Sat, Oct 3, 2015 at 3:41 AM, Robert Haas wrote: >>> On Fri, Oct 2, 2015 at 11:23 AM, Alvaro Herrera >>> wrote: > + /* all-frozen information is also cleared at the s

Re: [HACKERS] Freeze avoidance of very large table.

2015-10-08 Thread Fujii Masao
On Mon, Oct 5, 2015 at 7:31 PM, Masahiko Sawada wrote: > On Sat, Oct 3, 2015 at 3:41 AM, Robert Haas wrote: >> On Fri, Oct 2, 2015 at 11:23 AM, Alvaro Herrera >> wrote: + /* all-frozen information is also cleared at the same time */ PageClearAllVisible(page);

Re: [HACKERS] Freeze avoidance of very large table.

2015-10-05 Thread Masahiko Sawada
On Mon, Oct 5, 2015 at 11:03 PM, Fujii Masao wrote: > On Fri, Oct 2, 2015 at 8:14 PM, Masahiko Sawada wrote: >>> +#define Anum_pg_class_relallfrozen12 >>> Why is pg_class.relallfrozen necessary? ISTM that there is no user of it >>> now. >> >> The relallfrozen would be useful for user to

Re: [HACKERS] Freeze avoidance of very large table.

2015-10-05 Thread Simon Riggs
On 10 September 2015 at 01:58, Andres Freund wrote: > On 2015-09-04 23:35:42 +0100, Simon Riggs wrote: > > This looks OK. You saw that I was proposing to solve this problem a > > different way ("Summary of plans to avoid the annoyance of Freezing"), > > suggesting that we wait for a few CFs to se

Re: [HACKERS] Freeze avoidance of very large table.

2015-10-05 Thread Fujii Masao
On Fri, Oct 2, 2015 at 8:14 PM, Masahiko Sawada wrote: >> +#define Anum_pg_class_relallfrozen12 >> Why is pg_class.relallfrozen necessary? ISTM that there is no user of it now. > > The relallfrozen would be useful for user to estimate time to vacuum > freeze or anti-wrapping vacuum before

Re: [HACKERS] Freeze avoidance of very large table.

2015-10-05 Thread Masahiko Sawada
On Sat, Oct 3, 2015 at 3:41 AM, Robert Haas wrote: > On Fri, Oct 2, 2015 at 11:23 AM, Alvaro Herrera > wrote: >>> + /* all-frozen information is also cleared at the same time */ >>> PageClearAllVisible(page); >>> + PageClearAllFrozen(page); >> >> I wonder if

Re: [HACKERS] Freeze avoidance of very large table.

2015-10-02 Thread Robert Haas
On Fri, Oct 2, 2015 at 11:23 AM, Alvaro Herrera wrote: >> + /* all-frozen information is also cleared at the same time */ >> PageClearAllVisible(page); >> + PageClearAllFrozen(page); > > I wonder if it makes sense to have a macro to clear both in unison, > whi

Re: [HACKERS] Freeze avoidance of very large table.

2015-10-02 Thread Masahiko Sawada
On Sat, Oct 3, 2015 at 12:23 AM, Alvaro Herrera wrote: > Masahiko Sawada wrote: > Thank you for taking time to review this feature. Attached the latest version patch (v15). >> @@ -2972,10 +2981,15 @@ l1: >>*/ >> PageSetPrunable(page, xid); >> >> + /* clear PD_ALL_VISIBLE and P

Re: [HACKERS] Freeze avoidance of very large table.

2015-10-02 Thread Alvaro Herrera
Masahiko Sawada wrote: > @@ -2972,10 +2981,15 @@ l1: >*/ > PageSetPrunable(page, xid); > > + /* clear PD_ALL_VISIBLE and PD_ALL_FORZEN flags */ Typo "FORZEN". > if (PageIsAllVisible(page)) > { > all_visible_cleared = true; > + > + /* all-

Re: [HACKERS] Freeze avoidance of very large table.

2015-10-02 Thread Masahiko Sawada
On Fri, Oct 2, 2015 at 7:30 AM, Josh Berkus wrote: > On 10/01/2015 07:43 AM, Robert Haas wrote: >> On Thu, Oct 1, 2015 at 9:44 AM, Fujii Masao wrote: >>> I wonder how much it's worth renaming only the file extension while >>> there are many places where "visibility map" and "vm" are used, >>> for

Re: [HACKERS] Freeze avoidance of very large table.

2015-10-01 Thread Josh Berkus
On 10/01/2015 07:43 AM, Robert Haas wrote: > On Thu, Oct 1, 2015 at 9:44 AM, Fujii Masao wrote: >> I wonder how much it's worth renaming only the file extension while >> there are many places where "visibility map" and "vm" are used, >> for example, log messages, function names, variables, etc. >

Re: [HACKERS] Freeze avoidance of very large table.

2015-10-01 Thread Robert Haas
On Thu, Oct 1, 2015 at 9:44 AM, Fujii Masao wrote: > I wonder how much it's worth renaming only the file extension while > there are many places where "visibility map" and "vm" are used, > for example, log messages, function names, variables, etc. I'd be inclined to keep calling it the visibility

Re: [HACKERS] Freeze avoidance of very large table.

2015-10-01 Thread Fujii Masao
On Fri, Sep 18, 2015 at 8:14 PM, Masahiko Sawada wrote: > On Fri, Sep 18, 2015 at 6:13 PM, Fujii Masao wrote: >> On Fri, Sep 4, 2015 at 2:55 PM, Masahiko Sawada >> wrote: >>> On Fri, Sep 4, 2015 at 10:35 AM, Fujii Masao wrote: On Fri, Sep 4, 2015 at 2:23 AM, Masahiko Sawada wrote:

Re: [HACKERS] Freeze avoidance of very large table.

2015-09-18 Thread Masahiko Sawada
On Fri, Sep 18, 2015 at 6:13 PM, Fujii Masao wrote: > On Fri, Sep 4, 2015 at 2:55 PM, Masahiko Sawada wrote: >> On Fri, Sep 4, 2015 at 10:35 AM, Fujii Masao wrote: >>> On Fri, Sep 4, 2015 at 2:23 AM, Masahiko Sawada >>> wrote: On Thu, Aug 27, 2015 at 1:54 AM, Masahiko Sawada wrote:

Re: [HACKERS] Freeze avoidance of very large table.

2015-09-18 Thread Fujii Masao
On Fri, Sep 4, 2015 at 2:55 PM, Masahiko Sawada wrote: > On Fri, Sep 4, 2015 at 10:35 AM, Fujii Masao wrote: >> On Fri, Sep 4, 2015 at 2:23 AM, Masahiko Sawada >> wrote: >>> On Thu, Aug 27, 2015 at 1:54 AM, Masahiko Sawada >>> wrote: On Thu, Aug 20, 2015 at 11:46 PM, Alvaro Herrera

Re: [HACKERS] Freeze avoidance of very large table.

2015-09-09 Thread Andres Freund
On 2015-09-04 23:35:42 +0100, Simon Riggs wrote: > This looks OK. You saw that I was proposing to solve this problem a > different way ("Summary of plans to avoid the annoyance of Freezing"), > suggesting that we wait for a few CFs to see if a patch emerges for that - > then fall back to this patch

Re: [HACKERS] Freeze avoidance of very large table.

2015-09-08 Thread Robert Haas
On Mon, Sep 7, 2015 at 11:18 AM, Masahiko Sawada wrote: > On Sat, Sep 5, 2015 at 7:35 AM, Simon Riggs wrote: >> On 3 September 2015 at 18:23, Masahiko Sawada wrote: >>> The previous patch lacks some files for regression test. >>> Attached fixed v12 patch. >> >> This looks OK. You saw that I was

Re: [HACKERS] Freeze avoidance of very large table.

2015-09-07 Thread Masahiko Sawada
On Sat, Sep 5, 2015 at 7:35 AM, Simon Riggs wrote: > On 3 September 2015 at 18:23, Masahiko Sawada wrote: > >> >> The previous patch lacks some files for regression test. >> Attached fixed v12 patch. > > > This looks OK. You saw that I was proposing to solve this problem a > different way ("Summa

Re: [HACKERS] Freeze avoidance of very large table.

2015-09-04 Thread Simon Riggs
On 3 September 2015 at 18:23, Masahiko Sawada wrote: > The previous patch lacks some files for regression test. > Attached fixed v12 patch. > This looks OK. You saw that I was proposing to solve this problem a different way ("Summary of plans to avoid the annoyance of Freezing"), suggesting tha

Re: [HACKERS] Freeze avoidance of very large table.

2015-09-04 Thread Bruce Momjian
On Thu, Sep 3, 2015 at 11:56:52PM -0300, Alvaro Herrera wrote: > Bruce Momjian wrote: > > On Thu, Sep 3, 2015 at 11:37:09PM +0200, Petr Jelinek wrote: > > > >I don't understand. I'm just proposing that the source code for the > > > >extension to live in src/extensions/, and have the shared libra

Re: [HACKERS] Freeze avoidance of very large table.

2015-09-04 Thread Petr Jelinek
On 2015-09-04 02:11, Bruce Momjian wrote: On Thu, Sep 3, 2015 at 11:37:09PM +0200, Petr Jelinek wrote: I don't understand. I'm just proposing that the source code for the extension to live in src/extensions/, and have the shared library installed by toplevel make install; I'm not suggesting th

Re: [HACKERS] Freeze avoidance of very large table.

2015-09-03 Thread Masahiko Sawada
On Fri, Sep 4, 2015 at 10:35 AM, Fujii Masao wrote: > On Fri, Sep 4, 2015 at 2:23 AM, Masahiko Sawada wrote: >> On Thu, Aug 27, 2015 at 1:54 AM, Masahiko Sawada >> wrote: >>> On Thu, Aug 20, 2015 at 11:46 PM, Alvaro Herrera >>> wrote: Jim Nasby wrote: > I think things like pagein

Re: [HACKERS] Freeze avoidance of very large table.

2015-09-03 Thread Alvaro Herrera
Bruce Momjian wrote: > On Thu, Sep 3, 2015 at 11:37:09PM +0200, Petr Jelinek wrote: > > >I don't understand. I'm just proposing that the source code for the > > >extension to live in src/extensions/, and have the shared library > > >installed by toplevel make install; I'm not suggesting that the

Re: [HACKERS] Freeze avoidance of very large table.

2015-09-03 Thread Fujii Masao
On Fri, Sep 4, 2015 at 2:23 AM, Masahiko Sawada wrote: > On Thu, Aug 27, 2015 at 1:54 AM, Masahiko Sawada > wrote: >> On Thu, Aug 20, 2015 at 11:46 PM, Alvaro Herrera >> wrote: >>> Jim Nasby wrote: >>> I think things like pageinspect are very different; I really can't see any use for

Re: [HACKERS] Freeze avoidance of very large table.

2015-09-03 Thread Gavin Flower
On 04/09/15 12:11, Bruce Momjian wrote: On Thu, Sep 3, 2015 at 11:37:09PM +0200, Petr Jelinek wrote: I don't understand. I'm just proposing that the source code for the extension to live in src/extensions/, and have the shared library installed by toplevel make install; I'm not suggesting that

Re: [HACKERS] Freeze avoidance of very large table.

2015-09-03 Thread Josh Berkus
On 09/03/2015 05:11 PM, Bruce Momjian wrote: > On Thu, Sep 3, 2015 at 11:37:09PM +0200, Petr Jelinek wrote: >>> I don't understand. I'm just proposing that the source code for the >>> extension to live in src/extensions/, and have the shared library >>> installed by toplevel make install; I'm not

Re: [HACKERS] Freeze avoidance of very large table.

2015-09-03 Thread Bruce Momjian
On Thu, Sep 3, 2015 at 11:37:09PM +0200, Petr Jelinek wrote: > >I don't understand. I'm just proposing that the source code for the > >extension to live in src/extensions/, and have the shared library > >installed by toplevel make install; I'm not suggesting that the > >extension is installed aut

Re: [HACKERS] Freeze avoidance of very large table.

2015-09-03 Thread Petr Jelinek
On 2015-09-03 20:26, Alvaro Herrera wrote: Robert Haas wrote: On Thu, Aug 20, 2015 at 10:46 AM, Alvaro Herrera wrote: I don't think that necessarily means it must continue to be in contrib. Quite the contrary, I think it is a tool critical enough that it should not be relegated to be a secon

Re: [HACKERS] Freeze avoidance of very large table.

2015-09-03 Thread Robert Haas
On Thu, Sep 3, 2015 at 2:26 PM, Alvaro Herrera wrote: > Robert Haas wrote: >> On Thu, Aug 20, 2015 at 10:46 AM, Alvaro Herrera >> wrote: > >> > I don't think that necessarily means it must continue to be in contrib. >> > Quite the contrary, I think it is a tool critical enough that it should >> >

Re: [HACKERS] Freeze avoidance of very large table.

2015-09-03 Thread Alvaro Herrera
Robert Haas wrote: > On Thu, Aug 20, 2015 at 10:46 AM, Alvaro Herrera > wrote: > > I don't think that necessarily means it must continue to be in contrib. > > Quite the contrary, I think it is a tool critical enough that it should > > not be relegated to be a second-class citizen as it is now (le

Re: [HACKERS] Freeze avoidance of very large table.

2015-09-03 Thread Robert Haas
On Thu, Aug 20, 2015 at 10:46 AM, Alvaro Herrera wrote: > Jim Nasby wrote: >> I think things like pageinspect are very different; I really can't see any >> use for those beyond debugging (and debugging by an expert at that). > > I don't think that necessarily means it must continue to be in contri

Re: [HACKERS] Freeze avoidance of very large table.

2015-09-03 Thread Masahiko Sawada
On Thu, Aug 27, 2015 at 1:54 AM, Masahiko Sawada wrote: > On Thu, Aug 20, 2015 at 11:46 PM, Alvaro Herrera > wrote: >> Jim Nasby wrote: >> >>> I think things like pageinspect are very different; I really can't see any >>> use for those beyond debugging (and debugging by an expert at that). >> >>

Re: [HACKERS] Freeze avoidance of very large table.

2015-08-26 Thread Masahiko Sawada
On Thu, Aug 20, 2015 at 11:46 PM, Alvaro Herrera wrote: > Jim Nasby wrote: > >> I think things like pageinspect are very different; I really can't see any >> use for those beyond debugging (and debugging by an expert at that). > > I don't think that necessarily means it must continue to be in cont

Re: [HACKERS] Freeze avoidance of very large table.

2015-08-20 Thread Alvaro Herrera
Jim Nasby wrote: > I think things like pageinspect are very different; I really can't see any > use for those beyond debugging (and debugging by an expert at that). I don't think that necessarily means it must continue to be in contrib. Quite the contrary, I think it is a tool critical enough tha

Re: [HACKERS] Freeze avoidance of very large table.

2015-08-19 Thread Jim Nasby
On 8/19/15 2:56 AM, Masahiko Sawada wrote: The currently regression test for VM is that we just compare between the total number of all-visible and all-frozen in VM before and after VACUUM, and don't check particular a bit in VM. we could substitute it to the ANALYZE command with enough sampling

Re: [HACKERS] Freeze avoidance of very large table.

2015-08-19 Thread Masahiko Sawada
On Wed, Aug 19, 2015 at 1:28 AM, Robert Haas wrote: > On Tue, Aug 18, 2015 at 7:27 AM, Masahiko Sawada > wrote: >> I have encountered the much cases where pg_stat_statement, >> pgstattuples are required in production, so I basically agree with >> moving such extension into core. >> But IMO, the

Re: [HACKERS] Freeze avoidance of very large table.

2015-08-18 Thread Robert Haas
On Tue, Aug 18, 2015 at 7:27 AM, Masahiko Sawada wrote: > I have encountered the much cases where pg_stat_statement, > pgstattuples are required in production, so I basically agree with > moving such extension into core. > But IMO, the diagnostic tools for visibility map, heap (pageinspect) > and

Re: [HACKERS] Freeze avoidance of very large table.

2015-08-18 Thread Masahiko Sawada
On Mon, Aug 10, 2015 at 11:05 AM, Michael Paquier wrote: > On Mon, Aug 10, 2015 at 12:39 AM, Robert Haas wrote: >> On Thu, Aug 6, 2015 at 11:33 AM, Jim Nasby wrote: >>> They also provide a level of control over what is and isn't installed in a >>> cluster. Personally, I'd prefer that most users

Re: [HACKERS] Freeze avoidance of very large table.

2015-08-09 Thread Michael Paquier
On Mon, Aug 10, 2015 at 12:39 AM, Robert Haas wrote: > On Thu, Aug 6, 2015 at 11:33 AM, Jim Nasby wrote: >> They also provide a level of control over what is and isn't installed in a >> cluster. Personally, I'd prefer that most users not even be aware of the >> existence of things like pageinspec

Re: [HACKERS] Freeze avoidance of very large table.

2015-08-09 Thread Robert Haas
On Thu, Aug 6, 2015 at 11:33 AM, Jim Nasby wrote: > They also provide a level of control over what is and isn't installed in a > cluster. Personally, I'd prefer that most users not even be aware of the > existence of things like pageinspect. +1. If everybody feels that moving extensions currentl

Re: [HACKERS] Freeze avoidance of very large table.

2015-08-06 Thread Simon Riggs
On 5 August 2015 at 18:46, Alvaro Herrera wrote: > > What do others think? Wow, everything moves when you blink, eh? Sorry I was wasn't watching this. Mainly because I was working on some other related thoughts, separate post coming. 1. Most importantly, it needs to be somewhere where we can u

Re: [HACKERS] Freeze avoidance of very large table.

2015-08-06 Thread Jim Nasby
On 8/5/15 1:47 PM, Petr Jelinek wrote: On 2015-08-05 20:09, Alvaro Herrera wrote: Josh Berkus wrote: On 08/05/2015 10:46 AM, Alvaro Herrera wrote: 1. Add the functions as a builtins. This is what the current patch does. Simon seems to prefer this, because he wants the function to be a

Re: [HACKERS] Freeze avoidance of very large table.

2015-08-05 Thread Bruce Momjian
On Wed, Aug 5, 2015 at 11:57:48PM -0300, Alvaro Herrera wrote: > Bruce Momjian wrote: > > > This is why I suggested putting the new SQL function where it belongs > > for consistency and then open a separate thread to discuss the future of > > where we want diagnostic functions to be. It is too c

Re: [HACKERS] Freeze avoidance of very large table.

2015-08-05 Thread Alvaro Herrera
Bruce Momjian wrote: > This is why I suggested putting the new SQL function where it belongs > for consistency and then open a separate thread to discuss the future of > where we want diagnostic functions to be. It is too complicated to talk > about both issues in the same thread. Oh come on --

Re: [HACKERS] Freeze avoidance of very large table.

2015-08-05 Thread Bruce Momjian
On Wed, Aug 5, 2015 at 10:58:00AM -0700, Josh Berkus wrote: > On 08/05/2015 10:46 AM, Alvaro Herrera wrote: > > 1. Add the functions as a builtins. > >This is what the current patch does. Simon seems to prefer this, > >because he wants the function to be always available in production; >

Re: [HACKERS] Freeze avoidance of very large table.

2015-08-05 Thread Petr Jelinek
On 2015-08-05 20:09, Alvaro Herrera wrote: Josh Berkus wrote: On 08/05/2015 10:46 AM, Alvaro Herrera wrote: 1. Add the functions as a builtins. This is what the current patch does. Simon seems to prefer this, because he wants the function to be always available in production; but I

Re: [HACKERS] Freeze avoidance of very large table.

2015-08-05 Thread Alvaro Herrera
Josh Berkus wrote: > On 08/05/2015 10:46 AM, Alvaro Herrera wrote: > > 1. Add the functions as a builtins. > >This is what the current patch does. Simon seems to prefer this, > >because he wants the function to be always available in production; > >but I don't like this option because

Re: [HACKERS] Freeze avoidance of very large table.

2015-08-05 Thread Josh Berkus
On 08/05/2015 10:46 AM, Alvaro Herrera wrote: > 1. Add the functions as a builtins. >This is what the current patch does. Simon seems to prefer this, >because he wants the function to be always available in production; >but I don't like this option because adding functions as builtins

Re: [HACKERS] Freeze avoidance of very large table.

2015-08-05 Thread Alvaro Herrera
Josh Berkus wrote: > On 08/05/2015 10:26 AM, Bruce Momjian wrote: > > I don't care what we do, but I do think we should be consistent. > > Frankly I am unclear why I am even having to make this point, as cases > > where we have chosen expediency over consistency have served us badly in > > the pa

Re: [HACKERS] Freeze avoidance of very large table.

2015-08-05 Thread Josh Berkus
On 08/05/2015 10:26 AM, Bruce Momjian wrote: > On Wed, Aug 5, 2015 at 10:22:48AM -0700, Josh Berkus wrote: >> On 08/05/2015 10:00 AM, Alvaro Herrera wrote: >>> Anyway, the patch as proposed puts the new functions in core as builtins >>> (which is what Bruce seems to be objecting to). Maybe instea

Re: [HACKERS] Freeze avoidance of very large table.

2015-08-05 Thread Bruce Momjian
On Wed, Aug 5, 2015 at 10:22:48AM -0700, Josh Berkus wrote: > On 08/05/2015 10:00 AM, Alvaro Herrera wrote: > > Anyway, the patch as proposed puts the new functions in core as builtins > > (which is what Bruce seems to be objecting to). Maybe instead of > > proposing moving existing extensions in

Re: [HACKERS] Freeze avoidance of very large table.

2015-08-05 Thread Josh Berkus
On 08/05/2015 10:00 AM, Alvaro Herrera wrote: > Anyway, the patch as proposed puts the new functions in core as builtins > (which is what Bruce seems to be objecting to). Maybe instead of > proposing moving existing extensions in core, it would be better to have > this patch put those two new func

Re: [HACKERS] Freeze avoidance of very large table.

2015-08-05 Thread Alvaro Herrera
Robert Haas wrote: > On Wed, Aug 5, 2015 at 12:36 PM, Alvaro Herrera > wrote: > > Bruce Momjian wrote: > >> I understand the desire for a diagnostic function in core, but we have > >> to be consistent. Just because we are adding this function now doesn't > >> mean we should use different rules fr

Re: [HACKERS] Freeze avoidance of very large table.

2015-08-05 Thread Robert Haas
On Wed, Aug 5, 2015 at 12:36 PM, Alvaro Herrera wrote: > Bruce Momjian wrote: >> I understand the desire for a diagnostic function in core, but we have >> to be consistent. Just because we are adding this function now doesn't >> mean we should use different rules from what we did previously for >

Re: [HACKERS] Freeze avoidance of very large table.

2015-08-05 Thread Alvaro Herrera
Bruce Momjian wrote: > I understand the desire for a diagnostic function in core, but we have > to be consistent. Just because we are adding this function now doesn't > mean we should use different rules from what we did previously for > diagnostic functions. Either their is logic to why this fu

Re: [HACKERS] Freeze avoidance of very large table.

2015-08-05 Thread Bruce Momjian
On Wed, Jul 8, 2015 at 02:31:04PM +0100, Simon Riggs wrote: > On 7 July 2015 at 18:45, Sawada Masahiko wrote: > > On Wed, Jul 8, 2015 at 12:37 AM, Andres Freund wrote: > > On 2015-07-07 16:25:13 +0100, Simon Riggs wrote: > >> I don't think pg_freespacemap is the right place. > >

Re: [HACKERS] Freeze avoidance of very large table.

2015-07-29 Thread Masahiko Sawada
On Thu, Jul 16, 2015 at 8:51 PM, Sawada Masahiko wrote: > On Wed, Jul 15, 2015 at 3:07 AM, Sawada Masahiko > wrote: >> On Wed, Jul 15, 2015 at 12:55 AM, Simon Riggs wrote: >>> On 10 July 2015 at 15:11, Sawada Masahiko wrote: Oops, I had forgotten to add new file heapfuncs.c. >>>

Re: [HACKERS] Freeze avoidance of very large table.

2015-07-16 Thread Sawada Masahiko
On Wed, Jul 15, 2015 at 3:07 AM, Sawada Masahiko wrote: > On Wed, Jul 15, 2015 at 12:55 AM, Simon Riggs wrote: >> On 10 July 2015 at 15:11, Sawada Masahiko wrote: >>> >>> >>> Oops, I had forgotten to add new file heapfuncs.c. >>> Latest patch is attached. >> >> >> I think we've established the a

Re: [HACKERS] Freeze avoidance of very large table.

2015-07-14 Thread Sawada Masahiko
On Wed, Jul 15, 2015 at 12:55 AM, Simon Riggs wrote: > On 10 July 2015 at 15:11, Sawada Masahiko wrote: >> >> >> Oops, I had forgotten to add new file heapfuncs.c. >> Latest patch is attached. > > > I think we've established the approach is desirable and defined the way > forwards for this, so th

Re: [HACKERS] Freeze avoidance of very large table.

2015-07-14 Thread Alvaro Herrera
Michael Paquier wrote: > On Mon, Jul 13, 2015 at 9:03 PM, Sawada Masahiko > wrote: > > We use script file which are generated by pg_upgrade. > > I haven't followed this thread closely, but I am sure you recall that > vacuumdb has a parallel mode. I think having to vacuum the whole database dur

Re: [HACKERS] Freeze avoidance of very large table.

2015-07-14 Thread Simon Riggs
On 10 July 2015 at 15:11, Sawada Masahiko wrote: > > Oops, I had forgotten to add new file heapfuncs.c. > Latest patch is attached. > I think we've established the approach is desirable and defined the way forwards for this, so this is looking good. Some of my requests haven't been actioned yet

Re: [HACKERS] Freeze avoidance of very large table.

2015-07-13 Thread Andres Freund
On 2015-07-13 23:48:02 +0900, Sawada Masahiko wrote: > But please image the case where old cluster has table which is very > large, read-only and vacuum freeze is done. > In this case, the all-frozen bit of such table in new cluster will not > set, unless we do vacuum freeze again. > The informatio

Re: [HACKERS] Freeze avoidance of very large table.

2015-07-13 Thread Simon Riggs
On 13 July 2015 at 15:48, Sawada Masahiko wrote: > On Mon, Jul 13, 2015 at 9:22 PM, Andres Freund wrote: > > On 2015-07-13 21:03:07 +0900, Sawada Masahiko wrote: > >> Even If we implement rewriting tool for vm into pg_upgrade, it will > >> take time as much as revacuum because it need whole scan

Re: [HACKERS] Freeze avoidance of very large table.

2015-07-13 Thread Sawada Masahiko
On Mon, Jul 13, 2015 at 9:22 PM, Andres Freund wrote: > On 2015-07-13 21:03:07 +0900, Sawada Masahiko wrote: >> Even If we implement rewriting tool for vm into pg_upgrade, it will >> take time as much as revacuum because it need whole scanning table. > > Why would it? Sure, you can only set allvis

Re: [HACKERS] Freeze avoidance of very large table.

2015-07-13 Thread Michael Paquier
On Mon, Jul 13, 2015 at 9:03 PM, Sawada Masahiko wrote: > On Mon, Jul 13, 2015 at 7:46 PM, Amit Kapila wrote: >> On Mon, Jul 13, 2015 at 3:39 PM, Sawada Masahiko >> wrote: >>> >>> On Tue, Jul 7, 2015 at 9:07 PM, Simon Riggs wrote: >>> > On 6 July 2015 at 17:28, Simon Riggs wrote: >>> > >>> >>

Re: [HACKERS] Freeze avoidance of very large table.

2015-07-13 Thread Andres Freund
On 2015-07-13 21:03:07 +0900, Sawada Masahiko wrote: > Even If we implement rewriting tool for vm into pg_upgrade, it will > take time as much as revacuum because it need whole scanning table. Why would it? Sure, you can only set allvisible and not the frozen bit, but that's fine. That way the cos

Re: [HACKERS] Freeze avoidance of very large table.

2015-07-13 Thread Sawada Masahiko
On Mon, Jul 13, 2015 at 7:46 PM, Amit Kapila wrote: > On Mon, Jul 13, 2015 at 3:39 PM, Sawada Masahiko > wrote: >> >> On Tue, Jul 7, 2015 at 9:07 PM, Simon Riggs wrote: >> > On 6 July 2015 at 17:28, Simon Riggs wrote: >> > >> >> I think we need something for pg_upgrade to rewrite existing VMs.

Re: [HACKERS] Freeze avoidance of very large table.

2015-07-13 Thread Amit Kapila
On Mon, Jul 13, 2015 at 3:39 PM, Sawada Masahiko wrote: > > On Tue, Jul 7, 2015 at 9:07 PM, Simon Riggs wrote: > > On 6 July 2015 at 17:28, Simon Riggs wrote: > > > >> I think we need something for pg_upgrade to rewrite existing VMs. > >> Otherwise a large read only database would suddenly requi

Re: [HACKERS] Freeze avoidance of very large table.

2015-07-13 Thread Sawada Masahiko
On Tue, Jul 7, 2015 at 9:07 PM, Simon Riggs wrote: > On 6 July 2015 at 17:28, Simon Riggs wrote: > >> I think we need something for pg_upgrade to rewrite existing VMs. >> Otherwise a large read only database would suddenly require a massive >> revacuum after upgrade, which seems bad. That can wai

Re: [HACKERS] Freeze avoidance of very large table.

2015-07-10 Thread Jim Nasby
On 7/10/15 4:46 AM, Simon Riggs wrote: On 10 July 2015 at 09:49, Sawada Masahiko mailto:sawada.m...@gmail.com>> wrote: > It is a minor thing, but if there is no other use for this fourth > "bit-space", it seems a shame to waste it when there is some use for it. I > haven't looked

Re: [HACKERS] Freeze avoidance of very large table.

2015-07-10 Thread Sawada Masahiko
On Fri, Jul 10, 2015 at 10:43 PM, Fujii Masao wrote: > On Fri, Jul 10, 2015 at 2:41 PM, Sawada Masahiko > wrote: >> On Fri, Jul 10, 2015 at 3:05 AM, Sawada Masahiko >> wrote: >>> >>> Also something for pg_upgrade is also not yet. >>> >>> TODO >>> - Test case for this feature >>> - pg_upgrade s

<    1   2   3   >