Re: Impact of checkpointer during pg_upgrade

2023-09-10 Thread Dilip Kumar
On Fri, Sep 8, 2023 at 11:59 AM Amit Kapila wrote: > > On Fri, Sep 8, 2023 at 10:10 AM Michael Paquier wrote: > > > > On Fri, Sep 08, 2023 at 09:14:59AM +0530, Amit Kapila wrote: > > > On Fri, Sep 8, 2023 at 9:00 AM Zhijie Hou (Fujitsu) > > > wrote: > > >>> I > > >>> mean that doing the latter

Re: Impact of checkpointer during pg_upgrade

2023-09-08 Thread Amit Kapila
On Fri, Sep 8, 2023 at 10:10 AM Michael Paquier wrote: > > On Fri, Sep 08, 2023 at 09:14:59AM +0530, Amit Kapila wrote: > > On Fri, Sep 8, 2023 at 9:00 AM Zhijie Hou (Fujitsu) > > wrote: > >>> I > >>> mean that doing the latter is benefitial for the sake of any patch > >>> committed and > >>>

Re: Impact of checkpointer during pg_upgrade

2023-09-08 Thread Michael Paquier
On Fri, Sep 08, 2023 at 11:59:41AM +0530, Amit Kapila wrote: > I agree with doing something simple. So, to conclude, we agree on two > things in this thread (a) Use max_slot_wal_keep_size to -1 to start > postmaster for the old cluster during the upgrade; (b) Have an > elog(ERROR) to avoid

Re: Impact of checkpointer during pg_upgrade

2023-09-07 Thread Michael Paquier
On Fri, Sep 08, 2023 at 09:14:59AM +0530, Amit Kapila wrote: > On Fri, Sep 8, 2023 at 9:00 AM Zhijie Hou (Fujitsu) > wrote: >>> I >>> mean that doing the latter is benefitial for the sake of any patch >>> committed and >>> as a long-term method to rely on. > > What is your worry here? Are you

Re: Impact of checkpointer during pg_upgrade

2023-09-07 Thread Amit Kapila
On Fri, Sep 8, 2023 at 9:00 AM Zhijie Hou (Fujitsu) wrote: > > On Friday, September 8, 2023 10:58 AM Michael Paquier > wrote: > > > > On Fri, Sep 08, 2023 at 08:18:14AM +0530, Amit Kapila wrote: > > > This validation tries to ensure that we don't have any bugs/issues in > > > our patch. It may

Re: Impact of checkpointer during pg_upgrade

2023-09-07 Thread Michael Paquier
On Fri, Sep 08, 2023 at 03:30:23AM +, Zhijie Hou (Fujitsu) wrote: > I feel adding a check at pg_upgrade may not 100% detect the slot invalidation > if we check by querying the old cluster to get the slot info, because the > invalidation can happen before the first time we fetch the info or

RE: Impact of checkpointer during pg_upgrade

2023-09-07 Thread Zhijie Hou (Fujitsu)
On Friday, September 8, 2023 10:58 AM Michael Paquier wrote: > > On Fri, Sep 08, 2023 at 08:18:14AM +0530, Amit Kapila wrote: > > This validation tries to ensure that we don't have any bugs/issues in > > our patch. It may be a candidate for assert but I feel even if we > > encounter any bug it

Re: Impact of checkpointer during pg_upgrade

2023-09-07 Thread Michael Paquier
On Fri, Sep 08, 2023 at 08:18:14AM +0530, Amit Kapila wrote: > This validation tries to ensure that we don't have any bugs/issues in > our patch. It may be a candidate for assert but I feel even if we > encounter any bug it is better to fix the bug. My guess is that an elog-like error is more

Re: Impact of checkpointer during pg_upgrade

2023-09-07 Thread Amit Kapila
On Fri, Sep 8, 2023 at 5:37 AM Michael Paquier wrote: > > On Thu, Sep 07, 2023 at 03:33:52PM +0530, Amit Kapila wrote: > > I think if we just make max_slot_wal_keep_size to -1 that should be > > sufficient to not let any slots get invalidated during upgrade. Do you > > have anything else in mind?

Re: Impact of checkpointer during pg_upgrade

2023-09-07 Thread Michael Paquier
On Thu, Sep 07, 2023 at 03:33:52PM +0530, Amit Kapila wrote: > I think if we just make max_slot_wal_keep_size to -1 that should be > sufficient to not let any slots get invalidated during upgrade. Do you > have anything else in mind? Forcing wal_keep_size while on it may be a good thing. > If we

Re: Impact of checkpointer during pg_upgrade

2023-09-07 Thread Dilip Kumar
On Thu, Sep 7, 2023 at 3:34 PM Amit Kapila wrote: > > On Thu, Sep 7, 2023 at 12:55 PM Michael Paquier wrote: > > > > (Just dumping what I have in mind while reading the thread.) > > > > On Sat, Sep 02, 2023 at 10:08:51AM +0530, Amit Kapila wrote: > > > During pg_upgrade, we start the server for

Re: Impact of checkpointer during pg_upgrade

2023-09-07 Thread Amit Kapila
On Thu, Sep 7, 2023 at 12:55 PM Michael Paquier wrote: > > (Just dumping what I have in mind while reading the thread.) > > On Sat, Sep 02, 2023 at 10:08:51AM +0530, Amit Kapila wrote: > > During pg_upgrade, we start the server for the old cluster which can > > allow the checkpointer to remove

Re: Impact of checkpointer during pg_upgrade

2023-09-07 Thread Michael Paquier
(Just dumping what I have in mind while reading the thread.) On Sat, Sep 02, 2023 at 10:08:51AM +0530, Amit Kapila wrote: > During pg_upgrade, we start the server for the old cluster which can > allow the checkpointer to remove the WAL files. It has been noticed > that we do generate certain

Re: Impact of checkpointer during pg_upgrade

2023-09-05 Thread Dilip Kumar
On Tue, Sep 5, 2023 at 10:55 AM Amit Kapila wrote: > > > Earlier I was thinking that ERRORing out is better so that the user > > can take necessary action for the invalidated slots and then retry > > upgrade. But thinking again I could not find what are the advantages > > of this because if we

Re: Impact of checkpointer during pg_upgrade

2023-09-04 Thread Amit Kapila
On Tue, Sep 5, 2023 at 10:09 AM Dilip Kumar wrote: > > On Tue, Sep 5, 2023 at 9:38 AM Amit Kapila wrote: > > > > On Mon, Sep 4, 2023 at 4:19 PM Dilip Kumar wrote: > > > > > > Said that there is a possibility that some of the slots which got > > > invalidated even on the previous checkpoint

Re: Impact of checkpointer during pg_upgrade

2023-09-04 Thread Dilip Kumar
On Tue, Sep 5, 2023 at 9:38 AM Amit Kapila wrote: > > On Mon, Sep 4, 2023 at 4:19 PM Dilip Kumar wrote: > > > > Said that there is a possibility that some of the slots which got > > invalidated even on the previous checkpoint might get the same LSN as > > the slot which got invalidated later if

Re: Impact of checkpointer during pg_upgrade

2023-09-04 Thread Amit Kapila
On Mon, Sep 4, 2023 at 4:19 PM Dilip Kumar wrote: > > Said that there is a possibility that some of the slots which got > invalidated even on the previous checkpoint might get the same LSN as > the slot which got invalidated later if there is no activity between > these two checkpoints. So if we

Re: Impact of checkpointer during pg_upgrade

2023-09-04 Thread Dilip Kumar
On Mon, Sep 4, 2023 at 1:41 PM Dilip Kumar wrote: > > > > I think we can do better, like we can just read the latest > > > checkpoint's LSN before starting the old cluster. And now while > > > checking the slot can't we check if the the slot is invalidated then > > > their confirmed_flush_lsn >=

Re: Impact of checkpointer during pg_upgrade

2023-09-04 Thread Dilip Kumar
On Mon, Sep 4, 2023 at 11:18 AM Amit Kapila wrote: > > On Mon, Sep 4, 2023 at 10:33 AM Dilip Kumar wrote: > > > > On Mon, Sep 4, 2023 at 8:41 AM Amit Kapila wrote: > > > > > > On Sat, Sep 2, 2023 at 6:12 PM Dilip Kumar wrote: > > > > > > > > On Sat, Sep 2, 2023 at 10:09 AM Amit Kapila > > >

Re: Impact of checkpointer during pg_upgrade

2023-09-03 Thread Amit Kapila
On Mon, Sep 4, 2023 at 10:33 AM Dilip Kumar wrote: > > On Mon, Sep 4, 2023 at 8:41 AM Amit Kapila wrote: > > > > On Sat, Sep 2, 2023 at 6:12 PM Dilip Kumar wrote: > > > > > > On Sat, Sep 2, 2023 at 10:09 AM Amit Kapila > > > wrote: > > > > > > > The other possibilities apart from not allowing

Re: Impact of checkpointer during pg_upgrade

2023-09-03 Thread Dilip Kumar
On Mon, Sep 4, 2023 at 8:41 AM Amit Kapila wrote: > > On Sat, Sep 2, 2023 at 6:12 PM Dilip Kumar wrote: > > > > On Sat, Sep 2, 2023 at 10:09 AM Amit Kapila wrote: > > > > > The other possibilities apart from not allowing an upgrade in such a > > > case could be (a) Before starting the old

Re: Impact of checkpointer during pg_upgrade

2023-09-03 Thread Amit Kapila
On Sat, Sep 2, 2023 at 6:12 PM Dilip Kumar wrote: > > On Sat, Sep 2, 2023 at 10:09 AM Amit Kapila wrote: > > > The other possibilities apart from not allowing an upgrade in such a > > case could be (a) Before starting the old cluster, we fetch the slots > > directly from the disk using some tool

Re: Impact of checkpointer during pg_upgrade

2023-09-02 Thread Dilip Kumar
On Sat, Sep 2, 2023 at 10:09 AM Amit Kapila wrote: > > During pg_upgrade, we start the server for the old cluster which can > allow the checkpointer to remove the WAL files. It has been noticed > that we do generate certain types of WAL records (e.g > XLOG_RUNNING_XACTS, XLOG_CHECKPOINT_ONLINE,