Re: cleanup patches for incremental backup

2024-03-20 Thread Nathan Bossart
On Wed, Mar 20, 2024 at 02:53:01PM -0400, Robert Haas wrote: > On Wed, Mar 20, 2024 at 2:35 PM Nathan Bossart > wrote: >> Committed. > > Thanks. Sorry you had to clean up after me. :-( No worries. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: cleanup patches for incremental backup

2024-03-20 Thread Robert Haas
On Wed, Mar 20, 2024 at 2:35 PM Nathan Bossart wrote: > On Tue, Mar 19, 2024 at 01:15:02PM -0500, Nathan Bossart wrote: > > Assuming there are no objections or feedback, I plan to commit these two > > patches within the next couple of days. > > Committed. Thanks. Sorry you had to clean up after

Re: cleanup patches for incremental backup

2024-03-20 Thread Nathan Bossart
On Tue, Mar 19, 2024 at 01:15:02PM -0500, Nathan Bossart wrote: > Assuming there are no objections or feedback, I plan to commit these two > patches within the next couple of days. Committed. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: cleanup patches for incremental backup

2024-03-19 Thread Nathan Bossart
On Thu, Mar 14, 2024 at 08:52:55PM -0500, Nathan Bossart wrote: > Subject: [PATCH v1 1/2] Revert "Temporary patch to help debug pg_walsummary > test failures." > Subject: [PATCH v1 2/2] Fix possible overflow in MaybeRemoveOldWalSummaries(). Assuming there are no objections or feedback, I plan

Re: cleanup patches for incremental backup

2024-03-14 Thread Nathan Bossart
On Thu, Mar 14, 2024 at 04:00:10PM -0500, Nathan Bossart wrote: > Separately, I suppose it's probably time to revert the temporary debugging > code adding by commit 5ddf997. I can craft a patch for that, too. As promised... -- Nathan Bossart Amazon Web Services: https://aws.amazon.com >From

Re: cleanup patches for incremental backup

2024-03-14 Thread Nathan Bossart
On Wed, Jan 24, 2024 at 12:05:15PM -0600, Nathan Bossart wrote: > There might be an overflow risk in the cutoff time calculation, but I doubt > that's the root cause of these failures: > > /* >* Files should only be removed if the last modification time precedes > the >*

Re: cleanup patches for incremental backup

2024-01-31 Thread Robert Haas
On Tue, Jan 30, 2024 at 11:52 AM Robert Haas wrote: > Here's a patch for that. I now think > a7097ca630a25dd2248229f21ebce4968d85d10a was actually misguided, and > served only to mask some of the failures caused by waiting for the WAL > summary file. Committed. -- Robert Haas EDB:

Re: cleanup patches for incremental backup

2024-01-30 Thread Robert Haas
On Tue, Jan 30, 2024 at 10:51 AM Robert Haas wrote: > I think the solution here is to find a better way to wait for the > inserts to be summarized, one that actually does wait for that to > happen. Here's a patch for that. I now think a7097ca630a25dd2248229f21ebce4968d85d10a was actually

Re: cleanup patches for incremental backup

2024-01-30 Thread Robert Haas
On Mon, Jan 29, 2024 at 4:13 PM Nathan Bossart wrote: > On Mon, Jan 29, 2024 at 03:18:50PM -0500, Robert Haas wrote: > > I'm wondering if what we need to do is run pg_walsummary on both > > summary files in that case. If we just pick one or the other, how do > > we know which one to pick? > >

Re: cleanup patches for incremental backup

2024-01-29 Thread Nathan Bossart
On Mon, Jan 29, 2024 at 03:18:50PM -0500, Robert Haas wrote: > I'm wondering if what we need to do is run pg_walsummary on both > summary files in that case. If we just pick one or the other, how do > we know which one to pick? Even if we do that, isn't it possible that none of the summaries will

Re: cleanup patches for incremental backup

2024-01-29 Thread Robert Haas
On Mon, Jan 29, 2024 at 1:21 PM Nathan Bossart wrote: > > Ah, I think this query: > > > > SELECT tli, start_lsn, end_lsn from pg_available_wal_summaries() > > WHERE tli = $summarized_tli AND end_lsn > '$summarized_lsn' > > > > is returning more than one row in some cases. I

Re: cleanup patches for incremental backup

2024-01-29 Thread Nathan Bossart
On Sat, Jan 27, 2024 at 10:31:09AM -0600, Nathan Bossart wrote: > On Sat, Jan 27, 2024 at 11:00:01AM +0300, Alexander Lakhin wrote: >> I'm discouraged by "\n1" in the file name and in the >> "examining summary..." message. >> regress_log_002_blocks from the following successful test run on the

Re: cleanup patches for incremental backup

2024-01-27 Thread Nathan Bossart
On Sat, Jan 27, 2024 at 11:00:01AM +0300, Alexander Lakhin wrote: > 24.01.2024 20:46, Robert Haas wrote: >> This is weird. There's a little more detail in the log file, >> regress_log_002_blocks, e.g. from the first failure you linked: >> >> [11:18:20.683](96.787s) # before insert, summarized TLI

Re: cleanup patches for incremental backup

2024-01-27 Thread Alexander Lakhin
24.01.2024 20:46, Robert Haas wrote: This is weird. There's a little more detail in the log file, regress_log_002_blocks, e.g. from the first failure you linked: [11:18:20.683](96.787s) # before insert, summarized TLI 1 through 0/14E09D0 [11:18:21.188](0.505s) # after insert, summarized TLI 1

Re: cleanup patches for incremental backup

2024-01-26 Thread Alexander Lakhin
Hello Robert, 26.01.2024 21:37, Robert Haas wrote: On Fri, Jan 26, 2024 at 12:39 PM Nathan Bossart wrote: On Fri, Jan 26, 2024 at 11:04:37AM -0500, Robert Haas wrote: Here is v2 with that addition. Looks reasonable. Thanks for the report & review. I have committed that version. While

Re: cleanup patches for incremental backup

2024-01-26 Thread Robert Haas
On Fri, Jan 26, 2024 at 12:39 PM Nathan Bossart wrote: > On Fri, Jan 26, 2024 at 11:04:37AM -0500, Robert Haas wrote: > > Here is v2 with that addition. > > Looks reasonable. Thanks for the report & review. I have committed that version. -- Robert Haas EDB: http://www.enterprisedb.com

Re: cleanup patches for incremental backup

2024-01-26 Thread Nathan Bossart
On Fri, Jan 26, 2024 at 11:04:37AM -0500, Robert Haas wrote: > Here is v2 with that addition. Looks reasonable. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: cleanup patches for incremental backup

2024-01-26 Thread Robert Haas
On Thu, Jan 25, 2024 at 11:08 AM Nathan Bossart wrote: > On Thu, Jan 25, 2024 at 10:06:41AM -0500, Robert Haas wrote: > > On Wed, Jan 24, 2024 at 2:39 PM Nathan Bossart > > wrote: > >> That seems like a reasonable starting point. Even if it doesn't help > >> determine the root cause, it should

Re: cleanup patches for incremental backup

2024-01-25 Thread Nathan Bossart
On Thu, Jan 25, 2024 at 10:06:41AM -0500, Robert Haas wrote: > On Wed, Jan 24, 2024 at 2:39 PM Nathan Bossart > wrote: >> That seems like a reasonable starting point. Even if it doesn't help >> determine the root cause, it should at least help rule out concurrent >> summary removal. > > Here

Re: cleanup patches for incremental backup

2024-01-25 Thread Robert Haas
On Wed, Jan 24, 2024 at 2:39 PM Nathan Bossart wrote: > That seems like a reasonable starting point. Even if it doesn't help > determine the root cause, it should at least help rule out concurrent > summary removal. Here is a patch for that. -- Robert Haas EDB: http://www.enterprisedb.com

Re: cleanup patches for incremental backup

2024-01-24 Thread Nathan Bossart
On Wed, Jan 24, 2024 at 02:08:08PM -0500, Robert Haas wrote: > On Wed, Jan 24, 2024 at 1:05 PM Nathan Bossart > wrote: >> Otherwise, I think we'll probably need to add some additional logging to >> figure out what is happening... > > Where, though? I suppose we could: > > 1. Change the server

Re: cleanup patches for incremental backup

2024-01-24 Thread Robert Haas
On Wed, Jan 24, 2024 at 1:05 PM Nathan Bossart wrote: > There might be an overflow risk in the cutoff time calculation, but I doubt > that's the root cause of these failures: > > /* > * Files should only be removed if the last modification time > precedes the > * cutoff

Re: cleanup patches for incremental backup

2024-01-24 Thread Nathan Bossart
On Wed, Jan 24, 2024 at 12:46:16PM -0500, Robert Haas wrote: > The "examining summary" line is generated based on the output of > pg_available_wal_summaries(). The way that works is that the server > calls readdir(), disassembles the filename into a TLI and two LSNs, > and returns the result.

Re: cleanup patches for incremental backup

2024-01-24 Thread Robert Haas
On Wed, Jan 24, 2024 at 12:08 PM Nathan Bossart wrote: > I'm seeing some recent buildfarm failures for pg_walsummary: > > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=sungazer=2024-01-14%2006%3A21%3A58 > >

Re: cleanup patches for incremental backup

2024-01-24 Thread Nathan Bossart
I'm seeing some recent buildfarm failures for pg_walsummary: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=sungazer=2024-01-14%2006%3A21%3A58 https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=idiacanthus=2024-01-17%2021%3A10%3A36

Re: cleanup patches for incremental backup

2024-01-18 Thread Robert Haas
On Thu, Jan 18, 2024 at 4:50 AM Matthias van de Meent wrote: > Sure, that's fine with me. OK, committed that way. -- Robert Haas EDB: http://www.enterprisedb.com

Re: cleanup patches for incremental backup

2024-01-18 Thread Matthias van de Meent
On Wed, 17 Jan 2024 at 21:10, Robert Haas wrote: > > On Wed, Jan 17, 2024 at 1:42 PM Matthias van de Meent > wrote: > > Sure, added in attached. > > I think this mostly looks good now but I just realized that I think > this needs rephrasing: > > + To restore incremental backups the tool

Re: cleanup patches for incremental backup

2024-01-17 Thread Robert Haas
On Wed, Jan 17, 2024 at 1:42 PM Matthias van de Meent wrote: > Sure, added in attached. I think this mostly looks good now but I just realized that I think this needs rephrasing: + To restore incremental backups the tool + is used, which combines incremental backups with a base backup

Re: cleanup patches for incremental backup

2024-01-17 Thread Matthias van de Meent
On Tue, 16 Jan 2024 at 21:49, Robert Haas wrote: > > On Tue, Jan 16, 2024 at 3:22 PM Matthias van de Meent > wrote: > + A special base > backup > + that for some WAL-logged relations only contains the pages that were > + modified since a previous backup, as opposed to the full

Re: cleanup patches for incremental backup

2024-01-16 Thread Robert Haas
On Tue, Jan 16, 2024 at 3:22 PM Matthias van de Meent wrote: > > One other thought is that the incremental backup only replaces > > relation files with incremental files, and it never does anything > > about FSM files. So the statement that it only contains data that was > > potentially changed

Re: cleanup patches for incremental backup

2024-01-16 Thread Matthias van de Meent
On Tue, 16 Jan 2024 at 16:39, Robert Haas wrote: > > On Mon, Jan 15, 2024 at 3:31 PM Matthias van de Meent > wrote: > > Off-list I was notified that the new WAL summarizer process was not > > yet added to the glossary, so PFA a patch that does that. > > In passing, it also adds "incremental

Re: cleanup patches for incremental backup

2024-01-16 Thread Robert Haas
On Mon, Jan 15, 2024 at 3:31 PM Matthias van de Meent wrote: > Off-list I was notified that the new WAL summarizer process was not > yet added to the glossary, so PFA a patch that does that. > In passing, it also adds "incremental backup" to the glossary, and > updates the documented types of

Re: cleanup patches for incremental backup

2024-01-15 Thread Matthias van de Meent
On Mon, 15 Jan 2024 at 17:58, Robert Haas wrote: > > On Sat, Jan 13, 2024 at 1:00 PM Alexander Lakhin wrote: > > I've found one more typo in the sgml: > > summarized_pid > > And one in a comment: > > sumamry > > > > A trivial fix is attached. > > Thanks, committed. Off-list I was notified that

Re: cleanup patches for incremental backup

2024-01-15 Thread Robert Haas
On Sat, Jan 13, 2024 at 1:00 PM Alexander Lakhin wrote: > I've found one more typo in the sgml: > summarized_pid > And one in a comment: > sumamry > > A trivial fix is attached. Thanks, committed. -- Robert Haas EDB: http://www.enterprisedb.com

Re: cleanup patches for incremental backup

2024-01-13 Thread Alexander Lakhin
Hello Robert, 12.01.2024 17:50, Robert Haas wrote: On Thu, Jan 11, 2024 at 8:00 PM Shinoda, Noriyoshi (HPE Services Japan - FSIP) wrote: Thank you for developing the new tool. I have attached a patch that corrects the spelling of the --individual option in the SGML file. Thanks, committed.

Re: cleanup patches for incremental backup

2024-01-12 Thread Robert Haas
On Thu, Jan 11, 2024 at 8:00 PM Shinoda, Noriyoshi (HPE Services Japan - FSIP) wrote: > Thank you for developing the new tool. I have attached a patch that corrects > the spelling of the --individual option in the SGML file. Thanks, committed. -- Robert Haas EDB: http://www.enterprisedb.com

RE: cleanup patches for incremental backup

2024-01-11 Thread Shinoda, Noriyoshi (HPE Services Japan - FSIP)
Subject: Re: cleanup patches for incremental backup On Tue, Jan 9, 2024 at 1:18 PM Robert Haas wrote: > Here's v2. I plan to commit the rest of this fairly soon if there are > no comments. Done, with a minor adjustment to 0003. -- Robert Haas EDB: http://www.enterprise

Re: cleanup patches for incremental backup

2024-01-11 Thread Robert Haas
On Tue, Jan 9, 2024 at 1:18 PM Robert Haas wrote: > Here's v2. I plan to commit the rest of this fairly soon if there are > no comments. Done, with a minor adjustment to 0003. -- Robert Haas EDB: http://www.enterprisedb.com

Re: cleanup patches for incremental backup

2024-01-09 Thread Robert Haas
Hello again, I have now committed 0001. I got some off-list review of 0004; specifically, Michael Paquier said it looked OK, and Tom Lane found another bug. So I've added a fix for that in what's now 0003. Here's v2. I plan to commit the rest of this fairly soon if there are no comments.

cleanup patches for incremental backup

2024-01-05 Thread Robert Haas
Hi, I discovered that my patch to add WAL summarization added two new SQL-callable functions but failed to document them. 0001 fixes that. An outstanding item from the original thread was to write a better test for the not-yet-committed pg_walsummary utility. But I discovered that I couldn't do