Re: trying again to get incremental backup

2024-04-26 Thread Robert Haas
On Thu, Apr 25, 2024 at 6:44 PM Thom Brown wrote: > I would like to query the following: > > --tablespace-mapping=olddir=newdir > > Relocates the tablespace in directory olddir to newdir during the backup. > olddir is the absolute path of the tablespace as it exists in the first > backup

Re: trying again to get incremental backup

2024-04-25 Thread Thom Brown
On Wed, 3 Jan 2024 at 15:10, Robert Haas wrote: > On Fri, Dec 22, 2023 at 12:00 AM Alexander Lakhin > wrote: > > My quick experiment shows that that TimestampDifferenceMilliseconds call > > always returns zero, due to it's arguments swapped. > > Thanks. Tom already changed the unsigned -> int

Re: trying again to get incremental backup

2024-01-03 Thread Robert Haas
On Fri, Dec 22, 2023 at 12:00 AM Alexander Lakhin wrote: > My quick experiment shows that that TimestampDifferenceMilliseconds call > always returns zero, due to it's arguments swapped. Thanks. Tom already changed the unsigned -> int stuff in a separate commit, so I just pushed the fixes to

Re: trying again to get incremental backup

2024-01-02 Thread Robert Haas
On Wed, Dec 27, 2023 at 10:36 AM Nathan Bossart wrote: > On Wed, Dec 27, 2023 at 09:11:02AM -0500, Robert Haas wrote: > > Thanks. I don't think there's a real bug, but I pushed a fix, same as > > what you had. > > Thanks! I also noticed that WALSummarizerLock probably needs a mention in >

Re: trying again to get incremental backup

2023-12-27 Thread Nathan Bossart
On Wed, Dec 27, 2023 at 09:11:02AM -0500, Robert Haas wrote: > Thanks. I don't think there's a real bug, but I pushed a fix, same as > what you had. Thanks! I also noticed that WALSummarizerLock probably needs a mention in wait_event_names.txt. -- Nathan Bossart Amazon Web Services:

Re: trying again to get incremental backup

2023-12-27 Thread Robert Haas
On Sat, Dec 23, 2023 at 4:51 PM Nathan Bossart wrote: > My compiler has the following complaint: > > ../postgresql/src/backend/postmaster/walsummarizer.c: In function > ‘GetOldestUnsummarizedLSN’: > ../postgresql/src/backend/postmaster/walsummarizer.c:540:32: error: > ‘unsummarized_lsn’ may be

Re: trying again to get incremental backup

2023-12-23 Thread Nathan Bossart
My compiler has the following complaint: ../postgresql/src/backend/postmaster/walsummarizer.c: In function ‘GetOldestUnsummarizedLSN’: ../postgresql/src/backend/postmaster/walsummarizer.c:540:32: error: ‘unsummarized_lsn’ may be used uninitialized in this function [-Werror=maybe-uninitialized]

Re: trying again to get incremental backup

2023-12-21 Thread Alexander Lakhin
21.12.2023 23:43, Robert Haas wrote: There are also two deadcode.DeadStores complaints from clang. First one is about: /* * Align the wait time to prevent drift. This doesn't really matter, * but we'd like the warnings about how long we've been waiting to say

Re: trying again to get incremental backup

2023-12-21 Thread Robert Haas
On Thu, Dec 21, 2023 at 10:00 AM Alexander Lakhin wrote: > Please look at the attached patch; it corrects all 29 items ("recods" > fixed in two places), but maybe you find some substitutions wrong... Thanks, committed with a few additions. > I've also observed that those commits introduced new

Re: trying again to get incremental backup

2023-12-21 Thread Alexander Lakhin
21.12.2023 15:07, Robert Haas wrote: On Wed, Dec 20, 2023 at 11:00 PM Alexander Lakhin wrote: I've found several typos/inconsistencies introduced with 174c48050 and dc2123400. Maybe you would want to fix them, while on it?: That's an impressively long list of mistakes in something I thought

Re: trying again to get incremental backup

2023-12-21 Thread Robert Haas
On Wed, Dec 20, 2023 at 11:00 PM Alexander Lakhin wrote: > I've found several typos/inconsistencies introduced with 174c48050 and > dc2123400. Maybe you would want to fix them, while on it?: That's an impressively long list of mistakes in something I thought I'd been careful about. Sigh. I

Re: trying again to get incremental backup

2023-12-20 Thread Alexander Lakhin
Hello Robert, 20.12.2023 23:56, Robert Haas wrote: On Wed, Dec 20, 2023 at 8:11 AM Jakub Wartak wrote: the v15 patchset (posted yesterday) test results are GOOD: All right. I committed the main two patches, dropped the for-testing-only patch, and added a simple test to the remaining

Re: trying again to get incremental backup

2023-12-20 Thread Robert Haas
On Wed, Dec 20, 2023 at 8:11 AM Jakub Wartak wrote: > the v15 patchset (posted yesterday) test results are GOOD: All right. I committed the main two patches, dropped the for-testing-only patch, and added a simple test to the remaining pg_walsummary patch. That needs more work, but here's what I

Re: trying again to get incremental backup

2023-12-20 Thread Jakub Wartak
Hi Robert, On Tue, Dec 19, 2023 at 9:36 PM Robert Haas wrote: > > On Fri, Dec 15, 2023 at 5:36 AM Jakub Wartak > wrote: > > I've played with with initdb/pg_upgrade (17->17) and i don't get DBID > > mismatch (of course they do differ after initdb), but i get this > > instead: > > > > $

Re: trying again to get incremental backup

2023-12-18 Thread Robert Haas
On Fri, Dec 15, 2023 at 6:58 AM Peter Eisentraut wrote: > A separate bikeshedding topic: The GUC "summarize_wal", could that be > "wal_something" instead? (wal_summarize? wal_summarizer?) It would be > nice if these settings names group together a bit, both with existing > wal_* ones and also

Re: trying again to get incremental backup

2023-12-18 Thread Robert Haas
On Fri, Dec 15, 2023 at 6:53 AM Peter Eisentraut wrote: > The first fixes up some things in nls.mk related to a file move. The > second is some cleanup because some function you are using has been > removed in the meantime; you probably found that yourself while rebasing. Incorporated these. As

Re: trying again to get incremental backup

2023-12-18 Thread Peter Eisentraut
Another set of comments, about the patch that adds pg_combinebackup: Make sure all the options are listed in a consistent order. We have lately changed everything to be alphabetical. This includes: - reference page pg_combinebackup.sgml - long_options listing - getopt_long() argument -

Re: trying again to get incremental backup

2023-12-15 Thread Peter Eisentraut
A separate bikeshedding topic: The GUC "summarize_wal", could that be "wal_something" instead? (wal_summarize? wal_summarizer?) It would be nice if these settings names group together a bit, both with existing wal_* ones and also with the new ones you are adding (wal_summary_keep_time).

Re: trying again to get incremental backup

2023-12-15 Thread Peter Eisentraut
I have a couple of quick fixes here. The first fixes up some things in nls.mk related to a file move. The second is some cleanup because some function you are using has been removed in the meantime; you probably found that yourself while rebasing. The pg_walsummary patch doesn't have a

Re: trying again to get incremental backup

2023-12-15 Thread Jakub Wartak
Hi Robert, On Wed, Dec 13, 2023 at 2:16 PM Robert Haas wrote: > > > > > not even in case of an intervening > > > timeline switch. So, all of the errors in this function are warning > > > you that you've done something that you really should not have done. > > > In this particular case, you've

Re: trying again to get incremental backup

2023-12-13 Thread Robert Haas
On Wed, Dec 13, 2023 at 5:39 AM Jakub Wartak wrote: > > I can't exactly say that such a hint would be inaccurate, but I think > > the impulse to add it here is misguided. One of my design goals for > > this system is to make it so that you never have to take a new > > incremental backup "just

Re: trying again to get incremental backup

2023-12-13 Thread Jakub Wartak
Hi Robert, On Mon, Dec 11, 2023 at 6:08 PM Robert Haas wrote: > > On Fri, Dec 8, 2023 at 5:02 AM Jakub Wartak > wrote: > > While we are at it, maybe around the below in PrepareForIncrementalBackup() > > > > if (tlep[i] == NULL) > > ereport(ERROR, > > > >

Re: trying again to get incremental backup

2023-12-10 Thread Dilip Kumar
On Mon, Dec 11, 2023 at 11:44 AM Dilip Kumar wrote: > > On Tue, Dec 5, 2023 at 11:40 PM Robert Haas wrote: > > > > On Mon, Dec 4, 2023 at 3:58 PM Robert Haas wrote: > > > Considering all this, what I'm inclined to do is go and put > > > UPLOAD_MANIFEST back, instead of INCREMENTAL_WAL_RANGE,

Re: trying again to get incremental backup

2023-12-10 Thread Dilip Kumar
On Tue, Dec 5, 2023 at 11:40 PM Robert Haas wrote: > > On Mon, Dec 4, 2023 at 3:58 PM Robert Haas wrote: > > Considering all this, what I'm inclined to do is go and put > > UPLOAD_MANIFEST back, instead of INCREMENTAL_WAL_RANGE, and adjust > > accordingly. But first: does anybody see more

Re: trying again to get incremental backup

2023-12-08 Thread Jakub Wartak
On Thu, Dec 7, 2023 at 4:15 PM Robert Haas wrote: Hi Robert, > On Thu, Dec 7, 2023 at 9:42 AM Jakub Wartak > wrote: > > Comment: I was wondering if it wouldn't make some sense to teach > > pg_resetwal to actually delete all WAL summaries after any any > > WAL/controlfile alteration? > > I

Re: trying again to get incremental backup

2023-12-07 Thread Robert Haas
On Thu, Dec 7, 2023 at 9:42 AM Jakub Wartak wrote: > Comment: I was wondering if it wouldn't make some sense to teach > pg_resetwal to actually delete all WAL summaries after any any > WAL/controlfile alteration? I thought that this was a good idea so I decided to go implement it, only to

Re: trying again to get incremental backup

2023-12-07 Thread Jakub Wartak
On Tue, Dec 5, 2023 at 7:11 PM Robert Haas wrote: [..v13 patchset] The results with v13 patchset are following: * - requires checkpoint on primary when doing incremental on standby when it's too idle, this was explained by Robert in [1], something AKA too-fast-incremental backup due to

Re: trying again to get incremental backup

2023-12-04 Thread Robert Haas
On Thu, Nov 30, 2023 at 9:33 AM Robert Haas wrote: > 0005: Incremental backup itself. Changes: > - Remove UPLOAD_MANIFEST replication command and instead add > INCREMENTAL_WAL_RANGE replication command. Unfortunately, I think this change is going to need to be reverted. Jakub reported out a

Re: trying again to get incremental backup

2023-11-29 Thread Robert Haas
On Wed, Nov 15, 2023 at 9:14 AM Jakub Wartak wrote: > so I've spent some time playing still with patchset v8 (without the > 6/6 testing patch related to wal_level=minimal), with the exception of > - patchset v9 - marked otherwise. Thanks, as usual, for that. > 2. Usability thing: I hit the

Re: trying again to get incremental backup

2023-11-27 Thread Robert Haas
On Thu, Nov 23, 2023 at 11:18 PM Thomas Munro wrote: > Robert pinged me to see if I had any ideas. Thanks, Thomas. > The reason it fails on Windows is because there is a special code path > for WIN32 in the patch's src/bin/pg_combinebackup/copy_file.c, but it > is incomplete: it returns early

Re: trying again to get incremental backup

2023-11-23 Thread Thomas Munro
On Wed, Nov 22, 2023 at 3:14 AM Jakub Wartak wrote: > CFbot failed on two hosts this time, I haven't looked at the details > yet (https://cirrus-ci.com/task/6425149646307328 -> end of EOL? -> > LOG: WAL summarizer process (PID 71511) was terminated by signal 6: > Aborted?) Robert pinged me to

Re: trying again to get incremental backup

2023-11-21 Thread Jakub Wartak
On Mon, Nov 20, 2023 at 4:43 PM Robert Haas wrote: > > On Fri, Nov 17, 2023 at 5:01 AM Alvaro Herrera > wrote: > > I made a pass over pg_combinebackup for NLS. I propose the attached > > patch. > > This doesn't quite compile for me so I changed a few things and > incorporated it. Hopefully I

Re: trying again to get incremental backup

2023-11-20 Thread Robert Haas
On Mon, Nov 20, 2023 at 2:10 PM Robert Haas wrote: > > Is this meant to support multiple timelines each with non-overlapping > > adjacent ranges, rather than multiple non-adjacent ranges? > > Correct. I don't see how non-adjacent LSN ranges could ever be a > useful thing, but adjacent ranges on

Re: trying again to get incremental backup

2023-11-20 Thread Robert Haas
On Mon, Nov 20, 2023 at 2:03 PM Alvaro Herrera wrote: > That sounds good to me. Not having to parse the manifest server-side > sounds like a win, as does saving the transfer, for the cases where the > manifest is large. OK. I'll look into this next week, hopefully. > Is this meant to support

Re: trying again to get incremental backup

2023-11-20 Thread Alvaro Herrera
On 2023-Nov-16, Robert Haas wrote: > On Thu, Nov 16, 2023 at 12:23 PM Alvaro Herrera > wrote: > > I don't understand this point. Currently, the protocol is that > > UPLOAD_MANIFEST is used to send the manifest prior to requesting the > > backup. You seem to be saying that you're thinking of

Re: trying again to get incremental backup

2023-11-17 Thread Alvaro Herrera
I made a pass over pg_combinebackup for NLS. I propose the attached patch. -- Álvaro HerreraBreisgau, Deutschland — https://www.EnterpriseDB.com/ "Right now the sectors on the hard disk run clockwise, but I heard a rumor that you can squeeze 0.2% more throughput by running them

Re: trying again to get incremental backup

2023-11-16 Thread Robert Haas
On Thu, Nov 16, 2023 at 12:34 PM Alvaro Herrera wrote: > Putting those two thoughts together, I think pg_basebackup with > --progress could tell you "still waiting for the summary file up to LSN > %X/%X to appear, and the walsummarizer is currently handling lsn %X/%X" > or something like that.

Re: trying again to get incremental backup

2023-11-16 Thread Alvaro Herrera
On 2023-Nov-16, Alvaro Herrera wrote: > On 2023-Oct-04, Robert Haas wrote: > > - Right now, I have a hard-coded 60 second timeout for WAL > > summarization. If you try to take an incremental backup and the WAL > > summaries you need don't show up within 60 seconds, the backup times > > out. I

Re: trying again to get incremental backup

2023-11-16 Thread Alvaro Herrera
On 2023-Nov-16, Robert Haas wrote: > On Thu, Nov 16, 2023 at 5:21 AM Alvaro Herrera > wrote: > > I meant code like this > > > > memcpy(, rlocator, sizeof(RelFileLocator)); > > key.forknum = forknum; > > entry = blockreftable_lookup(brtab->hash, key); > > Ah, I hadn't

Re: trying again to get incremental backup

2023-11-16 Thread Alvaro Herrera
On 2023-Oct-04, Robert Haas wrote: > - I would like some feedback on the generation of WAL summary files. > Right now, I have it enabled by default, and summaries are kept for a > week. That means that, with no additional setup, you can take an > incremental backup as long as the reference backup

Re: trying again to get incremental backup

2023-11-16 Thread Robert Haas
On Thu, Nov 16, 2023 at 5:21 AM Alvaro Herrera wrote: > I meant code like this > > memcpy(, rlocator, sizeof(RelFileLocator)); > key.forknum = forknum; > entry = blockreftable_lookup(brtab->hash, key); Ah, I hadn't thought about that. Another way of handling that might be

Re: trying again to get incremental backup

2023-11-16 Thread Robert Haas
On Tue, Nov 14, 2023 at 8:12 AM Alvaro Herrera wrote: > 0001 looks OK to push, and since it stands on its own I would get it out > of the way soon rather than waiting for the rest of the series to be > further reviewed. All right, done. > 0003: > AmWalSummarizerProcess() is unused. Remove?

Re: trying again to get incremental backup

2023-11-16 Thread Alvaro Herrera
On 2023-Nov-13, Robert Haas wrote: > On Mon, Nov 13, 2023 at 11:25 AM Alvaro Herrera > wrote: > > Also, it would be good to provide and use a > > function to initialize a BlockRefTableKey from the RelFileNode and > > forknum components, and ensure that any padding bytes are zeroed. > >

Re: trying again to get incremental backup

2023-11-15 Thread Jakub Wartak
Hi Robert, [..spotted the v9 patchset..] so I've spent some time playing still with patchset v8 (without the 6/6 testing patch related to wal_level=minimal), with the exception of - patchset v9 - marked otherwise. 1. On compile time there were 2 warnings to shadowing variable (at least with gcc

Re: trying again to get incremental backup

2023-11-14 Thread Alvaro Herrera
0001 looks OK to push, and since it stands on its own I would get it out of the way soon rather than waiting for the rest of the series to be further reviewed. 0002: This moves bin/pg_verifybackup/parse_manifest.c to common/parse_manifest.c, where it's not clear that it's for backup manifests

Re: trying again to get incremental backup

2023-11-14 Thread Dilip Kumar
On Tue, Nov 14, 2023 at 2:10 AM Robert Haas wrote: > > On Mon, Nov 13, 2023 at 11:25 AM Alvaro Herrera > wrote: > > Great stuff you got here. I'm doing a first pass trying to grok the > > whole thing for more substantive comments, but in the meantime here are > > some cosmetic ones. > >

Re: trying again to get incremental backup

2023-11-13 Thread Dilip Kumar
On Tue, Nov 14, 2023 at 12:52 AM Robert Haas wrote: > > On Fri, Nov 10, 2023 at 6:27 AM Dilip Kumar wrote: > > - I think 0001 looks good improvement irrespective of the patch series. > > OK, perhaps that can be independently committed, then, if nobody objects. > > Thanks for the review; I've

Re: trying again to get incremental backup

2023-11-13 Thread Robert Haas
On Fri, Nov 10, 2023 at 6:27 AM Dilip Kumar wrote: > - I think 0001 looks good improvement irrespective of the patch series. OK, perhaps that can be independently committed, then, if nobody objects. Thanks for the review; I've fixed a bunch of things that you mentioned. I'll just comment on the

Re: trying again to get incremental backup

2023-11-13 Thread Alvaro Herrera
Great stuff you got here. I'm doing a first pass trying to grok the whole thing for more substantive comments, but in the meantime here are some cosmetic ones. I got the following warnings, both valid: ../../../../pgsql/source/master/src/common/blkreftable.c: In function 'WriteBlockRefTable':

Re: trying again to get incremental backup

2023-11-10 Thread Dilip Kumar
On Tue, Nov 7, 2023 at 2:06 AM Robert Haas wrote: > > On Mon, Oct 30, 2023 at 2:46 PM Andres Freund wrote: > > After playing with this for a while, I don't see a reason for > > wal_summarize_mb > > from a memory usage POV at least. > > Here's v8. Changes: Review comments, based on what I

Re: trying again to get incremental backup

2023-11-03 Thread Robert Haas
On Wed, Nov 1, 2023 at 8:57 AM Jakub Wartak wrote: > Thanks for answering! It all sounds like this > resync-standby-using-primary-incrbackup idea isn't fit for the current > pg_combinebackup, but rather for a new tool hopefully in future. It > could take the current LSN from stuck standby,

Re: trying again to get incremental backup

2023-11-01 Thread Jakub Wartak
On Mon, Oct 30, 2023 at 6:46 PM Robert Haas wrote: > > On Thu, Sep 28, 2023 at 6:22 AM Jakub Wartak > wrote: > > If that is still an area open for discussion: wouldn't it be better to > > just specify LSN as it would allow resyncing standby across major lag > > where the WAL to replay would be

Re: trying again to get incremental backup

2023-10-30 Thread Robert Haas
On Mon, Oct 30, 2023 at 2:46 PM Andres Freund wrote: > After playing with this for a while, I don't see a reason for wal_summarize_mb > from a memory usage POV at least. Cool! Thanks for testing. > I wonder if there are use cases that might like to consume WAL summaries > before the next

Re: trying again to get incremental backup

2023-10-30 Thread Andres Freund
Hi, On 2023-10-30 10:45:03 -0400, Robert Haas wrote: > On Tue, Oct 24, 2023 at 12:08 PM Robert Haas wrote: > > Note that whether to remove summaries is a separate question from > > whether to generate them in the first place. Right now, I have > > wal_summarize_mb controlling whether they get

Re: trying again to get incremental backup

2023-10-30 Thread Robert Haas
On Thu, Sep 28, 2023 at 6:22 AM Jakub Wartak wrote: > If that is still an area open for discussion: wouldn't it be better to > just specify LSN as it would allow resyncing standby across major lag > where the WAL to replay would be enormous? Given that we had > primary->standby where standby

Re: trying again to get incremental backup

2023-10-30 Thread Robert Haas
While reviewing this thread today, I realized that I never responded to this email. That was inadvertent; my apologies. On Wed, Jun 14, 2023 at 4:34 PM Matthias van de Meent wrote: > Nice, I like this idea. Cool. > Skimming through the 7th patch, I see claims that FSM is not fully > WAL-logged

Re: trying again to get incremental backup

2023-10-30 Thread Robert Haas
On Tue, Oct 24, 2023 at 12:08 PM Robert Haas wrote: > Note that whether to remove summaries is a separate question from > whether to generate them in the first place. Right now, I have > wal_summarize_mb controlling whether they get generated in the first > place, but as I noted in another recent

Re: trying again to get incremental backup

2023-10-26 Thread Robert Haas
On Thu, Oct 26, 2023 at 6:59 AM Andrew Dunstan wrote: > Because we won't be removing the RD parser. Ah, OK. -- Robert Haas EDB: http://www.enterprisedb.com

Re: trying again to get incremental backup

2023-10-26 Thread Andrew Dunstan
On 2023-10-25 We 15:19, Robert Haas wrote: On Wed, Oct 25, 2023 at 3:17 PM Andrew Dunstan wrote: OK, I'll go with that. It will actually be a bit less invasive than the patch I posted. Why's that? Because we won't be removing the RD parser. cheers andrew -- Andrew Dunstan EDB:

Re: trying again to get incremental backup

2023-10-25 Thread Robert Haas
On Wed, Oct 25, 2023 at 3:17 PM Andrew Dunstan wrote: > OK, I'll go with that. It will actually be a bit less invasive than the > patch I posted. Why's that? -- Robert Haas EDB: http://www.enterprisedb.com

Re: trying again to get incremental backup

2023-10-25 Thread Andrew Dunstan
On 2023-10-25 We 11:24, Robert Haas wrote: On Wed, Oct 25, 2023 at 10:33 AM Andrew Dunstan wrote: I'm not too worried about the maintenance burden. That said, I agree that JSON might not be the best format for backup manifests, but maybe that ship has sailed. I think it's a decision we

Re: trying again to get incremental backup

2023-10-25 Thread Robert Haas
On Wed, Oct 25, 2023 at 10:33 AM Andrew Dunstan wrote: > I'm not too worried about the maintenance burden. > > That said, I agree that JSON might not be the best format for backup > manifests, but maybe that ship has sailed. I think it's a decision we could walk back if we had a good enough

Re: trying again to get incremental backup

2023-10-25 Thread Andrew Dunstan
On 2023-10-25 We 09:05, Robert Haas wrote: On Wed, Oct 25, 2023 at 7:54 AM Andrew Dunstan wrote: Robert asked me to work on this quite some time ago, and most of this work was done last year. Here's my WIP for an incremental JSON parser. It works and passes all the usual json/b tests. It

Re: trying again to get incremental backup

2023-10-25 Thread Robert Haas
On Wed, Oct 25, 2023 at 7:54 AM Andrew Dunstan wrote: > Robert asked me to work on this quite some time ago, and most of this > work was done last year. > > Here's my WIP for an incremental JSON parser. It works and passes all > the usual json/b tests. It implements Algorithm 4.3 in the Dragon

Re: trying again to get incremental backup

2023-10-25 Thread Andrew Dunstan
On 2023-10-24 Tu 12:08, Robert Haas wrote: It looks like each file entry in the manifest takes about 150 bytes, so 1 GB would allow for 1024**3/150 = 7158278 files. That seems fine for now? I suspect a few people have more files than that. They'll just have to Maybe someone on the list can

Re: trying again to get incremental backup

2023-10-24 Thread Robert Haas
On Tue, Oct 24, 2023 at 10:53 AM Peter Eisentraut wrote: > The easiest answer is to have it off by default. Let people figure out > what works for them. There are various factors like storage, network, > server performance, RTO that will determine what combination of full > backup, incremental

Re: trying again to get incremental backup

2023-10-24 Thread Peter Eisentraut
On 04.10.23 22:08, Robert Haas wrote: - I would like some feedback on the generation of WAL summary files. Right now, I have it enabled by default, and summaries are kept for a week. That means that, with no additional setup, you can take an incremental backup as long as the reference backup was

Re: trying again to get incremental backup

2023-10-24 Thread Robert Haas
On Mon, Oct 23, 2023 at 7:56 PM David Steele wrote: > > I also think a lot of the use of the low-level API is > > driven by it being just too darn slow to copy the whole database, and > > incremental backup can help with that in some circumstances. > > I would argue that restore performance is

Re: trying again to get incremental backup

2023-10-23 Thread David Steele
On 10/23/23 11:44, Robert Haas wrote: On Fri, Oct 20, 2023 at 11:30 AM David Steele wrote: I don't plan to stand in your way on this feature. I'm reviewing what patches I can out of courtesy and to be sure that nothing adjacent to your work is being affected. My apologies if my reviews are

Re: trying again to get incremental backup

2023-10-23 Thread Robert Haas
On Fri, Oct 20, 2023 at 11:30 AM David Steele wrote: > Then I'm fine with it as is. OK, thanks. > In my view this feature puts the cart way before the horse. I'd think > higher priority features might be parallelism, a backup repository, > expiration management, archiving, or maybe even a

Re: trying again to get incremental backup

2023-10-20 Thread David Steele
On 10/19/23 16:00, Robert Haas wrote: On Thu, Oct 19, 2023 at 3:18 PM David Steele wrote: 0001 looks pretty good to me. The only thing I find a little troublesome is the repeated construction of file names with/without segment numbers in ResetUnloggedRelationsInDbspaceDir(), .e.g.: +

Re: trying again to get incremental backup

2023-10-20 Thread Jakub Wartak
Hi Robert, On Wed, Oct 4, 2023 at 10:09 PM Robert Haas wrote: > > On Tue, Oct 3, 2023 at 2:21 PM Robert Haas wrote: > > Here's a new patch set, also addressing Jakub's observation that > > MINIMUM_VERSION_FOR_WAL_SUMMARIES needed updating. > > Here's yet another new version.[..] Okay, so

Re: trying again to get incremental backup

2023-10-19 Thread Robert Haas
On Thu, Oct 19, 2023 at 3:18 PM David Steele wrote: > 0001 looks pretty good to me. The only thing I find a little troublesome > is the repeated construction of file names with/without segment numbers > in ResetUnloggedRelationsInDbspaceDir(), .e.g.: > > + if (segno == 0) >

Re: trying again to get incremental backup

2023-10-19 Thread David Steele
On 10/19/23 12:05, Robert Haas wrote: On Wed, Oct 4, 2023 at 4:08 PM Robert Haas wrote: Clearly there's a good amount of stuff to sort out here, but we've still got quite a bit of time left before feature freeze so I'd like to have a go at it. Please let me know your thoughts, if you have any.

Re: trying again to get incremental backup

2023-10-04 Thread Robert Haas
On Thu, Sep 28, 2023 at 6:22 AM Jakub Wartak wrote: > all those basic tests had GOOD results. Please find attached. I'll try > to schedule some more realistic (in terms of workload and sizes) test > in a couple of days + maybe have some fun with cross-backup-and > restores across standbys.

Re: trying again to get incremental backup

2023-10-03 Thread Robert Haas
On Fri, Sep 1, 2023 at 10:30 AM Robert Haas wrote: > > No objections to 0001/0002. > > Cool. Nobody else objected either, so I went ahead and committed those. I'll rebase the rest of the patches on top of the latest master and repost, hopefully after addressing some of the other review comments

Re: trying again to get incremental backup

2023-09-28 Thread Jakub Wartak
On Wed, Aug 30, 2023 at 4:50 PM Robert Haas wrote: [..] I've played a little bit more this second batch of patches on e8d74ad625f7344f6b715254d3869663c1569a51 @ 31Aug (days before wait events refactor): test_across_wallevelminimal.sh test_many_incrementals_dbcreate.sh test_many_incrementals.sh

Re: trying again to get incremental backup

2023-09-12 Thread Dilip Kumar
On Wed, Aug 30, 2023 at 9:20 PM Robert Haas wrote: > > Meanwhile, here's a rebased set of patches. The somewhat-primitive > attempts at writing tests are in 0009, but they don't work, for the > reasons explained above. I think I'd probably like to go ahead and > commit 0001 and 0002 soon if there

Re: trying again to get incremental backup

2023-09-05 Thread Robert Haas
On Mon, Sep 4, 2023 at 8:42 AM Dilip Kumar wrote: > Can't we think of comparing at the block level, like we can compare > each block but ignore the content of the hole? We could do that, but I don't think that's a full solution. I think I'd end up having to reimplement the equivalent of

Re: trying again to get incremental backup

2023-09-04 Thread Dilip Kumar
On Wed, Aug 30, 2023 at 9:20 PM Robert Haas wrote: > Unless someone has a brilliant idea that I lack, this suggests to me > that this whole line of testing is a dead end. I can, of course, write > tests that compare clusters *logically* -- do the correct relations > exist, are they accessible,

Re: trying again to get incremental backup

2023-09-01 Thread Robert Haas
Hey, thanks for the reply. On Thu, Aug 31, 2023 at 6:50 PM David Steele wrote: > pg_subtrans, at least, can be ignored since it is excluded from the > backup and not required for recovery. I agree... > Welcome to the club! Thanks for the welcome, but being a member feels *terrible*. :-) > I

Re: trying again to get incremental backup

2023-08-31 Thread David Steele
Hi Robert, On 8/30/23 10:49, Robert Haas wrote: In the limited time that I've had to work on this project lately, I've been trying to come up with a test case for this feature -- and since I've gotten completely stuck, I thought it might be time to post and see if anyone else has a better idea.

Re: trying again to get incremental backup

2023-06-19 Thread Andres Freund
Hi, On 2023-06-19 09:46:12 -0400, Robert Haas wrote: > On Wed, Jun 14, 2023 at 4:40 PM Andres Freund wrote: > > > But I'm not sure that's a great approach, because that LSN gap might be > > > large and then we're duplicating a lot of work that the summarizer has > > > probably already done most

Re: trying again to get incremental backup

2023-06-19 Thread Robert Haas
On Wed, Jun 14, 2023 at 4:40 PM Andres Freund wrote: > > But I'm not sure that's a great approach, because that LSN gap might be > > large and then we're duplicating a lot of work that the summarizer has > > probably already done most of. > > I guess that really depends on what the summary

Re: trying again to get incremental backup

2023-06-15 Thread Dilip Kumar
On Thu, Jun 15, 2023 at 2:11 AM Andres Freund wrote: > > > I'm not really sure. I expect Dilip would be happy to post his patch, > > and if you'd be willing to have a look at it and express your concerns > > or lack thereof, that would be super valuable. > > Will do. Adding me to CC: might help,

Re: trying again to get incremental backup

2023-06-14 Thread Andres Freund
Hi, On 2023-06-14 16:10:38 -0400, Robert Haas wrote: > On Wed, Jun 14, 2023 at 3:47 PM Andres Freund wrote: > > Could we just recompute the WAL summary for the [redo, end of chunk] for the > > relevant summary file? > > I'm not understanding how that would help. If we were going to compute > a

Re: trying again to get incremental backup

2023-06-14 Thread Matthias van de Meent
On Wed, 14 Jun 2023 at 20:47, Robert Haas wrote: > > A few years ago, I sketched out a design for incremental backup, but > no patch for incremental backup ever got committed. Instead, the whole > thing evolved into a project to add backup manifests, which are nice, > but not as nice as

Re: trying again to get incremental backup

2023-06-14 Thread Robert Haas
On Wed, Jun 14, 2023 at 3:47 PM Andres Freund wrote: > I assume this is "solely" required for keeping the incremental backups as > small as possible, rather than being required for correctness? I believe so. I want to spend some more time thinking about this to make sure I'm not missing

Re: trying again to get incremental backup

2023-06-14 Thread Andres Freund
Hi, On 2023-06-14 14:46:48 -0400, Robert Haas wrote: > A few years ago, I sketched out a design for incremental backup, but > no patch for incremental backup ever got committed. Instead, the whole > thing evolved into a project to add backup manifests, which are nice, > but not as nice as