Re: Reducing connection overhead in pg_upgrade compat check phase

2024-03-19 Thread Daniel Gustafsson
> On 19 Mar 2024, at 08:07, Peter Eisentraut wrote: > > On 18.03.24 13:11, Daniel Gustafsson wrote: >> Attached is a fresh rebase with only minor cosmetic touch-ups which I would >> like to go ahead with during this CF. >> Peter: does this address the comments you had on translation and code >>

Re: Reducing connection overhead in pg_upgrade compat check phase

2024-03-19 Thread Peter Eisentraut
On 18.03.24 13:11, Daniel Gustafsson wrote: Attached is a fresh rebase with only minor cosmetic touch-ups which I would like to go ahead with during this CF. Peter: does this address the comments you had on translation and code duplication? Yes, this looks good.

Re: Reducing connection overhead in pg_upgrade compat check phase

2024-03-18 Thread Daniel Gustafsson
Attached is a fresh rebase with only minor cosmetic touch-ups which I would like to go ahead with during this CF. Peter: does this address the comments you had on translation and code duplication? -- Daniel Gustafsson v15-0001-pg_upgrade-run-all-data-type-checks-per-connecti.patch

Re: Reducing connection overhead in pg_upgrade compat check phase

2024-02-09 Thread Daniel Gustafsson
> On 9 Feb 2024, at 00:04, Daniel Gustafsson wrote: > >> On 8 Feb 2024, at 15:16, Daniel Gustafsson wrote: > >> One option could perhaps be to include a version number for <= comparison, >> and >> if set to zero a function pointer to a version check function must be >> provided? >> That

Re: Reducing connection overhead in pg_upgrade compat check phase

2024-02-08 Thread Daniel Gustafsson
> On 8 Feb 2024, at 15:16, Daniel Gustafsson wrote: > One option could perhaps be to include a version number for <= comparison, and > if set to zero a function pointer to a version check function must be > provided? > That would handle the simple cases in a single place without messy logic,

Re: Reducing connection overhead in pg_upgrade compat check phase

2024-02-08 Thread Daniel Gustafsson
> On 8 Feb 2024, at 11:55, Peter Eisentraut wrote: > A few more quick comments: Thanks for reviewing! > I think the .report_text assignments also need a gettext_noop(), like the > .status assignments. Done in the attached. > The type DataTypesUsageChecks is only used in check.c, so doesn't

Re: Reducing connection overhead in pg_upgrade compat check phase

2024-02-08 Thread Peter Eisentraut
On 07.02.24 14:25, Daniel Gustafsson wrote: On 6 Feb 2024, at 17:47, Daniel Gustafsson wrote: On 6 Feb 2024, at 17:32, Nathan Bossart wrote: On Fri, Feb 02, 2024 at 12:18:25AM +0530, vignesh C wrote: With no update to the thread and the patch still not applying I'm marking this as returned

Re: Reducing connection overhead in pg_upgrade compat check phase

2024-02-07 Thread Daniel Gustafsson
> On 6 Feb 2024, at 17:47, Daniel Gustafsson wrote: > >> On 6 Feb 2024, at 17:32, Nathan Bossart wrote: >> >> On Fri, Feb 02, 2024 at 12:18:25AM +0530, vignesh C wrote: >>> With no update to the thread and the patch still not applying I'm >>> marking this as returned with feedback. Please

Re: Reducing connection overhead in pg_upgrade compat check phase

2024-02-06 Thread Nathan Bossart
On Tue, Feb 06, 2024 at 05:47:56PM +0100, Daniel Gustafsson wrote: >> On 6 Feb 2024, at 17:32, Nathan Bossart wrote: >> IMHO this patch is worth trying to get into v17. I'd be happy to take it >> forward if Daniel does not intend to work on it. > > I actually had the same thought yesterday and

Re: Reducing connection overhead in pg_upgrade compat check phase

2024-02-06 Thread Daniel Gustafsson
> On 6 Feb 2024, at 17:32, Nathan Bossart wrote: > > On Fri, Feb 02, 2024 at 12:18:25AM +0530, vignesh C wrote: >> With no update to the thread and the patch still not applying I'm >> marking this as returned with feedback. Please feel free to resubmit >> to the next CF when there is a new

Re: Reducing connection overhead in pg_upgrade compat check phase

2024-02-06 Thread Nathan Bossart
On Fri, Feb 02, 2024 at 12:18:25AM +0530, vignesh C wrote: > With no update to the thread and the patch still not applying I'm > marking this as returned with feedback. Please feel free to resubmit > to the next CF when there is a new version of the patch. IMHO this patch is worth trying to get

Re: Reducing connection overhead in pg_upgrade compat check phase

2024-02-01 Thread vignesh C
On Sat, 27 Jan 2024 at 09:10, vignesh C wrote: > > On Fri, 27 Oct 2023 at 18:50, Daniel Gustafsson wrote: > > > > Attached is a v10 rebase of this patch which had undergone significant > > bitrot > > due to recent changes in the pg_upgrade check phase. This brings in the > > changes into the

Re: Reducing connection overhead in pg_upgrade compat check phase

2024-01-26 Thread vignesh C
On Fri, 27 Oct 2023 at 18:50, Daniel Gustafsson wrote: > > Attached is a v10 rebase of this patch which had undergone significant bitrot > due to recent changes in the pg_upgrade check phase. This brings in the > changes into the proposed structure without changes to queries, with no >

Re: Reducing connection overhead in pg_upgrade compat check phase

2023-10-27 Thread Daniel Gustafsson
Attached is a v10 rebase of this patch which had undergone significant bitrot due to recent changes in the pg_upgrade check phase. This brings in the changes into the proposed structure without changes to queries, with no additional changes to the proposed functionality. Testing with a

Re: Reducing connection overhead in pg_upgrade compat check phase

2023-09-14 Thread Daniel Gustafsson
> On 13 Sep 2023, at 16:12, Peter Eisentraut wrote: > The alignment of this output looks a bit funny: > > ... > Checking for prepared transactionsok > Checking for contrib/isn with bigint-passing mismatch ok > Checking for data type usage

Re: Reducing connection overhead in pg_upgrade compat check phase

2023-09-13 Thread Peter Eisentraut
On 31.08.23 23:34, Daniel Gustafsson wrote: On 12 Jul 2023, at 01:36, Nathan Bossart wrote: On Wed, Jul 12, 2023 at 12:43:14AM +0200, Daniel Gustafsson wrote: I did have coffee before now, but only found time to actually address this now so here is a v7 with just that change and a fresh

Re: Reducing connection overhead in pg_upgrade compat check phase

2023-08-31 Thread Daniel Gustafsson
> On 12 Jul 2023, at 01:36, Nathan Bossart wrote: > > On Wed, Jul 12, 2023 at 12:43:14AM +0200, Daniel Gustafsson wrote: >> I did have coffee before now, but only found time to actually address this >> now >> so here is a v7 with just that change and a fresh rebase. > > Thanks. I think the

Re: Reducing connection overhead in pg_upgrade compat check phase

2023-07-11 Thread Nathan Bossart
On Wed, Jul 12, 2023 at 12:43:14AM +0200, Daniel Gustafsson wrote: > I did have coffee before now, but only found time to actually address this now > so here is a v7 with just that change and a fresh rebase. Thanks. I think the patch is in decent shape. -- Nathan Bossart Amazon Web Services:

Re: Reducing connection overhead in pg_upgrade compat check phase

2023-07-11 Thread Daniel Gustafsson
> On 11 Jul 2023, at 01:26, Daniel Gustafsson wrote: >> On 11 Jul 2023, at 01:09, Nathan Bossart wrote: >> I think we need to tmp++ somewhere here. > > Yuk, yes, will fix when caffeinated. Thanks. I did have coffee before now, but only found time to actually address this now so here is a v7

Re: Reducing connection overhead in pg_upgrade compat check phase

2023-07-10 Thread Daniel Gustafsson
> On 11 Jul 2023, at 01:09, Nathan Bossart wrote: > On Mon, Jul 10, 2023 at 04:43:23PM +0200, Daniel Gustafsson wrote: >>> +static int n_data_types_usage_checks = 7; >>> >>> Can we determine this programmatically so that folks don't need to remember >>> to update it? >> >> Fair point, I've

Re: Reducing connection overhead in pg_upgrade compat check phase

2023-07-10 Thread Nathan Bossart
On Mon, Jul 10, 2023 at 04:43:23PM +0200, Daniel Gustafsson wrote: >> On 8 Jul 2023, at 23:43, Nathan Bossart wrote: >> Since "script" will be NULL and "db_used" will be false in the first >> iteration of the loop, couldn't we move this stuff to before the loop? > > We could. It's done this way

Re: Reducing connection overhead in pg_upgrade compat check phase

2023-07-10 Thread Daniel Gustafsson
> On 8 Jul 2023, at 23:43, Nathan Bossart wrote: Thanks for reviewing! > Since "script" will be NULL and "db_used" will be false in the first > iteration of the loop, couldn't we move this stuff to before the loop? We could. It's done this way to match how all the other checks are performing

Re: Reducing connection overhead in pg_upgrade compat check phase

2023-07-08 Thread Nathan Bossart
Thanks for the new patch. On Thu, Jul 06, 2023 at 05:58:33PM +0200, Daniel Gustafsson wrote: >> On 4 Jul 2023, at 21:08, Nathan Bossart wrote: >> Also, I think it would be worth breaking check_for_data_types_usage() into >> a few separate functions (or doing some other similar refactoring) to >>

Re: Reducing connection overhead in pg_upgrade compat check phase

2023-07-06 Thread Daniel Gustafsson
> On 4 Jul 2023, at 21:08, Nathan Bossart wrote: > > I put together a rebased version of the patch for cfbot. Thanks for doing that, much appreciated! I was busy looking at other peoples patches and hadn't gotten to my own yet =) > On 13 Mar 2023, at 19:21, Nathan Bossart wrote: > I noticed

Re: Reducing connection overhead in pg_upgrade compat check phase

2023-07-04 Thread Nathan Bossart
I put together a rebased version of the patch for cfbot. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com >From ee5805dc450f081b77ae3a7df315ceafb6ccc5e1 Mon Sep 17 00:00:00 2001 From: Daniel Gustafsson Date: Mon, 13 Mar 2023 14:46:24 +0100 Subject: [PATCH v4 1/1] pg_upgrade: run

Re: Reducing connection overhead in pg_upgrade compat check phase

2023-03-13 Thread Nathan Bossart
On Mon, Mar 13, 2023 at 03:10:58PM +0100, Daniel Gustafsson wrote: > The attached v3 is a rebase to handle conflicts and with the above comments > adressed. Thanks for the new version of the patch. I noticed that git-am complained when I applied the patch: Applying: pg_upgrade: run all data

Re: Reducing connection overhead in pg_upgrade compat check phase

2023-03-13 Thread Daniel Gustafsson
> On 23 Feb 2023, at 15:12, Daniel Gustafsson wrote: > >> On 22 Feb 2023, at 20:20, Nathan Bossart wrote: > >> One thing I noticed is that the >> "failed check" log is only printed once, even if multiple data type checks >> failed. I believe this is because this message uses PG_STATUS. If I

Re: Reducing connection overhead in pg_upgrade compat check phase

2023-02-23 Thread Daniel Gustafsson
> On 22 Feb 2023, at 20:20, Nathan Bossart wrote: > One thing I noticed is that the > "failed check" log is only printed once, even if multiple data type checks > failed. I believe this is because this message uses PG_STATUS. If I > change it to PG_REPORT, all of the "failed check" messages

Re: Reducing connection overhead in pg_upgrade compat check phase

2023-02-22 Thread Nathan Bossart
On Wed, Feb 22, 2023 at 10:37:35AM +0100, Daniel Gustafsson wrote: >> On 18 Feb 2023, at 06:46, Nathan Bossart wrote: >> The last 4 are for supported versions and, from a very >> quick glance, seem possible to consolidate. That would bring us to a total >> of 11 separate loops that we could

Re: Reducing connection overhead in pg_upgrade compat check phase

2023-02-22 Thread Daniel Gustafsson
> On 18 Feb 2023, at 06:46, Nathan Bossart wrote: > > On Fri, Feb 17, 2023 at 10:44:49PM +0100, Daniel Gustafsson wrote: >> When adding a check to pg_upgrade a while back I noticed in a profile that >> the >> cluster compatibility check phase spend a lot of time in connectToServer. >> Some >>

Re: Reducing connection overhead in pg_upgrade compat check phase

2023-02-18 Thread Justin Pryzby
On Fri, Feb 17, 2023 at 10:44:49PM +0100, Daniel Gustafsson wrote: > When adding a check to pg_upgrade a while back I noticed in a profile that the > cluster compatibility check phase spend a lot of time in connectToServer. > Some > of this can be attributed to data type checks which each run

Re: Reducing connection overhead in pg_upgrade compat check phase

2023-02-17 Thread Nathan Bossart
On Fri, Feb 17, 2023 at 10:44:49PM +0100, Daniel Gustafsson wrote: > When adding a check to pg_upgrade a while back I noticed in a profile that the > cluster compatibility check phase spend a lot of time in connectToServer. > Some > of this can be attributed to data type checks which each run

Re: Reducing connection overhead in pg_upgrade compat check phase

2023-02-17 Thread Nathan Bossart
On Fri, Feb 17, 2023 at 10:44:49PM +0100, Daniel Gustafsson wrote: > In the trivial case, a single database, I don't see a reduction of performance > over the current approach. In a cluster with 100 (empty) databases there is a > ~15% reduction in time to run a --check pass. While it won't move

Reducing connection overhead in pg_upgrade compat check phase

2023-02-17 Thread Daniel Gustafsson
When adding a check to pg_upgrade a while back I noticed in a profile that the cluster compatibility check phase spend a lot of time in connectToServer. Some of this can be attributed to data type checks which each run serially in turn connecting to each database to run the check, and this seemed