Re: Can we get rid of repeated queries from pg_dump?

2021-08-31 Thread hubert depesz lubaczewski
On Tue, Aug 31, 2021 at 03:06:25PM -0400, Tom Lane wrote: > Agreed, but as I said upthread, fixing that looks like it will be > rather invasive. Meanwhile, I went ahead and pushed the two > simple improvements discussed so far. Great. Thank you very much. Best regards, depesz

RE: Added schema level support for publication.

2021-08-31 Thread tanghy.f...@fujitsu.com
> On Monday, August 30, 2021 11:28 PM vignesh C wrote: > > I have fixed these comments as part of v23 patch attached at [1]. > [1] - https://www.postgresql.org/message- > id/CALDaNm0xmqJeQEfV5Wnj2BawM%3DsdFdfOXz5N%2BgGG3WB6k9%3Dtdw > %40mail.gmail.com > Thanks for your new patch. Here are some

Re: Possible missing segments in archiving on standby

2021-08-31 Thread Fujii Masao
On 2021/09/01 12:12, Kyotaro Horiguchi wrote: Putting aside the issue C, it would work as far as recovery is not paused or delayed. Although simply doing that means we run additional and a bit) wasteful XLogArchiveCheckDone() in most cases, It's hard to imagine moving the responsibility to

Re: Multi-Column List Partitioning

2021-08-31 Thread Amit Langote
Hi Nitin, On Tue, Aug 31, 2021 at 8:02 PM Nitin Jadhav wrote: > The attached patch also fixes the above comments. I noticed that multi-column list partitions containing NULLs don't work correctly with partition pruning yet. create table p0 (a int, b text, c bool) partition by list (a, b, c);

Re: Allow escape in application_name (was: [postgres_fdw] add local pid to fallback_application_name)

2021-08-31 Thread Fujii Masao
On 2021/08/31 16:11, kuroda.hay...@fujitsu.com wrote: Dear Fujii-san, I attached new version, that almost all codes moved from libpq to postgres_fdw. Thanks for updating the patch! Can we split the patch into two as follows? If so, we can review and commit them one by one. #1. Add

Re: pg_receivewal starting position

2021-08-31 Thread Bharath Rupireddy
On Mon, Aug 30, 2021 at 3:26 PM Ronan Dunklau wrote: > > 7) How about we let pg_receivewal use READ_REPLICATION_SLOT as an option? > > From my point of view, I already expected it to use something like that when > using a replication slot. Maybe an option to turn it off could be useful ? IMO,

Re: pgstat_send_connstats() introduces unnecessary timestamp and UDP overhead

2021-08-31 Thread Andres Freund
On 2021-08-27 13:57:45 +0900, Michael Paquier wrote: > On Wed, Aug 25, 2021 at 01:20:03AM -0700, Andres Freund wrote: > > On 2021-08-25 12:51:58 +0900, Michael Paquier wrote: > > As I said before, this ship has long sailed: > > > > typedef struct PgStat_MsgTabstat > > { > > PgStat_MsgHdr

Re: pgstat_send_connstats() introduces unnecessary timestamp and UDP overhead

2021-08-31 Thread Andres Freund
On 2021-09-01 05:39:14 +0200, Laurenz Albe wrote: > On Tue, 2021-08-31 at 18:55 -0700, Andres Freund wrote: > > > > On Tue, Aug 31, 2021 at 04:55:35AM +0200, Laurenz Albe wrote:In the > > > > view of that, how about doubling PGSTAT_STAT_INTERVAL to 1000 > > > > > > > milliseconds?  That would

Re: prevent immature WAL streaming

2021-08-31 Thread Kyotaro Horiguchi
At Tue, 31 Aug 2021 20:15:24 -0700, Andres Freund wrote in > Hi, > > On 2021-09-01 11:34:34 +0900, Fujii Masao wrote: > > On 2021/09/01 0:53, Andres Freund wrote: > > > Of course, we need to be careful to not weaken WAL validity checking too > > > much. How about the following: > > > > > > If

Re: improve pg_receivewal code

2021-08-31 Thread Bharath Rupireddy
On Mon, Aug 30, 2021 at 1:02 PM Bharath Rupireddy wrote: > > Hi, > > I see there's a scope to do following improvements to pg_receivewal.c: > > 1) Fetch the server system identifier in the first RunIdentifySystem > call and use it to identify(via pg_receivewal's ReceiveXlogStream) any >

Re: pgstat_send_connstats() introduces unnecessary timestamp and UDP overhead

2021-08-31 Thread Laurenz Albe
On Wed, 2021-09-01 at 10:33 +0900, Michael Paquier wrote: > On Tue, Aug 31, 2021 at 04:55:35AM +0200, Laurenz Albe wrote: > > In the view of that, how about doubling PGSTAT_STAT_INTERVAL to 1000 > > milliseconds? > > Perhaps we could do that.  Now, increasing an interval for the sake of >

Re: pgstat_send_connstats() introduces unnecessary timestamp and UDP overhead

2021-08-31 Thread Laurenz Albe
On Tue, 2021-08-31 at 18:55 -0700, Andres Freund wrote: > > > On Tue, Aug 31, 2021 at 04:55:35AM +0200, Laurenz Albe wrote:In the view > > > of that, how about doubling PGSTAT_STAT_INTERVAL to 1000 > > > > > milliseconds?  That would mean slightly less up-to-date statistics, but > > > I doubt

Re: Replication slot drop message is sent after pgstats shutdown.

2021-08-31 Thread Andres Freund
Hi, On 2021-09-01 10:05:18 +0900, Masahiko Sawada wrote: > On Wed, Sep 1, 2021 at 2:39 AM Andres Freund wrote: > > On 2021-08-31 18:34:12 +0900, Kyotaro Horiguchi wrote: > > > At Tue, 31 Aug 2021 17:14:45 +0900, Masahiko Sawada > > > wrote in > > > > On Tue, Aug 31, 2021 at 2:34 PM Andres

Re: Added schema level support for publication.

2021-08-31 Thread Greg Nancarrow
On Tue, Aug 31, 2021 at 8:57 PM Amit Kapila wrote: > > I find the way it is implemented to be more intuitive as that gives > users more flexibility to retain certain tables from the schema and > appears to be exactly what users intended by the command. I don't > think finding duplicates among

Re: prevent immature WAL streaming

2021-08-31 Thread Andres Freund
Hi, On 2021-09-01 11:34:34 +0900, Fujii Masao wrote: > On 2021/09/01 0:53, Andres Freund wrote: > > Of course, we need to be careful to not weaken WAL validity checking too > > much. How about the following: > > > > If we're "aborting" a continued record, we set XLP_FIRST_IS_ABORTED_PARTIAL > >

Re: Possible missing segments in archiving on standby

2021-08-31 Thread Kyotaro Horiguchi
At Tue, 31 Aug 2021 23:23:27 +0900, Fujii Masao wrote in > > > On 2021/08/31 16:35, Kyotaro Horiguchi wrote: > > I'm not sure which is simpler, but it works except for B, the case of > > a long-jump by a segment switch. When a segment switch happens, > > walsender sends filling zero-pages

Re: Is it worth pushing conditions to sublink/subplan?

2021-08-31 Thread shawn wang
I tested it the way you said and increased the number of sub-tables. I created a hash partition table of 1000 sub-tables. Test according to your first SQL, the optimizer cuts the unnecessary sub-tables well. You can see the plan: postgres=# explain analyze postgres-# select temp.p1 from

Re: prevent immature WAL streaming

2021-08-31 Thread Fujii Masao
On 2021/09/01 0:53, Andres Freund wrote: I was thinking that on a normal WAL write we'd do nothing. Instead we would have dedicated code at the end of recovery that, if the WAL ends in a partial record, changes the page following the "valid" portion of the WAL to indicate that an incomplete

Re: archive status ".ready" files may be created too early

2021-08-31 Thread alvhe...@alvh.no-ip.org
On 2021-Sep-01, Michael Paquier wrote: > That's about 515e3d8, right? Yes. > I have not looked in details at what you have here, but this produces > a compilation warning on Windows for me with this part of the patch: This seems a tiny speck in a sea of bogosity. If you want to silence the

Re: pgstat_send_connstats() introduces unnecessary timestamp and UDP overhead

2021-08-31 Thread Andres Freund
Hi, On August 31, 2021 6:33:15 PM PDT, Michael Paquier wrote: >On Tue, Aug 31, 2021 at 04:55:35AM +0200, Laurenz Albe wrote: >> In the view of that, how about doubling PGSTAT_STAT_INTERVAL to 1000 >> milliseconds? That would mean slightly less up-to-date statistics, but >> I doubt that that

Re: archive status ".ready" files may be created too early

2021-08-31 Thread Michael Paquier
On Tue, Aug 31, 2021 at 08:52:05PM -0400, alvhe...@alvh.no-ip.org wrote: > Yeah, that's becoming my conclusion too -- undo that, and start from > scratch using the other idea. That's about 515e3d8, right? I have not looked in details at what you have here, but this produces a compilation warning

Re: Postgres Win32 build broken?

2021-08-31 Thread Michael Paquier
On Tue, Aug 31, 2021 at 07:49:40PM -0300, Ranier Vilela wrote: > I'm not a perl specialist and it seems to me that the Win32 build is broken. > The Win32 build is still important because of the 32-bit clients still in > use. > I'm investigating the problem. Being able to see the command you are

Re: pgstat_send_connstats() introduces unnecessary timestamp and UDP overhead

2021-08-31 Thread Michael Paquier
On Tue, Aug 31, 2021 at 04:55:35AM +0200, Laurenz Albe wrote: > In the view of that, how about doubling PGSTAT_STAT_INTERVAL to 1000 > milliseconds? That would mean slightly less up-to-date statistics, but > I doubt that that will be a problem. And it should even out the increase > in statistics

RE: Added schema level support for publication.

2021-08-31 Thread houzj.f...@fujitsu.com
From Mon, Aug 30, 2021 11:26 PM vignesh C wrote: > On Mon, Aug 30, 2021 at 9:10 AM houzj.f...@fujitsu.com > wrote: > > > > 5) > > + if (list_length(pubobj->name) == 1 && > > + (strcmp(relname, "CURRENT_SCHEMA") == > 0)) > > +

Re: Replication slot drop message is sent after pgstats shutdown.

2021-08-31 Thread Masahiko Sawada
On Wed, Sep 1, 2021 at 2:39 AM Andres Freund wrote: > > Hi, > > On 2021-08-31 18:34:12 +0900, Kyotaro Horiguchi wrote: > > At Tue, 31 Aug 2021 17:14:45 +0900, Masahiko Sawada > > wrote in > > > On Tue, Aug 31, 2021 at 2:34 PM Andres Freund wrote: > > > > > > > > On 2021-08-31 14:22:39 +0900,

Re: archive status ".ready" files may be created too early

2021-08-31 Thread alvhe...@alvh.no-ip.org
On 2021-Aug-31, Andres Freund wrote: > Maybe, but this is getting uglier and uglier. > > I think patch should be reverted. It's not in a state that's appropriate for > the backbranches. Yeah, that's becoming my conclusion too -- undo that, and start from scratch using the other idea. --

Re: Add jsonlog log_destination for JSON server logs

2021-08-31 Thread Michael Paquier
On Tue, Aug 31, 2021 at 11:34:56AM -0400, Sehrope Sarkuni wrote: > The second commit adds a TAP test for log_destination "csvlog". This was > done to both confirm that the previous change didn't break anything and as > a skeleton for the test in the next commit. +note "Before sleep";

Re: Proposal: More structured logging

2021-08-31 Thread Michael Paquier
On Tue, Aug 31, 2021 at 10:46:30AM -0400, Sehrope Sarkuni wrote: > It needed some cleanup due to bit rot, but it now builds and works atop > master. I'll post it in its own thread. Thanks. -- Michael signature.asc Description: PGP signature

Re: pg_receivewal starting position

2021-08-31 Thread Michael Paquier
On Mon, Aug 30, 2021 at 11:55:42AM +0200, Ronan Dunklau wrote: > Le vendredi 27 août 2021, 05:44:32 CEST Michael Paquier a écrit : >> + if (slot == NULL || !slot->in_use) >> >>

Re: archive status ".ready" files may be created too early

2021-08-31 Thread Andres Freund
On 2021-08-31 23:31:15 +, Bossart, Nathan wrote: > On 8/31/21, 1:30 PM, "Andres Freund" wrote: > > On 2021-08-31 18:09:36 +, Bossart, Nathan wrote: > >> What appears to happen in this case is that bgwriter eventually creates a > >> xl_running_xacts record and nudges walwriter to flush it

Postgres Win32 build broken?

2021-08-31 Thread Ranier Vilela
Hi hackers, I'm not a perl specialist and it seems to me that the Win32 build is broken. The Win32 build is still important because of the 32-bit clients still in use. I'm investigating the problem. ---

Re: RFC: Improve CPU cache locality of syscache searches

2021-08-31 Thread Andres Freund
Hi, On 2021-08-31 15:06:32 -0400, John Naylor wrote: > Were you thinking in terms of passing the type oid in parameters, like this? > > HeapTuple > SearchCatCache1(CatCache *cache, Datum v1, Oid t1) > { > return SearchCatCacheInternal(cache, 1, v1, t1, 0, 0, 0, 0, 0, 0); > } > > And then

Re: [PoC] Federated Authn/z with OAUTHBEARER

2021-08-31 Thread Jacob Champion
On Fri, 2021-08-27 at 11:32 +0900, Michael Paquier wrote: > Now if you'd really wish to > stress that without relying on the backend, one simple way is to use > pgbench -C -n with a mostly-empty script (one meta-command) coupled > with some profiling. Ah, thanks! I'll add that to the toolbox.

Re: log_autovacuum in Postgres 14 -- ordering issue

2021-08-31 Thread Peter Geoghegan
On Wed, Aug 25, 2021 at 5:23 PM Alvaro Herrera wrote: > Ooh, this was illuminating -- thanks for explaining. TBH I would have > been very confused if asked to explain what that log line meant; and now > that I know what it means, I am even more convinced that we need to work > harder at it :-) >

Re: archive status ".ready" files may be created too early

2021-08-31 Thread Andres Freund
Hi, On 2021-08-31 18:09:36 +, Bossart, Nathan wrote: > On 8/31/21, 10:21 AM, "Andres Freund" wrote: > > What would trigger the flushing? We don't write out partially filled pages > > unless > > a) we're explicitly flushing an LSN on the partial page (e.g. because a > >synchronous commit

[PATCH] Support pg_ident mapping for LDAP

2021-08-31 Thread Jacob Champion
Hello, There was a brief discussion [1] back in February on allowing user mapping for LDAP, in order to open up some more complex authorization logic (and slightly reduce the need for LDAP-to-Postgres user synchronization). Attached is an implementation of this that separates the LDAP

Re: mark the timestamptz variant of date_bin() as stable

2021-08-31 Thread Tom Lane
John Naylor writes: > On Tue, Aug 31, 2021 at 3:07 PM Tom Lane wrote: >> Yeah, you need to bump catversion. > Done, thanks for confirming. For future reference --- I think it's potentially confusing to use the same catversion number in different branches, except for the short time after a new

Re: CFM for september commitfest

2021-08-31 Thread Daniel Gustafsson
> On 27 Aug 2021, at 01:16, Jaime Casanova wrote: > Do we already have a ${subject}? Otherwise I could offer my self. AFAICT from searching the archive there has been no other volunteers, and the CF starts quite soon so if you’re still up for it then thanks for picking up the task! > If anyone

Re: mark the timestamptz variant of date_bin() as stable

2021-08-31 Thread John Naylor
On Tue, Aug 31, 2021 at 3:07 PM Tom Lane wrote: > > John Naylor writes: > > (Starting a new thread for greater visibility) > > The attached is a fairly straightforward correction. I did want to make > > sure it was okay to bump the catversion in the PG14 branch also. I've seen > > fixes where

Re: mark the timestamptz variant of date_bin() as stable

2021-08-31 Thread Tom Lane
John Naylor writes: > (Starting a new thread for greater visibility) > The attached is a fairly straightforward correction. I did want to make > sure it was okay to bump the catversion in the PG14 branch also. I've seen > fixes where doing that during beta was in question. Yeah, you need to bump

Re: RFC: Improve CPU cache locality of syscache searches

2021-08-31 Thread John Naylor
On Fri, Aug 27, 2021 at 3:42 PM Andres Freund wrote: > > Hi, > > On 2021-08-19 19:10:37 -0400, John Naylor wrote: > > I've made a small step in this direction in the attached. It uses a > > template approach to generate type-specific SearchCatCache* functions, for > > now only the 4-key ones.

Re: Can we get rid of repeated queries from pg_dump?

2021-08-31 Thread Tom Lane
hubert depesz lubaczewski writes: > My point in here is that potential optimizations regarding queries for > pg_proc might speed up dumps for more people - as they might use things > like postgis, but never realized that it can be much faster. Agreed, but as I said upthread, fixing that looks

Re: Bug fix for cache lookup failure for statistic_ext type

2021-08-31 Thread Mark Dilger
> On Aug 31, 2021, at 10:50 AM, Tomas Vondra > wrote: > > I've pushed a fix for this. And then a fix for the fix :-( because I forgot > about the rule that role names in regression tests should start with regress_ > prefix, so animals enforcing this failed. Thanks! — Mark Dilger

mark the timestamptz variant of date_bin() as stable

2021-08-31 Thread John Naylor
(Starting a new thread for greater visibility) The attached is a fairly straightforward correction. I did want to make sure it was okay to bump the catversion in the PG14 branch also. I've seen fixes where doing that during beta was in question. -- John Naylor EDB: http://www.enterprisedb.com

Re: archive status ".ready" files may be created too early

2021-08-31 Thread Bossart, Nathan
On 8/31/21, 10:21 AM, "Andres Freund" wrote: > What would trigger the flushing? We don't write out partially filled pages > unless > a) we're explicitly flushing an LSN on the partial page (e.g. because a >synchronous commit record resides on it) > b) there's an async commit (i.e. commit with

Re: Bug fix for cache lookup failure for statistic_ext type

2021-08-31 Thread Tomas Vondra
On 8/5/21 12:32 AM, Tomas Vondra wrote: On 8/5/21 12:03 AM, Mark Dilger wrote: Hackers, You can easily get a cache lookup failure by changing the regression tests as included in this small patch.  The failure looks thus: +COMMENT ON STATISTICS ab1_a_b_stats IS 'new comment'; +CREATE ROLE

Re: Patch: shouldn't timezone(text, timestamp[tz]) be STABLE?

2021-08-31 Thread Tom Lane
John Naylor writes: > I believe it's been discussed before that it'd be safer if pg_proc.dat had > the same defaults as CREATE FUNCTION, and this is further evidence for that. Yeah, maybe so. It'd make the .dat file quite a bit bigger, but maybe less mistake-prone.

Re: Replication slot drop message is sent after pgstats shutdown.

2021-08-31 Thread Andres Freund
Hi, On 2021-08-31 18:34:12 +0900, Kyotaro Horiguchi wrote: > At Tue, 31 Aug 2021 17:14:45 +0900, Masahiko Sawada > wrote in > > On Tue, Aug 31, 2021 at 2:34 PM Andres Freund wrote: > > > > > > On 2021-08-31 14:22:39 +0900, Masahiko Sawada wrote: > > > > You mean to move only the part of

Re: Patch: shouldn't timezone(text, timestamp[tz]) be STABLE?

2021-08-31 Thread John Naylor
On Mon, Aug 30, 2021 at 12:58 PM Tom Lane wrote: > > Aleksander Alekseev writes: > > Got it. But in this case, what's your opinion on the differences between > > date_trunc() and timezone()? Shouldn't date_trunc() be always IMMUTABLE as > > well? > > No, because date_trunc depends on the current

Re: suboverflowed subtransactions concurrency performance optimize

2021-08-31 Thread Zhihong Yu
On Mon, Aug 30, 2021 at 1:43 AM Pengchengliu wrote: > Hi hackers, > I wrote a patch to resolve the subtransactions concurrency performance > problems when suboverflowed. > > When we use more than PGPROC_MAX_CACHED_SUBXIDS(64) subtransactions per > transaction concurrency, it will lead to

Re: archive status ".ready" files may be created too early

2021-08-31 Thread Andres Freund
Hi, On 2021-08-31 17:01:31 +, Bossart, Nathan wrote: > > If the record ending at s4 + 10 isn't an async commit (and thus > > XLogCtl->asyncXactLSN is smaller), and there are no further records, we can > > end up waiting effectively forever for s2 (and s3) to be archived. If all > > other

Re: psql: \dl+ to list large objects privileges

2021-08-31 Thread Pavel Luzanov
On 31.08.2021 17:35, Daniel Gustafsson wrote: Please do, if it was interesting enough for you to write it, it’s interesting enough to be in the commitfest. Thanks, added to the commitfest. Pavel Luzanov Postgres Professional: https://postgrespro.com The Russian Postgres Company

Re: Improve logging when using Huge Pages

2021-08-31 Thread Fujii Masao
On 2021/08/31 15:57, Julien Rouhaud wrote: On Tue, Aug 31, 2021 at 1:37 PM Shinoda, Noriyoshi (PN Japan FSIP) wrote: In the current version, when GUC huge_pages=try, which is the default setting, no log is output regardless of the success or failure of the HugePages acquisition. If you

Re: Returning to Postgres community work

2021-08-31 Thread Gurjeet Singh
On Tue, Aug 31, 2021 at 8:04 AM Alvaro Herrera wrote: > > On 2021-Aug-30, Gurjeet Singh wrote: > > > I'm very happy to announce that I now work for Supabase [1]. They > > have hired me so that I can participate in, and contribute to the > > Postgres community. > > Hey Gurjeet, welcome back.

Re: archive status ".ready" files may be created too early

2021-08-31 Thread Bossart, Nathan
On 8/31/21, 12:44 AM, "Andres Freund" wrote: > If there's now a flush request including all of s3, we'll have the following > sequence of notifies: > > NotifySegmentsReadyForArchive(s1) > nothing happens, smaller than s1+10 > > NotifySegmentsReadyForArchive(s2) > earliestSegBoundary = s4 >

Re: [PATCH] pg_permissions

2021-08-31 Thread Peter Eisentraut
On 11.03.21 08:00, Joel Jacobson wrote: Do we prefer "pg_permissions" or "pg_privileges"? pg_privileges would be better. "Permissions" is not an SQL term.

Re: Fix around conn_duration in pgbench

2021-08-31 Thread Fujii Masao
On 2021/08/31 16:56, Fabien COELHO wrote: I would think we should leave as it is for pg13 and before to not surprise users. Ok. Thank you for your opinion. I also agree with not changing the behavior of long-stable branches, and I think this is the same opinion as Fujii-san. Attached is

Re: prevent immature WAL streaming

2021-08-31 Thread Andres Freund
Hi, On 2021-08-31 09:56:30 -0400, Alvaro Herrera wrote: > On 2021-Aug-30, Andres Freund wrote: > > I think a better approach might be to handle this on the WAL layout > > level. What if we never overwrite partial records but instead just > > skipped over them during decoding? > > Maybe this is a

Add jsonlog log_destination for JSON server logs

2021-08-31 Thread Sehrope Sarkuni
Hi, This patch adds a new log_destination, "jsonlog", that writes log entries as lines of JSON. It was originally started by David Fetter using the jsonlog module by Michael Paquier ( https://github.com/michaelpq/pg_plugins/blob/master/jsonlog/jsonlog.c) as a basis for how to serialize the log

Re: Added missing invalidations for all tables publication

2021-08-31 Thread vignesh C
On Tue, Aug 31, 2021 at 2:00 PM Kyotaro Horiguchi wrote: > > At Tue, 31 Aug 2021 08:31:05 +0530, vignesh C wrote in > > On Tue, Aug 31, 2021 at 7:40 AM houzj.f...@fujitsu.com > > wrote: > > Thanks for the comment, I have slightly modified the test case which > > will fail without the patch.

Re: [PATCH] Allow multiple recursive self-references

2021-08-31 Thread Denis Hirn
The documentation was not up to date anymore with the most recent changes. This version of the patch fixes that. Best, –– Denis 0008-Allow-multiple-recursive-self-references.patch Description: Binary data

Re: Proposal: More structured logging

2021-08-31 Thread Sehrope Sarkuni
On Tue, Aug 24, 2021 at 7:22 PM Michael Paquier wrote: > From a code perspective, and while on it, we could split a bit elog.c > and move the log entries generated for each format into their own > file. That would be cleaner for CSV and JSON. As a whole I don't > have an objection with moving

Re: psql: \dl+ to list large objects privileges

2021-08-31 Thread Daniel Gustafsson
> On 31 Aug 2021, at 16:14, Pavel Luzanov wrote: > If it's interesting, I can add the patch to commitfest. Please do, if it was interesting enough for you to write it, it’s interesting enough to be in the commitfest. -- Daniel Gustafsson https://vmware.com/

Re: Possible missing segments in archiving on standby

2021-08-31 Thread Fujii Masao
On 2021/08/31 16:35, Kyotaro Horiguchi wrote: I'm not sure which is simpler, but it works except for B, the case of a long-jump by a segment switch. When a segment switch happens, walsender sends filling zero-pages but even if walreceiver is terminated before the segment is completed,

psql: \dl+ to list large objects privileges

2021-08-31 Thread Pavel Luzanov
Hello, While working through the documentation I found an empty cell in the table for the large objects privilege display with the psql command [1]. And indeed the \dl command does not show privileges. And there is no modifier + for it. This patch adds a + modifier to the \dl command and

Re: Added schema level support for publication.

2021-08-31 Thread vignesh C
On Tue, Aug 31, 2021 at 4:27 PM Amit Kapila wrote: > > On Tue, Aug 31, 2021 at 10:50 AM Greg Nancarrow wrote: > > > > On Tue, Aug 31, 2021 at 1:41 PM vignesh C wrote: > > > > > > > > > > > I notice that "CREATE PUBLICATION pub1 FOR ALL TABLES IN SCHEMA sc1, > > > > TABLE sc1.test;" maintains

Re: prevent immature WAL streaming

2021-08-31 Thread Alvaro Herrera
On 2021-Aug-30, Andres Freund wrote: > I'm doubtful that the approach of adding awareness of record boundaries > is a good path to go down: Honestly, I do not like it one bit and if I can avoid relying on them while making the whole thing work correctly, I am happy. Clearly it wasn't a problem

Re: pgsql: Avoid using ambiguous word "positive" in error message.

2021-08-31 Thread Andrew Dunstan
On 8/30/21 10:19 AM, Robert Haas wrote: > On Mon, Aug 30, 2021 at 10:16 AM Tom Lane wrote: >> Robert Haas writes: >>> The new style seems good, but I don't really agree that "positive" and >>> "non-negative" are ambiguous. "positive" means >0 and "non-negative" >>> means >= 0, because 0 is

Re: pg_receivewal starting position

2021-08-31 Thread Ronan Dunklau
Le mardi 31 août 2021, 13:17:22 CEST Bharath Rupireddy a écrit : > On Mon, Aug 30, 2021 at 3:26 PM Ronan Dunklau wrote: > > Thank you for this review ! Please see the other side of the thread where > > I > > answered Michael Paquier with a new patchset, which includes some of your > > proposed

Re: Added schema level support for publication.

2021-08-31 Thread Amit Kapila
On Tue, Aug 31, 2021 at 9:15 AM vignesh C wrote: > > On Mon, Aug 30, 2021 at 12:12 PM Amit Kapila wrote: > > > > > > Okay, I got it but let's add few comments in the code related to it. > > Also, I noticed that the code in InvalidatePublicationRels() already > > exists in

Re: Failure of subscription tests with topminnow

2021-08-31 Thread Ajin Cherian
On Tue, Aug 31, 2021 at 3:47 PM Masahiko Sawada wrote: > > On Tue, Aug 31, 2021 at 12:11 PM Amit Kapila wrote: > > > > On Mon, Aug 30, 2021 at 5:48 PM Ajin Cherian wrote: > > > > > > On Mon, Aug 30, 2021 at 7:52 PM Amit Kapila > > > wrote: > > > > > > I have made the above changes on HEAD. >

Re: pg_receivewal starting position

2021-08-31 Thread Bharath Rupireddy
On Tue, Aug 31, 2021 at 4:47 PM Bharath Rupireddy wrote: > > On Mon, Aug 30, 2021 at 3:26 PM Ronan Dunklau wrote: > > Thank you for this review ! Please see the other side of the thread where I > > answered Michael Paquier with a new patchset, which includes some of your > > proposed

Re: Returning to Postgres community work

2021-08-31 Thread Joe Conway
On 8/31/21 1:53 AM, Gurjeet Singh wrote: I'm very happy to announce that I now work for Supabase [1]. They have hired me so that I can participate in, and contribute to the Postgres community. Welcome back! :-) Joe -- Crunchy Data - http://crunchydata.com PostgreSQL Support for Secure

Re: Pg stuck at 100% cpu, for multiple days

2021-08-31 Thread hubert depesz lubaczewski
On Tue, Aug 31, 2021 at 04:00:14PM +0530, Amit Kapila wrote: > One possibility could be there are quite a few DDLs happening in this > application at some particular point in time which can lead to high While not impossible, I'd rather say it's not very likely. We don't use temporary tables, and

Re: pg_receivewal starting position

2021-08-31 Thread Bharath Rupireddy
On Mon, Aug 30, 2021 at 3:26 PM Ronan Dunklau wrote: > Thank you for this review ! Please see the other side of the thread where I > answered Michael Paquier with a new patchset, which includes some of your > proposed modifications. Thanks for the updated patches. Here are some comments on

Re: Added schema level support for publication.

2021-08-31 Thread Amit Kapila
On Tue, Aug 31, 2021 at 10:50 AM Greg Nancarrow wrote: > > On Tue, Aug 31, 2021 at 1:41 PM vignesh C wrote: > > > > > > > > I notice that "CREATE PUBLICATION pub1 FOR ALL TABLES IN SCHEMA sc1, > > > TABLE sc1.test;" maintains the table separately and results in the > > > following in the \dRp+

Re: Pg stuck at 100% cpu, for multiple days

2021-08-31 Thread Amit Kapila
On Tue, Aug 31, 2021 at 11:41 AM hubert depesz lubaczewski wrote: > > On Mon, Aug 30, 2021 at 09:09:20PM +0200, Laurenz Albe wrote: > > On Mon, 2021-08-30 at 17:18 +0200, hubert depesz lubaczewski wrote: > > > The thing is - I can't close it with pg_terminate_backend(), and I'd > > > rather not

Re: perlcritic: prohibit map and grep in void conext

2021-08-31 Thread Dagfinn Ilmari Mannsåker
On Tue, 31 Aug 2021, at 10:30, Daniel Gustafsson wrote: > > On 31 Aug 2021, at 06:19, Julien Rouhaud wrote: > > > > On Tue, Aug 31, 2021 at 9:23 AM Michael Paquier wrote: > >> > >> On Mon, Aug 30, 2021 at 02:27:09PM -0400, Andrew Dunstan wrote: > >>> There doesn't seem to have been much

Re: Separate out FileSet from SharedFileSet (was Re: pgsql: pgstat: Bring up pgstat in BaseInit() to fix uninitialized use o)

2021-08-31 Thread Amit Kapila
On Fri, Aug 27, 2021 at 12:04 PM Dilip Kumar wrote: > Few comments on v6-0002* = 1. -BufFileDeleteFileSet(FileSet *fileset, const char *name) +BufFileDeleteFileSet(FileSet *fileset, const char *name, bool missing_ok) { char segment_name[MAXPGPATH]; int segment = 0;

Re: Replication slot drop message is sent after pgstats shutdown.

2021-08-31 Thread Kyotaro Horiguchi
At Tue, 31 Aug 2021 17:14:45 +0900, Masahiko Sawada wrote in > On Tue, Aug 31, 2021 at 2:34 PM Andres Freund wrote: > > > > On 2021-08-31 14:22:39 +0900, Masahiko Sawada wrote: > > > You mean to move only the part of sending the message to its own > > > before_shmem_exit() callback? or move

Re: perlcritic: prohibit map and grep in void conext

2021-08-31 Thread Daniel Gustafsson
> On 31 Aug 2021, at 06:19, Julien Rouhaud wrote: > > On Tue, Aug 31, 2021 at 9:23 AM Michael Paquier wrote: >> >> On Mon, Aug 30, 2021 at 02:27:09PM -0400, Andrew Dunstan wrote: >>> There doesn't seem to have been much pushback, so let's try it and see. >> >> Okay, fine by me. > > +1 Since

Re: [PATCH] Tab completion for ALTER TABLE … ADD …

2021-08-31 Thread Dagfinn Ilmari Mannsåker
On Tue, 31 Aug 2021, at 04:20, Michael Paquier wrote: > On Mon, Aug 30, 2021 at 02:38:19PM +0100, Dagfinn Ilmari Mannsåker wrote: > > That was because I forgot to exclude all the other object types that can > > come after ADD. Attached is a patch that does that. I also moved it > > right next to

Re: Added missing invalidations for all tables publication

2021-08-31 Thread Kyotaro Horiguchi
At Tue, 31 Aug 2021 08:31:05 +0530, vignesh C wrote in > On Tue, Aug 31, 2021 at 7:40 AM houzj.f...@fujitsu.com > wrote: > Thanks for the comment, I have slightly modified the test case which > will fail without the patch. Attached v2 patch which has the changes > for the same. The test works

Re: [PATCH] Allow multiple recursive self-references

2021-08-31 Thread Denis Hirn
> I am not sure if this patch should introduce such a rewriting. I have thought about this again. I think it is reasonable that this patch introduces such a rewriting. > This well-formedness check apparently needs to be enhanced to allow for more > than two branches in the union. The new

Re: Replication slot drop message is sent after pgstats shutdown.

2021-08-31 Thread Masahiko Sawada
On Tue, Aug 31, 2021 at 2:34 PM Andres Freund wrote: > > On 2021-08-31 14:22:39 +0900, Masahiko Sawada wrote: > > You mean to move only the part of sending the message to its own > > before_shmem_exit() callback? or move ReplicationSlotRelease() and > > ReplicationSlotCleanup() from ProcKill() to

Re: Fix around conn_duration in pgbench

2021-08-31 Thread Fabien COELHO
I would think we should leave as it is for pg13 and before to not surprise users. Ok. Thank you for your opinion. I also agree with not changing the behavior of long-stable branches, and I think this is the same opinion as Fujii-san. Attached is the patch to fix to measure disconnection

Re: archive status ".ready" files may be created too early

2021-08-31 Thread Andres Freund
Hi On 2021-08-31 06:45:06 +, Bossart, Nathan wrote: > On 8/30/21, 7:39 PM, "Andres Freund" wrote: > > On 2021-08-30 22:39:04 +, Bossart, Nathan wrote: > >> If we called NotifySegmentsReadyForArchive() before we updated the > >> flush location in shared memory, we might skip nudging the

Re: Possible missing segments in archiving on standby

2021-08-31 Thread Kyotaro Horiguchi
At Tue, 31 Aug 2021 01:54:36 +0900, Fujii Masao wrote in > > > On 2020/06/30 16:55, Kyotaro Horiguchi wrote: > > Hello. > > While looking a patch, I found that a standby with archive_mode=always > > fails to archive segments under certain conditions. > > I encountered this issue, too. > >

RE: Allow escape in application_name (was: [postgres_fdw] add local pid to fallback_application_name)

2021-08-31 Thread kuroda.hay...@fujitsu.com
Dear Fujii-san, I attached new version, that almost all codes moved from libpq to postgres_fdw. Now we can accept four types of escapes. All escapes will be rewritten to connection souce's information: * application_name, * user name, * database name, and * backend's pid. These are cannot be

Re: Fix around conn_duration in pgbench

2021-08-31 Thread Yugo NAGATA
On Tue, 31 Aug 2021 15:39:18 +0900 (JST) Tatsuo Ishii wrote: > >> >> > My 0.02€: From a benchmarking perspective, ISTM that it makes sense to > >> >> > include disconnection times, which are clearly linked to connections, > >> >> > especially with -C. So I'd rather have the more meaningful

Re: Improve logging when using Huge Pages

2021-08-31 Thread Julien Rouhaud
On Tue, Aug 31, 2021 at 1:37 PM Shinoda, Noriyoshi (PN Japan FSIP) wrote: > > In the current version, when GUC huge_pages=try, which is the default > setting, no log is output regardless of the success or failure of the > HugePages acquisition. If you want to output logs, you need to set >

Re: .ready and .done files considered harmful

2021-08-31 Thread Bossart, Nathan
On 8/25/21, 4:11 AM, "Dipesh Pandit" wrote: > Please find attached patch v11. Apologies for the delay. I still intend to review this. Nathan

RE: suboverflowed subtransactions concurrency performance optimize

2021-08-31 Thread Pengchengliu
Hi Andrey, Thanks a lot for your replay and reference information. The default NUM_SUBTRANS_BUFFERS is 32. My implementation is local_cache_subtrans_pages can be adjusted dynamically. If we configure local_cache_subtrans_pages as 64, every backend use only extra 64*8192=512KB memory.

Re: Fix around conn_duration in pgbench

2021-08-31 Thread Tatsuo Ishii
>> >> > My 0.02€: From a benchmarking perspective, ISTM that it makes sense to >> >> > include disconnection times, which are clearly linked to connections, >> >> > especially with -C. So I'd rather have the more meaningful figure even >> >> > at the price of a small change in an undocumented

Re: Pg stuck at 100% cpu, for multiple days

2021-08-31 Thread hubert depesz lubaczewski
On Mon, Aug 30, 2021 at 08:15:24PM -0400, Joe Conway wrote: > It would be interesting to step through a few times to see if it is really > stuck in that loop. That would be consistent with 100% CPU and not checking > for interrupts I think. If the problem will happen again, will do my best to get

Re: Pg stuck at 100% cpu, for multiple days

2021-08-31 Thread hubert depesz lubaczewski
On Mon, Aug 30, 2021 at 09:09:20PM +0200, Laurenz Albe wrote: > On Mon, 2021-08-30 at 17:18 +0200, hubert depesz lubaczewski wrote: > > The thing is - I can't close it with pg_terminate_backend(), and I'd > > rather not kill -9, as it will, I think, close all other connections, > > and this is

Re: Can we get rid of repeated queries from pg_dump?

2021-08-31 Thread hubert depesz lubaczewski
On Mon, Aug 30, 2021 at 08:11:00PM -0400, Tom Lane wrote: > [ redirecting to -hackers ] > > I wrote: > > I experimented with the attached, very quick-n-dirty patch to collect > > format_type results during the initial scan of pg_type, instead. On the > > regression database in HEAD, it reduces

Re: Fix around conn_duration in pgbench

2021-08-31 Thread Yugo NAGATA
On Tue, 31 Aug 2021 14:46:42 +0900 (JST) Tatsuo Ishii wrote: > >> > My 0.02€: From a benchmarking perspective, ISTM that it makes sense to > >> > include disconnection times, which are clearly linked to connections, > >> > especially with -C. So I'd rather have the more meaningful figure even >