Re: Improving LWLock wait events

2020-12-22 Thread Craig Ringer
On Wed, 23 Dec 2020 at 15:51, Craig Ringer wrote: > > I've struggled with this quite a bit myself. > > By the way, I sent in a patch to enhance the static tracepoints available for LWLocks. See

Re: Improving LWLock wait events

2020-12-22 Thread Craig Ringer
On Mon, 21 Dec 2020 at 05:27, Andres Freund wrote: > Hi, > > The current wait events are already pretty useful. But I think we could > make them more informative without adding real runtime overhead. > > All 1-3 sound pretty sensible to me. I also think there's a 4, but I think the tradeoffs

Re: Perform COPY FROM encoding conversions in larger chunks

2020-12-22 Thread Heikki Linnakangas
On 22/12/2020 22:01, John Naylor wrote: In 0004, it seems you have some doubts about upgrade compatibility. Is that because user-defined conversions would no longer have the right signature? Exactly. If you have an extension that adds a custom conversion function and does CREATE CONVERSION,

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-12-22 Thread Michael Paquier
On Tue, Dec 22, 2020 at 03:15:37PM -0600, Justin Pryzby wrote: > Now, I really think utility.c ought to pass in a pointer to a local > ReindexOptions variable to avoid all the memory context, which is unnecessary > and prone to error. Yeah, it sounds right to me to just bite the bullet and do

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-12-22 Thread k.jami...@fujitsu.com
On Tuesday, December 22, 2020 9:11 PM, Amit Kapila wrote: > On Tue, Dec 22, 2020 at 2:55 PM Amit Kapila > wrote: > > Next, I'll look into DropRelFileNodesAllBuffers() > > optimization patch. > > > > Review of v35-0004-Optimize-DropRelFileNodesAllBuffers-in-recovery [1] >

TAP PostgresNode function to gdb stacks and optional cores for all backends

2020-12-22 Thread Craig Ringer
Hi all I recently wrote a utility that adds a $node->gdb_backends() method to PostgresNode instances - figured I'd share it here in case anyone finds it useful, or wants to adopt it into the features of the TAP tools. This function provides a one-line way to dump stacks for all running backends

Logical decoding without slots: decoding in lockstep with recovery

2020-12-22 Thread Craig Ringer
Hi all I want to share an idea I've looked at a few times where I've run into situations where logical slots were inadvertently dropped, or where it became necessary to decode changes in the past on a slot. As most of you will know you can't just create a logical slot in the past. Even if it was

Re: Better client reporting for "immediate stop" shutdowns

2020-12-22 Thread Bharath Rupireddy
On Tue, Dec 22, 2020 at 11:02 PM Tom Lane wrote: > Magnus Hagander writes: > > On Tue, Dec 22, 2020 at 2:29 AM Bharath Rupireddy > > wrote: > >> If I'm correct, quickdie() doesn't access any shared memory because > >> one of the reason we can be in quickdie() is when the shared memory > >>

Re: Single transaction in the tablesync worker?

2020-12-22 Thread Peter Smith
Hi Amit. PSA my v7 WIP patch for the Solution1. This patch still applies onto the v30 patch set [1] from other 2PC thread: [1] https://www.postgresql.org/message-id/CAFPTHDYA8yE6tEmQ2USYS68kNt%2BkM%3DSwKgj%3Djy4AvFD5e9-UTQ%40mail.gmail.com (I understand you would like this to be delivered as a

proposal - support tsv output format for psql

2020-12-22 Thread Pavel Stehule
Hi I am playing with clipboard on Linux, and I found a way, how to redirect psql output to clipboard via wl-copy or xclip and then to Libre Office. Now it looks so best format is tsv select * from pg_database \g (format=tsv) | wl-paste -t application/x-libreoffice-tsvc Implementation of tsv

Re: libpq compression

2020-12-22 Thread Konstantin Knizhnik
On 22.12.2020 22:03, Tom Lane wrote: Tomas Vondra writes: I don't see aby benchmark results in this thread, allowing me to make that conclusion, and I find it hard to believe that 200MB/client is a sensible trade-off. It assumes you have that much memory, and it may allow easy DoS attack

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-12-22 Thread Zhihong Yu
Hi, It is possible to come out of the nested loop without goto. + boolcached = true; ... +* to that fork during recovery. +*/ + for (i = 0; i < n && cached; i++) ... + if (!cached) +. break; Here I changed the initial value for cached to true so that we

Re: Reduce the number of special cases to build contrib modules on windows

2020-12-22 Thread Michael Paquier
On Tue, Dec 22, 2020 at 11:24:40PM +1300, David Rowley wrote: > On Wed, 11 Nov 2020 at 13:44, Michael Paquier wrote: >> It seems to me that your patch is doing the right thing for adminpack >> and that its Makefile has no need to include a reference to libpq >> source path, no? > > Yeah. Likely

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-12-22 Thread Justin Pryzby
On Tue, Dec 22, 2020 at 03:22:19PM -0800, Zhihong Yu wrote: > Justin: > For reindex_index() : > > + if (options->tablespaceOid == MyDatabaseTableSpace) > + options->tablespaceOid = InvalidOid; > ... > + oldTablespaceOid = iRel->rd_rel->reltablespace; > + if (set_tablespace && > +

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-12-22 Thread Kyotaro Horiguchi
At Wed, 23 Dec 2020 04:22:19 +, "tsunakawa.ta...@fujitsu.com" wrote in > From: Amit Kapila > > + /* Get the number of blocks for a relation's fork */ > > + block[i][j] = smgrnblocks(smgr_reln[i], j, ); > > + > > + if (!cached) > > + goto buffer_full_scan; > > > > Why do we need to use

Cleanup some -I$(libpq_srcdir) in makefiles

2020-12-22 Thread Michael Paquier
Hi all, While looking at a patch from David, I have noticed $subject: https://www.postgresql.org/message-id/CAApHDvpgB+vxk=w6opkidwzzeo6knifqidnomzr8p4rotyk...@mail.gmail.com adminpack and old_snapshot have no need for those references as they don't use libpq. Any objections to a small-ish

Re: Preventing hangups in bgworker start/stop during DB shutdown

2020-12-22 Thread Craig Ringer
On Wed, 23 Dec 2020 at 05:40, Tom Lane wrote: > Here's an attempt at closing the race condition discussed in [1] > (and in some earlier threads, though I'm too lazy to find them). > > The core problem is that the bgworker management APIs were designed > without any thought for exception

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-12-22 Thread tsunakawa.ta...@fujitsu.com
From: Amit Kapila > + /* Get the number of blocks for a relation's fork */ > + block[i][j] = smgrnblocks(smgr_reln[i], j, ); > + > + if (!cached) > + goto buffer_full_scan; > > Why do we need to use goto here? We can simply break from the loop and > then check if (cached && nBlocksToInvalidate <

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-12-22 Thread Amit Kapila
On Tue, Dec 22, 2020 at 5:41 PM Amit Kapila wrote: > > On Tue, Dec 22, 2020 at 2:55 PM Amit Kapila wrote: > > > > Apart from tests, do let me know if you are happy with the changes in > > the patch? Next, I'll look into DropRelFileNodesAllBuffers() > > optimization patch. > > > > Review of

RE: Consider Parallelism While Planning For REFRESH MATERIALIZED VIEW

2020-12-22 Thread Hou, Zhijie
> Thanks for taking a look at the patch. > > The intention of the patch is to just enable the parallel mode while planning > the select part of the materialized view, but the insertions do happen in > the leader backend itself. That way even if there's temporary tablespace > gets created, we have

Re: Parallel bitmap index scan

2020-12-22 Thread Dilip Kumar
On Wed, 23 Dec 2020 at 4:15 AM, Tomas Vondra wrote: > On 11/11/20 8:52 PM, Tomas Vondra wrote: > > Hi, > > > > I took a look at this today, doing a bit of stress-testing, and I can > > get it to crash because of segfaults in pagetable_create (not sure if > > the issue is there, it might be just

Re: Proposed patch for key managment

2020-12-22 Thread Bruce Momjian
On Tue, Dec 22, 2020 at 10:40:17AM -0500, Bruce Momjian wrote: > On Mon, Dec 21, 2020 at 10:07:48PM -0500, Bruce Momjian wrote: > > Attached is the script patch. It is also at: > > > > > > https://github.com/postgres/postgres/compare/master...bmomjian:cfe-sh.diff > > > > I think it still

Re: Parallel INSERT (INTO ... SELECT ...)

2020-12-22 Thread Amit Kapila
On Wed, Dec 23, 2020 at 7:52 AM Hou, Zhijie wrote: > > Hi > > > > I may be wrong, and if I miss sth in previous mails, please give me some > > hints. > > > IMO, serial insertion with underlying parallel SELECT can be > > > considered for foreign table or temporary table, as the insertions only >

RE: Parallel INSERT (INTO ... SELECT ...)

2020-12-22 Thread Hou, Zhijie
Hi > > I may be wrong, and if I miss sth in previous mails, please give me some > hints. > > IMO, serial insertion with underlying parallel SELECT can be > > considered for foreign table or temporary table, as the insertions only > happened in the leader process. > > > > I don't think we support

Re: Confused about stream replication protocol documentation

2020-12-22 Thread Li Japin
On Dec 22, 2020, at 11:13 PM, Fujii Masao mailto:masao.fu...@oss.nttdata.com>> wrote: ‘B’ means a backend and ‘F’ means a frontend. Maybe as [1] does, we should add the note like "Each is marked to indicate that it can be sent by a frontend (F) and a backend (B)" into the description about each

Re: Parallel INSERT (INTO ... SELECT ...)

2020-12-22 Thread Amit Kapila
On Wed, Dec 23, 2020 at 7:15 AM Hou, Zhijie wrote: > > Hi > > I have an issue about the parallel-safety checks. > > If target table is foreign table or temporary table, > rel_max_parallel_hazard_for_modify will return PROPARALLEL_UNSAFE, > which not only disable parallel insert but also disable

RE: Parallel INSERT (INTO ... SELECT ...)

2020-12-22 Thread Hou, Zhijie
Hi I have an issue about the parallel-safety checks. If target table is foreign table or temporary table, rel_max_parallel_hazard_for_modify will return PROPARALLEL_UNSAFE, which not only disable parallel insert but also disable underlying parallel SELECT. +create temporary table temp_names

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-12-22 Thread Amit Kapila
On Wed, Dec 23, 2020 at 6:30 AM k.jami...@fujitsu.com wrote: > > On Tuesday, December 22, 2020 6:25 PM, Amit Kapila wrote: > > > Apart from tests, do let me know if you are happy with the changes in the > > patch? Next, I'll look into DropRelFileNodesAllBuffers() optimization patch. > > Thank

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-12-22 Thread k.jami...@fujitsu.com
On Tuesday, December 22, 2020 6:25 PM, Amit Kapila wrote: > Attached, please find the updated patch with the following modifications, (a) > updated comments at various places especially to tell why this is a safe > optimization, (b) merged the patch for extending the smgrnblocks and > vacuum

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-12-22 Thread Zhihong Yu
Justin: For reindex_index() : + if (options->tablespaceOid == MyDatabaseTableSpace) + options->tablespaceOid = InvalidOid; ... + if (set_tablespace && + (options->tablespaceOid != oldTablespaceOid || + (options->tablespaceOid == MyDatabaseTableSpace &&

New IndexAM API controlling index vacuum strategies

2020-12-22 Thread Masahiko Sawada
Hi all, I've started this separate thread from [1] for discussing the general API design of index vacuum. Summary: * Call ambulkdelete and amvacuumcleanup even when INDEX_CLEANUP is false, and leave it to the index AM whether or not skip them. * Add a new index AM API amvacuumstrategy(), asking

Re: Parallel bitmap index scan

2020-12-22 Thread Tomas Vondra
On 11/11/20 8:52 PM, Tomas Vondra wrote: > Hi, > > I took a look at this today, doing a bit of stress-testing, and I can > get it to crash because of segfaults in pagetable_create (not sure if > the issue is there, it might be just a symptom of an issue elsewhere). > > Attached is a shell script

Re: Proposed patch for key managment

2020-12-22 Thread Bruce Momjian
On Tue, Dec 22, 2020 at 04:13:06PM -0500, Bruce Momjian wrote: > On Tue, Dec 22, 2020 at 08:15:27PM +, Alastair Turner wrote: > > Hi Bruce > > > > In ckey_passphrase.sh.sample > > > > + > > +echo "$PASS" | sha256sum | cut -d' ' -f1 > > + > > > > Under the threat model discussed, a copy of

Re: libpq compression

2020-12-22 Thread Tom Lane
I wrote: > Robert Haas writes: >> But there is a privilege boundary between the sender and the receiver. >> What's alleged here is that the sender can do a thing which causes the >> receiver to burn through tons of memory. It doesn't help anything to >> say, well, the sender ought to use a window

Preventing hangups in bgworker start/stop during DB shutdown

2020-12-22 Thread Tom Lane
Here's an attempt at closing the race condition discussed in [1] (and in some earlier threads, though I'm too lazy to find them). The core problem is that the bgworker management APIs were designed without any thought for exception conditions, notably "we're not gonna launch any more workers

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-12-22 Thread Justin Pryzby
On Tue, Dec 22, 2020 at 06:57:41PM +0900, Michael Paquier wrote: > On Tue, Dec 22, 2020 at 02:32:05AM -0600, Justin Pryzby wrote: > > Also, this one is going to be subsumed by ExecReindex(), so the palloc will > > go > > away (otherwise I would ask to pass it in from the caller): > > Yeah,

Re: Proposed patch for key managment

2020-12-22 Thread Bruce Momjian
On Tue, Dec 22, 2020 at 08:15:27PM +, Alastair Turner wrote: > Hi Bruce > > In ckey_passphrase.sh.sample > > + > +echo "$PASS" | sha256sum | cut -d' ' -f1 > + > > Under the threat model discussed, a copy of the keyfile could be > attacked offline. So getting from passphrase to DEKs should

Re: [PATCH] Automatic HASH and LIST partition creation

2020-12-22 Thread Fabien COELHO
BTW could you tell me a couple of words about pros and cons of c-code syntax parsing comparing to parsing using gram.y trees? I'd rather use an automatic tool (lexer/parser) if possible instead of doing it by hand if I can. If you want a really nice syntax with clever tricks, then you may

Re: libpq compression

2020-12-22 Thread Tom Lane
Robert Haas writes: > On Tue, Dec 22, 2020 at 2:33 PM Tom Lane wrote: >> I'd assume that there's a direct correlation between the compression level >> setting and the window size; but I've not studied the libzstd docs in >> enough detail to know what it is. > But there is a privilege boundary

Re: libpq compression

2020-12-22 Thread Robert Haas
On Tue, Dec 22, 2020 at 2:33 PM Tom Lane wrote: > I'd assume that there's a direct correlation between the compression level > setting and the window size; but I've not studied the libzstd docs in > enough detail to know what it is. But there is a privilege boundary between the sender and the

Re: Proposed patch for key managment

2020-12-22 Thread Alastair Turner
Hi Bruce In ckey_passphrase.sh.sample + +echo "$PASS" | sha256sum | cut -d' ' -f1 + Under the threat model discussed, a copy of the keyfile could be attacked offline. So getting from passphrase to DEKs should be as resource intensive as possible to slow down brute-force attempts. Instead of

Re: Perform COPY FROM encoding conversions in larger chunks

2020-12-22 Thread John Naylor
On Wed, Dec 16, 2020 at 8:18 AM Heikki Linnakangas wrote: > > Currently, COPY FROM parses the input one line at a time. Each line is > converted to the database encoding separately, or if the file encoding > matches the database encoding, we just check that the input is valid for > the encoding.

Re: libpq compression

2020-12-22 Thread Tom Lane
Tomas Vondra writes: > On 12/22/20 8:03 PM, Tom Lane wrote: >> The link Ken pointed at suggests that restricting the window size to >> 8MB is a common compromise. It's not clear to me what that does to >> the achievable compression ratio. Even 8MB could be an annoying cost >> if it's being paid

Re: libpq compression

2020-12-22 Thread Tomas Vondra
On 12/22/20 8:03 PM, Tom Lane wrote: Tomas Vondra writes: I don't see aby benchmark results in this thread, allowing me to make that conclusion, and I find it hard to believe that 200MB/client is a sensible trade-off. It assumes you have that much memory, and it may allow easy DoS attack

Re: [HACKERS] [PATCH] Generic type subscripting

2020-12-22 Thread Tom Lane
Pavel Stehule writes: > But maybe we try to design some that are designed already. Is there some > info about index specification in SQL/JSON? We do have precedent for this, it's the rules about resolving argument types for overloaded functions. But the conclusion that that precedent leads to

Re: [HACKERS] [PATCH] Generic type subscripting

2020-12-22 Thread Pavel Stehule
út 22. 12. 2020 v 18:35 odesílatel Dmitry Dolgov <9erthali...@gmail.com> napsal: > > On Tue, Dec 22, 2020 at 11:57:13AM -0500, Tom Lane wrote: > > Dmitry Dolgov <9erthali...@gmail.com> writes: > > > On Tue, Dec 22, 2020 at 12:19:26PM +0100, Pavel Stehule wrote: > > >> I expect behave like > > >>

Re: libpq compression

2020-12-22 Thread Tom Lane
Tomas Vondra writes: > I don't see aby benchmark results in this thread, allowing me to make > that conclusion, and I find it hard to believe that 200MB/client is a > sensible trade-off. > It assumes you have that much memory, and it may allow easy DoS attack > (although maybe it's not worse

Re: libpq compression

2020-12-22 Thread Tomas Vondra
On 12/22/20 7:31 PM, Andrey Borodin wrote: 22 дек. 2020 г., в 23:15, Tomas Vondra написал(а): On 12/22/20 6:56 PM, Robert Haas wrote: On Tue, Dec 22, 2020 at 6:24 AM Daniil Zakhlystov wrote: When using bidirectional compression, Postgres resource usage correlates with the selected

Re: libpq compression

2020-12-22 Thread Andrey Borodin
> 22 дек. 2020 г., в 23:15, Tomas Vondra > написал(а): > > > > On 12/22/20 6:56 PM, Robert Haas wrote: >> On Tue, Dec 22, 2020 at 6:24 AM Daniil Zakhlystov >> wrote: >>> When using bidirectional compression, Postgres resource usage correlates >>> with the selected compression level. For

Re: Fix typo about generate_gather_paths

2020-12-22 Thread Tomas Vondra
On 12/9/20 3:21 AM, Hou, Zhijie wrote: Hi Since ba3e76c, the optimizer call generate_useful_gather_paths instead of generate_gather_paths() outside. But I noticed that some comment still talking about generate_gather_paths not generate_useful_gather_paths. I think we should fix these

Re: libpq compression

2020-12-22 Thread Kenneth Marshall
On Tue, Dec 22, 2020 at 07:15:23PM +0100, Tomas Vondra wrote: > > > On 12/22/20 6:56 PM, Robert Haas wrote: > >On Tue, Dec 22, 2020 at 6:24 AM Daniil Zakhlystov > > wrote: > >>When using bidirectional compression, Postgres resource usage correlates > >>with the selected compression level. For

Re: On login trigger: take three

2020-12-22 Thread Pavel Stehule
út 22. 12. 2020 v 12:42 odesílatel Konstantin Knizhnik < k.knizh...@postgrespro.ru> napsal: > > > On 22.12.2020 12:25, Pavel Stehule wrote: > > > regress tests fails > > sysviews ... FAILED 112 ms > test event_trigger... FAILED (test process exited

Re: libpq compression

2020-12-22 Thread Tomas Vondra
On 12/22/20 6:56 PM, Robert Haas wrote: On Tue, Dec 22, 2020 at 6:24 AM Daniil Zakhlystov wrote: When using bidirectional compression, Postgres resource usage correlates with the selected compression level. For example, here is the Postgresql application memory usage: No compression -

Re: libpq compression

2020-12-22 Thread Robert Haas
On Tue, Dec 22, 2020 at 6:24 AM Daniil Zakhlystov wrote: > When using bidirectional compression, Postgres resource usage correlates with > the selected compression level. For example, here is the Postgresql > application memory usage: > > No compression - 1.2 GiB > > ZSTD > zstd:1 - 1.4 GiB >

HOT chain bug in latestRemovedXid calculation

2020-12-22 Thread Peter Geoghegan
ISTM that heap_compute_xid_horizon_for_tuples() calculates latestRemovedXid for index deletion callers without sufficient care. The function only follows line pointer redirects, which is necessary but not sufficient to visit all relevant heap tuple headers -- it also needs to traverse HOT chains,

Re: libpq compression

2020-12-22 Thread Robert Haas
On Mon, Dec 14, 2020 at 12:53 PM Daniil Zakhlystov wrote: > > On Dec 10, 2020, at 1:39 AM, Robert Haas wrote: > > Good points. I guess you need to arrange to "flush" at the compression > > layer as well as the libpq layer so that you don't end up with data > > stuck in the compression buffers. >

Re: [HACKERS] [PATCH] Generic type subscripting

2020-12-22 Thread Dmitry Dolgov
> On Tue, Dec 22, 2020 at 11:57:13AM -0500, Tom Lane wrote: > Dmitry Dolgov <9erthali...@gmail.com> writes: > > On Tue, Dec 22, 2020 at 12:19:26PM +0100, Pavel Stehule wrote: > >> I expect behave like > >> > >> update x set test[1] = 10; --> "[10]"; > >> update x set test['1'] = 10; --> "{"1":

Re: [HACKERS] [PATCH] Generic type subscripting

2020-12-22 Thread Pavel Stehule
út 22. 12. 2020 v 17:57 odesílatel Tom Lane napsal: > Dmitry Dolgov <9erthali...@gmail.com> writes: > > On Tue, Dec 22, 2020 at 12:19:26PM +0100, Pavel Stehule wrote: > >> I expect behave like > >> > >> update x set test[1] = 10; --> "[10]"; > >> update x set test['1'] = 10; --> "{"1": 10}" > >

Re: Better client reporting for "immediate stop" shutdowns

2020-12-22 Thread Tom Lane
Magnus Hagander writes: > On Tue, Dec 22, 2020 at 2:29 AM Bharath Rupireddy > wrote: >> If I'm correct, quickdie() doesn't access any shared memory because >> one of the reason we can be in quickdie() is when the shared memory >> itself is corrupted(the comment down below on why we don't call >>

Re: [HACKERS] [PATCH] Generic type subscripting

2020-12-22 Thread Tom Lane
Dmitry Dolgov <9erthali...@gmail.com> writes: > On Tue, Dec 22, 2020 at 12:19:26PM +0100, Pavel Stehule wrote: >> I expect behave like >> >> update x set test[1] = 10; --> "[10]"; >> update x set test['1'] = 10; --> "{"1": 10}" > Yes, I also was thinking about this because such behaviour is more

Re: [HACKERS] [PATCH] Generic type subscripting

2020-12-22 Thread Dmitry Dolgov
> On Tue, Dec 22, 2020 at 12:19:26PM +0100, Pavel Stehule wrote: > > > Here is the new version of jsonb subscripting rebased on the committed > > infrastructure patch. I hope it will not introduce any confusion with > > the previously posted patched in this thread (about alter type subscript > >

Re: [PATCH] Automatic HASH and LIST partition creation

2020-12-22 Thread Pavel Borisov
> > Why? We could accept anything in the list? i.e.: > > (ident =? value[, ident =? value]*) > > > I don't against this but as far as I've heard there is some > > opposition among PG community against new keywords. Maybe I am wrong. > > the ident is a keyword that can be interpreted later on,

Re: Proposed patch for key managment

2020-12-22 Thread Bruce Momjian
On Mon, Dec 21, 2020 at 10:07:48PM -0500, Bruce Momjian wrote: > Attached is the script patch. It is also at: > > > https://github.com/postgres/postgres/compare/master...bmomjian:cfe-sh.diff > > I think it still needs docs but those will have to be done after the > code doc patch is

Re: [PATCH] Automatic HASH and LIST partition creation

2020-12-22 Thread Fabien COELHO
HEllo. CREATE TABLE foo(a int) PARTITION BY LIST(a) CONFIGURATION (FOR VALUES IN (1,2),(3,4) DEFAULT PARTITION foo_def); I would like to disagree with this syntactic approach because it would very specific to each partition method. IMHO the syntax should be as generic as possible. I'd

Re: Confused about stream replication protocol documentation

2020-12-22 Thread Fujii Masao
On 2020/12/22 18:07, Li Japin wrote: Hi, all In Stream Replication Protocol [1], the documentation of `START_REPLICATION` message is XLogData (B)   … Primary keepalive message (B)   … Standby status update (F)   … Hot Standby feedback message (F)   ... I’m confused about the means of

Re: [PATCH] Automatic HASH and LIST partition creation

2020-12-22 Thread Pavel Borisov
> > > CREATE TABLE foo(a int) PARTITION BY LIST(a) CONFIGURATION (FOR VALUES > IN > > (1,2),(3,4) DEFAULT PARTITION foo_def); > > I would like to disagree with this syntactic approach because it would > very specific to each partition method. IMHO the syntax should be as > generic as possible. I'd

Re: Deadlock between backend and recovery may not be detected

2020-12-22 Thread Fujii Masao
On 2020/12/22 20:42, Fujii Masao wrote: On 2020/12/22 10:25, Masahiko Sawada wrote: On Fri, Dec 18, 2020 at 6:36 PM Fujii Masao wrote: On 2020/12/17 2:15, Fujii Masao wrote: On 2020/12/16 23:28, Drouvot, Bertrand wrote: Hi, On 12/16/20 2:36 PM, Victor Yegorov wrote: *CAUTION*:

Re: [PATCH] Automatic HASH and LIST partition creation

2020-12-22 Thread Fabien COELHO
CREATE TABLE foo(a int) PARTITION BY LIST(a) CONFIGURATION (FOR VALUES IN (1,2),(3,4) DEFAULT PARTITION foo_def); I would like to disagree with this syntactic approach because it would very specific to each partition method. IMHO the syntax should be as generic as possible. I'd suggest

EXPLAIN/EXPLAIN ANALYZE REFRESH MATERIALIZED VIEW

2020-12-22 Thread Bharath Rupireddy
Hi, Currently, $subject is not allowed. We do plan the mat view query before every refresh. I propose to show the explain/explain analyze of the select part of the mat view in case of Refresh Mat View(RMV). It will be useful for the user to know what exactly is being planned and executed as part

Re: proposal: schema variables

2020-12-22 Thread Pavel Stehule
>> +* both variants, and returns InvalidOid with not_uniq flag, >> when >> >> 'and return' (no s) >> >> + return InvalidOid; >> + } >> + else if (OidIsValid(varoid_without_attr)) >> >> 'else' is not needed (since the if block ends with return). >> >> For clean_cache_callback(), >> >> + if (hash_search(schemavarhashtab, >> + (void *) >varid, >> + HASH_REMOVE, >> + NULL) == NULL) >> + elog(DEBUG1, "hash table corrupted"); >> >> Maybe add more information to the debug, such as var name. >> Should we come out of the while loop in this scenario ? >> >> Cheers >> > schema-variables-20201222.patch.gz Description: application/gzip

Re: Implementing Incremental View Maintenance

2020-12-22 Thread Yugo NAGATA
Hi hackers, I heard the opinion that this patch is too big and hard to review. So, I wander that we should downsize the patch by eliminating some features and leaving other basic features. If there are more opinions this makes it easer for reviewers to look at this patch, I would like do it. If

Re: Implementing Incremental View Maintenance

2020-12-22 Thread Yugo NAGATA
Hi, Attached is the revised patch (v20) to add support for Incremental Materialized View Maintenance (IVM). In according with Konstantin's suggestion, I made a few optimizations. 1. Creating an index on the matview automatically When creating incremental maintainable materialized view (IMMV)s,

Re: Consider Parallelism While Planning For REFRESH MATERIALIZED VIEW

2020-12-22 Thread Bharath Rupireddy
On Tue, Dec 22, 2020 at 4:53 PM Hou, Zhijie wrote: > I have an issue about the safety of enable parallel select. > > I checked the [parallel insert into select] patch. > https://commitfest.postgresql.org/31/2844/ > It seems parallel select is not allowed when target table is temporary table. > >

回复:Re: Cache relation sizes?

2020-12-22 Thread 陈佳昕(步真)
Hi Thomas: I studied your patch these days and found there might be a problem. When execute 'drop database', the smgr shared pool will not be removed because of no call 'smgr_drop_sr'. Function 'dropdb' in dbcommands.c remove the buffer from bufferpool and unlink the real files by 'rmtree', It

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-12-22 Thread Amit Kapila
On Tue, Dec 22, 2020 at 2:55 PM Amit Kapila wrote: > > Apart from tests, do let me know if you are happy with the changes in > the patch? Next, I'll look into DropRelFileNodesAllBuffers() > optimization patch. > Review of v35-0004-Optimize-DropRelFileNodesAllBuffers-in-recovery [1]

Re: proposal: schema variables

2020-12-22 Thread Pavel Stehule
Hi ne 20. 12. 2020 v 20:24 odesílatel Zhihong Yu napsal: > Hi, > I took a look at the rebased patch. > > + varisnotnull > + boolean > + > + > + True if the schema variable doesn't allow null value. The default > value is false. > > I wonder whether the field can be

Re: On login trigger: take three

2020-12-22 Thread Konstantin Knizhnik
On 22.12.2020 12:25, Pavel Stehule wrote: regress tests fails      sysviews                     ... FAILED      112 ms test event_trigger                ... FAILED (test process exited with exit code 2)      447 ms test fast_default                 ... FAILED      392 ms test stats        

Re: Deadlock between backend and recovery may not be detected

2020-12-22 Thread Fujii Masao
On 2020/12/22 10:25, Masahiko Sawada wrote: On Fri, Dec 18, 2020 at 6:36 PM Fujii Masao wrote: On 2020/12/17 2:15, Fujii Masao wrote: On 2020/12/16 23:28, Drouvot, Bertrand wrote: Hi, On 12/16/20 2:36 PM, Victor Yegorov wrote: *CAUTION*: This email originated from outside of the

Re: Deadlock between backend and recovery may not be detected

2020-12-22 Thread Fujii Masao
On 2020/12/19 1:43, Drouvot, Bertrand wrote: Hi, On 12/18/20 10:35 AM, Fujii Masao wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. On 2020/12/17 2:15, Fujii

Re: Single transaction in the tablesync worker?

2020-12-22 Thread Peter Smith
On Mon, Dec 21, 2020 at 11:36 PM Amit Kapila wrote: > > On Mon, Dec 21, 2020 at 3:17 PM Peter Smith wrote: > > > > On Mon, Dec 21, 2020 at 4:23 PM Amit Kapila wrote: > > > > > Few other comments: > > > == > > > > Thanks for your feedback. > > > > > 1. > > > * FIXME 3 - Crashed

Re: libpq compression

2020-12-22 Thread Daniil Zakhlystov
Hi! I’ve fixed an issue with compression level parsing in this PR https://github.com/postgrespro/libpq_compression/pull/4 Also, did a couple of pgbenchmarks to measure database resource usage with different compression levels. Firstly, I measured the bidirectional compression scenario, i.e.

RE: Consider Parallelism While Planning For REFRESH MATERIALIZED VIEW

2020-12-22 Thread Hou, Zhijie
Hi > Added this to commitfest, in case it is useful - > https://commitfest.postgresql.org/31/2856/ I have an issue about the safety of enable parallel select. I checked the [parallel insert into select] patch. https://commitfest.postgresql.org/31/2844/ It seems parallel select is not allowed

Re: [HACKERS] [PATCH] Generic type subscripting

2020-12-22 Thread Pavel Stehule
út 22. 12. 2020 v 11:24 odesílatel Dmitry Dolgov <9erthali...@gmail.com> napsal: > > On Fri, Dec 18, 2020 at 08:59:25PM +0100, Dmitry Dolgov wrote: > > > On Thu, Dec 17, 2020 at 03:29:35PM -0500, Tom Lane wrote: > > > Dmitry Dolgov <9erthali...@gmail.com> writes: > > > > On Thu, Dec 17, 2020 at

Re: Single transaction in the tablesync worker?

2020-12-22 Thread Peter Smith
Hi Amit. PSA my v6 WIP patch for the Solution1. This patch still applies onto the v30 patch set [1] from other 2PC thread: [1] https://www.postgresql.org/message-id/CAFPTHDYA8yE6tEmQ2USYS68kNt%2BkM%3DSwKgj%3Djy4AvFD5e9-UTQ%40mail.gmail.com (I understand you would like this to be delivered as a

Re: pg_preadv() and pg_pwritev()

2020-12-22 Thread Thomas Munro
On Mon, Dec 21, 2020 at 11:40 AM Andres Freund wrote: > On 2020-12-20 16:26:42 +1300, Thomas Munro wrote: > > > 1. port.h cannot assume that has already been included; > > > nor do I want to fix that by including there. Do we > > > really need to define a fallback value of IOV_MAX? If so, > >

Re: postgres_fdw - cached connection leaks if the associated user mapping/foreign server is dropped

2020-12-22 Thread Bharath Rupireddy
On Fri, Dec 18, 2020 at 6:46 PM Bharath Rupireddy wrote: > On Fri, Dec 18, 2020 at 6:39 PM Bharath Rupireddy > wrote: > > > > On Fri, Dec 18, 2020 at 5:06 PM Hou, Zhijie > > wrote: > > > I have an issue about the existing testcase. > > > > > > """ > > > -- Test that alteration of server

Re: Reduce the number of special cases to build contrib modules on windows

2020-12-22 Thread David Rowley
On Wed, 11 Nov 2020 at 13:44, Michael Paquier wrote: > > On Wed, Nov 11, 2020 at 11:01:57AM +1300, David Rowley wrote: > > I'm still working through some small differences in some of the > > .vcxproj files. I've been comparing these by copying *.vcxproj out to > > another directory with patched

Re: [HACKERS] [PATCH] Generic type subscripting

2020-12-22 Thread Dmitry Dolgov
> On Fri, Dec 18, 2020 at 08:59:25PM +0100, Dmitry Dolgov wrote: > > On Thu, Dec 17, 2020 at 03:29:35PM -0500, Tom Lane wrote: > > Dmitry Dolgov <9erthali...@gmail.com> writes: > > > On Thu, Dec 17, 2020 at 01:49:17PM -0500, Tom Lane wrote: > > >> We can certainly reconsider the API for the

Re: [HACKERS] logical decoding of two-phase transactions

2020-12-22 Thread Amit Kapila
On Tue, Dec 22, 2020 at 2:51 PM Ajin Cherian wrote: > > On Sat, Dec 19, 2020 at 2:13 PM Amit Kapila wrote: > > > Okay, I have changed the rollback_prepare API as discussed above and > > accordingly handle the case where rollback is received without prepare > > in apply_handle_rollback_prepared.

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-12-22 Thread Michael Paquier
On Tue, Dec 22, 2020 at 02:32:05AM -0600, Justin Pryzby wrote: > Also, this one is going to be subsumed by ExecReindex(), so the palloc will go > away (otherwise I would ask to pass it in from the caller): Yeah, maybe. Still you need to be very careful if you have any allocated variables like a

Re: Fail Fast In CTAS/CMV If Relation Already Exists To Avoid Unnecessary Rewrite, Planning Costs

2020-12-22 Thread Bharath Rupireddy
On Tue, Dec 22, 2020 at 2:07 PM Michael Paquier wrote: > I was looking at your patch today, and I actually found the conclusion > to output an empty plan while issuing a NOTICE to be quite intuitive > if the caller uses IF NOT EXISTS with EXPLAIN. Thanks! > Thanks for adding some test cases!

Re: On login trigger: take three

2020-12-22 Thread Pavel Stehule
Hi po 21. 12. 2020 v 11:06 odesílatel Konstantin Knizhnik < k.knizh...@postgrespro.ru> napsal: > > > On 20.12.2020 10:04, Pavel Stehule wrote: > > > > čt 17. 12. 2020 v 19:30 odesílatel Pavel Stehule > napsal: > >> >> >> čt 17. 12. 2020 v 14:04 odesílatel Konstantin Knizhnik < >>

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-12-22 Thread Amit Kapila
On Tue, Dec 22, 2020 at 8:30 AM Kyotaro Horiguchi wrote: > > At Tue, 22 Dec 2020 02:48:22 +, "tsunakawa.ta...@fujitsu.com" > wrote in > > From: Amit Kapila > > > Why would all client backends wait for AccessExclusive lock on this > > > relation? Say, a client needs a buffer for some other

Re: [HACKERS] logical decoding of two-phase transactions

2020-12-22 Thread Ajin Cherian
On Sat, Dec 19, 2020 at 2:13 PM Amit Kapila wrote: > Okay, I have changed the rollback_prepare API as discussed above and > accordingly handle the case where rollback is received without prepare > in apply_handle_rollback_prepared. I have reviewed and tested your new patchset, I agree with all

Re: Feature Proposal: Add ssltermination parameter for SNI-based LoadBalancing

2020-12-22 Thread Lukas Meisegeier
Hey, whats the state of this? Can we start working out a plan to remove the inital SSLRequest from the connection protocol or is there any reason to keep it? I would start by removing the need of the SSLRequest in the psql-server if its started with a special parameter(ssl-only or so).

Confused about stream replication protocol documentation

2020-12-22 Thread Li Japin
Hi, all In Stream Replication Protocol [1], the documentation of `START_REPLICATION` message is XLogData (B) … Primary keepalive message (B) … Standby status update (F) … Hot Standby feedback message (F) ... I’m confused about the means of ‘B’ and ‘F’? If it doesn't make sense, why we

Re: Better client reporting for "immediate stop" shutdowns

2020-12-22 Thread Magnus Hagander
On Tue, Dec 22, 2020 at 2:29 AM Bharath Rupireddy wrote: > > On Tue, Dec 22, 2020 at 3:13 AM Tom Lane wrote: > > Up to now, if you shut down the database with "pg_ctl stop -m immediate" > > then clients get a scary message claiming that something has crashed, > > because backends can't tell

Re: Fail Fast In CTAS/CMV If Relation Already Exists To Avoid Unnecessary Rewrite, Planning Costs

2020-12-22 Thread Michael Paquier
On Mon, Dec 21, 2020 at 12:01:38PM +0530, Bharath Rupireddy wrote: > On Fri, Dec 18, 2020 at 8:15 AM Bharath Rupireddy >> I tried to make it consistent by issuing NOTICE (not an error) even >> for EXPLAIN/EXPLAIN ANALYZE IF NOT EXISTS case. If issue notice and >> exit from the ExplainOneUtility,

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-12-22 Thread Justin Pryzby
On Tue, Dec 22, 2020 at 03:47:57PM +0900, Michael Paquier wrote: > On Wed, Dec 16, 2020 at 10:01:11AM +0900, Michael Paquier wrote: > > On Tue, Dec 15, 2020 at 09:45:17PM -0300, Alvaro Herrera wrote: > > > I don't like this idea too much, because adding an option causes an ABI > > > break. I

Re: ModifyTable overheads in generic plans

2020-12-22 Thread Amit Langote
On Mon, Dec 7, 2020 at 3:53 PM Amit Langote wrote: > > On Thu, Nov 12, 2020 at 5:04 PM Amit Langote wrote: > > Attached new 0002 which does these adjustments. I went with > > ri_RootTargetDesc to go along with ri_RelationDesc. > > > > Also, I have updated the original 0002 (now 0003) to make >

  1   2   >