Re: Replace remaining StrNCpy() by strlcpy()

2020-08-07 Thread Peter Eisentraut
On 2020-08-05 17:49, Tom Lane wrote: However I do see one remaining nit to pick, in CreateInitDecodingContext: /* register output plugin name with slot */ SpinLockAcquire(>mutex); - StrNCpy(NameStr(slot->data.plugin), plugin, NAMEDATALEN); + namestrcpy(>data.plugin,

Re: Creating a function for exposing memory usage of backend process

2020-08-07 Thread Michael Paquier
On Fri, Jul 31, 2020 at 03:23:52PM -0400, Robert Haas wrote: > On Fri, Jul 31, 2020 at 4:25 AM torikoshia wrote: >> And as Fujii-san told me in person, exposing memory address seems >> not preferable considering there are security techniques like >> address space layout randomization. > > Yeah,

Re: walsender waiting_for_ping spuriously set

2020-08-07 Thread Alvaro Herrera
On 2020-Aug-07, Alvaro Herrera wrote: > I'm thinking in keeping the sentences that were added in that commit, > maybe like so: > > > * We only send regular messages to the client for full decoded > > * transactions, but a synchronous replication and walsender > >

Re: walsender waiting_for_ping spuriously set

2020-08-07 Thread Alvaro Herrera
I just noticed that part of this comment I'm modifying: > @@ -1444,17 +1444,13 @@ WalSndWaitForWal(XLogRecPtr loc) >* We only send regular messages to the client for full decoded >* transactions, but a synchronous replication and walsender > shutdown >

Re: Improve handling of pg_stat_statements handling of bind "IN" variables

2020-08-07 Thread Pavel Trukhanov
Hey, let me know if there's any way I can help. I would argue that making even a small improvement here would be beneficial to many. On Tue, Jul 21, 2020 at 11:59 AM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > On Thu, Oct 3, 2019 at 3:33 AM Pavel Trukhanov < > pavel.trukha...@gmail.com>

Re: Should the nbtree page split REDO routine's locking work more like the locking on the primary?

2020-08-07 Thread Peter Geoghegan
On Thu, Aug 6, 2020 at 7:00 PM Peter Geoghegan wrote: > On Thu, Aug 6, 2020 at 6:08 PM Tom Lane wrote: > > +1 for making this more like what happens in original execution ("on the > > primary", to use your wording). Perhaps what you suggest here is still > > not enough like the original

Re: PROC_IN_ANALYZE stillborn 13 years ago

2020-08-07 Thread Andres Freund
Hi, On 2020-08-06 18:02:26 -0400, Tom Lane wrote: > Andres Freund writes: > > In fact using conceptually like a new snapshot for each sample tuple > > actually seems like it'd be somewhat of an improvement over using a > > single snapshot. > > Dunno, that feels like a fairly bad idea to me. It

Re: PROC_IN_ANALYZE stillborn 13 years ago

2020-08-07 Thread Alvaro Herrera
On 2020-Aug-05, Andres Freund wrote: > I'm mildly against that, because I'd really like to start making use of > the flag. Not so much for cancellations, but to avoid the drastic impact > analyze has on bloat. In OLTP workloads with big tables, and without > disabled cost limiting for analyze

Re: Issue with cancel_before_shmem_exit while searching to remove a particular registered exit callbacks

2020-08-07 Thread Andres Freund
Hi, On 2020-08-07 12:29:03 -0400, Robert Haas wrote: > On Thu, Aug 6, 2020 at 11:46 PM Bharath Rupireddy > wrote: > > I sent the patch previously[1], but attaching here again, modifies > > cancel_before_shmem_exit() function comment to reflect the safe usage > > of before_shmem_exit_list

Re: Add LWLock blocker(s) information

2020-08-07 Thread David Zhang
Hi, This is a very interesting topic. I did apply the 2nd patch to master branch and performed a quick test. I can observe below information, postgres=# select * from pg_lwlock_blocking_pid(26925); requested_mode | last_holder_pid | last_holder_mode | nb_holders

Re: PROC_IN_ANALYZE stillborn 13 years ago

2020-08-07 Thread Tom Lane
Robert Haas writes: > Thinking about it more, there are really two ways to think about an > estimated row count. > On the one hand, if you think of the row count estimate as the number > of rows that are going to pop out of a node, then it's always right to > think of a unique index as limiting

Re: Parallel worker hangs while handling errors.

2020-08-07 Thread Robert Haas
On Fri, Aug 7, 2020 at 2:00 PM Tom Lane wrote: > The point of the sigdelset is that if somewhere later on, we install > the BlockSig mask, then SIGQUIT will remain unblocked. I mean, we're just repeating the same points here, but that's not what the comment says. > You asserted > upthread that

Re: PROC_IN_ANALYZE stillborn 13 years ago

2020-08-07 Thread Robert Haas
On Thu, Aug 6, 2020 at 5:35 PM Tom Lane wrote: > > It's not clear to me that it would even be correct to categorize those > > somewhat-different results as "less accurate." > > Estimating two rows where the correct answer is one row is clearly > "less accurate" [ than estimating one row ].

Re: Parallel worker hangs while handling errors.

2020-08-07 Thread Tom Lane
Robert Haas writes: > On Fri, Aug 7, 2020 at 12:56 PM Tom Lane wrote: >> That SETMASK call will certainly unblock SIGQUIT, so I don't see what >> your point is. > I can't figure out if you're trolling me here or what. It's true that > the PG_SETMASK() call will certainly unblock SIGQUIT, but

Re: Issue with cancel_before_shmem_exit while searching to remove a particular registered exit callbacks

2020-08-07 Thread Robert Haas
On Fri, Aug 7, 2020 at 1:12 PM Tom Lane wrote: > That's a meaningless statement for any one caller. So it needs to be more > like "we expect callers to add and remove temporary before_shmem_exit > callbacks in strict LIFO order". Sure, that seems fine. > I wonder whether we ought to change the

Re: Parallel worker hangs while handling errors.

2020-08-07 Thread Robert Haas
On Fri, Aug 7, 2020 at 12:56 PM Tom Lane wrote: > > I don't think that your analysis here is correct. The sigdelset call > > is manipulating BlockSig, and the subsequent PG_SETMASK call is > > working with UnblockSig, so it doesn't make sense to view one as a > > preparatory step for the other. >

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-08-07 Thread Robert Haas
On Fri, Aug 7, 2020 at 12:52 PM Tom Lane wrote: > At least in the case of segment zero, the file will still exist. It'll > have been truncated to zero length, and if the filesystem is stupid about > holes in files then maybe a write to a high block number would consume > excessive disk space,

Re: Issue with cancel_before_shmem_exit while searching to remove a particular registered exit callbacks

2020-08-07 Thread Tom Lane
Robert Haas writes: > ... So I suggest something more like: > - * callback. For simplicity, only the latest entry can be > - * removed. (We could work harder but there is no need for > - * current uses.) > + * callback. We only look at the latest entry for removal, as we > + * expect the

Re: Parallel worker hangs while handling errors.

2020-08-07 Thread Tom Lane
Robert Haas writes: > On Fri, Aug 7, 2020 at 11:36 AM Tom Lane wrote: >> The sigdelset call, just like the adjacent pqsignal calls, is >> preparatory setup; it does not intend to allow anything to happen >> immediately. > I don't think that your analysis here is correct. The sigdelset call > is

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-08-07 Thread Tom Lane
Robert Haas writes: > On Fri, Aug 7, 2020 at 12:09 PM Tom Lane wrote: >> ... it's not very clear what to do in >> backends that haven't got access to that table, but maybe we could just >> accept that backends that are forced to flush dirty buffers might do some >> useless writes in such cases.

Re: Issue with cancel_before_shmem_exit while searching to remove a particular registered exit callbacks

2020-08-07 Thread Robert Haas
On Thu, Aug 6, 2020 at 11:46 PM Bharath Rupireddy wrote: > I sent the patch previously[1], but attaching here again, modifies > cancel_before_shmem_exit() function comment to reflect the safe usage > of before_shmem_exit_list callback mechanism and also removes the > point "For simplicity, only

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-08-07 Thread Robert Haas
On Fri, Aug 7, 2020 at 12:09 PM Tom Lane wrote: > As for (1), maybe we don't need to keep the info in shmem. I'll just > point out that the checkpointer has *already got* a complete list of all > recently-dropped relations, cf pendingUnlinks in sync.c. So you could > imagine looking aside at

Re: Parallel worker hangs while handling errors.

2020-08-07 Thread Robert Haas
On Fri, Aug 7, 2020 at 11:36 AM Tom Lane wrote: > I think that code is the way it is intentionally: the idea is to not > accept any signals until we reach the explicit "PG_SETMASK();" > call further down, between the sigsetjmp stanza and the main loop. > The sigdelset call, just like the adjacent

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-08-07 Thread Tom Lane
Robert Haas writes: > On Fri, Aug 7, 2020 at 12:03 AM Tom Lane wrote: >> Yeah, there is no room for "good enough" here. If a dirty buffer remains >> in the system, the checkpointer will eventually try to flush it, and fail >> (because there's no file to write it to), and then checkpointing will

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-08-07 Thread Robert Haas
On Thu, Aug 6, 2020 at 9:23 AM Ashutosh Sharma wrote: > Attached v4 patch fixes the latest comments from Robert and Masahiko-san. Compiler warning: heap_surgery.c:136:13: error: comparison of unsigned expression < 0 is always false [-Werror,-Wtautological-compare] if (blkno < 0

Re: Parallel worker hangs while handling errors.

2020-08-07 Thread Tom Lane
Robert Haas writes: > On Fri, Aug 7, 2020 at 5:05 AM Bharath Rupireddy > wrote: >> We intend to unblock SIGQUIT before sigsetjmp() in places like >> bgwriter, checkpointer, walwriter and walreceiver, but we only call >> sigdelset(, SIGQUIT);, Without PG_SETMASK();, seems >> like we are not

Re: Handing off SLRU fsyncs to the checkpointer

2020-08-07 Thread Robert Haas
On Wed, Aug 5, 2020 at 2:01 AM Thomas Munro wrote: > * Master is around 11% faster than last week before commit c5315f4f > "Cache smgrnblocks() results in recovery." > * This patch gives a similar speedup, bringing the total to around 25% > faster than last week (the time is ~20% less, the WAL

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-08-07 Thread Robert Haas
On Fri, Aug 7, 2020 at 12:03 AM Tom Lane wrote: > Yeah, there is no room for "good enough" here. If a dirty buffer remains > in the system, the checkpointer will eventually try to flush it, and fail > (because there's no file to write it to), and then checkpointing will be > stuck. So we cannot

Re: Parallel worker hangs while handling errors.

2020-08-07 Thread Robert Haas
On Fri, Aug 7, 2020 at 5:05 AM Bharath Rupireddy wrote: > We intend to unblock SIGQUIT before sigsetjmp() in places like > bgwriter, checkpointer, walwriter and walreceiver, but we only call > sigdelset(, SIGQUIT);, Without PG_SETMASK();, seems > like we are not actually unblocking SIQUIT and

Re: [PG13] Planning (time + buffers) data structure in explain plan (format text)

2020-08-07 Thread Pierre Giraud
Le 07/08/2020 à 14:52, Julien Rouhaud a écrit : > Hi, > > On Fri, Aug 7, 2020 at 2:30 PM Pierre Giraud wrote: >> >> Hi all, >> >> As far as I understand, in the upcoming version 13, information about >> buffers used during planning is now available in the explain plan. > > Indeed. > >> […]

Re: problem with RETURNING and update row movement

2020-08-07 Thread Etsuro Fujita
On Mon, Aug 3, 2020 at 4:39 PM Amit Langote wrote: > On Mon, Aug 3, 2020 at 2:54 PM Amit Langote wrote: > > By the way, you'll need two adjustments to even get this example > > working, one of which is a reported problem that system columns in > > RETURNING list during an operation on

Re: Yet another issue with step generation in partition pruning

2020-08-07 Thread Etsuro Fujita
On Fri, Aug 7, 2020 at 2:55 PM Etsuro Fujita wrote: > On Thu, Aug 6, 2020 at 12:20 AM Etsuro Fujita wrote: > > Will push the patch tomorrow. > > Done. (I didn't have time for this, because I was terribly busy with > other stuff.) I mean I didn't have time for this *yesterday*. Best regards,

Re: [PG13] Planning (time + buffers) data structure in explain plan (format text)

2020-08-07 Thread Julien Rouhaud
Hi, On Fri, Aug 7, 2020 at 2:30 PM Pierre Giraud wrote: > > Hi all, > > As far as I understand, in the upcoming version 13, information about > buffers used during planning is now available in the explain plan. Indeed. > […] > Planning Time: 0.203 ms >Buffers: shared hit=14 > […] > > In

Re: LSM tree for Postgres

2020-08-07 Thread Alexander Korotkov
ср, 5 авг. 2020 г., 09:13 Konstantin Knizhnik : > Concerning degrade of basic index - B-Tree itself is balanced tree. Yes, > insertion of random keys can cause split of B-Tree page. > In the worst case half of B-Tree page will be empty. So B-Tree size will > be two times larger than ideal tree. >

[PG13] Planning (time + buffers) data structure in explain plan (format text)

2020-08-07 Thread Pierre Giraud
Hi all, As far as I understand, in the upcoming version 13, information about buffers used during planning is now available in the explain plan. […] Planning Time: 0.203 ms Buffers: shared hit=14 […] In the JSON format, the data structure is a bit different: […] "Planning": {

Re: ModifyTable overheads in generic plans

2020-08-07 Thread Amit Langote
On Tue, Aug 4, 2020 at 3:15 PM Amit Langote wrote: > On Sat, Aug 1, 2020 at 4:46 AM Robert Haas wrote: > > On Fri, Jun 26, 2020 at 8:36 AM Amit Langote > > wrote: > > > 0001 and 0002 are preparatory patches. > > > > I read through these patches a bit but it's really unclear what the > > point

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-08-07 Thread Tomas Vondra
On Fri, Aug 07, 2020 at 10:08:23AM +0300, Konstantin Knizhnik wrote: On 07.08.2020 00:33, Tomas Vondra wrote: Unfortunately Konstantin did not share any details about what workloads he tested, what config etc. But I find the "no regression" hypothesis rather hard to believe, because we're

[PATCH] Covering SPGiST index

2020-08-07 Thread Pavel Borisov
Hi, hackers! I'd like to propose a patch which introduces a functionality to include additional columns to SPGiST index to increase speed of queries containing them due to making the scans index only in this case. To date this functionality was available in GiSt and btree, I suppose the same is

Re: [Proposal] Global temporary tables

2020-08-07 Thread movead...@highgo.ca
>I find this is the most latest mail with an attachment, so I test and reply on >this thread, several points as below: >1. I notice it produces new relfilenode when new session login and some >data insert. But the relfilenode column in pg_class still the one when create >the global temp table. I

Re: walsender waiting_for_ping spuriously set

2020-08-07 Thread Ashutosh Bapat
The patch looks good to me. Thanks for improving comments around that code. I like the change to set waiting_for_ping_response in WalSndKeepalive. Thanks. On Fri, 7 Aug 2020 at 04:26, Alvaro Herrera wrote: > Ashutosh Bapat noticed that WalSndWaitForWal() is setting > waiting_for_ping_response

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-08-07 Thread Amit Langote
On Mon, Aug 3, 2020 at 8:38 PM Amit Langote wrote: > On Wed, Jul 29, 2020 at 5:36 PM Andrey V. Lepikhov > wrote: > > > Will send more comments after reading the v5 patch. > > > > > Ok. I'll be waiting for the end of your review. > > Sorry about the late reply. > > If you'd like to send a new

Re: Parallel worker hangs while handling errors.

2020-08-07 Thread Bharath Rupireddy
On Fri, Aug 7, 2020 at 1:34 AM Robert Haas wrote: > > On Tue, Jul 28, 2020 at 5:35 AM Bharath Rupireddy > wrote: > > The v4 patch looks good to me. Hang is not seen, make check and make > > check-world passes. I moved this to the committer for further review > > in

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-08-07 Thread k.jami...@fujitsu.com
On Friday, August 7, 2020 12:38 PM, Amit Kapila wrote: Hi, > On Thu, Aug 6, 2020 at 6:53 AM k.jami...@fujitsu.com > wrote: > > > > On Saturday, August 1, 2020 5:24 AM, Andres Freund wrote: > > > > Hi, > > Thank you for your constructive review and comments. > > Sorry for the late reply. > > > > >

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-08-07 Thread Dilip Kumar
On Thu, Aug 6, 2020 at 2:43 PM Amit Kapila wrote: > > On Wed, Aug 5, 2020 at 7:37 PM Dilip Kumar wrote: > > > > On Wed, Aug 5, 2020 at 6:25 PM Amit Kapila wrote: > > > > > > > > Can we add a test for incomplete changes (probably with toast > > > insertion but we can do it for spec_insert case

Re: [Proposal] Global temporary tables

2020-08-07 Thread 曾文旌
> 2020年7月31日 上午4:57,Robert Haas 写道: > > On Thu, Jul 30, 2020 at 8:09 AM wenjing zeng wrote: >> Please continue to review the code. > > This patch is pretty light on comments. Many of the new functions have > no header comments, for example. There are comments here and there in > the body of

Re: [Proposal] Global temporary tables

2020-08-07 Thread 曾文旌
Thank you very much for reviewing this patch. This is very important to improve the GTT. > 2020年8月3日 下午3:09,movead...@highgo.ca 写道: > > > >Fixed in global_temporary_table_v29-pg13.patch > >Please check. > > I find this is the most latest mail with an attachment, so I test and reply on > this

Re: Creating a function for exposing memory usage of backend process

2020-08-07 Thread Kasahara Tatsuhito
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: not tested Documentation:tested, passed I tested the latest

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-08-07 Thread Ashutosh Sharma
Thanks Rajkumar for testing the patch. Here are some of the additional test-cases that I would suggest you to execute, if possible: 1) You may try running the test-cases that you have executed so far with SR setup and see if the changes are getting reflected on the standby. 2) You may also try

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-08-07 Thread Konstantin Knizhnik
On 07.08.2020 00:33, Tomas Vondra wrote: Unfortunately Konstantin did not share any details about what workloads he tested, what config etc. But I find the "no regression" hypothesis rather hard to believe, because we're adding non-trivial amount of code to a place that can be quite hot.

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-08-07 Thread Ashutosh Sharma
On Thu, Aug 6, 2020 at 9:19 PM Robert Haas wrote: > > On Thu, Aug 6, 2020 at 2:11 AM Ashutosh Sharma wrote: > > Okay, If you want I can remove the restriction on a toast table, but, > > then that means a user can directly remove the data chunks from the > > toast table without changing anything

Re: Switch to multi-inserts for pg_depend

2020-08-07 Thread Michael Paquier
On Fri, Aug 07, 2020 at 03:16:19PM +0900, Michael Paquier wrote: > I am adding this thread to the next commit fest. Thoughts are > welcome. Forgot to mention that this is based on some initial work from Daniel, and that we have discussed the issue before I worked on it. -- Michael

Switch to multi-inserts for pg_depend

2020-08-07 Thread Michael Paquier
Hi all, This is a continuation of the work that has been previously discussed here, resulting mainly in e3931d0 for pg_attribute and pg_shdepend: https://www.postgresql.org/message-id/20190213182737.mxn6hkdxwrzgx...@alap3.anarazel.de I have been looking at the amount of work that could be done

Re: [PATCH] - Provide robust alternatives for replace_string

2020-08-07 Thread Asim Praveen
> On 05-Aug-2020, at 7:01 PM, Alvaro Herrera wrote: > > On 2020-Aug-05, Asim Praveen wrote: > >> Please find attached a StringInfo based solution to this problem. It >> uses fgetln instead of fgets such that a line is read in full, without >> ever splitting it. > > never heard of fgetln, my

Re: Amcheck: do rightlink verification with lock coupling

2020-08-07 Thread Andrey M. Borodin
> 6 авг. 2020 г., в 21:38, Peter Geoghegan написал(а): > > On Wed, Aug 5, 2020 at 9:50 PM Andrey M. Borodin wrote: >> Sounds great! Thanks! > > I'm afraid that there is another problem, this time with > btree_xlog_split(). It's possible to get false positives when running > the new test