回复:Re: Regarding the necessity of RelationGetNumberOfBlocks for every rescan / bitmap heap scan.

2021-05-30 Thread 陈佳昕(步真)
+1, This would be an nice improvement even the lseek is fast usually, it is a system call after all Buzhen-- 发件人:Andy Fan 日 期:2021年05月31日 13:46:22 收件人:PostgreSQL Hackers 主 题:Re: Regarding the necessity of RelationGetNumberOfBlocks

Re: Regarding the necessity of RelationGetNumberOfBlocks for every rescan / bitmap heap scan.

2021-05-30 Thread Andy Fan
On Sat, May 29, 2021 at 11:23 AM Andy Fan wrote: > Hi: > > I'm always confused about the following codes. > > static void > initscan(HeapScanDesc scan, ScanKey key, bool keep_startblock) > { > ParallelBlockTableScanDesc bpscan = NULL; > bool allow_strat; > bool allow_sync; > > /* > * Determine

Re: Parallel Full Hash Join

2021-05-30 Thread Greg Nancarrow
On Sat, Mar 6, 2021 at 12:31 PM Thomas Munro wrote: > > On Tue, Mar 2, 2021 at 11:27 PM Thomas Munro wrote: > > On Fri, Feb 12, 2021 at 11:02 AM Melanie Plageman > > wrote: > > > I just attached the diff. > > > > Squashed into one patch for the cfbot to chew on, with a few minor > > adjustments

Re: Bracket, brace, parenthesis

2021-05-30 Thread Kyotaro Horiguchi
At Mon, 31 May 2021 11:36:23 +0900, Michael Paquier wrote in > On Fri, May 28, 2021 at 03:25:40PM +0900, Kyotaro Horiguchi wrote: > > Mmm. Thanks. So the test doesn't a check for the case of trailing > > garbage. Looking the discussion about trailing garbage of integer > > values, we might

Re: Multiple hosts in connection string failed to failover in non-hot standby mode

2021-05-30 Thread Tom Lane
Michael Paquier writes: > In my recent quest to look at GSSAPI builds on Windows, I have bumped > into another failure that's related to this thread. hamerkop > summarizes the situation here: > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hamerkop=2021-05-29%2010%3A15%3A42 > There are

Re: Fdw batch insert error out when set batch_size > 65535

2021-05-30 Thread Bharath Rupireddy
On Mon, May 31, 2021 at 1:21 AM Tomas Vondra wrote: > > Hi, > > I took at this patch today. I did some minor changes, mostly: > > 1) change the code limiting batch_size from > > if (fmstate->p_nums > 0 && >(batch_size * fmstate->p_nums > PQ_QUERY_PARAM_MAX_LIMIT)) > { >

Re: pg_get_wal_replay_pause_state() should not return 'paused' while a promotion is ongoing.

2021-05-30 Thread Fujii Masao
On 2021/05/19 19:24, Fujii Masao wrote: On 2021/05/19 16:43, Kyotaro Horiguchi wrote: +1 for adding some tests for pg_wal_replay_pause() but the test seems like checking only that pg_get_wal_replay_pause_state() returns the expected state value.  Don't we need to check that the recovery is

Re: Multiple hosts in connection string failed to failover in non-hot standby mode

2021-05-30 Thread Michael Paquier
On Mon, May 31, 2021 at 11:00:55AM +0900, Michael Paquier wrote: > Yeah. On the contrary, it could be confusing if one sees an error > message but there is nothing to worry about, because things are > working in the scope of what the user wanted at connection time. In my recent quest to look at

Re: Forget close an open relation in ReorderBufferProcessTXN()

2021-05-30 Thread Amit Langote
On Thu, May 27, 2021 at 3:36 PM Amit Kapila wrote: > On Fri, May 21, 2021 at 1:12 PM Amit Langote wrote: > > > > Hmm, maybe get_rel_syn_entry() should explicitly set map to NULL when > > first initializing an entry. It's possible that without doing so, the > > map remains set to a garbage

Re: Decoding speculative insert with toast leaks memory

2021-05-30 Thread Dilip Kumar
On Mon, 31 May 2021 at 8:21 AM, Amit Kapila wrote: > On Sat, May 29, 2021 at 5:45 PM Tomas Vondra > wrote: > > > > On 5/29/21 6:29 AM, Amit Kapila wrote: > > > On Fri, May 28, 2021 at 5:16 PM Tomas Vondra > > > wrote: > > >> > > >> I wonder if there's a way to free the TOASTed data earlier,

Re: Race condition in recovery?

2021-05-30 Thread Tatsuro Yamada
Hi Horiguchi-san, (Why me?) Because the story was also related to PG-REX, which you are also involved in developing. Perhaps off-list instead of -hackers would have been better, but I emailed -hackers because the same problem could be encountered by PostgreSQL users who do not use PG-REX.

Re: Decoding speculative insert with toast leaks memory

2021-05-30 Thread Amit Kapila
On Sat, May 29, 2021 at 5:45 PM Tomas Vondra wrote: > > On 5/29/21 6:29 AM, Amit Kapila wrote: > > On Fri, May 28, 2021 at 5:16 PM Tomas Vondra > > wrote: > >> > >> I wonder if there's a way to free the TOASTed data earlier, instead of > >> waiting until the end of the transaction (as this patch

Re: Bracket, brace, parenthesis

2021-05-30 Thread Michael Paquier
On Fri, May 28, 2021 at 03:25:40PM +0900, Kyotaro Horiguchi wrote: > Mmm. Thanks. So the test doesn't a check for the case of trailing > garbage. Looking the discussion about trailing garbage of integer > values, we might need one for the case. > > The atached second file adds a test for

RE: [BUG]Update Toast data failure in logical replication

2021-05-30 Thread tanghy.f...@fujitsu.com
On Friday, May 28, 2021 6:51 PM, Dilip Kumar wrote: > Seems you did not set the replica identity for updating the tuple. > Try this before updating, and it should work. Thanks for your reply. I tried it. > ALTER TABLE toasted_key REPLICA IDENTITY USING INDEX toasted_key_pkey; This didn't work.

Re: Multiple hosts in connection string failed to failover in non-hot standby mode

2021-05-30 Thread Michael Paquier
On Sun, May 30, 2021 at 08:25:00PM -0500, Justin Pryzby wrote: > ..But I think it's not useful to put details into errorMessage on success, > unless you're going to document that. It would never have occurred to me to > look there, or that it would even be safe. Yeah. On the contrary, it could

Re: Teaching users how they can get the most out of HOT in Postgres 14

2021-05-30 Thread Masahiko Sawada
On Fri, May 28, 2021 at 9:53 AM Peter Geoghegan wrote: > > On Sun, May 23, 2021 at 11:34 PM Masahiko Sawada > wrote: > > I think the possible side effect of this hard-coded > > BYPASS_THRESHOLD_PAGES would be that by default, bulkdelete is not > > called for a long term and the index becomes

Re: Multiple hosts in connection string failed to failover in non-hot standby mode

2021-05-30 Thread Justin Pryzby
On Thu, May 06, 2021 at 01:22:27PM -0400, Tom Lane wrote: > Justin Pryzby writes: > > 52a10224 broke sqlsmith, of all things. > > > It was using errmsg as a test of success, instead of checking if the > > connection > > result wasn't null: > > > conn = PQconnectdb(conninfo.c_str()); > >

Re: be-secure-gssapi.c and auth.c with setenv() not compatible on Windows

2021-05-30 Thread Michael Paquier
On Sat, May 29, 2021 at 10:44:14AM -0400, Tom Lane wrote: > What I had in mind was to *only* add pgwin32_setenv, not setenv.c. > There's no evidence that any other modern platform lacks setenv. > Moreover, there's no issue in these branches unless your platform > lacks setenv yet has GSS support.

Re: postgres_fdw batching vs. (re)creating the tuple slots

2021-05-30 Thread Tom Lane
Andres Freund writes: > On 2021-05-30 17:10:59 -0400, Tom Lane wrote: >> But it does seem like the hashing scheme somebody added to resowners >> is a bit too simplistic. It ought to be able to cope with lots of >> refs to the same object, or at least not be extra-awful for that case. > It's not

Re: O_DIRECT on macOS

2021-05-30 Thread Thomas Munro
On Mon, May 31, 2021 at 4:19 AM Justin Pryzby wrote: > Should there be an "else" to warn/error in the case that "direct" is requested > but not supported? The way we use O_DIRECT currently is extremely minimal, it's just "if you've got it, we'll use it, but otherwise not complain", and I wasn't

Re: O_DIRECT on macOS

2021-05-30 Thread Thomas Munro
On Mon, May 31, 2021 at 8:12 AM Andres Freund wrote: > On 2021-05-30 16:39:48 +1200, Thomas Munro wrote: > > I thought about a few different ways to encapsulate this API > > difference in PostgreSQL, and toyed with two: > > > > 1. We could define our own fake O_DIRECT flag, and translate that to

Re: Performance degradation of REFRESH MATERIALIZED VIEW

2021-05-30 Thread Tomas Vondra
Hi, Based on the investigation and (lack of) progress so far, I'll revert part of the COPY FREEZE improvements shortly. I'll keep the initial 7db0cd2145 changes, tweaking heap_multi_insert, and remove most of 39b66a91bd (except for the heap_xlog_multi_insert bit). This should address the small

Re: storing an explicit nonce

2021-05-30 Thread Andres Freund
Hi, On 2021-05-27 17:00:23 -0400, Bruce Momjian wrote: > If you go in that direction, you should make sure pg_upgrade preserves > what you use (it does not preserve relfilenode, just pg_class.oid) Is there a reason for pg_upgrade not to maintain relfilenode, aside from implementation simplicity

Re: postgres_fdw batching vs. (re)creating the tuple slots

2021-05-30 Thread Andres Freund
Hi, On 2021-05-30 17:10:59 -0400, Tom Lane wrote: > But it does seem like the hashing scheme somebody added to resowners > is a bit too simplistic. It ought to be able to cope with lots of > refs to the same object, or at least not be extra-awful for that case. It's not really the hashing

Re: postgres_fdw batching vs. (re)creating the tuple slots

2021-05-30 Thread Tom Lane
Andres Freund writes: > On 2021-05-30 22:22:10 +0200, Tomas Vondra wrote: >> The other problem is that ResourceArrayAdd/Remove seem to behave a bit >> poorly with very many elements - I'm not sure if it's O(N^2) or worse, >> but growing the array and linear searches seem to be a bit expensive. >

Re: Clear empty space in a page.

2021-05-30 Thread Andres Freund
Hi, On 2021-05-30 03:10:26 +0300, Yura Sokolov wrote: > While this result is not directly applied to stock PostgreSQL, I believe > page compression is important for full_page_writes with wal_compression > enabled. And probably when PostgreSQL is used on filesystem with > compression enabled

Re: postgres_fdw batching vs. (re)creating the tuple slots

2021-05-30 Thread Andres Freund
Hi, On 2021-05-30 22:22:10 +0200, Tomas Vondra wrote: > There are two problems at play, here. Firstly, the way it's coded now > the slots are pretty much re-created for each batch. So with 1M rows and > batches of 32k rows, that's ~30x drop/create. That seems a bit wasteful, > and it shouldn't be

Re: GISTSTATE is too large

2021-05-30 Thread Andres Freund
Hi, On 2021-05-30 15:14:33 +0200, Andreas Karlsson wrote: > I did the first part since it seemed easy enough and an obvious win for all > workloads. Cool! > +typedef struct GIST_COL_STATE > +{ > + FmgrInfoconsistentFn; > + FmgrInfounionFn; > + FmgrInfo

postgres_fdw batching vs. (re)creating the tuple slots

2021-05-30 Thread Tomas Vondra
Hi, while looking at the other thread related to postgres_fdw batching [1] and testing with very large batches, I noticed this disappointing behavior when inserting 1M rows (just integers, nothing fancy): no batching: 64782 ms 100 rows: 2118 ms 32767 rows: 41115 ms Pretty nice improvement

Re: O_DIRECT on macOS

2021-05-30 Thread Andres Freund
Hi, Thanks for starting the discussion on this! On 2021-05-30 16:39:48 +1200, Thomas Munro wrote: > I thought about a few different ways to encapsulate this API > difference in PostgreSQL, and toyed with two: > > 1. We could define our own fake O_DIRECT flag, and translate that to > the right

Re: list of extended statistics on psql (\dX)

2021-05-30 Thread Tomas Vondra
On 5/30/21 7:24 PM, Justin Pryzby wrote: > On Wed, Jan 20, 2021 at 11:00:50PM +0100, Tomas Vondra wrote: >> Thanks, I've pushed this. I had to tweak the regression tests a bit, for two >> reasons: > > \dX isn't checking schema visibility rules, so accidentally shows stats > objects > outside

Re: Fdw batch insert error out when set batch_size > 65535

2021-05-30 Thread Tomas Vondra
Hi, I took at this patch today. I did some minor changes, mostly: 1) change the code limiting batch_size from if (fmstate->p_nums > 0 && (batch_size * fmstate->p_nums > PQ_QUERY_PARAM_MAX_LIMIT)) { batch_size = PQ_QUERY_PARAM_MAX_LIMIT / fmstate->p_nums; } to if

Re: list of extended statistics on psql (\dX)

2021-05-30 Thread Justin Pryzby
On Wed, Jan 20, 2021 at 11:00:50PM +0100, Tomas Vondra wrote: > Thanks, I've pushed this. I had to tweak the regression tests a bit, for two > reasons: \dX isn't checking schema visibility rules, so accidentally shows stats objects outside of the search path. I noticed after installing the

Reducing memory consumption for pending inval messages

2021-05-30 Thread Tom Lane
I got interested in $SUBJECT as a result of the thread at [1]. It turns out that the existing implementation in inval.c is quite inefficient when a lot of individual commands each register just a few invalidations --- but a few invalidations per command is pretty typical. As an example, consider

Re: Clear empty space in a page.

2021-05-30 Thread Omar Kilani
Hi, I happened to be running some postgres on zfs on Linux/aarch64 tests and tested this patch. Kernel: 4.18.0-305.el8.aarch64 CPU: 16x3.0GHz Ampere Alta / Arm Neoverse N1 cores ZFS: 2.1.0-rc6 ZFS options: options spl spl_kmem_cache_slab_limit=65536 (see:

Re: O_DIRECT on macOS

2021-05-30 Thread Justin Pryzby
On Sun, May 30, 2021 at 04:39:48PM +1200, Thomas Munro wrote: > +BasicOpenFilePermDirect(const char *fileName, int fileFlags, mode_t fileMode, > + bool direct) > ... > +#if !defined(O_DIRECT) && defined(F_NOCACHE) > + /* macOS requires an extra

Re: GISTSTATE is too large

2021-05-30 Thread Zhihong Yu
On Sun, May 30, 2021 at 6:14 AM Andreas Karlsson wrote: > On 4/26/21 12:20 AM, Andres Freund wrote: > > It seems pretty clear that this should be changed to be something more > > like > > > > [...] > > > > with initGISTstate allocating based on > > IndexRelationGetNumberOfKeyAttributes() instead

Re: GISTSTATE is too large

2021-05-30 Thread Andreas Karlsson
On 4/26/21 12:20 AM, Andres Freund wrote: It seems pretty clear that this should be changed to be something more like [...] with initGISTstate allocating based on IndexRelationGetNumberOfKeyAttributes() instead of using a constant. And then subsequently change GIST_COL_STATE to embed the

Re: [PATCH] expand the units that pg_size_pretty supports on output

2021-05-30 Thread Asif Rehman
The following review has been posted through the commitfest application: make installcheck-world: not tested Implements feature: not tested Spec compliant: not tested Documentation:not tested Hi David, I was reviewing this patch and the compilation failed with

Re: [PATCH] More docs on what to do and not do in extension code

2021-05-30 Thread Laurenz Albe
On Mon, 2021-01-18 at 15:56 +0800, Craig Ringer wrote: > The attached patch expands the xfunc docs and bgworker docs a little, > providing a starting point for developers > to learn how to do some common tasks the Postgres Way. I like these changes! Here is a review: + + See for

psql - factor out echo code

2021-05-30 Thread Fabien COELHO
While working on something in "psql/common.c" I noticed some triplicated code, including a long translatable string. This minor patch refactors this in one function. -- Fabien.diff --git a/src/bin/psql/common.c b/src/bin/psql/common.c index 7a95465111..4fd80ec6bb 100644 ---

Re: Add ZSON extension to /contrib/

2021-05-30 Thread Joel Jacobson
On Tue, May 25, 2021, at 22:10, Tom Lane wrote: > Magnus Hagander mailto:magnus%40hagander.net>> writes: > > On Tue, May 25, 2021 at 12:55 PM Aleksander Alekseev > > mailto:aleksander%40timescale.com>> wrote: > >> Back in 2016 while being at PostgresPro I developed the ZSON extension > >> [1].

Re: security_definer_search_path GUC

2021-05-30 Thread Pavel Stehule
ne 30. 5. 2021 v 8:52 odesílatel Joel Jacobson napsal: > On Sat, May 29, 2021, at 22:10, Marko Tiikkaja wrote: > > On Sat, May 29, 2021 at 11:06 PM Joel Jacobson wrote: > > > Glad you bring this problem up for discussion, something should be done to > improve the situation. > > Personally, as I

Re: security_definer_search_path GUC

2021-05-30 Thread Joel Jacobson
On Sun, May 30, 2021, at 08:51, Joel Jacobson wrote: > Maybe this is out of scope for extensions, since I guess extensions are > supposed to add features? > > If so, how about a new separate command `CREATE REDUCTION` specifically to > remove unwanted core features, > which then wouldn't need

Re: security_definer_search_path GUC

2021-05-30 Thread Joel Jacobson
On Sat, May 29, 2021, at 22:10, Marko Tiikkaja wrote: > On Sat, May 29, 2021 at 11:06 PM Joel Jacobson wrote: >> __ >> Glad you bring this problem up for discussion, something should be done to >> improve the situation. >> >> Personally, as I really dislike search_path and consider using it an