RE: Disable WAL logging to speed up data loading

2020-11-18 Thread tsunakawa.ta...@fujitsu.com
(1) #define RelationNeedsWAL(relation) \ + (wal_level != WAL_LEVEL_NONE && \ ((relation)->rd_rel->relpersistence ==

Re: Disable WAL logging to speed up data loading

2020-11-18 Thread Laurenz Albe
On Thu, 2020-11-19 at 05:24 +, osumi.takami...@fujitsu.com wrote: > > > > ereport(WARNING, > > > > (errmsg("WAL was generated with wal_level=minimal, data may > > > > be missing"), > > > > errhint("This happens if you temporarily set > > > > wal_level=minimal without taking a

Use macros for calculating LWLock offset

2020-11-18 Thread japin
Hi, In the lwlock.c, InitializeLWLocks() calculate the LWLock offset by itself (c319991bcad), however, there are macros defined in lwlock.h, I think, we can use the macros. diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c index 2fa90cc095..108e652179

Re: Is postgres ready for 2038?

2020-11-18 Thread Tom Lane
Pavel Borisov writes: > чт, 19 нояб. 2020 г. в 09:29, Greg Stark : >> Wait, is configuring with a Perl that has 32-bit time_t driving the >> rest of Postgres to use 32-bit timestamps? That seems like the tail >> wagging the dog. >> It seems like a sensible compromise would be to have Postgres's

Re: Add statistics to pg_stat_wal view for wal related parameter tuning

2020-11-18 Thread Masahiro Ikeda
On 2020-11-17 11:46, Fujii Masao wrote: On 2020/11/16 16:35, Masahiro Ikeda wrote: On 2020-11-12 14:58, Fujii Masao wrote: On 2020/11/06 10:25, Masahiro Ikeda wrote: On 2020-10-30 11:50, Fujii Masao wrote: On 2020/10/29 17:03, Masahiro Ikeda wrote: Hi, Thanks for your comments and advice.

Re: Is postgres ready for 2038?

2020-11-18 Thread Pavel Borisov
чт, 19 нояб. 2020 г. в 09:29, Greg Stark : > On Wed, 18 Nov 2020 at 12:22, Tom Lane wrote: > > > > Andrew Dunstan writes: > > > On 11/18/20 9:44 AM, Tom Lane wrote: > > >> Hmm. Digging around, I see that Mkvcbuild.pm intentionally absorbs > > >> _USE_32BIT_TIME_T when building with a Perl that

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-11-18 Thread tsunakawa.ta...@fujitsu.com
From: Andres Freund > DropRelFileNodeBuffers() in recovery? The most common path is > DropRelationFiles()->smgrdounlinkall()->DropRelFileNodesAllBuffers(), > which 3/4 doesn't address and 4/4 doesn't mention. > > 4/4 seems to address DropRelationFiles(), but only talks about TRUNCATE? Yes.

Re: Should we document IS [NOT] OF?

2020-11-18 Thread Tom Lane
"David G. Johnston" writes: > Is there a feature code? I skimmed the standard and non-standard tables in > our appendix and couldn’t find this in either. a19d9d3c4 seems to have thought it was S151. regards, tom lane

Re: Should we document IS [NOT] OF?

2020-11-18 Thread David G. Johnston
On Wednesday, November 18, 2020, Tom Lane wrote: > I wrote: > > So my vote would be to rip it out, not document it. Somebody can try > > again in future, perhaps. But if we document it we're just locking > > ourselves into a SQL incompatibility. > > Apparently, somebody already had that

Re: don't allocate HashAgg hash tables when running explain only

2020-11-18 Thread Heikki Linnakangas
On 19/11/2020 07:20, Greg Stark wrote: On Wed, 18 Nov 2020 at 05:40, Heikki Linnakangas wrote: On 13/11/2020 18:10, Alexey Bashtanov wrote: I would appreciate if someone could have a look at the patch attached, which makes executor skip initializing hash tables when doing explain only.

Re: Should we document IS [NOT] OF?

2020-11-18 Thread Tom Lane
I wrote: > So my vote would be to rip it out, not document it. Somebody can try > again in future, perhaps. But if we document it we're just locking > ourselves into a SQL incompatibility. Apparently, somebody already had that thought. See func.sgml lines 765-782, which were commented out by

Re: Protect syscache from bloating with negative cache entries

2020-11-18 Thread Kyotaro Horiguchi
At Wed, 18 Nov 2020 21:42:02 -0800, Andres Freund wrote in > Hi, > > On 2020-11-19 14:25:36 +0900, Kyotaro Horiguchi wrote: > > # Creation, searching and expiration > > master : 6393.23(100.0) > > patched-off: 6527.94(102.1) > > patched-on : 15880.01(248.4) > > What's the

Re: [doc] adding way to examine the plan type of prepared statements

2020-11-18 Thread torikoshia
On 2020-11-18 11:04, torikoshia wrote: Hi, Currently, EXPLAIN is the only way to know whether the plan is generic or custom according to the manual of PREPARE. https://www.postgresql.org/docs/devel/sql-prepare.html After commit d05b172, we can also use pg_prepared_statements view to

Re: Should we document IS [NOT] OF?

2020-11-18 Thread Tom Lane
"David G. Johnston" writes: > Over in news [1] Josh Drake and Eric Ridge discovered the undocumented > feature "IS [NOT] OF"; introduced seemingly as an "oh-by-the-way" in 2002 > via commit eb121ba2cfe [2]. > Is there any reason not to document this back to 9.5, As far as I can tell from

Re: [HACKERS] logical decoding of two-phase transactions

2020-11-18 Thread Amit Kapila
On Thu, Nov 19, 2020 at 11:27 AM Ajin Cherian wrote: > > > Why did you make a change in stream_start API? I think it should be > > *_change and *_truncate APIs because the concurrent abort can happen > > while decoding any intermediate change. If you agree then you can > > probably take that code

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-11-18 Thread Amit Kapila
On Wed, Nov 18, 2020 at 11:43 PM Andres Freund wrote: > > Hi, > > On 2020-11-18 17:34:31 +0530, Amit Kapila wrote: > > Yeah, that won't be a bad idea especially because the patch being > > discussed in the thread you referred is still in an exploratory phase. > > I haven't tested or done a

Should we document IS [NOT] OF?

2020-11-18 Thread David G. Johnston
Hackers, Over in news [1] Josh Drake and Eric Ridge discovered the undocumented feature "IS [NOT] OF"; introduced seemingly as an "oh-by-the-way" in 2002 via commit eb121ba2cfe [2]. Is there any reason not to document this back to 9.5, probably with a note nearby to pg_typeof(any), which is a

Re: Protect syscache from bloating with negative cache entries

2020-11-18 Thread Andres Freund
Hi, On 2020-11-19 14:25:36 +0900, Kyotaro Horiguchi wrote: > # Creation, searching and expiration > master : 6393.23(100.0) > patched-off: 6527.94(102.1) > patched-on : 15880.01(248.4) What's the deal with this massive increase here? Greetings, Andres Freund

Re: [doc] plan invalidation when statistics are update

2020-11-18 Thread torikoshia
On 2020-11-18 11:35, Fujii Masao wrote: Thanks for your comment! On 2020/11/18 11:04, torikoshia wrote: Hi, AFAIU, when the planner statistics are updated, generic plans are invalidated and PostgreSQL recreates. However, the manual doesn't seem to explain it explicitly.  

Re: Is postgres ready for 2038?

2020-11-18 Thread Greg Stark
On Wed, 18 Nov 2020 at 12:22, Tom Lane wrote: > > Andrew Dunstan writes: > > On 11/18/20 9:44 AM, Tom Lane wrote: > >> Hmm. Digging around, I see that Mkvcbuild.pm intentionally absorbs > >> _USE_32BIT_TIME_T when building with a Perl that defines that. > >> I don't know what the state of play

Re: Protect syscache from bloating with negative cache entries

2020-11-18 Thread Kyotaro Horiguchi
Thank you for the comments. At Tue, 17 Nov 2020 16:22:54 -0500, Robert Haas wrote in > On Tue, Nov 17, 2020 at 10:46 AM Heikki Linnakangas wrote: > > 0.7% degradation is probably acceptable. > > I haven't looked at this patch in a while and I'm pleased with the way > it seems to have been

RE: Disable WAL logging to speed up data loading

2020-11-18 Thread osumi.takami...@fujitsu.com
Hello On Thursday, Nov 19, 2020 12:45 PM Tsunakawa, Takayuki/綱川 貴之 > > On Tuesday, Nov 3, 2020 3:02 AM Stephen Frost > > wrote: > > > Checking the WAL level certainly seems critical to anything that's > > > reading the WAL. We certainly do this already when running as a > > > replica: > > >

Re: Protect syscache from bloating with negative cache entries

2020-11-18 Thread Kyotaro Horiguchi
At Tue, 17 Nov 2020 17:46:25 +0200, Heikki Linnakangas wrote in > On 09/11/2020 11:34, Kyotaro Horiguchi wrote: > > At Fri, 6 Nov 2020 10:42:15 +0200, Heikki Linnakangas > > wrote in > >> Do you need the "ntaccess == 2" test? You could always increment the > >> counter, and in the code that

Re: don't allocate HashAgg hash tables when running explain only

2020-11-18 Thread Greg Stark
On Wed, 18 Nov 2020 at 05:40, Heikki Linnakangas wrote: > > On 13/11/2020 18:10, Alexey Bashtanov wrote: >> > I would appreciate if someone could have a look at the patch attached, > > which makes executor skip initializing hash tables when doing explain only. > > Makes sense. Committed, thanks

Re: Cache relation sizes?

2020-11-18 Thread Thomas Munro
On Tue, Nov 17, 2020 at 10:48 PM Amit Kapila wrote: > Yeah, it is good to verify VACUUM stuff but I have another question > here. What about queries having functions that access the same > relation (SELECT c1 FROM t1 WHERE c1 <= func(); assuming here function > access the relation t1)? Now, here

Re: pl/pgsql feature request: shorthand for argument and local variable references

2020-11-18 Thread Pavel Stehule
Hi this is less invasive, and probably more correct work with ns items patch. čt 19. 11. 2020 v 1:54 odesílatel Vik Fearing napsal: > On 11/18/20 9:21 PM, Pavel Stehule wrote: > > postgres=# create or replace function bubu(a int, b int) > > returns void as $$ > > #routine_label b > > begin > >

Re: [PATCH] postgres_fdw connection caching - cause remote sessions linger till the local session exit

2020-11-18 Thread Bharath Rupireddy
On Wed, Nov 18, 2020 at 10:32 PM Alexey Kondratov wrote: > > > Below is what I have in my mind, mostly inline with yours: > > > > a) Have a server level option (keep_connetion true/false, with the > > default being true), when set to false the connection that's made with > > this foreign server

Re: Optimising latch signals

2020-11-18 Thread Thomas Munro
On Fri, Nov 13, 2020 at 12:42 PM Thomas Munro wrote: > 1. It's a bit clunky that pqinitmask() takes a new argument to say > whether SIGURG should be blocked; that's because the knowledge of > which latch implementation we're using is private to latch.c, and only > the epoll version needs to

RE: Disable WAL logging to speed up data loading

2020-11-18 Thread tsunakawa.ta...@fujitsu.com
From: Osumi, Takamichi/大墨 昂道 > there were some ideas to trace the change of wal_level, > in other words, *stronger mechanism* to check wal_level. > I agree with the idea to have a new monitoring item > and would like to implement those kind of, or one of those ideas for the next > patch. > But

Re: "as quickly as possible" (was: remove spurious CREATE INDEX CONCURRENTLY wait)

2020-11-18 Thread Michael Paquier
On Thu, Nov 19, 2020 at 12:13:44PM +0900, Michael Paquier wrote: > That still looks useful for debugging, so DEBUG1 sounds fine to me. By the way, it strikes me that you could just do nothing as long as (log_min_messages > DEBUG1), so you could encapsulate most of the logic that plays with the

Re: Move OpenSSL random under USE_OPENSSL_RANDOM

2020-11-18 Thread Michael Paquier
On Wed, Nov 18, 2020 at 10:43:35AM +0100, Daniel Gustafsson wrote: > While it does simplify configure.ac, I'm just not a fan of the strict ordering > which is required without the labels even implying it. But that might just be > my personal preference. I just looked at that, and the attached

Re: POC: Cleaning up orphaned files using undo logs

2020-11-18 Thread Amit Kapila
On Wed, Nov 18, 2020 at 4:03 PM Antonin Houska wrote: > > Amit Kapila wrote: > > > On Fri, Nov 13, 2020 at 6:02 PM Antonin Houska wrote: > > > > > > Amit Kapila wrote: > > > > > > > On Thu, Nov 12, 2020 at 2:45 PM Antonin Houska wrote: > > > > > > > > > > > > > > > No background undo > > > >

Re: "as quickly as possible" (was: remove spurious CREATE INDEX CONCURRENTLY wait)

2020-11-18 Thread Michael Paquier
On Wed, Nov 18, 2020 at 02:48:40PM -0800, Andres Freund wrote: > On 2020-11-18 18:41:27 -0300, Alvaro Herrera wrote: >> We could make this more concurrent by copying lock->tag to a local >> variable, releasing the lock, then doing all the string formatting and >> printing. See attached

RE: POC: postgres_fdw insert batching

2020-11-18 Thread tsunakawa.ta...@fujitsu.com
From: Tomas Vondra > Unfortunately, this does not compile for me, because nodeModifyTable calls > ExecGetTouchedPartitions, which is not defined anywhere. Not sure what's > that about, so I simply commented-out this. That probably fails the > partitioned > cases, but it allowed me to do some

Re: ResourceOwner refactoring

2020-11-18 Thread Michael Paquier
On Wed, Nov 18, 2020 at 10:50:08AM +0200, Heikki Linnakangas wrote: > If RESOWNER_ARRAY_STATS is increased to 16, all the lookups fit in the > array. But I haven't done any benchmarking to see which is faster. My gut tells me that your guess is right, but it would be better to be sure. > BTW, I

RE: Disable WAL logging to speed up data loading

2020-11-18 Thread osumi.takami...@fujitsu.com
Hello In the past discussion of wal_level=none, there were some ideas to trace the change of wal_level, in other words, *stronger mechanism* to check wal_level. I agree with the idea to have a new monitoring item and would like to implement those kind of, or one of those ideas for the next

Re: CREATE AGGREGATE array_cat

2020-11-18 Thread Andres Freund
Hi, On 2020-11-18 19:54:52 -0500, Chapman Flack wrote: > On 11/18/20 19:46, David G. Johnston wrote: > > > I doubt there is any substantial resistance to including such a function > > but it would have to be written in C. > > Would anything have to be written at all, save the CREATE AGGREGATE >

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-11-18 Thread Michael Paquier
On Wed, Nov 18, 2020 at 11:09:28AM -0800, Andres Freund wrote: > Uh, wait a second. The acquisition of this lock hasn't been affected by > the snapshot scalability changes, and therefore are unrelated to > ->pgxactoff changing or not. > > In 13 this is: >

Re: Sloppiness around failure handling of parsePGArray in pg_dump

2020-11-18 Thread Michael Paquier
On Wed, Nov 18, 2020 at 10:19:40AM +0100, Daniel Gustafsson wrote: > I agree that we should fix this even if it will have quite limited impact in > production settings. Patch LGTM, +1. Thanks. I have reviewed that again this morning and applied it. > Another thing caught my eye here (while not

Re: CREATE AGGREGATE array_cat

2020-11-18 Thread Tom Lane
Vik Fearing writes: > On 11/19/20 1:54 AM, Chapman Flack wrote: >> Would anything have to be written at all, save the CREATE AGGREGATE >> suggested in the original message, using the existing array_cat as the >> state transition function? > Nope. As my example showed. But by the same token,

speaking of CREATE AGGREGATE ...

2020-11-18 Thread Chapman Flack
I saw in the docs for CREATE AGGREGATE, under argmode, "Aggregate functions do not support OUT arguments". And because I tend to poke things I'm told not to poke, I thought "but wait, the CREATE AGGREGATE doesn't have much to say about the return type anyway, that's just the FINALFUNC's return

Re: CREATE AGGREGATE array_cat

2020-11-18 Thread Vik Fearing
On 11/19/20 1:54 AM, Chapman Flack wrote: > On 11/18/20 19:46, David G. Johnston wrote: > >> I doubt there is any substantial resistance to including such a function >> but it would have to be written in C. > > Would anything have to be written at all, save the CREATE AGGREGATE > suggested in

Re: CREATE AGGREGATE array_cat

2020-11-18 Thread David G. Johnston
On Wed, Nov 18, 2020 at 5:54 PM Chapman Flack wrote: > On 11/18/20 19:46, David G. Johnston wrote: > > > I doubt there is any substantial resistance to including such a function > > but it would have to be written in C. > > Would anything have to be written at all, save the CREATE AGGREGATE >

Re: CREATE AGGREGATE array_cat

2020-11-18 Thread Chapman Flack
On 11/18/20 19:46, David G. Johnston wrote: > I doubt there is any substantial resistance to including such a function > but it would have to be written in C. Would anything have to be written at all, save the CREATE AGGREGATE suggested in the original message, using the existing array_cat as

Re: pl/pgsql feature request: shorthand for argument and local variable references

2020-11-18 Thread Vik Fearing
On 11/18/20 9:21 PM, Pavel Stehule wrote: > postgres=# create or replace function bubu(a int, b int) > returns void as $$ > #routine_label b > begin > raise notice '% %', b.a, b.b; > end; > $$ language plpgsql; Why not use the block labeling syntax we already have? create or replace function

Re: CREATE AGGREGATE array_cat

2020-11-18 Thread David G. Johnston
On Wed, Nov 18, 2020 at 5:37 PM Vik Fearing wrote: > On 11/18/20 11:19 PM, David G. Johnston wrote: > > On Wednesday, November 18, 2020, Vlad Bokov wrote: > > > >> Hi, I wonder why there's no function to aggregate arrays by > >> concatenation out of the box? > >> > > > > See array_agg(...) > >

Re: CREATE AGGREGATE array_cat

2020-11-18 Thread Vik Fearing
On 11/18/20 11:19 PM, David G. Johnston wrote: > On Wednesday, November 18, 2020, Vlad Bokov wrote: > >> Hi, I wonder why there's no function to aggregate arrays by >> concatenation out of the box? >> > > See array_agg(...) Why? That doesn't do what is wanted. vik=# select array_agg(a)

Re: PATCH: Batch/pipelining support for libpq

2020-11-18 Thread Andres Freund
Hi On Wed, Nov 18, 2020, at 09:51, Alvaro Herrera wrote: > On 2020-Nov-14, Daniel Verite wrote: > > > The patch I posted in [1] was pretty simple, but at the time, query > > results were always discarded. Now that pgbench can instantiate > > variables from query results, a script can do: > >

Re: "as quickly as possible" (was: remove spurious CREATE INDEX CONCURRENTLY wait)

2020-11-18 Thread Andres Freund
Hi, On 2020-11-18 18:41:27 -0300, Alvaro Herrera wrote: > The amount of stuff that this is doing with ProcArrayLock held > exclusively seems a bit excessive; it sounds like we could copy the > values we need first, release the lock, and *then* do all that memory > allocation and string printing

Re: Additional improvements to extended statistics

2020-11-18 Thread Tomas Vondra
On 11/17/20 4:35 PM, Dean Rasheed wrote: > On Thu, 12 Nov 2020 at 14:18, Tomas Vondra > wrote: >> >> Here is an improved WIP version of the patch series, modified to address >> the issue with repeatedly applying the extended statistics, as discussed >> with Dean in this thread. It's a bit

Re: Allow matching whole DN from a client certificate

2020-11-18 Thread Daniel Gustafsson
> On 18 Nov 2020, at 19:01, Andrew Dunstan wrote: > OK, here's a new patch, including docco and tests. Looks good on a quick skim, the only thing that stood out was: + This option is probably best used in comjunction with a username map. s/comjunction/conjunction/ Will do more testing

Re: More time spending with "delete pending"

2020-11-18 Thread Tom Lane
Alexander Lakhin writes: > 18.11.2020 23:39, Tom Lane wrote: >> Now, the ones on the 10 and 11 branches are all from pg_ctl, which >> does not use pgwin32_open() in those branches, only native open(). >> So those aren't fair to count against it. But we have nearly as >> many similar failures in

Re: PATCH: Batch/pipelining support for libpq

2020-11-18 Thread Alvaro Herrera
On 2020-Nov-14, Daniel Verite wrote: > The patch I posted in [1] was pretty simple, but at the time, query > results were always discarded. Now that pgbench can instantiate > variables from query results, a script can do: > select 1 as var \gset > select :var; > This kind of sequence wouldn't

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-11-18 Thread Alvaro Herrera
On 2020-Nov-18, Michael Paquier wrote: > On Tue, Nov 17, 2020 at 02:14:53PM -0300, Alvaro Herrera wrote: > > diff --git a/src/backend/replication/logical/logical.c > > b/src/backend/replication/logical/logical.c > > index f1f4df7d70..4324e32656 100644 > > ---

"as quickly as possible" (was: remove spurious CREATE INDEX CONCURRENTLY wait)

2020-11-18 Thread Alvaro Herrera
> On 2020-11-17 12:55:01 -0300, Alvaro Herrera wrote: > > ... ah, but I realize now that this means that we can use shared lock > > here, not exclusive, which is already an enormous improvement. That's > > because ->pgxactoff can only be changed with exclusive lock held; so as > > long as we hold

Re: CREATE AGGREGATE array_cat

2020-11-18 Thread David G. Johnston
On Wednesday, November 18, 2020, Vlad Bokov wrote: > Hi, I wonder why there's no function to aggregate arrays by > concatenation out of the box? > See array_agg(...) David J.

CREATE AGGREGATE array_cat

2020-11-18 Thread Vlad Bokov
Hi, I wonder why there's no function to aggregate arrays by concatenation out of the box? There is a usual function `array_cat(anyarray, anyarray)`, but it doesn't seamlessly work with grouping. Wouldn't it be natural to have this: CREATE AGGREGATE array_cat (anyarray) (     sfunc = array_cat,  

Re: WIP: WAL prefetch (another approach)

2020-11-18 Thread Stephen Frost
Greetings, * Thomas Munro (thomas.mu...@gmail.com) wrote: > On Sat, Nov 14, 2020 at 4:13 AM Stephen Frost wrote: > > * Tomas Vondra (tomas.von...@enterprisedb.com) wrote: > > > On 11/13/20 3:20 AM, Thomas Munro wrote: > > > > I'm not really sure what to do about achive restore scripts that > > >

Re: More time spending with "delete pending"

2020-11-18 Thread Alexander Lakhin
Hello Tom, 18.11.2020 23:39, Tom Lane wrote: > BTW ... scraping the buildfarm logs for "could not open ... Permission > denied" failures suggests that pgwin32_open() isn't the pinnacle of > perfection either. In the last three months I found these instances: > > dory | REL_11_STABLE |

Re: More time spending with "delete pending"

2020-11-18 Thread Tom Lane
BTW ... scraping the buildfarm logs for "could not open ... Permission denied" failures suggests that pgwin32_open() isn't the pinnacle of perfection either. In the last three months I found these instances: dory | REL_11_STABLE | 2020-08-21 22:15:14 | Check | pg_ctl: could

Re: pl/pgsql feature request: shorthand for argument and local variable references

2020-11-18 Thread Pavel Stehule
st 18. 11. 2020 v 21:21 odesílatel Pavel Stehule napsal: > > > st 18. 11. 2020 v 6:58 odesílatel Pavel Stehule > napsal: > >> >> >> út 17. 11. 2020 v 21:45 odesílatel Chapman Flack >> napsal: >> >>> On 11/17/20 15:18, Jack Christensen wrote: >>> >> CREATE OR REPLACE FUNCTION

Re: pl/pgsql feature request: shorthand for argument and local variable references

2020-11-18 Thread Pavel Stehule
st 18. 11. 2020 v 6:58 odesílatel Pavel Stehule napsal: > > > út 17. 11. 2020 v 21:45 odesílatel Chapman Flack > napsal: > >> On 11/17/20 15:18, Jack Christensen wrote: >> >> CREATE OR REPLACE FUNCTION very_long_name(par1 int) >> >> RETURNS int AS $$ >> >> #routine_label lnm >> >> BEGIN >> >>

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-11-18 Thread Andres Freund
Hi, On 2020-11-17 12:55:01 -0300, Alvaro Herrera wrote: > ... ah, but I realize now that this means that we can use shared lock > here, not exclusive, which is already an enormous improvement. That's > because ->pgxactoff can only be changed with exclusive lock held; so as > long as we hold

Re: cutting down the TODO list thread

2020-11-18 Thread Tom Lane
John Naylor writes: > Here are the next couple of sections with items proposed to be moved to the > "not worth doing" page. As before, if there are any objections, let me > know. I'll make the move in a few days. > - Fix /contrib/ltree operator > Bug from 2007 with zero followup Actually, I

Re: POC: postgres_fdw insert batching

2020-11-18 Thread Tomas Vondra
On 11/17/20 10:11 AM, tsunakawa.ta...@fujitsu.com wrote: > Hello, > > > Modified the patch as I talked with Tomas-san. The performance > results of loading one million records into a hash-partitioned table > with 8 partitions are as follows: > > unpatched, local: 8.6 seconds unpatched, fdw:

Re: More time spending with "delete pending"

2020-11-18 Thread Tom Lane
Alexander Lakhin writes: > 15.11.2020 04:11, Justin Pryzby wrote: >> Your patch introduces a "loops", but doesn't use it to escape the loop. > Indeed, this is my mistake. Please look at the corrected patch (now that > code corresponds to the pgwin32_open() as intended). So what you're saying

Re: cutting down the TODO list thread

2020-11-18 Thread Bruce Momjian
On Wed, Nov 18, 2020 at 02:26:46PM -0400, John Naylor wrote: > Here are the next couple of sections with items proposed to be moved to the > "not worth doing" page. As before, if there are any objections, let me know. > I'll make the move in a few days.  > > Also, since 13 has been released, I'll

Re: cutting down the TODO list thread

2020-11-18 Thread John Naylor
Here are the next couple of sections with items proposed to be moved to the "not worth doing" page. As before, if there are any objections, let me know. I'll make the move in a few days. Also, since 13 has been released, I'll change the explanation of Done items to "will appear in the PostgreSQL

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-11-18 Thread Andres Freund
Hi, On 2020-11-18 17:34:31 +0530, Amit Kapila wrote: > Yeah, that won't be a bad idea especially because the patch being > discussed in the thread you referred is still in an exploratory phase. > I haven't tested or done a detailed review but I feel there shouldn't > be many problems if we agree

Re: Allow matching whole DN from a client certificate

2020-11-18 Thread Andrew Dunstan
On 11/12/20 4:21 PM, Andrew Dunstan wrote: > On 11/12/20 8:37 AM, Daniel Gustafsson wrote: >>> On 11 Nov 2020, at 21:44, Andrew Dunstan wrote: >>> If people like this idea I'll add tests and docco and add it to the next CF. >> Sounds like a good idea, please do. >> >> Can this case really happen

Re: VACUUM (DISABLE_PAGE_SKIPPING on)

2020-11-18 Thread Robert Haas
On Wed, Nov 18, 2020 at 12:54 PM Simon Riggs wrote: > Patches attached. > 1. vacuum_anti_wraparound.v2.patch > 2. vacuumdb_anti_wrap.v1.patch - depends upon (1) I don't like the use of ANTI_WRAPAROUND as a name for this new option. Wouldn't it make more sense to call it AGGRESSIVE? Or maybe

Re: VACUUM (DISABLE_PAGE_SKIPPING on)

2020-11-18 Thread Simon Riggs
On Wed, 18 Nov 2020 at 10:28, Masahiko Sawada wrote: > > So we have 3 ways to reset relfrozenxid by a user action: > > VACUUM (DISABLE_PAGE_SKIPPING ON) - scans all blocks, deliberately > > ignoring the ones it could have skipped. This certainly slows it down. > > VACUUM (FREEZE ON) - freezes

Re: Is postgres ready for 2038?

2020-11-18 Thread Tom Lane
Andrew Dunstan writes: > On 11/18/20 9:44 AM, Tom Lane wrote: >> Hmm. Digging around, I see that Mkvcbuild.pm intentionally absorbs >> _USE_32BIT_TIME_T when building with a Perl that defines that. >> I don't know what the state of play is in terms of Windows Perl >> distributions getting off of

Re: [PATCH] postgres_fdw connection caching - cause remote sessions linger till the local session exit

2020-11-18 Thread Alexey Kondratov
On 2020-11-18 16:39, Bharath Rupireddy wrote: Thanks for the interest shown! On Wed, Nov 18, 2020 at 1:07 AM Alexey Kondratov wrote: Regarding the initial issue I prefer point #3, i.e. foreign server option. It has a couple of benefits IMO: 1) it may be set separately on per foreign server

Re: Is postgres ready for 2038?

2020-11-18 Thread Andrew Dunstan
On 11/18/20 9:44 AM, Tom Lane wrote: > Pavel Borisov writes: >>> Maybe we need to dig a little more to see what's going on here. >> How about just a mention in the future documentation to never ever define >> _USE_32BIT_TIME_T when compiling PG under Windows? Should be enough, I >> suppose. >

Re: pg_stat_statements and "IN" conditions

2020-11-18 Thread Dmitry Dolgov
> On Wed, Aug 12, 2020 at 06:19:02PM +0200, Dmitry Dolgov wrote: > > I would like to start another thread to follow up on [1], mostly to bump up > the > topic. Just to remind, it's about how pg_stat_statements jumbling ArrayExpr in > queries like: > > SELECT something FROM table WHERE col IN

Re: Tab complete for CREATE OR REPLACE TRIGGER statement

2020-11-18 Thread Tom Lane
Tels writes: > On 2020-11-18 06:06, Michael Paquier wrote: >> On Mon, Nov 16, 2020 at 10:14:10PM -0500, Tom Lane wrote: >>> Agreed, I'm not trying to block this patch. Just wishing >>> there were a better way. > To me the code looks like a prime candidate for "data-driven" > refactoring. > It

Re: Is postgres ready for 2038?

2020-11-18 Thread Pavel Borisov
Yes, I agree. ср, 18 нояб. 2020 г. в 18:44, Tom Lane : > Pavel Borisov writes: > >> Maybe we need to dig a little more to see what's going on here. > > > How about just a mention in the future documentation to never ever define > > _USE_32BIT_TIME_T when compiling PG under Windows? Should be

Re: Tab complete for CREATE OR REPLACE TRIGGER statement

2020-11-18 Thread Tels
On 2020-11-18 06:06, Michael Paquier wrote: On Mon, Nov 16, 2020 at 10:14:10PM -0500, Tom Lane wrote: Michael Paquier writes: I don't think that this is a requirement for this thread, though. Agreed, I'm not trying to block this patch. Just wishing there were a better way. Okay. I have

Re: proposal: possibility to read dumped table's name from file

2020-11-18 Thread Stephen Frost
Greetings, * Justin Pryzby (pry...@telsasoft.com) wrote: > On Wed, Nov 11, 2020 at 06:49:43AM +0100, Pavel Stehule wrote: > > >> Perhaps this feature could co-exist with a full blown configuration for > > >> pg_dump, but even then there's certainly issues with what's proposed- > > >> how would

Re: Is postgres ready for 2038?

2020-11-18 Thread Tom Lane
Pavel Borisov writes: >> Maybe we need to dig a little more to see what's going on here. > How about just a mention in the future documentation to never ever define > _USE_32BIT_TIME_T when compiling PG under Windows? Should be enough, I > suppose. Hmm. Digging around, I see that Mkvcbuild.pm

Re: [PATCH] postgres_fdw connection caching - cause remote sessions linger till the local session exit

2020-11-18 Thread Bharath Rupireddy
Thanks for the interest shown! On Wed, Nov 18, 2020 at 1:07 AM Alexey Kondratov wrote: > > I had a look on the initial patch and discussed options [1] to proceed > with this issue. I agree with Bruce about idle_session_timeout, it would > be a nice to have in-core feature on its own. However,

Re: Is postgres ready for 2038?

2020-11-18 Thread Pavel Borisov
> > But it does: "time_t is, by default, equivalent to __time64_t." See > > < > https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/time-time32-time64?view=msvc-160 > > > > > Maybe we need to dig a little more to see what's going on here. > How about just a mention in the future

Re: Is postgres ready for 2038?

2020-11-18 Thread Andrew Dunstan
On 11/17/20 11:04 AM, Tom Lane wrote: > =?UTF-8?B?5pa55b6z6Lyd?= writes: >> Is there any road map for 2038 problems in Postgres? > Postgres has no problem with post-2038 dates as long as you are using a > system with 64-bit time_t. In other words, the road map is "get off > Windows, or press

Re: Devel docs on website reloading

2020-11-18 Thread Magnus Hagander
On Wed, Nov 18, 2020 at 1:31 PM Alvaro Herrera wrote: > > On 2020-Nov-18, Magnus Hagander wrote: > > > It would be trivial to change this so that it only actually updates > > pages if they have been changed. > > I think this means we could also check much more frequently whether a > rebuild is

Re: Devel docs on website reloading

2020-11-18 Thread Alvaro Herrera
On 2020-Nov-18, Magnus Hagander wrote: > It would be trivial to change this so that it only actually updates > pages if they have been changed. I think this means we could also check much more frequently whether a rebuild is needed, right? We could do that every 30 mins or so, since most of the

Re: Use standard SIGHUP and SIGTERM handlers in autoprewarm module

2020-11-18 Thread Fujii Masao
On 2020/11/17 21:18, Bharath Rupireddy wrote: Thanks Craig! On Fri, Oct 23, 2020 at 9:37 AM Craig Ringer wrote: src/test/modules/test_shm_mq/worker.c appears to do the right thing the wrong way - it has its own custom handler instead of using die() or SignalHandlerForShutdownRequest().

Re: [HACKERS] make async slave to wait for lsn to be replayed

2020-11-18 Thread a . pervushina
Hello, I've changed the BEGIN WAIT FOR LSN statement to core functions pg_waitlsn, pg_waitlsn_infinite and pg_waitlsn_no_wait. Currently the functions work inside repeatable read transactions, but waitlsn creates a snapshot if called first in a transaction block, which can possibly lead the

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-11-18 Thread Amit Kapila
On Wed, Nov 18, 2020 at 2:34 PM k.jami...@fujitsu.com wrote: > > On Thursday, November 12, 2020 1:14 PM, Tsunakawa-san wrote: > I forgot to reply. > Thank you very much Tsunakawa-san for testing and to everyone > who has provided their reviews and insights as well. > > Now thinking about

Re: Devel docs on website reloading

2020-11-18 Thread Magnus Hagander
On Wed, Nov 18, 2020 at 12:09 PM Heikki Linnakangas wrote: > > On 18/11/2020 12:56, Magnus Hagander wrote: > > Today, we build and update the docs on > > https://www.postgresql.org/docs/devel/ are rebuilt and deployed by the > > buildfarm automatically every 4 hours. > > > > If there are no

Re: Devel docs on website reloading

2020-11-18 Thread Heikki Linnakangas
On 18/11/2020 12:56, Magnus Hagander wrote: Today, we build and update the docs on https://www.postgresql.org/docs/devel/ are rebuilt and deployed by the buildfarm automatically every 4 hours. If there are no changes at all made to the docs, they are *still* kicked out of all caches and the

Devel docs on website reloading

2020-11-18 Thread Magnus Hagander
Today, we build and update the docs on https://www.postgresql.org/docs/devel/ are rebuilt and deployed by the buildfarm automatically every 4 hours. If there are no changes at all made to the docs, they are *still* kicked out of all caches and the search indexes are rebuilt, because we change the

Re: don't allocate HashAgg hash tables when running explain only

2020-11-18 Thread Heikki Linnakangas
On 13/11/2020 18:10, Alexey Bashtanov wrote: Hi, I got somewhat scared when my explain took a few seconds to complete and used a few gigs of RAM. To reproduce try the following: discard temp; create temp table a as select to_timestamp(generate_series(1, 7000)) i; analyze a; set work_mem to

Re: POC: Cleaning up orphaned files using undo logs

2020-11-18 Thread Antonin Houska
Amit Kapila wrote: > On Fri, Nov 13, 2020 at 6:02 PM Antonin Houska wrote: > > > > Amit Kapila wrote: > > > > > On Thu, Nov 12, 2020 at 2:45 PM Antonin Houska wrote: > > > > > > > > > > > > No background undo > > > > -- > > > > > > > > Reduced complexity of the patch seems to

Re: VACUUM (DISABLE_PAGE_SKIPPING on)

2020-11-18 Thread Masahiko Sawada
On Tue, Nov 17, 2020 at 8:27 PM Simon Riggs wrote: > > On Mon, 16 Nov 2020 at 22:53, Masahiko Sawada wrote: > > On Tue, Nov 17, 2020 at 5:52 AM Simon Riggs wrote: > > > I don't think the doc is wrong. If DISABLE_PAGE_SKIPPING is specified, > > we not only set aggressive = true but also skip

Re: Detecting File Damage & Inconsistencies

2020-11-18 Thread Simon Riggs
On Wed, 18 Nov 2020 at 06:42, Craig Ringer wrote: > > On Fri, Nov 13, 2020 at 7:24 PM Simon Riggs wrote: >> >> >> What I'm proposing is an option to add 16 bytes onto each COMMIT >> record > > > Would it make sense to write this at the time we write a topxid assignment to > WAL instead? > >

Re: Parallel copy

2020-11-18 Thread vignesh C
On Sat, Nov 7, 2020 at 7:01 PM vignesh C wrote: > > On Thu, Nov 5, 2020 at 6:33 PM Hou, Zhijie wrote: > > > > Hi > > > > > > > > my $bytes = $ARGV[0]; > > > for(my $i = 0; $i < $bytes; $i+=8){ > > > print "longdata"; > > > } > > > print "\n"; > > > > > > > > > postgres=# copy

Re: Parallel copy

2020-11-18 Thread vignesh C
On Sat, Oct 31, 2020 at 2:07 AM Tomas Vondra wrote: > > Hi, > > I've done a bit more testing today, and I think the parsing is busted in > some way. Consider this: > > test=# create extension random; > CREATE EXTENSION > > test=# create table t (a text); > CREATE TABLE > >

Re: Parallel copy

2020-11-18 Thread vignesh C
On Fri, Nov 13, 2020 at 2:25 PM Amit Kapila wrote: > > On Wed, Nov 11, 2020 at 10:42 PM vignesh C wrote: > > > > On Tue, Nov 10, 2020 at 7:27 PM Amit Kapila wrote: > > > > > > On Tue, Nov 10, 2020 at 7:12 PM vignesh C wrote: > > > > > > > > On Tue, Nov 3, 2020 at 2:28 PM Amit Kapila > > > >

Re: Parallel copy

2020-11-18 Thread vignesh C
On Thu, Oct 29, 2020 at 2:26 PM Daniel Westermann (DWE) wrote: > > On 27/10/2020 15:36, vignesh C wrote: > >> Attached v9 patches have the fixes for the above comments. > > >I did some testing: > > I did some testing as well and have a cosmetic remark: > > postgres=# copy t1 from

  1   2   >