Re: Use standard SIGHUP and SIGTERM handlers in autoprewarm module

2020-11-09 Thread Kyotaro Horiguchi
At Sat, 7 Nov 2020 19:31:21 +0530, Bharath Rupireddy wrote in > The main reason for having SetLatch() in > SignalHandlerForConfigReload() is to wake up the calling process if > waiting in WaitLatchOrSocket() or WaitLatch() and reload the new > config file and use the reloaded config variables.

Re: Temporary tables versus wraparound... again

2020-11-09 Thread Masahiko Sawada
On Mon, Nov 9, 2020 at 3:23 PM Greg Stark wrote: > > On Mon, 9 Nov 2020 at 00:17, Noah Misch wrote: > > > > > 2) adding the dependency on heapam.h to heap.c makes sense because of > > > heap_inplace_update bt it may be a bit annoying because I suspect > > > that's a useful sanity check that the

Re: abstract Unix-domain sockets

2020-11-09 Thread Michael Paquier
On Mon, Nov 09, 2020 at 09:04:24AM +0100, Peter Eisentraut wrote: > On 2020-11-09 07:08, Michael Paquier wrote: > The @ is the standard way of representing this in the user interface and the > configuration, so it seems sensible to me that way. Ok. > Can you sketch how you would structure this?

Re: PATCH: Report libpq version and configuration

2020-11-09 Thread Craig Ringer
On Tue, Nov 10, 2020 at 12:33 AM Tom Lane wrote: > Alvaro Herrera writes: > > Well, if we can make it run in more systems than just Linux, then it > > seems worth having. The submitted patch seems a little bit on the > > naughty side. > > I agree that the facility seems possibly useful, as

Re: MultiXact\SLRU buffers configuration

2020-11-09 Thread Andrey Borodin
> 10 нояб. 2020 г., в 05:13, Tomas Vondra > написал(а): > After the issue reported in [1] got fixed, I've restarted the multi-xact > stress test, hoping to reproduce the issue. But so far no luck :-( Tomas, many thanks for looking into this. I figured out that to make multixact sets bigger

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-11-09 Thread tsunakawa.ta...@fujitsu.com
From: Jamison, Kirk/ジャミソン カーク > So I proceeded to update the patches using the "cached" parameter and > updated the corresponding comments to it in 0002. OK, I'm in favor of the name "cached" now, although I first agreed with Horiguchi-san in that it's better to use a name that represents the

Re: logical streaming of xacts via test_decoding is broken

2020-11-09 Thread Dilip Kumar
On Tue, Nov 10, 2020 at 10:52 AM Amit Kapila wrote: > > On Tue, Nov 10, 2020 at 10:26 AM Dilip Kumar wrote: > > > > On Tue, Nov 10, 2020 at 8:14 AM Amit Kapila wrote: > > > > > > On Mon, Nov 9, 2020 at 6:00 PM Dilip Kumar wrote: > > > > > > > > On Mon, Nov 9, 2020 at 5:37 PM Dilip Kumar > >

Re: [HACKERS] logical decoding of two-phase transactions

2020-11-09 Thread Peter Smith
FYI - I have cross-checked all the v18 patch code against the v18 code coverage [1] resulting from running the tests. The purpose of this study was to identify where there may be any gaps in the testing of this patch - e.g is there some v18 code not currently getting executed by the tests? I

Add docs stub for recovery.conf

2020-11-09 Thread Craig Ringer
Hi all I noticed that when recovery.conf was removed in 2dedf4d9a8 (yay!) the docs for it were removed completely as well. That's largely sensible, but is confusing when users have upgraded and are trying to find out what happened, or how to configure equivalent functionality.

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-11-09 Thread Thomas Munro
On Tue, Nov 10, 2020 at 6:18 PM Amit Kapila wrote: > Do we always truncate all the blocks? What if the vacuum has cleaned > last N (say 100) blocks then how do we handle it? Oh, hmm. Yeah, that idea only works for DROP, not for truncate last N.

[PATCH] TAP test showing that pg_replication_slot_advance() works on standby

2020-11-09 Thread Craig Ringer
Hi all The attached patch adds a test to the TAP logical decoding suite to show that pg_replication_slot_advance() works on a server in standby mode. I didn't add a full demonstration of how to do failover with logical slots because we're still missing a way to "sync" a logical slot from a

Re: [BUG]: segfault during update

2020-11-09 Thread Tom Lane
Andres Freund writes: > On 2020-11-08 12:46:44 -0500, Tom Lane wrote: >> In v12 we end up using the junkfilter's output >> slot, which does not have a sufficiently accurate tupdesc to deal with >> an on-disk tuple rather than one constructed by the executor. > I really wonder if we ought to

Re: logical streaming of xacts via test_decoding is broken

2020-11-09 Thread Amit Kapila
On Tue, Nov 10, 2020 at 10:26 AM Dilip Kumar wrote: > > On Tue, Nov 10, 2020 at 8:14 AM Amit Kapila wrote: > > > > On Mon, Nov 9, 2020 at 6:00 PM Dilip Kumar wrote: > > > > > > On Mon, Nov 9, 2020 at 5:37 PM Dilip Kumar wrote: > > > > > > > > On Mon, Nov 9, 2020 at 4:21 PM Amit Kapila > > >

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-11-09 Thread Amit Kapila
On Tue, Nov 10, 2020 at 10:00 AM Thomas Munro wrote: > > On Sat, Nov 7, 2020 at 12:40 AM Amit Kapila wrote: > > I think one of the problems is returning fewer rows and that too > > without any warning or error, so maybe that is a bigger problem but we > > seem to be okay with it as that is

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-11-09 Thread k.jami...@fujitsu.com
On Tuesday, November 10, 2020 12:27 PM, Horiguchi-san wrote: > To: amit.kapil...@gmail.com > Cc: Jamison, Kirk/ジャミソン カーク ; Tsunakawa, > Takayuki/綱川 貴之 ; t...@sss.pgh.pa.us; > and...@anarazel.de; robertmh...@gmail.com; > tomas.von...@2ndquadrant.com; pgsql-hack...@postgresql.org > Subject: Re:

Re: [BUG]: segfault during update

2020-11-09 Thread Andres Freund
Hi, On 2020-11-08 12:46:44 -0500, Tom Lane wrote: > This logic is entirely gone in v12, which confirms my instinct that > there was something about Andres' slot-manipulation changes that > broke this scenario. Entirely possible :(. In my defense, it wasn't exactly obvious or documented that the

Re: logical streaming of xacts via test_decoding is broken

2020-11-09 Thread Dilip Kumar
On Tue, Nov 10, 2020 at 8:14 AM Amit Kapila wrote: > > On Mon, Nov 9, 2020 at 6:00 PM Dilip Kumar wrote: > > > > On Mon, Nov 9, 2020 at 5:37 PM Dilip Kumar wrote: > > > > > > On Mon, Nov 9, 2020 at 4:21 PM Amit Kapila > > > wrote: > > > > > > The bigger question is do we want to give users

Re: Allow some recovery parameters to be changed with reload

2020-11-09 Thread Kyotaro Horiguchi
At Sat, 07 Nov 2020 00:36:33 +0300, Sergei Kornilov wrote in > Hello > > > I'm wondering if it's safe to allow restore_command to be emptied during > > archive recovery. Even when it's emptied, archive recovery can proceed > > by reading WAL files from pg_wal directory. This is the same

Re: Online verification of checksums

2020-11-09 Thread Michael Paquier
On Thu, Nov 05, 2020 at 10:57:16AM +0900, Michael Paquier wrote: > I was referring to the patch I sent on this thread that fixes the > detection of a corruption for the zero-only case and where pd_lsn > and/or pg_upper are trashed by a corruption of the page header. Both > cases allow a base

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-11-09 Thread David Rowley
On Tue, 10 Nov 2020 at 15:38, Andy Fan wrote: > While I have interest about what caused the tiny difference, I admit that > what direction > this patch should go is more important. Not sure if anyone is convinced that > v8 and v9 have a similar performance. The current data show it is

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-11-09 Thread Thomas Munro
On Sat, Nov 7, 2020 at 12:40 AM Amit Kapila wrote: > I think one of the problems is returning fewer rows and that too > without any warning or error, so maybe that is a bigger problem but we > seem to be okay with it as that is already a known thing though I > think that is not documented

Re: [HACKERS] logical decoding of two-phase transactions

2020-11-09 Thread Masahiko Sawada
On Mon, Nov 9, 2020 at 8:21 PM Amit Kapila wrote: > > On Mon, Nov 9, 2020 at 1:38 PM Masahiko Sawada wrote: > > > > On Mon, Nov 9, 2020 at 3:23 PM Peter Smith wrote: > > > > > > > I've looked at the patches and done some tests. Here is my comment and > > question I realized during testing and

Re: Refactor MD5 implementations and switch to EVP for OpenSSL

2020-11-09 Thread Michael Paquier
On Fri, Nov 06, 2020 at 04:34:34PM +0900, Michael Paquier wrote: > The attached patch set does a bit of rework to make the Postgres code > more consistent with OpenSSL, similarly to the work I did for all the > SHA2 implementations with EVP in [1]: > - 0001 is something stolen from the SHA2 set,

Re: upcoming API changes for LLVM 12

2020-11-09 Thread Tom Lane
Andres Freund writes: > I pushed the change to master. Thanks! > If that doesn't show any problems, I'll > backpatch in a week or so. Seawasp runs only on master, so it should > satisfy the buildfarm at least. Yeah, sounds like a good plan. FWIW, master builds clean for me.

Re: upcoming API changes for LLVM 12

2020-11-09 Thread Andres Freund
Hi, On 2020-11-08 18:22:50 -0500, Tom Lane wrote: > Andres Freund writes: > > Yea, I'll try to do that in the next few days I pushed the change to master. If that doesn't show any problems, I'll backpatch in a week or so. Seawasp runs only on master, so it should satisfy the buildfarm at least.

Re: list of extended statistics on psql

2020-11-09 Thread Tatsuro Yamada
Hi Tomas, I took a look at this today, and I think the code is ready, but the regression test needs a bit more work: Thanks for taking your time. :-D 1) It's probably better to use somewhat more specific names for the objects, especially when created in public schema. It decreases the

Re: warn_unused_results

2020-11-09 Thread Michael Paquier
On Mon, Nov 09, 2020 at 08:23:31AM +0100, Peter Eisentraut wrote: > On 2020-11-09 07:56, Michael Paquier wrote: >> This is accepted by clang, and MSVC has visibly an equivalent for >> that, as of VS 2012: >> #elif defined(_MSC_VER) && (_MSC_VER >= 1700) >> #define pg_nodiscard _Check_return_ >> We

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-11-09 Thread Kyotaro Horiguchi
At Tue, 10 Nov 2020 08:33:26 +0530, Amit Kapila wrote in > On Tue, Nov 10, 2020 at 8:19 AM k.jami...@fujitsu.com > wrote: > > > > I repeated the recovery performance test for vacuum. (I made a mistake > > previously in NBuffers/128) > > The 3 kinds of thresholds are almost equally performant.

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-11-09 Thread Tom Lane
Alvaro Herrera writes: > Yeah ... it would be much better if we can make it use atomics instead. I was thinking more like "do we need any locking at all". Assuming that a proc's vacuumFlags can be set by only the process itself, there's no write conflicts to worry about. On the read side,

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-11-09 Thread Amit Kapila
On Tue, Nov 10, 2020 at 8:19 AM k.jami...@fujitsu.com wrote: > > > From: k.jami...@fujitsu.com > > On Thursday, October 22, 2020 3:15 PM, Kyotaro Horiguchi > > wrote: > > > I'm not sure about the exact steps of the test, but it can be expected > > > if we have many small relations to truncate.

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-11-09 Thread k.jami...@fujitsu.com
> From: k.jami...@fujitsu.com > On Thursday, October 22, 2020 3:15 PM, Kyotaro Horiguchi > wrote: > > I'm not sure about the exact steps of the test, but it can be expected > > if we have many small relations to truncate. > > > > Currently BUF_DROP_FULL_SCAN_THRESHOLD is set to Nbuffers / 512, >

Re: logical streaming of xacts via test_decoding is broken

2020-11-09 Thread Amit Kapila
On Mon, Nov 9, 2020 at 6:00 PM Dilip Kumar wrote: > > On Mon, Nov 9, 2020 at 5:37 PM Dilip Kumar wrote: > > > > On Mon, Nov 9, 2020 at 4:21 PM Amit Kapila wrote: > > > > The bigger question is do we want to give users an option > > > for skip_empty_streams similar to skip_empty_xacts? I would

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-11-09 Thread Michael Paquier
On Mon, Nov 09, 2020 at 11:31:15PM -0300, Alvaro Herrera wrote: > Yeah ... it would be much better if we can make it use atomics instead. > Currently it's an uint8, and in PGPROC itself it's probably not a big > deal to enlarge that, but I fear that quadrupling the size of the > mirroring array in

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-11-09 Thread Andy Fan
On Tue, Nov 10, 2020 at 7:55 AM David Rowley wrote: > On Tue, 10 Nov 2020 at 12:49, Tom Lane wrote: > > > > Alvaro Herrera writes: > > > Are you taking into account the possibility that generated machine code > > > is a small percent slower out of mere bad luck? I remember someone > > >

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-11-09 Thread Alvaro Herrera
On 2020-Nov-09, Tom Lane wrote: > Michael Paquier writes: > > On Mon, Nov 09, 2020 at 08:32:13PM -0500, Tom Lane wrote: > >> Do we really need exclusive lock on the ProcArray to make this flag > >> change? That seems pretty bad from a concurrency standpoint. > > > Any place where we update

Avoiding useless SHA256 initialization with backup manifests, breaking base backups with FIPS

2020-11-09 Thread Michael Paquier
Hi all, Trying to use OpenSSL with FIPS breaks if one attempts to call the low-level SHA2 routines we currently use in sha2_openssl.c (upstream calls that OpenSSLDie()), forcing a crash of PG. The actual way to fix that is to use EVP as I solved here: https://commitfest.postgresql.org/30/2762/

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-11-09 Thread Tom Lane
Michael Paquier writes: > On Mon, Nov 09, 2020 at 08:32:13PM -0500, Tom Lane wrote: >> Do we really need exclusive lock on the ProcArray to make this flag >> change? That seems pretty bad from a concurrency standpoint. > Any place where we update vacuumFlags acquires an exclusive LWLock on >

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-11-09 Thread Michael Paquier
On Mon, Nov 09, 2020 at 08:32:13PM -0500, Tom Lane wrote: > Do we really need exclusive lock on the ProcArray to make this flag > change? That seems pretty bad from a concurrency standpoint. Any place where we update vacuumFlags acquires an exclusive LWLock on ProcArrayLock. That's held for a

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-11-09 Thread Tom Lane
Michael Paquier writes: >> +LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE); >> +MyProc->vacuumFlags |= PROC_IN_SAFE_IC; >> +ProcGlobal->vacuumFlags[MyProc->pgxactoff] = >> MyProc->vacuumFlags; >> +LWLockRelease(ProcArrayLock); > I can't help noticing

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-11-09 Thread Michael Paquier
On Mon, Nov 09, 2020 at 04:47:43PM +0100, Dmitry Dolgov wrote: > > On Tue, Nov 03, 2020 at 07:14:47PM +0100, Dmitry Dolgov wrote: > > > On Thu, Aug 20, 2020 at 03:11:19PM +0900, Michael Paquier wrote: > > > On Wed, Aug 19, 2020 at 02:16:46PM -0400, Alvaro Herrera wrote: > > > > I did not set the

Re: pg_upgrade analyze script

2020-11-09 Thread Michael Paquier
On Mon, Nov 09, 2020 at 03:47:22PM +0100, Peter Eisentraut wrote: > You should just remove those calls. There is no need to replace them with > vacuumdb calls. The reason those calls were there is that they were testing > the generated script itself. If the script is gone, there is no more

Adding an aminsert() hint that triggers bottom-up index deletion for UPDATEs that can't use HOT

2020-11-09 Thread Peter Geoghegan
I am working on a project called "bottom-up index deletion" (I was calling it "delete deduplication" until recently). This is another nbtree project. The patch series adds a mechanism that controls old duplicate versions caused by non-HOT UPDATEs. This is very effective. In fact, it prevents

Re: Disable WAL logging to speed up data loading

2020-11-09 Thread Kyotaro Horiguchi
At Mon, 9 Nov 2020 10:18:08 -0500, Stephen Frost wrote in > Greetings, > > * osumi.takami...@fujitsu.com (osumi.takami...@fujitsu.com) wrote: > > When I consider the use case is the system of data warehouse > > as described upthread, the size of each table can be large. > > Thus, changing the

RE: POC: postgres_fdw insert batching

2020-11-09 Thread tsunakawa.ta...@fujitsu.com
Hello, The attached patch implements the new bulk insert routine for postgres_fdw and the executor utilizing it. It passes make check-world. I measured performance in a basic non-partitioned case by modifying Tomas-san's scripts. They perform an INSERT SELECT statement that copies one

Make Append Cost aware of some run time partition prune case

2020-11-09 Thread Andy Fan
Currently the cost model of append path sums the cost/rows for all the subpaths, it usually works well until we run into the run-time partition prune case. The first result is that generic plans will rarely be used for some cases. For instance, SELECT * FROM p WHERE pkey = $1; The custom plan

Re: MultiXact\SLRU buffers configuration

2020-11-09 Thread Tomas Vondra
Hi, After the issue reported in [1] got fixed, I've restarted the multi-xact stress test, hoping to reproduce the issue. But so far no luck :-( I've started slightly different tests on two machines - on one machine I've done this: a) init.sql create table t (a int); insert into t select

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-11-09 Thread Peter Geoghegan
On Mon, Nov 9, 2020 at 3:49 PM Tom Lane wrote: > Alvaro Herrera writes: > > Are you taking into account the possibility that generated machine code > > is a small percent slower out of mere bad luck? I remember someone > > suggesting that they can make code 2% faster or so by inserting random >

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-11-09 Thread David Rowley
On Tue, 10 Nov 2020 at 12:49, Tom Lane wrote: > > Alvaro Herrera writes: > > Are you taking into account the possibility that generated machine code > > is a small percent slower out of mere bad luck? I remember someone > > suggesting that they can make code 2% faster or so by inserting random

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-11-09 Thread Tom Lane
Alvaro Herrera writes: > Are you taking into account the possibility that generated machine code > is a small percent slower out of mere bad luck? I remember someone > suggesting that they can make code 2% faster or so by inserting random > no-op instructions in the binary, or something like

Re: Add important info about ANALYZE after create Functional Index

2020-11-09 Thread Fabrízio de Royes Mello
On Mon, 9 Nov 2020 at 20:27 Bruce Momjian wrote: > > I see REINDEX CONCURRENTLY was fixed in head, but the docs didn't get > updated to mention the need to run ANALYZE or wait for autovacuum before > expression indexes can be fully used by the optimizer. Instead of > putting this mention in the

Re: Add important info about ANALYZE after create Functional Index

2020-11-09 Thread Bruce Momjian
On Tue, Oct 27, 2020 at 12:12:00AM -0700, Nikolay Samokhvalov wrote: > On Mon, Oct 26, 2020 at 3:08 PM Fabrízio de Royes Mello < > fabriziome...@gmail.com> wrote: > > Would be nice if add some information about it into our docs but not sure > where. I'm thinking about: > -

Re: Additional Chapter for Tutorial

2020-11-09 Thread David G. Johnston
On Sun, Nov 8, 2020 at 8:56 AM Jürgen Purtz wrote: > > Good catches. Everything applied. > Reviewed the first three sections. template0 - I would remove the schema portions of this and simply note this as being a pristine recovery database in the diagram. I would drop the word "more" and just

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-11-09 Thread Alvaro Herrera
On 2020-Nov-10, David Rowley wrote: > On Mon, 9 Nov 2020 at 16:29, Andy Fan wrote: > > However I believe v9 > > should be no worse than v8 all the time, Is there any theory to explain > > your result? > > Nothing jumps out at me from looking at profiles. The only thing I > noticed was the

Re: -Wformat-signedness

2020-11-09 Thread Thomas Munro
On Tue, Nov 10, 2020 at 4:25 AM Tom Lane wrote: > Peter Eisentraut writes: > > 1. enums are unsigned by default in gcc, so all those internal error > > messages "unrecognized blah kind: %d" need to be changed to %u. > > Do we have reason to think that that is true in every C compiler? > My own

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-11-09 Thread David Rowley
On Mon, 9 Nov 2020 at 16:29, Andy Fan wrote: > I think either version is OK for me and I like this patch overall. That's good to know. Thanks. > However I believe v9 > should be no worse than v8 all the time, Is there any theory to explain > your result? Nothing jumps out at me from looking

Re: Error on failed COMMIT

2020-11-09 Thread Dave Cramer
On Wed, 30 Sep 2020 at 18:14, Andrew Dunstan wrote: > > On 8/4/20 12:19 PM, Dave Cramer wrote: > > Attached is the rebased patch for consideration. > > > > > > > It's a bit sad this has been hanging around so long without attention. > > > The previous discussion seems to give the patch a clean

Re: Useless string ouput in error message

2020-11-09 Thread Fabien COELHO
I think I found a typo for the output of an error message which may cause building warning. Please refer to the attachment for the detail. Indeed. Thanks for the fix! -- Fabien.

Re: Misuse of TimestampDifference() in the autoprewarm feature of pg_prewarm

2020-11-09 Thread Tom Lane
Alexey Kondratov writes: > On 2020-11-09 21:53, Tom Lane wrote: >> 0002 seems like a pretty clear bug fix, though I wonder if this is exactly >> what we want to do going forward. It seems like a very large fraction of >> the callers of TimestampDifference would like to have the value in msec, >>

Re: Misuse of TimestampDifference() in the autoprewarm feature of pg_prewarm

2020-11-09 Thread Alexey Kondratov
On 2020-11-09 21:53, Tom Lane wrote: Alexey Kondratov writes: After fixing this issue I have noticed that it still dumps blocks twice at each timeout (here I set autoprewarm_interval to 15s): ... This happens because at timeout time we were using continue, but actually we still have to wait

Re: public schema default ACL

2020-11-09 Thread Bruce Momjian
On Mon, Nov 2, 2020 at 01:41:09PM -0500, Stephen Frost wrote: > At least from seeing the users that start out with PG and then come to > the Slack or IRC channel asking questions, the on-boarding experience > today typically consists of 'apt install postgresql' and then complaints > that they

Re: public schema default ACL

2020-11-09 Thread Bruce Momjian
On Mon, Nov 2, 2020 at 11:05:15PM -0800, Noah Misch wrote: > On Mon, Nov 02, 2020 at 12:42:26PM -0500, Tom Lane wrote: > > Robert Haas writes: > > > On Mon, Nov 2, 2020 at 5:51 AM Peter Eisentraut > > > wrote: > > >> I'm not convinced, however, that this would would really move the needle > >

Re: Skip ExecCheckRTPerms in CTAS with no data

2020-11-09 Thread Anastasia Lubennikova
On 29.09.2020 14:39, Bharath Rupireddy wrote: On Mon, Sep 28, 2020 at 7:48 PM Tom Lane wrote: Bharath Rupireddy writes: In case of CTAS with no data, we actually do not insert the tuples into the created table, so we can skip checking for the insert permissions. Anyways, the insert

Re: Disable WAL logging to speed up data loading

2020-11-09 Thread Stephen Frost
Greetings, * David G. Johnston (david.g.johns...@gmail.com) wrote: > On Mon, Nov 9, 2020 at 10:36 AM Stephen Frost wrote: > > * David G. Johnston (david.g.johns...@gmail.com) wrote: > > > If the commit doesn't complete all of the newly created pages are junk. > > > Otherwise, you have a

Re: WIP: BRIN multi-range indexes

2020-11-09 Thread John Naylor
On Mon, Nov 9, 2020 at 1:39 PM Tomas Vondra wrote: > > > While investigating the failures, I've tried increasing the values a > lot, without observing any measurable increase in runtime. IIRC I've > even used (10 * target_partlen) or something like that. That tells me > it's not very sensitive

Re: Misuse of TimestampDifference() in the autoprewarm feature of pg_prewarm

2020-11-09 Thread Tom Lane
Alexey Kondratov writes: > After fixing this issue I have noticed that it still dumps blocks twice > at each timeout (here I set autoprewarm_interval to 15s): > ... > This happens because at timeout time we were using continue, but > actually we still have to wait the entire

Re: PATCH: Attempt to make dbsize a bit more consistent

2020-11-09 Thread Soumyadeep Chakraborty
Hey Georgios, Thanks for looking for more avenues to invoke tableAM APIS! Please find my review below: On Tue, Oct 13, 2020 at 6:28 AM wrote: 1. > /* > - * heap size, including FSM and VM > + * table size, including FSM and VM > */ We should not mention FSM and VM in dbsize.c at all as

Re: RE: Delaying/avoiding BTreeTupleGetNAtts() call within _bt_compare()

2020-11-09 Thread Peter Geoghegan
On Mon, Nov 2, 2020 at 1:04 PM Peter Geoghegan wrote: > if Andres cannot spend any time on this in the foreseeable future then > I'll withdraw the patch. I intend to formally withdraw the patch on > November 9th, provided no new information comes to light. I have now formally withdrawn the patch

Re: Disable WAL logging to speed up data loading

2020-11-09 Thread David G. Johnston
On Mon, Nov 9, 2020 at 10:36 AM Stephen Frost wrote: > * David G. Johnston (david.g.johns...@gmail.com) wrote: > > > If the commit doesn't complete all of the newly created pages are junk. > > Otherwise, you have a crash-recoverable state for those tables as regards > > those specific pages. > >

Re: automatic analyze: readahead - add "IO read time" log message

2020-11-09 Thread Tomas Vondra
On 11/9/20 7:06 PM, Stephen Frost wrote: > Greetings, > > * Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: >> On 11/4/20 5:02 PM, Stephen Frost wrote: >>> * Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: > If you highlight "738754560" in the output it appears to duplicate the >

Re: automatic analyze: readahead - add "IO read time" log message

2020-11-09 Thread Stephen Frost
Greetings, * Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: > On 11/4/20 5:02 PM, Stephen Frost wrote: > >* Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: > >>>If you highlight "738754560" in the output it appears to duplicate the > >>>syscalls issued until it preads() - in case of

Re: Misuse of TimestampDifference() in the autoprewarm feature of pg_prewarm

2020-11-09 Thread Jeevan Ladhe
Hi, On Mon, Nov 9, 2020 at 10:15 PM Alexey Kondratov wrote: > Hi Hackers, > > Today I have accidentally noticed that autoprewarm feature of pg_prewarm > used TimestampDifference()'s results in a wrong way. > > First, it used *seconds* result from it as a *milliseconds*. It was > causing it to

Re: WIP: BRIN multi-range indexes

2020-11-09 Thread Tomas Vondra
On 11/9/20 3:29 PM, John Naylor wrote: > On Sat, Nov 7, 2020 at 4:38 PM Tomas Vondra > mailto:tomas.von...@enterprisedb.com>> > wrote: > >> Overall, I think there's very little difference, particularly in the >> "match" cases when we're searching for a value that we know is in the >> table.

Re: Disable WAL logging to speed up data loading

2020-11-09 Thread Stephen Frost
Greetings, * David G. Johnston (david.g.johns...@gmail.com) wrote: > On Mon, Nov 9, 2020 at 8:18 AM Stephen Frost wrote: > > Presently, my feeling is that we could address this use-case without > > having to introduce a new cluster-wide WAL level, and that's the > > direction I'd want to see

Re: Disable WAL logging to speed up data loading

2020-11-09 Thread David G. Johnston
On Mon, Nov 9, 2020 at 8:18 AM Stephen Frost wrote: > Presently, my feeling is that we could address this use-case without > having to introduce a new cluster-wide WAL level, and that's the > direction I'd want to see this going. Perhaps I'm missing something > about why the approach I've set

Misuse of TimestampDifference() in the autoprewarm feature of pg_prewarm

2020-11-09 Thread Alexey Kondratov
Hi Hackers, Today I have accidentally noticed that autoprewarm feature of pg_prewarm used TimestampDifference()'s results in a wrong way. First, it used *seconds* result from it as a *milliseconds*. It was causing it to make dump file autoprewarm.blocks ~every second with default setting of

Re: PATCH: Report libpq version and configuration

2020-11-09 Thread Tom Lane
Alvaro Herrera writes: > Well, if we can make it run in more systems than just Linux, then it > seems worth having. The submitted patch seems a little bit on the > naughty side. I agree that the facility seems possibly useful, as long as we can minimize its platform dependency. Just embedding

Re: Reduce the time required for a database recovery from archive.

2020-11-09 Thread Stephen Frost
Greetings, * Dmitry Shulga (d.shu...@postgrespro.ru) wrote: > > On 19 Oct 2020, at 23:25, Stephen Frost wrote: > Implementation of this approach assumes running several background > processes (bgworkers) > each of which runs a shell command specified by the parameter >

Re: PATCH: Report libpq version and configuration

2020-11-09 Thread Alvaro Herrera
On 2020-Oct-27, Craig Ringer wrote: > On Tue, Oct 27, 2020 at 12:56 AM Tom Lane wrote: > > > +1. Are we concerned about translatability of these strings? I think > > I'd vote against, as it would complicate applications, but it's worth > > thinking about it now not later. > > It's necessary

Re: abstract Unix-domain sockets

2020-11-09 Thread Andreas Karlsson
On 11/9/20 9:04 AM, Peter Eisentraut wrote: On 2020-11-09 07:08, Michael Paquier wrote: As abstract namespaces don't have permissions, anyone knowing the name of the path, which should be unique, can have an access to the server. Do you think that the documentation should warn the user about

Re: -O switch

2020-11-09 Thread Tom Lane
Magnus Hagander writes: > On Wed, Nov 4, 2020 at 2:10 AM Tom Lane wrote: >> ... looking at this again, BackendRun certainly looks ridiculously >> over-engineered for what it still does. > Yeah, looking at it again, I agree. PFA an updated patch, which I'll > go ahead and push shortly. LGTM. >

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-11-09 Thread Dmitry Dolgov
> On Tue, Nov 03, 2020 at 07:14:47PM +0100, Dmitry Dolgov wrote: > > On Thu, Aug 20, 2020 at 03:11:19PM +0900, Michael Paquier wrote: > > On Wed, Aug 19, 2020 at 02:16:46PM -0400, Alvaro Herrera wrote: > > > I did not set the flag in REINDEX CONCURRENTLY, but as I understand it > > > can be done

Re: Reduce the time required for a database recovery from archive.

2020-11-09 Thread Dmitry Shulga
Hello Stephen, > On 19 Oct 2020, at 23:25, Stephen Frost wrote: > > Greetings, > Implementation of this approach assumes running several background processes (bgworkers) each of which runs a shell command specified by the parameter restore_command to deliver an

Re: -Wformat-signedness

2020-11-09 Thread Tom Lane
Peter Eisentraut writes: > 1. enums are unsigned by default in gcc, so all those internal error > messages "unrecognized blah kind: %d" need to be changed to %u. Do we have reason to think that that is true in every C compiler? My own preference for this would be to leave the messages as-is and

Re: Disable WAL logging to speed up data loading

2020-11-09 Thread Stephen Frost
Greetings, * osumi.takami...@fujitsu.com (osumi.takami...@fujitsu.com) wrote: > On Tuesday, Nov 3, 2020 3:02 AM Stephen Frost wrote: > > I'm not sure that wal_level=none is really the right way to address this > > use-case. We already have unlogged tables and that's pretty clean and > > meets

Re: Prevent printing "next step instructions" in initdb and pg_upgrade

2020-11-09 Thread Alvaro Herrera
On 2020-Nov-09, Magnus Hagander wrote: > On Mon, Nov 9, 2020 at 3:29 PM Alvaro Herrera wrote: > > > > How about a switch like "--with-scripts=" where the list can be > > "all" to include everything (default), "none" to include nothing, or a > > comma-separated list of things to include? (Also

Re: vacuum -vs reltuples on insert only index

2020-11-09 Thread Jehan-Guillaume de Rorthais
On Wed, 4 Nov 2020 18:44:03 -0800 Peter Geoghegan wrote: > On Mon, Nov 2, 2020 at 10:03 AM Peter Geoghegan wrote: > > Actually, it seems better to always count num_index_tuples the old way > > during cleanup-only index VACUUMs, despite the inaccuracy that that > > creates with posting list

Re: pg_upgrade analyze script

2020-11-09 Thread Peter Eisentraut
On 2020-11-09 11:22, Magnus Hagander wrote: I have spotted one small-ish thing. This patch is missing to update the following code in vcregress.pl: print "\nSetting up stats on new cluster\n\n"; system(".\\analyze_new_cluster.bat") == 0 or exit 1; Ah, nice catch -- thanks! I guess

Re: Rethinking LOCK TABLE's behavior on views

2020-11-09 Thread Alvaro Herrera
On 2020-Nov-07, Noah Misch wrote: > On Sat, Nov 07, 2020 at 11:57:20AM -0500, Tom Lane wrote: > > A completely different approach we could consider is to weaken the > > permissions requirements for LOCK on a view, say "allow it if either > > the calling user or the view owner has the needed

Re: Prevent printing "next step instructions" in initdb and pg_upgrade

2020-11-09 Thread Magnus Hagander
On Mon, Nov 9, 2020 at 3:29 PM Alvaro Herrera wrote: > > On 2020-Nov-09, Magnus Hagander wrote: > > > But for usability that makes less sense. For the delete script, the > > wrapper (that the switch is intended for) knows more than pg_upgrade > > about how to delete it, so it can do a better job,

Re: WIP: BRIN multi-range indexes

2020-11-09 Thread John Naylor
On Sat, Nov 7, 2020 at 4:38 PM Tomas Vondra wrote: > Overall, I think there's very little difference, particularly in the > "match" cases when we're searching for a value that we know is in the > table. The one-hash variant seems to perform a bit better, but the > difference is fairly small. > >

Re: Prevent printing "next step instructions" in initdb and pg_upgrade

2020-11-09 Thread Alvaro Herrera
On 2020-Nov-09, Magnus Hagander wrote: > But for usability that makes less sense. For the delete script, the > wrapper (that the switch is intended for) knows more than pg_upgrade > about how to delete it, so it can do a better job, and thus it makes > sense to silence it. But for something like

Re: Reduce the number of special cases to build contrib modules on windows

2020-11-09 Thread Alvaro Herrera
On 2020-Nov-06, David Rowley wrote: > +# Handle makefile rules for when file to be added to the project > +# does not exist. Returns 1 when the original file add should be > +# skipped. > +sub FindAndAddAdditionalFiles > +{ > + my $self = shift; > + my $fname = shift; > + my ($ext) =

Re: Making cancellations safe

2020-11-09 Thread Dave Cramer
On Wed, 4 Nov 2020 at 10:50, Shay Rojansky wrote: > Hi all. > > Back in 2016 I started a thread about making cancellations safer[1], I'd > like to try to pick this up again. Here a summary of the previous > conversation: > > The main ask here is to allow clients to specify which command to

Re: document pg_settings view doesn't display custom options

2020-11-09 Thread John Naylor
On Mon, Nov 9, 2020 at 2:12 AM Fujii Masao wrote: > Pushed. Thanks! > Thank you! -- John Naylor EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: Prevent printing "next step instructions" in initdb and pg_upgrade

2020-11-09 Thread Magnus Hagander
On Mon, Nov 9, 2020 at 2:18 PM Anastasia Lubennikova wrote: > > On 02.11.2020 16:23, Magnus Hagander wrote: > > On Tue, Oct 27, 2020 at 11:35:25AM +0100, Peter Eisentraut wrote: > > On 2020-10-06 12:26, Magnus Hagander wrote: > > I went with the name --no-instructions to have the same name for

Re: Prevent printing "next step instructions" in initdb and pg_upgrade

2020-11-09 Thread Anastasia Lubennikova
On 02.11.2020 16:23, Magnus Hagander wrote: On Tue, Oct 27, 2020 at 11:35:25AM +0100, Peter Eisentraut wrote: On 2020-10-06 12:26, Magnus Hagander wrote: I went with the name --no-instructions to have the same name for both initdb and pg_upgrade. The downside is that "no-instructions" also

Re: logical streaming of xacts via test_decoding is broken

2020-11-09 Thread Dilip Kumar
On Mon, Nov 9, 2020 at 5:37 PM Dilip Kumar wrote: > > On Mon, Nov 9, 2020 at 4:21 PM Amit Kapila wrote: > > > > On Mon, Nov 9, 2020 at 3:01 PM Dilip Kumar wrote: > > > > > > On Mon, Nov 9, 2020 at 1:34 PM Dilip Kumar wrote: > > > > > > > > On Mon, Nov 9, 2020 at 11:31 AM Amit Kapila > > > >

Re: logical streaming of xacts via test_decoding is broken

2020-11-09 Thread Dilip Kumar
On Mon, Nov 9, 2020 at 4:21 PM Amit Kapila wrote: > > On Mon, Nov 9, 2020 at 3:01 PM Dilip Kumar wrote: > > > > On Mon, Nov 9, 2020 at 1:34 PM Dilip Kumar wrote: > > > > > > On Mon, Nov 9, 2020 at 11:31 AM Amit Kapila > > > wrote: > > > > > > > > On Mon, Nov 9, 2020 at 11:21 AM Dilip Kumar

Re: Prevent printing "next step instructions" in initdb and pg_upgrade

2020-11-09 Thread Magnus Hagander
On Mon, Nov 2, 2020 at 2:23 PM Magnus Hagander wrote: > > On Tue, Oct 27, 2020 at 12:40 PM Bruce Momjian wrote: > > > > On Tue, Oct 27, 2020 at 12:35:56PM +0100, Peter Eisentraut wrote: > > > On 2020-10-27 11:53, Bruce Momjian wrote: > > > > On Tue, Oct 27, 2020 at 11:35:25AM +0100, Peter

Re: [HACKERS] logical decoding of two-phase transactions

2020-11-09 Thread Amit Kapila
On Mon, Nov 9, 2020 at 1:38 PM Masahiko Sawada wrote: > > On Mon, Nov 9, 2020 at 3:23 PM Peter Smith wrote: > > > > I've looked at the patches and done some tests. Here is my comment and > question I realized during testing and reviewing. > > +static void > +DecodePrepare(LogicalDecodingContext

  1   2   >