Re: pg_attribute.attname inconsistency when renaming primary key columns

2021-02-23 Thread Kyotaro Horiguchi
At Mon, 22 Feb 2021 21:42:44 +0100, "Joel Jacobson" wrote in > I solved my problem by using attnum::text instead of attname for > pg_class.relkind = ‘i’ as a work-around to avoid a diff. For your information, note that the attname of an index relation is not the name of the target column in

Re: Improvements and additions to COPY progress reporting

2021-02-23 Thread Justin Pryzby
On Sun, Feb 21, 2021 at 08:10:09PM +0100, Matthias van de Meent wrote: > Subject: [PATCH v9 1/3] Add progress-reported components for COPY progress > reporting > /* Increment amount of processed tuples and update the > progress */ > /* Increment amount of processed

Re: Improvements and additions to COPY progress reporting

2021-02-23 Thread Michael Paquier
On Tue, Feb 23, 2021 at 10:27:24AM +0100, Matthias van de Meent wrote: > Note, I'm happy to be proven wrong here, in which case I don't > disagree, but according to my limited knowledge, these outputs should > be stable. I am planning to look more at 0001 and 0003, but for now I have been looking

Re: Is Recovery actually paused?

2021-02-23 Thread Dilip Kumar
On Wed, Feb 24, 2021 at 12:39 PM Kyotaro Horiguchi wrote: > > At Tue, 23 Feb 2021 12:03:32 +0530, Dilip Kumar wrote > in > > On Fri, Feb 12, 2021 at 3:26 AM Robert Haas wrote: > > > There might be some more to say here, but those are things I notice on > > > a first read-through. > > > > Okay.

Re: doc review for v14

2021-02-23 Thread Justin Pryzby
On Wed, Feb 24, 2021 at 04:18:51PM +0900, Michael Paquier wrote: > On Mon, Feb 22, 2021 at 02:03:45AM -0600, Justin Pryzby wrote: > > Rebased, with a few additions. > > Thanks. I have done a pass through this series, and applied most of > this stuff with a backpatch for the doc portions. > > +

Re: doc review for v14

2021-02-23 Thread Michael Paquier
On Mon, Feb 22, 2021 at 02:03:45AM -0600, Justin Pryzby wrote: > Rebased, with a few additions. Thanks. I have done a pass through this series, and applied most of this stuff with a backpatch for the doc portions. +The status of each kind of extended statistics is shown in a column +

Re: Single transaction in the tablesync worker?

2021-02-23 Thread Masahiko Sawada
On Fri, Feb 12, 2021 at 2:49 PM Amit Kapila wrote: > > On Fri, Feb 12, 2021 at 10:08 AM Ajin Cherian wrote: > > > > On Fri, Feb 12, 2021 at 2:46 PM Amit Kapila wrote: > > > > > > > > Thanks, I have pushed the patch but getting one failure: > > >

Re: About to add WAL write/fsync statistics to pg_stat_wal view

2021-02-23 Thread Fujii Masao
On 2021/02/15 11:59, Masahiro Ikeda wrote: On 2021-02-10 00:51, David G. Johnston wrote: On Thu, Feb 4, 2021 at 4:45 PM Masahiro Ikeda wrote: I pgindented the patches. ... XLogWrite, which is invoked during an XLogFlush request (see ...).  This is also incremented by the WAL receiver

Re: Is Recovery actually paused?

2021-02-23 Thread Kyotaro Horiguchi
At Tue, 23 Feb 2021 12:03:32 +0530, Dilip Kumar wrote in > On Fri, Feb 12, 2021 at 3:26 AM Robert Haas wrote: > > There might be some more to say here, but those are things I notice on > > a first read-through. > > Okay. It seems to me all the suggestions are addressed in this version. +

RE: Refactor ECPGconnect and allow IPv6 connection

2021-02-23 Thread kuroda.hay...@fujitsu.com
Dear Wang, Thank you for giving comments! I forgot to write that parse functions imitates libpq's functios, but you understood that immediately. Genius! > So, I think parse_options() is not need to be refactored. OK. > I think we can use the message as same as the message in fe-connect.c: > >

Re: Asynchronous and "direct" IO support for PostgreSQL.

2021-02-23 Thread Thomas Munro
On Tue, Feb 23, 2021 at 11:03 PM Andres Freund wrote: > over the last ~year I spent a lot of time trying to figure out how we could > add AIO (asynchronous IO) and DIO (direct IO) support to postgres. While > there's still a *lot* of open questions, I think I now have a decent handle on > most of

Re: a misbehavior of partition row movement (?)

2021-02-23 Thread Rahila Syed
Hi Amit, Sorry for the late reply. I assume these are comments for the v3-0001 & v3-0002 patches... > > Yes, those were comments for patches on master. > > The partition-key-update-1.spec test fails with the following error > message appearing in the diffs. > > > > step s1u3pc: UPDATE

RE: Refactor ECPGconnect and allow IPv6 connection

2021-02-23 Thread wangsh.f...@fujitsu.com
Hi, Kuroda-san: Kuroda, Hayato/黒田 隼人 wrote: > * parse_options() was not refactored because > it does not affect to parsing the host. > I will try it if should be. It seems host only can be the name of server, please refer [1]. And if I use command: ./bin/psql

Re: repeated decoding of prepared transactions

2021-02-23 Thread Ajin Cherian
On Tue, Feb 23, 2021 at 8:54 PM Amit Kapila wrote: > 1. With respect to SQL APIs, currently 'two-phase-commit' is a plugin > option so it is possible that the first time when it gets changes > (pg_logical_slot_get_changes) *without* 2PC enabled it will not get > the prepared even though prepare

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-23 Thread Amit Kapila
On Wed, Feb 24, 2021 at 8:41 AM Greg Nancarrow wrote: > > On Tue, Feb 23, 2021 at 10:53 PM Amit Kapila wrote: > > > > > But the non-parallel plan was chosen (instead of a parallel plan) > > > because of parallel-safety checks on the partitions, which found > > > attributes of the partitions

RE: libpq debug log

2021-02-23 Thread k.jami...@fujitsu.com
> From: alvhe...@alvh.no-ip.org > I'll give this another look tomorrow, but I wanted to pass along that I prefer > libpq-trace.{c,h} instead of libpq-logging. I also renamed variable "pin" and > pgindented. I don't have any major reservations about this patch now, so I'll > mark it

Re: [PATCH] Feature improvement for TRUNCATE tab completion.

2021-02-23 Thread Fujii Masao
On 2021/02/23 1:44, Muhammad Usama wrote: The following review has been posted through the commitfest application: make installcheck-world: not tested Implements feature: tested, passed Spec compliant: tested, passed Documentation:not tested Other than "Hunk #1

contrib/cube - binary input/output handlers

2021-02-23 Thread Kohei KaiGai
Hello, I noticed that contrib/cube data type does not support binary input/output handler when I tried to dump a table with cube columns, using a tool [*1] that uses binary data over libpq. $ pg2arrow -d postgres -t my_table ../utils/pgsql_client.c:351 SQL execution failed: ERROR: no binary

Re: Improvements and additions to COPY progress reporting

2021-02-23 Thread Bharath Rupireddy
On Tue, Feb 23, 2021 at 2:57 PM Matthias van de Meent wrote: > On Mon, 22 Feb 2021 at 05:49, Bharath Rupireddy > wrote: > > > > On Mon, Feb 22, 2021 at 12:40 AM Matthias van de Meent > > wrote: > > > > > > On Sat, 20 Feb 2021 at 07:09, Bharath Rupireddy > > > wrote: > > > > > > > > For COPY

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-23 Thread Greg Nancarrow
On Tue, Feb 23, 2021 at 10:53 PM Amit Kapila wrote: > > On Tue, Feb 23, 2021 at 4:47 PM Greg Nancarrow wrote: > > > > On Tue, Feb 23, 2021 at 2:30 PM Amit Kapila wrote: > > > > > > On Tue, Feb 23, 2021 at 6:37 AM Greg Nancarrow > > > wrote: > > > > > > > > On Tue, Feb 23, 2021 at 12:33 AM

Re: Fallback table AM for relkinds without storage

2021-02-23 Thread Michael Paquier
On Mon, Feb 22, 2021 at 05:19:37PM -0800, Andres Freund wrote: > This doesn't seem like an advantage to me. Isn't this just pushing logic > away from a fairly obvious point into an AM that one would expect to > never actually get called? > > If we want to go down this path what's the

Re: Some regular-expression performance hacking

2021-02-23 Thread Tom Lane
Here's another little piece of regex performance hacking. This is based on looking at a slow regexp I found in Tcl's bug tracker: -- Adapted from http://core.tcl.tk/tcl/tktview?name=446565 select regexp_matches( repeat(' 123 345 123 ', 10),

Re: some pointless HeapTupleHeaderIndicatesMovedPartitions calls

2021-02-23 Thread Michael Paquier
On Mon, Feb 22, 2021 at 05:15:57PM -0300, Álvaro Herrera wrote: > I changed my mind on this after noticing that > ItemPointerIndicatesMovedPartitions has a few callers; leaving the > interface incomplete/asymmetric would be worse. So I propose to do > this. Doing that looks fine to me as well.

RE: libpq debug log

2021-02-23 Thread tsunakawa.ta...@fujitsu.com
From: alvhe...@alvh.no-ip.org > I'll give this another look tomorrow, but I wanted to pass along that I prefer > libpq-trace.{c,h} instead of libpq-logging. I also renamed variable "pin" and > pgindented. Ah, you're right, because the function names are PQtrace() and PQuntrace(). > I don't

Re: [PoC] Non-volatile WAL buffer

2021-02-23 Thread Takashi Menjo
Hi, I had a performance test in another environment. The steps, setup, and postgresql.conf of the test are same as the ones sent by me on Feb 17 [1], except the following items: # Setup - Distro: Red Hat Enterprise Linux release 8.2 (Ootpa) - C compiler: gcc-8.3.1-5.el8.x86_64 - libc:

Refactor ECPGconnect and allow IPv6 connection

2021-02-23 Thread kuroda.hay...@fujitsu.com
Dear Hackers, In the previous discussion [1], we noticed that ECPG cannot accept IPv6 connection string, it means the following statement does not work well: EXEC SQL CONNECT TO 'tcp:postgresql://::1/postgres'; This is caused because colons are gotten entangled in the ECPGconnect(), and Wang

Re: libpq debug log

2021-02-23 Thread alvhe...@alvh.no-ip.org
I'll give this another look tomorrow, but I wanted to pass along that I prefer libpq-trace.{c,h} instead of libpq-logging. I also renamed variable "pin" and pgindented. I don't have any major reservations about this patch now, so I'll mark it ready-for-committer in case somebody else wants to

Re: Support for NSS as a libpq TLS backend

2021-02-23 Thread Jacob Champion
On Mon, 2021-02-22 at 14:31 +0100, Daniel Gustafsson wrote: > The attached fixes that as well as implements the sslcrldir > support that was committed recently. The crldir parameter isn't applicable to > NSS per se since all CRL's are loaded into the NSS database, but it does need > to be

Re: Fix typo about generate_gather_paths

2021-02-23 Thread Alvaro Herrera
On 2020-Dec-22, Tomas Vondra wrote: > Thanks. I started looking at this a bit more closely, and I think most of > the changes are fine - the code was changed to call a different function, > but the comments still reference generate_gather_paths(). Hi, this was forgotten. It seemed better to fix

Re: partial heap only tuples

2021-02-23 Thread Bossart, Nathan
On 2/10/21, 2:43 PM, "Bruce Momjian" wrote: > I wonder if you should create a Postgres wiki page to document all of > this. I agree PG 15 makes sense. I would like to help with this if I > can. I will need to study this email more later. I've started the wiki page for this:

Re: Asynchronous and "direct" IO support for PostgreSQL.

2021-02-23 Thread Andres Freund
Hi, On 2021-02-23 14:58:32 -0500, Greg Stark wrote: > So firstly this is all just awesome work and I have questions but I > don't want them to come across in any way as criticism or as a demand > for more work. I posted it to get argued with ;). > The callbacks make me curious about two

Re: Faulty HEAP_XMAX_LOCK_ONLY & HEAP_KEYS_UPDATED hintbit combination

2021-02-23 Thread Alvaro Herrera
On 2021-Feb-05, Julien Rouhaud wrote: > Thanks, that's way better, copied in v3. Thank you, pushed. The code changes are only relevant in master, but I did back-patch the README.tuplock to all live branches. > I'm still a bit worried about that description though, as that flag > isn't

Re: Asynchronous and "direct" IO support for PostgreSQL.

2021-02-23 Thread Greg Stark
On Tue, 23 Feb 2021 at 05:04, Andres Freund wrote: > > ## Callbacks > > In the core AIO pieces there are two different types of callbacks at the > moment: > > Shared callbacks, which can be invoked by any backend (normally the issuing > backend / the AIO workers, but can be other backends if they

Re: Some regular-expression performance hacking

2021-02-23 Thread Tom Lane
Andres Freund writes: > On 2021-02-23 13:09:18 -0500, Tom Lane wrote: >> Oddly, I see no such warning with Fedora's current compiler, >> gcc version 10.2.1 20201125 (Red Hat 10.2.1-9) (GCC) >> Are you using any special compiler switches? > A few. At first I didn't see any relevant ones - but I

Re: Some regular-expression performance hacking

2021-02-23 Thread Tom Lane
Andres Freund writes: > On 2021-02-23 12:52:28 -0500, Tom Lane wrote: >> ... It is annoying to have to expend >> an always-on check for a can't-happen case, though. > Wouldn't quite work like that because of the restrictions of what pg > infrastructure we want to expose the regex engine to, but

Re: Some regular-expression performance hacking

2021-02-23 Thread Andres Freund
On 2021-02-23 13:09:18 -0500, Tom Lane wrote: > Andres Freund writes: > > One of the recent commits have introduce a new warning with gcc 10, when > > building with optimizations: > > Oddly, I see no such warning with Fedora's current compiler, > gcc version 10.2.1 20201125 (Red Hat 10.2.1-9)

Re: Some regular-expression performance hacking

2021-02-23 Thread Tom Lane
Andres Freund writes: > One of the recent commits have introduce a new warning with gcc 10, when > building with optimizations: Oddly, I see no such warning with Fedora's current compiler, gcc version 10.2.1 20201125 (Red Hat 10.2.1-9) (GCC) Are you using any special compiler switches?

Re: Some regular-expression performance hacking

2021-02-23 Thread Andres Freund
Hi, On 2021-02-23 12:52:28 -0500, Tom Lane wrote: > I wrote: > > Hmph. There's an "assert(depth >= 0)" immediately in front of that, > > so I'm not looking too kindly on the compiler thinking it's smarter > > than I am. Do you have a suggestion on how to shut it up? gcc can't see the assert

Re: Some regular-expression performance hacking

2021-02-23 Thread Tom Lane
I wrote: > Hmph. There's an "assert(depth >= 0)" immediately in front of that, > so I'm not looking too kindly on the compiler thinking it's smarter > than I am. Do you have a suggestion on how to shut it up? On reflection, maybe the thing to do is convert the assert into an always-on check,

Re: Some regular-expression performance hacking

2021-02-23 Thread Tom Lane
Andres Freund writes: > One of the recent commits have introduce a new warning with gcc 10, when > building with optimizations: > In file included from > /home/andres/src/postgresql/src/backend/regex/regcomp.c:2304: > /home/andres/src/postgresql/src/backend/regex/regc_nfa.c: In function >

Re: new heapcheck contrib module

2021-02-23 Thread Mark Dilger
> On Feb 17, 2021, at 12:56 PM, Robert Haas wrote: > > On Wed, Feb 17, 2021 at 1:46 PM Mark Dilger > wrote: >> It will reconnect and retry a command one time on error. That should cover >> the case that the connection to the database was merely lost. If the second >> attempt also fails,

Re: Some regular-expression performance hacking

2021-02-23 Thread Andres Freund
Hi, One of the recent commits have introduce a new warning with gcc 10, when building with optimizations: In file included from /home/andres/src/postgresql/src/backend/regex/regcomp.c:2304: /home/andres/src/postgresql/src/backend/regex/regc_nfa.c: In function ‘checkmatchall’:

Re: Bizarre behavior of \w in a regular expression bracket construct

2021-02-23 Thread Tom Lane
I wrote: > Alvaro Herrera writes: >> It looks like the interpretation of these other engines is that [\d-a] >> is the set of \d, the literal character "-", and the literal character >> "a". In other words, the - preceded by \d or \w (or any other character >> class, I guess?) loses its special

Re: libpq compression

2021-02-23 Thread Konstantin Knizhnik
On 22.02.2021 08:38, Craig Ringer wrote: On Thu, 11 Feb 2021, 21:09 Daniil Zakhlystov, mailto:usernam...@yandex-team.ru>> wrote:: 3. Chunked compression allows to compress only well compressible messages and save the CPU cycles by not compressing the others 4. Chunked

Re: pg_upgrade version checking questions

2021-02-23 Thread Daniel Gustafsson
> On 27 Jul 2019, at 08:42, Peter Eisentraut > wrote: > > On 2019-07-25 16:33, Daniel Gustafsson wrote: >> Fair enough, those are both excellent points. I’ve shuffled the code around >> to >> move back the check for exec_path to setup (albeit earlier than before due to >> where we perform

Re: [PATCH]: Allow errors in parameter values to be reported during the BIND phase itself..

2021-02-23 Thread Justin Pryzby
On Mon, Feb 22, 2021 at 11:15:25AM -0600, Justin Pryzby wrote: > On Sun, Feb 21, 2021 at 11:05:26PM -0600, Justin Pryzby wrote: > > On Mon, Jan 04, 2021 at 11:09:39AM -0600, Justin Pryzby wrote: > > > For example: > > > > > > $ python3.5 -c "import pg; db=pg.DB(); q = db.query(\"SET > > >

Re: {CREATE INDEX, REINDEX} CONCURRENTLY improvements

2021-02-23 Thread Álvaro Herrera
On 2021-Feb-23, Masahiko Sawada wrote: > I've looked at the v3 patch and it looks good to me. A minor comment is: > > + /* Catalog tables need to consider all backends. */ > + h->catalog_oldest_nonremovable = > +

Re: libpq compression

2021-02-23 Thread Konstantin Knizhnik
On 22.02.2021 23:44, Tom Lane wrote: Robert Haas writes: So at the end of the day I'm not really quite sure what is best here. I agree with all of Craig's points about the advantages of packet-level compression, so I'd really prefer to make that approach work if we can. However, it also

INSERT ... ON CONFLICT ... : expose INSERT vs UPDATE status

2021-02-23 Thread George MacKerron
Hi all. I have a library that helps with querying Postgres from TypeScript, and a user just filed this issue: https://github.com/jawj/zapatos/issues/74 The library uses the xmax method (ubiquitous on Stack Overflow) to detect whether an upsert query resulted in an insert or an update. It

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-23 Thread Amit Kapila
On Tue, Feb 23, 2021 at 4:47 PM Greg Nancarrow wrote: > > On Tue, Feb 23, 2021 at 2:30 PM Amit Kapila wrote: > > > > On Tue, Feb 23, 2021 at 6:37 AM Greg Nancarrow wrote: > > > > > > On Tue, Feb 23, 2021 at 12:33 AM Amit Kapila > > > wrote: > > > > > > > > On Mon, Feb 22, 2021 at 8:41 AM Greg

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-23 Thread Greg Nancarrow
On Tue, Feb 23, 2021 at 2:30 PM Amit Kapila wrote: > > On Tue, Feb 23, 2021 at 6:37 AM Greg Nancarrow wrote: > > > > On Tue, Feb 23, 2021 at 12:33 AM Amit Kapila > > wrote: > > > > > > On Mon, Feb 22, 2021 at 8:41 AM Greg Nancarrow > > > wrote: > > > > > > > > On Fri, Feb 19, 2021 at 9:38 PM

Re: {CREATE INDEX, REINDEX} CONCURRENTLY improvements

2021-02-23 Thread Masahiko Sawada
On Tue, Feb 23, 2021 at 9:15 AM Álvaro Herrera wrote: > > On 2021-Feb-22, Álvaro Herrera wrote: > > > Here's an updated patch. > > > > I haven't added commentary or documentation to account for the new > > assumption, per Matthias' comment and Robert's discussion thereof. > > Done that. I also

RE: A reloption for partitioned tables - parallel_workers

2021-02-23 Thread houzj.f...@fujitsu.com
> > It seems the patch does not include the code that get the > > parallel_workers from new struct " PartitionedTableRdOptions ", Did I miss > something ? > > Aren't the following hunks in the v2 patch what you meant? > > diff --git a/src/backend/access/common/reloptions.c >

Asynchronous and "direct" IO support for PostgreSQL.

2021-02-23 Thread Andres Freund
Hi, over the last ~year I spent a lot of time trying to figure out how we could add AIO (asynchronous IO) and DIO (direct IO) support to postgres. While there's still a *lot* of open questions, I think I now have a decent handle on most of the bigger architectural questions. Thus this long

Re: repeated decoding of prepared transactions

2021-02-23 Thread Amit Kapila
On Mon, Feb 22, 2021 at 2:57 PM Andres Freund wrote: > > > Yeah, we need to probably store this new point as slot's persistent > > information. > > Should be fine I think... > So, we are in agreement that the above solution will work and we won't need to resend the prepare after the restart. I

Re: repeated decoding of prepared transactions

2021-02-23 Thread Ajin Cherian
On Mon, Feb 22, 2021 at 8:27 PM Andres Freund wrote: > > Yeah, we need to probably store this new point as slot's persistent > > information. > > Should be fine I think... This idea looks convincing. I'll write up a patch incorporating these changes. regards, Ajin Cherian Fujitsu Australia

Re: A reloption for partitioned tables - parallel_workers

2021-02-23 Thread Amit Langote
Hi, On Tue, Feb 23, 2021 at 3:12 PM houzj.f...@fujitsu.com wrote: > > Here is an updated version of the Seamus' patch that takes into account > > these > > and other comments received on this thread so far. > > Maybe warrants adding some tests too but I haven't. > > > > Seamus, please register

Re: Improvements and additions to COPY progress reporting

2021-02-23 Thread Matthias van de Meent
On Mon, 22 Feb 2021 at 05:49, Bharath Rupireddy wrote: > > On Mon, Feb 22, 2021 at 12:40 AM Matthias van de Meent > wrote: > > > > On Sat, 20 Feb 2021 at 07:09, Bharath Rupireddy > > wrote: > > > > > > For COPY TO the name "source_type" column and for COPY FROM the name > > >

Re: computing dT from an interval

2021-02-23 Thread Michael J. Baars
On Mon, 2021-02-22 at 10:52 -0500, Tom Lane wrote: > "Michael J. Baars" writes: > > So how do you compute the number of seconds in 8 years? > > IMO, that's a meaningless computation, because the answer is not fixed. > Before you claim otherwise, think about the every-four-hundred-years > leap

Re: [PATCH] pg_hba.conf error messages for logical replication connections

2021-02-23 Thread Amit Kapila
On Mon, Feb 22, 2021 at 6:08 PM Euler Taveira wrote: > > On Sat, Feb 20, 2021, at 7:33 AM, Amit Kapila wrote: > > I have used a bit of different wording here to make things clear. > > Let me know what you think of the attached? > > WFM. > Thanks, Pushed! -- With Regards, Amit Kapila.

Re: [HACKERS] logical decoding of two-phase transactions

2021-02-23 Thread Amit Kapila
On Tue, Feb 23, 2021 at 7:43 AM Amit Kapila wrote: > > On Mon, Feb 22, 2021 at 11:04 PM Markus Wanner wrote: > > > > On 04.01.21 09:18, Amit Kapila wrote: > > > Thanks, I have pushed the 0001* patch after making the above and a few > > > other cosmetic modifications. > > > > That commit added

Re: pg_temp_%d namespace creation can invalidate all the cached plan in other backends

2021-02-23 Thread Andy Fan
On Tue, Feb 23, 2021 at 1:50 PM Tom Lane wrote: > Andy Fan writes: > > Planning is expensive and we use plancache to bypass its effect. I find > the > > $subject recently which is caused by we register NAMESPACEOID > invalidation > > message for pg_temp_%s as well as other normal namespaces.