Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-07-14 Thread Peter Geoghegan
On Thu, Jul 9, 2020 at 12:06 PM Jonathan S. Katz wrote: > On 7/2/20 11:47 AM, James Coleman wrote: > > It seems like the consensus over at another discussion on this topic > > [1] is that we ought to go ahead and print the zeros [for machine > > readable output formats], even though that creates

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-14 Thread Andrey M. Borodin
Hi! > 14 июля 2020 г., в 02:12, Robert Haas написал(а): > > So I have these questions: > > - Do people think it would me smart/good/useful to include something > like this in PostgreSQL? > > - If so, how? I would propose a new contrib module that we back-patch > all the way My 0.05₽. At

Re: proposal: possibility to read dumped table's name from file

2020-07-14 Thread Pavel Stehule
po 13. 7. 2020 v 15:33 odesílatel vignesh C napsal: > On Mon, Jul 13, 2020 at 1:51 PM Pavel Stehule > wrote: > >> Can this be changed to dump objects and data based on the filter > >> expressions from the filter file. > > > > > > I am sorry, I don't understand. This should work for data from

Re: Reducing WaitEventSet syscall churn

2020-07-14 Thread Thomas Munro
On Sun, Jul 12, 2020 at 7:22 AM Tom Lane wrote: > [...complaints about 0005 and 0006...] We already have > PGEVT_CONNRESET and PGEVT_CONNDESTROY as application-visible connection > state change events, and I don't see why those aren't sufficient. I think Horiguchi-san's general idea of using

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-07-14 Thread Peter Geoghegan
On Thu, Jul 2, 2020 at 8:47 AM James Coleman wrote: > It seems like the consensus over at another discussion on this topic > [1] is that we ought to go ahead and print the zeros [for machine > readable output formats], even though that creates some interesting > scenarios like the fact that disk

Re: Parallel Seq Scan vs kernel read ahead

2020-07-14 Thread Thomas Munro
On Fri, Jun 26, 2020 at 3:33 AM Robert Haas wrote: > On Tue, Jun 23, 2020 at 11:53 PM David Rowley wrote: > > In summary, based on these tests, I don't think we're making anything > > worse in regards to synchronize_seqscans if we cap the maximum number > > of blocks to allocate to each worker

Re: proposal: possibility to read dumped table's name from file

2020-07-14 Thread Pavel Stehule
po 13. 7. 2020 v 20:00 odesílatel Justin Pryzby napsal: > On Mon, Jul 13, 2020 at 08:15:42AM +0200, Pavel Stehule wrote: > > > Do you want to add any more documentation ? > > > > > > > done > > Thanks - I think the documentation was maybe excessive. See attached. > I merged your patch - thank

Re: replication_origin and replication_origin_lsn usage on subscriber

2020-07-14 Thread Dilip Kumar
On Tue, Jul 14, 2020 at 11:14 AM Amit Kapila wrote: > > On Tue, Jul 14, 2020 at 11:00 AM Dilip Kumar wrote: > > > > On Thu, Jul 9, 2020 at 6:55 PM Amit Kapila wrote: > > > > > > On Thu, Jul 9, 2020 at 6:14 PM Petr Jelinek wrote: > > > > > > > > > > > > If we were to support the origin

Re: Improvements in Copy From

2020-07-14 Thread vignesh C
On Tue, Jul 14, 2020 at 11:13 AM David Rowley wrote: > > On Tue, 14 Jul 2020 at 17:22, David Rowley wrote: > > > > On Thu, 2 Jul 2020 at 00:46, vignesh C wrote: > > > b) CopyMultiInsertInfoNextFreeSlot had an unused function parameter > > > that is not being used, it can be removed. > > > >

Re: INSERT INTO SELECT, Why Parallelism is not selected?

2020-07-14 Thread Dilip Kumar
On Mon, Jul 13, 2020 at 4:23 PM Amit Kapila wrote: > > On Sat, Jul 11, 2020 at 6:07 PM Dilip Kumar wrote: > > > > I have just notice that the parallelism is off even for the select > > part of the query mentioned in the $subject. I see the only reason it > > is not getting parallel because we

Re: replication_origin and replication_origin_lsn usage on subscriber

2020-07-14 Thread Amit Kapila
On Tue, Jul 14, 2020 at 12:05 PM Dilip Kumar wrote: > > On Tue, Jul 14, 2020 at 11:14 AM Amit Kapila wrote: > > > > On Tue, Jul 14, 2020 at 11:00 AM Dilip Kumar wrote: > > > > > > On Thu, Jul 9, 2020 at 6:55 PM Amit Kapila > > > wrote: > > > > > > > > On Thu, Jul 9, 2020 at 6:14 PM Petr

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-14 Thread Magnus Hagander
On Tue, Jul 14, 2020 at 3:26 AM Tom Lane wrote: > Robert Haas writes: > > On Mon, Jul 13, 2020 at 8:58 PM Tom Lane wrote: > >> The more that I think about it, the more I think that the proposed > >> functions are tools for wizards only, and so I'm getting hesitant > >> about having them in

Re: Online checksums verification in the backend

2020-07-14 Thread Julien Rouhaud
On Sun, Jul 12, 2020 at 12:34:03PM -0500, Justin Pryzby wrote: > Small language fixes in comments and user-facing docs. Thanks a lot! I just fixed a small issue (see below), PFA updated v10. > > diff --git a/src/backend/storage/page/checksum.c > b/src/backend/storage/page/checksum.c > index

Re: replication_origin and replication_origin_lsn usage on subscriber

2020-07-14 Thread Petr Jelinek
Hi, On 14/07/2020 10:29, Amit Kapila wrote: On Tue, Jul 14, 2020 at 12:05 PM Dilip Kumar wrote: On Tue, Jul 14, 2020 at 11:14 AM Amit Kapila wrote: On Tue, Jul 14, 2020 at 11:00 AM Dilip Kumar wrote: On Thu, Jul 9, 2020 at 6:55 PM Amit Kapila wrote: On Thu, Jul 9, 2020 at 6:14 PM

Re: replication_origin and replication_origin_lsn usage on subscriber

2020-07-14 Thread Petr Jelinek
On 14/07/2020 11:36, Dilip Kumar wrote: On Tue, Jul 14, 2020 at 2:47 PM Petr Jelinek wrote: Hi, On 14/07/2020 10:29, Amit Kapila wrote: On Tue, Jul 14, 2020 at 12:05 PM Dilip Kumar wrote: On Tue, Jul 14, 2020 at 11:14 AM Amit Kapila wrote: On Tue, Jul 14, 2020 at 11:00 AM Dilip Kumar

Re: [PATCH] postgres_fdw connection caching - cause remote sessions linger till the local session exit

2020-07-14 Thread Bharath Rupireddy
Thanks all for the ideas. There have been various points/approaches discussed in the entire email chain so far. I would like to summarize all of them here, so that we can agree on one of the options and proceed further with this feature. The issue this feature is trying to solve: In postgres_fdw,

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2020-07-14 Thread Atsushi Torikoshi
Hi, v9 patch fails to apply to HEAD, could you check and rebase it? And here are minor typos. 79 +* utility statements. Note that we don't compute a queryId for prepared 80 +* statemets related utility, as those will inherit from the underlying 81 +* statements's one

Re: Don't choke on files that are removed while pg_rewind runs.

2020-07-14 Thread Daniel Gustafsson
> On 13 Jul 2020, at 14:18, Michael Paquier wrote: > That sounds like a good idea with an extra qual in the first part of > the inner CTE, if coupled with a check to make sure that we never > get a NULL result. Now there is IMO an argument to not complicate > more this query as it is not like a

Re: Transactions involving multiple postgres foreign servers, take 2

2020-07-14 Thread Masahiro Ikeda
I've attached the latest version patches. I've incorporated the review comments I got so far and improved locking strategy. I want to ask a question about streaming replication with 2PC. Are you going to support 2PC with streaming replication? I tried streaming replication using v23 patches. I

Re: Avoid useless retrieval of defaults and check constraints in pg_dump -a

2020-07-14 Thread Julien Rouhaud
On Sun, Jul 12, 2020 at 4:29 PM Justin Pryzby wrote: > > On Sun, Jul 12, 2020 at 07:48:50AM +0200, Julien Rouhaud wrote: > > Currently, getTableAttrs() always retrieves info about columns defaults and > > check constraints, while this will never be used if --data-only option if > > used. > >

Re: replication_origin and replication_origin_lsn usage on subscriber

2020-07-14 Thread Dilip Kumar
On Tue, Jul 14, 2020 at 2:47 PM Petr Jelinek wrote: > > Hi, > > On 14/07/2020 10:29, Amit Kapila wrote: > > On Tue, Jul 14, 2020 at 12:05 PM Dilip Kumar wrote: > >> > >> On Tue, Jul 14, 2020 at 11:14 AM Amit Kapila > >> wrote: > >>> > >>> On Tue, Jul 14, 2020 at 11:00 AM Dilip Kumar > >>>

Re: INSERT INTO SELECT, Why Parallelism is not selected?

2020-07-14 Thread Rushabh Lathia
On Sat, Jul 11, 2020 at 6:07 PM Dilip Kumar wrote: > I have just notice that the parallelism is off even for the select > part of the query mentioned in the $subject. I see the only reason it > is not getting parallel because we block the parallelism if the query > type is not SELECT. I don't

Re: Issue with cancel_before_shmem_exit while searching to remove a particular registered exit callbacks

2020-07-14 Thread Bharath Rupireddy
On Tue, Jul 7, 2020 at 10:24 PM Andres Freund wrote: > > On 2020-07-07 09:44:41 -0400, Tom Lane wrote: > > Bharath Rupireddy writes: > > > Firstly, pg_start_backup registers nonexclusive_base_backup_cleanup as > > > on_shmem_exit call back which will > > > add this function to the

Re: replication_origin and replication_origin_lsn usage on subscriber

2020-07-14 Thread Amit Kapila
On Tue, Jul 14, 2020 at 2:47 PM Petr Jelinek wrote: > > Hi, > > On 14/07/2020 10:29, Amit Kapila wrote: > > On Tue, Jul 14, 2020 at 12:05 PM Dilip Kumar wrote: > >> > >> On Tue, Jul 14, 2020 at 11:14 AM Amit Kapila > >> wrote: > >>> > >>> On Tue, Jul 14, 2020 at 11:00 AM Dilip Kumar > >>>

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-14 Thread Robert Haas
On Tue, Jul 14, 2020 at 4:59 AM Magnus Hagander wrote: > The countersable of this is pg_resetwal. The number of people who have broken > their database with that tool is not small. Very true. > That said, we could have a separate "class" of extensions/tools in the > distribution, and

Re: DROP relation IF EXISTS Docs and Tests - Bug Fix

2020-07-14 Thread Pavel Stehule
út 14. 7. 2020 v 16:09 odesílatel David G. Johnston < david.g.johns...@gmail.com> napsal: > On Tue, Jul 14, 2020 at 6:56 AM Pavel Stehule > wrote: > >> út 14. 7. 2020 v 15:55 odesílatel David G. Johnston < >> david.g.johns...@gmail.com> napsal: >> >>> Further comments welcome so I'm putting it

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-14 Thread Magnus Hagander
On Tue, Jul 14, 2020 at 4:09 PM Robert Haas wrote: > On Tue, Jul 14, 2020 at 8:25 AM Magnus Hagander > wrote: > > I don't think that it necessarily has to be. As long as we're talking > about adding something and not actually changing their existing packages, > getting this into both yum and

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-14 Thread Stephen Frost
Greetings, * Magnus Hagander (mag...@hagander.net) wrote: > On Tue, Jul 14, 2020 at 4:09 PM Robert Haas wrote: > > On Tue, Jul 14, 2020 at 8:25 AM Magnus Hagander > > wrote: > > > I don't think that it necessarily has to be. As long as we're talking > > about adding something and not actually

Re: DROP relation IF EXISTS Docs and Tests - Bug Fix

2020-07-14 Thread Pavel Stehule
út 14. 7. 2020 v 15:55 odesílatel David G. Johnston < david.g.johns...@gmail.com> napsal: > On Tue, Jul 14, 2020 at 5:40 AM Justin Pryzby > wrote: > >> On Tue, Jul 14, 2020 at 07:25:56AM +0200, Pavel Stehule wrote: >> > út 14. 7. 2020 v 0:37 odesílatel David G. Johnston < >>

Re: [Proposal] Global temporary tables

2020-07-14 Thread Pavel Stehule
pá 10. 7. 2020 v 11:04 odesílatel wenjing zeng napsal: > HI all > > I started using my personal email to respond to community issue. > > > > 2020年7月7日 下午6:05,Pavel Stehule 写道: > > Hi > > >> GTT Merge the latest PGMaster and resolves conflicts. >> >> >> > I tested it and it looks fine. I think

Re: Creating foreign key on partitioned table is too slow

2020-07-14 Thread Daniel Gustafsson
> On 24 Mar 2020, at 16:26, Alvaro Herrera wrote: > It > would be great if Kato Sho can try the original test case with my latest > patch (the one in https://postgr.es/m/20191113214544.GA16060@alvherre.pgsql ) > and let us know if it improves things. Hi!, Are you able to test Alvaros latest

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-14 Thread Magnus Hagander
On Tue, Jul 14, 2020 at 1:52 PM Robert Haas wrote: > On Tue, Jul 14, 2020 at 4:59 AM Magnus Hagander > wrote: > > The countersable of this is pg_resetwal. The number of people who have > broken their database with that tool is not small. > > Very true. > > > That said, we could have a separate

Re: DROP relation IF EXISTS Docs and Tests - Bug Fix

2020-07-14 Thread Justin Pryzby
On Tue, Jul 14, 2020 at 07:25:56AM +0200, Pavel Stehule wrote: > út 14. 7. 2020 v 0:37 odesílatel David G. Johnston > napsal: > > On Mon, Jul 13, 2020 at 2:12 AM Pavel Stehule > > wrote: > I think so now all changes are correct and valuable. I''l mark this patch > as ready for commit This is

Re: Retry Cached Remote Connections for postgres_fdw in case remote backend gets killed/goes away

2020-07-14 Thread Bharath Rupireddy
On Tue, Jul 14, 2020 at 6:13 PM Ashutosh Bapat wrote: > > Has this been added to CF, possibly next CF? > I have not added yet. Request to get it done in this CF, as the final patch for review(v3 patch) is ready and shared. We can target it to the next CF if there are major issues with the patch.

Re: [PATCH] Performance Improvement For Copy From Binary Files

2020-07-14 Thread vignesh C
On Tue, Jul 14, 2020 at 11:19 AM Amit Langote wrote: > > > Sounds fine to me. Although CopyLoadRawBuf() does not seem to a > candidate for rigorous code optimization as it does not get called > that often. > I thought we could include that change as we are making changes around that code. Rest

Re: DROP relation IF EXISTS Docs and Tests - Bug Fix

2020-07-14 Thread David G. Johnston
On Tue, Jul 14, 2020 at 5:40 AM Justin Pryzby wrote: > On Tue, Jul 14, 2020 at 07:25:56AM +0200, Pavel Stehule wrote: > > út 14. 7. 2020 v 0:37 odesílatel David G. Johnston < > david.g.johns...@gmail.com> napsal: > > > On Mon, Jul 13, 2020 at 2:12 AM Pavel Stehule > wrote: > > I think so now

Re: [Proposal] Global temporary tables

2020-07-14 Thread Pavel Stehule
po 13. 7. 2020 v 13:59 odesílatel wenjing zeng napsal: > > > 2020年7月10日 下午5:03,wenjing zeng 写道: > > HI all > > I started using my personal email to respond to community issue. > > > > 2020年7月7日 下午6:05,Pavel Stehule 写道: > > Hi > > >> GTT Merge the latest PGMaster and resolves conflicts. >> >>

Re: Partitioning and postgres_fdw optimisations for multi-tenancy

2020-07-14 Thread Alexey Kondratov
On 2020-07-14 15:27, Ashutosh Bapat wrote: On Tue, Jul 14, 2020 at 12:48 AM Alexey Kondratov wrote: I built a simple two node multi-tenant schema for tests, which can be easily set up with attached scripts. It creates three tables (companies, users, documents) distributed over two nodes.

Re: DROP relation IF EXISTS Docs and Tests - Bug Fix

2020-07-14 Thread David G. Johnston
On Tue, Jul 14, 2020 at 7:21 AM Pavel Stehule wrote: > út 14. 7. 2020 v 16:09 odesílatel David G. Johnston < > david.g.johns...@gmail.com> napsal: > >> On Tue, Jul 14, 2020 at 6:56 AM Pavel Stehule >> wrote: >> >>> út 14. 7. 2020 v 15:55 odesílatel David G. Johnston < >>>

Re: [PATCH] postgres_fdw connection caching - cause remote sessions linger till the local session exit

2020-07-14 Thread Robert Haas
On Tue, Jul 14, 2020 at 6:09 AM Bharath Rupireddy wrote: > Thanks all for the ideas. There have been various points/approaches > discussed in the entire email chain so far. > I would like to summarize all of them here, so that we can agree on > one of the options and proceed further with this

Re: Partitioning and postgres_fdw optimisations for multi-tenancy

2020-07-14 Thread Ashutosh Bapat
On Tue, Jul 14, 2020 at 12:48 AM Alexey Kondratov wrote: > > Hi Hackers, > > The idea of achieving Postgres scaling via sharding using postgres_fdw + > partitioning got a lot of attention last years. Many optimisations have > been done in this direction: partition pruning, partition-wise >

Re: Improvements in Copy From

2020-07-14 Thread vignesh C
On Tue, Jul 14, 2020 at 12:17 PM vignesh C wrote: > > On Tue, Jul 14, 2020 at 11:13 AM David Rowley wrote: > > > > On Tue, 14 Jul 2020 at 17:22, David Rowley wrote: > > > > > > On Thu, 2 Jul 2020 at 00:46, vignesh C wrote: > > > > b) CopyMultiInsertInfoNextFreeSlot had an unused function

Re: DROP relation IF EXISTS Docs and Tests - Bug Fix

2020-07-14 Thread David G. Johnston
On Tue, Jul 14, 2020 at 6:56 AM Pavel Stehule wrote: > út 14. 7. 2020 v 15:55 odesílatel David G. Johnston < > david.g.johns...@gmail.com> napsal: > >> Further comments welcome so I'm putting it back into needs review for the >> moment while I work on the refactor. >> > > attached fixed patch >

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-14 Thread Robert Haas
On Mon, Jul 13, 2020 at 9:29 PM Fujii Masao wrote: > But updating this tool can fit to the release schedule and > policy of PostgreSQL? > > While investigating the problem by using this tool, we may want to > add new feature into the tool because it's necessary for the investigation. > But users

Re: [patch] demote

2020-07-14 Thread Amul Sul
On Mon, Jul 13, 2020 at 8:35 PM Jehan-Guillaume de Rorthais wrote: > > Hi, > > Another summary + patch + tests. > > This patch supports 2PC. The goal is to keep them safe during demote/promote > actions so they can be committed/rollbacked later on a primary. See tests. > Wondering is it

Re: Retry Cached Remote Connections for postgres_fdw in case remote backend gets killed/goes away

2020-07-14 Thread Ashutosh Bapat
Has this been added to CF, possibly next CF? On Tue, Jul 14, 2020 at 7:27 AM Bharath Rupireddy wrote: > > > > > You could get a backend's PID using PQbackendPID and then kill it by > > calling pg_terminate_backend() to kill the remote backend to automate > > scenario of remote backend being

Re: Binary support for pgoutput plugin

2020-07-14 Thread Dave Cramer
On Tue, 14 Jul 2020 at 09:26, Daniel Gustafsson wrote: > > On 13 Jul 2020, at 15:11, Dave Cramer wrote: > > I took another look at the updated version today. Since there now were > some > unused variables and (I believe) unnecessary checks (int size and > endianness > etc) left, I took the

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-14 Thread Robert Haas
On Tue, Jul 14, 2020 at 8:25 AM Magnus Hagander wrote: > I don't think that it necessarily has to be. As long as we're talking about > adding something and not actually changing their existing packages, getting > this into both yum and apt shouldn't be *that* hard, if it's coordinated well >

Re: Is it useful to record whether plans are generic or custom?

2020-07-14 Thread torikoshia
On 2020-07-10 10:49, torikoshia wrote: On 2020-07-08 16:41, Fujii Masao wrote: On 2020/07/08 10:14, torikoshia wrote: On 2020-07-06 22:16, Fujii Masao wrote: On 2020/06/11 14:59, torikoshia wrote: On 2020-06-10 18:00, Kyotaro Horiguchi wrote: +    TupleDescInitEntry(tupdesc, (AttrNumber)

Re: Binary support for pgoutput plugin

2020-07-14 Thread Daniel Gustafsson
> On 13 Jul 2020, at 15:11, Dave Cramer wrote: I took another look at the updated version today. Since there now were some unused variables and (I believe) unnecessary checks (int size and endianness etc) left, I took the liberty to fix those. I also fixed some markup in the catalog docs, did

RE: Index Skip Scan (new UniqueKeys)

2020-07-14 Thread Floris Van Nee
> > > - the uniquekeys that is built, still contains some redundant keys, that are > normally eliminated from the path keys lists. > > I guess you're talking about: > > + if (EC_MUST_BE_REDUNDANT(ec)) > + continue; > > Can you add some test cases to your changes to show the effect of it? It

Re: Binary support for pgoutput plugin

2020-07-14 Thread Andres Freund
Hi, On 2020-07-14 14:08:53 -0400, Dave Cramer wrote: > On Tue, 14 Jul 2020 at 12:59, Tom Lane wrote: > > > So I started looking through this seriously, and my first question > > is why do the docs and code keep saying that "base types" are sent > > in binary? Why not just "data"? Are there

Re: Index Skip Scan (new UniqueKeys)

2020-07-14 Thread Dmitry Dolgov
> On Sun, Jul 12, 2020 at 12:48:47PM +, Floris Van Nee wrote: > > > > Good point, thanks for looking at this. With the latest planner version > > there > > are indeed more possibilities to use skipping. It never occured to me that > > some of those paths will still rely on index scan

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-14 Thread Alvaro Herrera
On 2020-Jul-13, Andres Freund wrote: > Hi, > > On 2020-07-13 17:12:18 -0400, Robert Haas wrote: > > 1. There's nothing to identify the tuple that has the problem, and no > > way to know how many more of them there might be. Back-patching > > b61d161c146328ae6ba9ed937862d66e5c8b035a would help

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-14 Thread Andres Freund
Hi, On 2020-07-13 21:18:10 -0400, Robert Haas wrote: > On Mon, Jul 13, 2020 at 9:10 PM Andres Freund wrote: > > > What if clog has been truncated so that the xmin can't be looked up? > > > > That's possible, but probably only in cases where xmin actually > > committed. > > Isn't that the normal

Re: [patch] demote

2020-07-14 Thread Andres Freund
Hi, On 2020-07-14 17:26:37 +0530, Amul Sul wrote: > On Mon, Jul 13, 2020 at 8:35 PM Jehan-Guillaume de Rorthais > wrote: > > > > Hi, > > > > Another summary + patch + tests. > > > > This patch supports 2PC. The goal is to keep them safe during demote/promote > > actions so they can be

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-14 Thread Robert Haas
On Tue, Jul 14, 2020 at 3:42 PM Andres Freund wrote: > The "found xmin ... from before relfrozenxid ..." cases should all be > fixable without needing such a function, and without it making fixing > them significantly easier, no? As far as I understand your suggested > solution, you need the

Re: [PATCH] postgres_fdw connection caching - cause remote sessions linger till the local session exit

2020-07-14 Thread Bruce Momjian
On Tue, Jul 14, 2020 at 03:38:49PM +0530, Bharath Rupireddy wrote: > Approach #4: > A postgres_fdw foreign server level option: connection idle time, the > amount of idle time for that server cached entry, after which the > cached entry goes away. Probably the backend, before itself going to >

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-14 Thread Andres Freund
Hi, On 2020-07-14 13:20:25 -0400, Alvaro Herrera wrote: > On 2020-Jul-13, Andres Freund wrote: > > > Hi, > > > > On 2020-07-13 17:12:18 -0400, Robert Haas wrote: > > > 1. There's nothing to identify the tuple that has the problem, and no > > > way to know how many more of them there might be.

Re: Rethinking opclass member checks and dependency strength

2020-07-14 Thread Anastasia Lubennikova
On 31.03.2020 23:45, Tom Lane wrote: I wrote: Still haven't got a better naming idea, but in the meantime here's a rebase to fix a conflict with 612a1ab76. Maybe "amadjustmembers" will work? I've looked through the patch and noticed this comment: +            default: +                /*

Re: Binary support for pgoutput plugin

2020-07-14 Thread Dave Cramer
On Tue, 14 Jul 2020 at 12:59, Tom Lane wrote: > So I started looking through this seriously, and my first question > is why do the docs and code keep saying that "base types" are sent > in binary? Why not just "data"? Are there any cases where we > don't use binary format, if the subscription

Re: Multi-byte character case-folding

2020-07-14 Thread Bruce Momjian
On Mon, Jul 6, 2020 at 08:32:22PM -0400, Tom Lane wrote: > Yes, a GUC changing this would be a headache. It would be just as much of > a compatibility and security hazard as standard_conforming_strings (which > indeed I've been thinking of proposing that we get rid of; it's hung > around long

Re: INSERT INTO SELECT, Why Parallelism is not selected?

2020-07-14 Thread Robert Haas
On Sat, Jul 11, 2020 at 8:37 AM Dilip Kumar wrote: > I have just notice that the parallelism is off even for the select > part of the query mentioned in the $subject. I see the only reason it > is not getting parallel because we block the parallelism if the query > type is not SELECT. I don't

Re: Default setting for enable_hashagg_disk

2020-07-14 Thread Robert Haas
On Mon, Jul 13, 2020 at 2:50 PM Peter Geoghegan wrote: > Primarily in favor of escape hatch: > > Jeff, > DavidR, > Pavel, > Andres, > Robert ??, > Amit ?? > > Primarily in favor of hash_mem/hash_mem_multiplier: > > PeterG, > Tom, > Alvaro, > Tomas, > Justin, > DavidG, > Jonathan Katz > > There

Re: Binary support for pgoutput plugin

2020-07-14 Thread Tom Lane
Dave Cramer writes: > On Tue, 14 Jul 2020 at 12:59, Tom Lane wrote: >> So I started looking through this seriously, and my first question >> is why do the docs and code keep saying that "base types" are sent >> in binary? Why not just "data"? Are there any cases where we >> don't use binary

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

2020-07-14 Thread Andres Freund
Hi, On 2020-07-10 19:01:49 -0400, Alvaro Herrera wrote: > Totally unasked for, here's a rebase of this patch series. I didn't do > anything other than rebasing to current master, solving a couple of very > trivial conflicts, fixing some whitespace complaints by git apply, and > running tests to

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-14 Thread Alvaro Herrera
On 2020-Jul-14, Andres Freund wrote: > Hi, > > On 2020-07-14 13:20:25 -0400, Alvaro Herrera wrote: > > Just having the block number is already a tremendous step forward; with > > that you can ask the customer to set a pageinspect dump of tuple > > headers, and then the problem is obvious. Now

Re: Binary support for pgoutput plugin

2020-07-14 Thread Tom Lane
So I started looking through this seriously, and my first question is why do the docs and code keep saying that "base types" are sent in binary? Why not just "data"? Are there any cases where we don't use binary format, if the subscription requests it? If there's not a concrete reason to use

Re: COPY FREEZE and setting PD_ALL_VISIBLE/visibility map bits

2020-07-14 Thread Anastasia Lubennikova
On 01.07.2020 12:38, Daniel Gustafsson wrote: This patch incurs a compiler warning, which probably is quite simple to fix: heapam.c: In function ‘heap_multi_insert’: heapam.c:2349:4: error: ‘recptr’ may be used uninitialized in this function [-Werror=maybe-uninitialized]

Re: warnings for invalid function casts

2020-07-14 Thread Peter Eisentraut
On 2020-07-07 18:08, Tom Lane wrote: Peter Eisentraut writes: On 2020-07-04 16:16, Tom Lane wrote: I'm for a typedef. There is *nothing* readable about "(void (*) (void))", and the fact that it's theoretically incorrect for the purpose doesn't exactly aid intelligibility either. With a

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-14 Thread Andres Freund
Hi, On 2020-07-14 07:51:27 -0400, Robert Haas wrote: > On Tue, Jul 14, 2020 at 3:08 AM Peter Eisentraut > wrote: > > In the meantime, if you're wizard enough to deal with this kind of > > thing, you could also clone the module from the PG14 tree and build it > > against older versions manually.

Re: Towards easier AMs: Cleaning up inappropriate use of name "relkind"

2020-07-14 Thread Alvaro Herrera
This patch is way too large. Probably in part explained by the fact that you seem to have run pgindent and absorbed a lot of unrelated changes. This makes the patch essentially unreviewable. I think you should define a RelationGetRelkind() static function that returns the appropriate datatype

Re: Parallel Seq Scan vs kernel read ahead

2020-07-14 Thread David Rowley
On Tue, 14 Jul 2020 at 19:13, Thomas Munro wrote: > > On Fri, Jun 26, 2020 at 3:33 AM Robert Haas wrote: > > On Tue, Jun 23, 2020 at 11:53 PM David Rowley wrote: > > > In summary, based on these tests, I don't think we're making anything > > > worse in regards to synchronize_seqscans if we cap

Re: Binary support for pgoutput plugin

2020-07-14 Thread Andres Freund
Hi, On 2020-07-14 19:46:52 -0400, Tom Lane wrote: > Andres Freund writes: > > There's also send/receive functions that do not work across systems, > > unfortunately :(. In particular record and array send functions embed > > type oids and their receive functions verify that they match the local

Re: Postgres is not able to handle more than 4k tables!?

2020-07-14 Thread Bruce Momjian
On Fri, Jul 10, 2020 at 02:10:20AM +, tsunakawa.ta...@fujitsu.com wrote: > > There were many proposed patches which help to improve this > > situation. But as far as this patches increase performance only > > at huge servers with large number of cores and show almost no > > improvement (or

Re: Cache lookup errors with functions manipulation object addresses

2020-07-14 Thread Michael Paquier
On Tue, Jul 07, 2020 at 11:08:30AM -0400, Alvaro Herrera wrote: g> That's a holdover from old times, when we thought functions were > procedures. That's no longer the case. Thanks, so "routine" it is. I have done an extra round of review of this patch, and noticed two things: - In

Re: Default setting for enable_hashagg_disk

2020-07-14 Thread Peter Geoghegan
On Tue, Jul 14, 2020 at 12:46 PM Robert Haas wrote: > - I thought the problem we were trying to solve here was that, in v12, > if the planner thinks that your hashagg will fit in memory when really > it doesn't, you will get good performance because we'll cheat; in v13, > you'll get VERY bad

Re: [patch] demote

2020-07-14 Thread Jehan-Guillaume de Rorthais
On Tue, 14 Jul 2020 12:49:51 -0700 Andres Freund wrote: > Hi, > > On 2020-07-14 17:26:37 +0530, Amul Sul wrote: > > On Mon, Jul 13, 2020 at 8:35 PM Jehan-Guillaume de Rorthais > > wrote: > > > > > > Hi, > > > > > > Another summary + patch + tests. > > > > > > This patch supports 2PC. The

Refactoring relkind as an enum

2020-07-14 Thread Mark Dilger
> On Jul 14, 2020, at 4:12 PM, Alvaro Herrera wrote: > > This patch is way too large. Probably in part explained by the fact > that you seem to have run pgindent and absorbed a lot of unrelated > changes. This makes the patch essentially unreviewable. I did not run pgindent, but when

Re: SQL/JSON: functions

2020-07-14 Thread Andrew Dunstan
On 7/14/20 1:00 PM, Andrew Dunstan wrote: > On 7/5/20 1:29 PM, Justin Pryzby wrote: >> On Mon, Mar 23, 2020 at 08:28:52PM +0300, Nikita Glukhov wrote: >>> Attached 47th version of the patches. >> The patch checker/cfbot says this doesn't apply ; could you send a rebasified >> version ? >> > To

Re: Binary support for pgoutput plugin

2020-07-14 Thread Tom Lane
Andres Freund writes: > There's also send/receive functions that do not work across systems, > unfortunately :(. In particular record and array send functions embed > type oids and their receive functions verify that they match the local > system. Which basically means that if there's any

Re: Postgres is not able to handle more than 4k tables!?

2020-07-14 Thread Alvaro Herrera
On 2020-Jul-10, Konstantin Knizhnik wrote: > @@ -3076,6 +3080,10 @@ relation_needs_vacanalyze(Oid relid, > instuples = tabentry->inserts_since_vacuum; > anltuples = tabentry->changes_since_analyze; > > + rel = RelationIdGetRelation(relid); > +

Re: avoid bitmapOR-ing indexes with scan condition inconsistent with partition constraint

2020-07-14 Thread Justin Pryzby
Added here: https://commitfest.postgresql.org/29/2644/ And updated tests to pass following: |commit 689696c7110f148ede8004aae50d7543d05b5587 |Author: Tom Lane |Date: Tue Jul 14 18:56:49 2020 -0400 | |Fix bitmap AND/OR scans on the inside of a nestloop partition-wise join. >From

Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)

2020-07-14 Thread Justin Pryzby
On Sun, Jun 21, 2020 at 08:53:25PM -0500, Justin Pryzby wrote: > I'm still waiting to hear feedback from a commiter if this is a good idea to > put this into the system catalog. Right now, ts_debug is the only nontrivial > function. I'm still missing feedback from committers about the foundation

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-07-14 Thread Dilip Kumar
On Mon, Jul 13, 2020 at 4:00 PM Amit Kapila wrote: > > On Mon, Jul 13, 2020 at 3:04 PM Dilip Kumar wrote: > > > > On Mon, Jul 13, 2020 at 2:56 PM Amit Kapila wrote: > > > > > > On Mon, Jul 13, 2020 at 2:32 PM Dilip Kumar wrote: > > > > > > > > On Mon, Jul 13, 2020 at 11:10 AM Amit Kapila > >

Re: WIP: BRIN multi-range indexes

2020-07-14 Thread Alexander Korotkov
On Mon, Jul 13, 2020 at 5:59 PM Tomas Vondra wrote: > >> > If we really want to print something nicer, I'd say it needs to be a > >> > special function in the BRIN opclass. > >> > >> If that can be done, then +1. We just need to ensure that the function > >> knows and can verify the type of

Re: GSSENC'ed connection stalls while reconnection attempts.

2020-07-14 Thread Kyotaro Horiguchi
At Tue, 14 Jul 2020 13:31:31 +0900 (JST), Kyotaro Horiguchi wrote in > Agreed to separate the change from this issue. I also don't think > that change in behavior dramatically improve the situation since we > should have had a bunch of trouble when a write actually failed in the > normal case.

Re: Binary support for pgoutput plugin

2020-07-14 Thread Andres Freund
Hi, On 2020-07-14 22:28:48 -0400, Tom Lane wrote: > Andres Freund writes: > > What is the gain in having these checks? recv functions need to be safe > > against arbitrary input, so a type crosscheck doesn't buy additional > > safety in that regard. Not that a potential attacker couldn't just >

Re: Parallel Seq Scan vs kernel read ahead

2020-07-14 Thread David Rowley
On Wed, 15 Jul 2020 at 14:51, Amit Kapila wrote: > > On Wed, Jul 15, 2020 at 5:55 AM David Rowley wrote: > > If we've not seen any performance regressions within 1 week, then I > > propose that we (pending final review) push this to allow wider > > testing. > > I think Soumyadeep has reported a

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-07-14 Thread Dilip Kumar
On Mon, Jul 13, 2020 at 10:47 AM Dilip Kumar wrote: > > On Sun, Jul 12, 2020 at 9:56 PM Dilip Kumar wrote: > > > > On Mon, Jul 6, 2020 at 11:43 AM Dilip Kumar wrote: > > > > > > On Mon, Jul 6, 2020 at 11:31 AM Amit Kapila > > > wrote: > > > > > > > > On Sun, Jul 5, 2020 at 4:47 PM Dilip Kumar

Re: Default setting for enable_hashagg_disk

2020-07-14 Thread Peter Geoghegan
On Mon, Jul 13, 2020 at 9:47 AM Alvaro Herrera wrote: > I'm in favor of hash_mem_multiplier. I think a >1 default is more > sensible than =1 in the long run, but if strategic vote is what we're > doing, then I support the =1 option. Attached is a WIP patch implementing hash_mem_multiplier, with

Re: Parallel copy

2020-07-14 Thread Amit Kapila
On Sun, Jul 12, 2020 at 5:48 PM Bharath Rupireddy wrote: > > > > > Hi Bharath, > > > > I was looking forward to review this patch-set but unfortunately it is > > showing a reject in copy.c, and might need a rebase. > > I was applying on master over the commit- > >

Re: Binary support for pgoutput plugin

2020-07-14 Thread Tom Lane
Andres Freund writes: > What is the gain in having these checks? recv functions need to be safe > against arbitrary input, so a type crosscheck doesn't buy additional > safety in that regard. Not that a potential attacker couldn't just > change the content anyways? You're confusing security

Re: replication_origin and replication_origin_lsn usage on subscriber

2020-07-14 Thread Amit Kapila
On Tue, Jul 14, 2020 at 3:37 PM Petr Jelinek wrote: > > On 14/07/2020 11:36, Dilip Kumar wrote: > > On Tue, Jul 14, 2020 at 2:47 PM Petr Jelinek wrote: > >> > >> I am not sure if I can follow the discussion here very well, but if I > >> understand correctly I'd like to clarify two things: > >> -

Re: Is it useful to record whether plans are generic or custom?

2020-07-14 Thread Fujii Masao
On 2020/07/14 21:24, torikoshia wrote: On 2020-07-10 10:49, torikoshia wrote: On 2020-07-08 16:41, Fujii Masao wrote: On 2020/07/08 10:14, torikoshia wrote: On 2020-07-06 22:16, Fujii Masao wrote: On 2020/06/11 14:59, torikoshia wrote: On 2020-06-10 18:00, Kyotaro Horiguchi wrote: +   

Re: [PATCH] Performance Improvement For Copy From Binary Files

2020-07-14 Thread vignesh C
On Wed, Jul 15, 2020 at 8:03 AM Amit Langote wrote: > > Hi Vignesh, > > On Tue, Jul 14, 2020 at 10:23 PM vignesh C wrote: > > On Tue, Jul 14, 2020 at 11:19 AM Amit Langote > > wrote: > > > Sounds fine to me. Although CopyLoadRawBuf() does not seem to a > > > candidate for rigorous code

Re: Parallel Seq Scan vs kernel read ahead

2020-07-14 Thread Amit Kapila
On Wed, Jul 15, 2020 at 5:55 AM David Rowley wrote: > > On Tue, 14 Jul 2020 at 19:13, Thomas Munro wrote: > > > > On Fri, Jun 26, 2020 at 3:33 AM Robert Haas wrote: > > > On Tue, Jun 23, 2020 at 11:53 PM David Rowley > > > wrote: > > > > In summary, based on these tests, I don't think we're

Re: [PATCH] Performance Improvement For Copy From Binary Files

2020-07-14 Thread Amit Langote
Hi Vignesh, On Tue, Jul 14, 2020 at 10:23 PM vignesh C wrote: > On Tue, Jul 14, 2020 at 11:19 AM Amit Langote wrote: > > Sounds fine to me. Although CopyLoadRawBuf() does not seem to a > > candidate for rigorous code optimization as it does not get called > > that often. > > I thought we could

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-14 Thread Jochem van Dieten
On Tue, Jul 14, 2020 at 3:26 AM Tom Lane wrote: > I think you're attacking a straw man. I'm well aware of how open source > works, thanks. What I'm saying is that contrib is mostly seen to be > reasonably harmless stuff. Sure, you can overwrite data you didn't want > to with adminpack's

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2020-07-14 Thread Julien Rouhaud
On Tue, Jul 14, 2020 at 07:11:02PM +0900, Atsushi Torikoshi wrote: > Hi, > > v9 patch fails to apply to HEAD, could you check and rebase it? Thanks for the notice, v10 attached! > And here are minor typos. > > 79 +* utility statements. Note that we don't compute a queryId > for

  1   2   >