Re: [HACKERS] Slow synchronous logical replication

2017-10-10 Thread Masahiko Sawada
On Mon, Oct 9, 2017 at 4:37 PM, Konstantin Knizhnik wrote: > Thank you for explanations. > > On 08.10.2017 16:00, Craig Ringer wrote: >> >> I think it'd be helpful if you provided reproduction instructions, >> test programs, etc, making it very clear when things are / aren't >> related to your cha

Re: [HACKERS] How does postgres store the join predicate for a relation in a given query

2017-10-10 Thread Ashutosh Bapat
On Tue, Oct 10, 2017 at 7:29 PM, Gourav Kumar wrote: > Hi all, > > When you fire a query in postgresql, it will first parse the query and > create the data structures for storing various aspects of the query and > executing the query. (Like RangeTblEntry, PlannerInfo, RangeOptInfo etc.). > > I wan

Re: [HACKERS] show precise repos version for dev builds?

2017-10-10 Thread Jeremy Schneider
On Sun, Oct 1, 2017 at 8:10 AM, Tom Lane wrote: > > configure --with-extra-version=whateveryouwant I see that this build option has been around since 9.4; is anyone using it to mark patched production builds? EnterpriseDB or 2ndQuadrant? How about the cloud providers? -Jeremy -- http://about.

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

2017-10-10 Thread Andres Freund
Hi, On 2017-10-03 13:58:37 -0400, Robert Haas wrote: > On Tue, Oct 3, 2017 at 12:23 PM, Andres Freund wrote: > > Makes sense? > > Yes. Here's an updated version of this patchset. Changes: - renamed pq_send$type_pre to pq_write*type - renamed pq_beginmessage_pre/pq_beginmessage_keep to the _reu

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-10-10 Thread Michael Paquier
On Wed, Oct 11, 2017 at 11:31 AM, Wood, Dan wrote: > I found one glitch with our merge of the original dup row fix. With that > corrected AND Alvaro’s Friday fix things are solid. > No dup’s. No index corruption. > > Thanks so much. Nice to hear that! You guys seem to be doing extensive testin

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-10-10 Thread Wood, Dan
I found one glitch with our merge of the original dup row fix. With that corrected AND Alvaro’s Friday fix things are solid. No dup’s. No index corruption. Thanks so much. On 10/10/17, 7:25 PM, "Michael Paquier" wrote: On Tue, Oct 10, 2017 at 11:14 PM, Alvaro Herrera wrote: > I

Re: [HACKERS] pg_regress help output

2017-10-10 Thread Joe Conway
On 10/10/2017 07:06 PM, Tom Lane wrote: > Joe Conway writes: >> I have been annoyed at least twice now by the lack of pg_regress command >> line help output for the "--bindir=" option. In passing I noted >> that there was no output for "--help" or "--version" options either. > >> Any objections t

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-10-10 Thread Michael Paquier
On Tue, Oct 10, 2017 at 11:14 PM, Alvaro Herrera wrote: > I was seeing just the reindex problem. I don't see any more dups. > > But I've tried to reproduce it afresh now, and let it run for a long > time and nothing happened. Maybe I made a mistake last week and > ran an unfixed version. I don'

Re: [HACKERS] pg_regress help output

2017-10-10 Thread Tom Lane
Joe Conway writes: > I have been annoyed at least twice now by the lack of pg_regress command > line help output for the "--bindir=" option. In passing I noted > that there was no output for "--help" or "--version" options either. > Any objections to the attached? +1 for documenting it, but the

Re: [HACKERS] Is it time to kill support for very old servers?

2017-10-10 Thread Michael Paquier
On Wed, Oct 11, 2017 at 10:46 AM, Andres Freund wrote: > I'm not following. The "D" is in the 'dispchar' field, not the value > field, no? The default value is NULL? Oops, yes. I misread the code. Other debug options are not documented, so fine for me to not provide any documentation then. -- Mi

Re: [HACKERS] Is it time to kill support for very old servers?

2017-10-10 Thread Andres Freund
Hi, On 2017-10-11 10:40:11 +0900, Michael Paquier wrote: > >> + if (conn->forced_protocol_version != NULL) > >> + { > >> + conn->pversion = atoi(conn->forced_protocol_version); > >> + } > >> This should check for strlen > 0 as well. > > > > Why? Note that we don't do elsehwere in fe-co

Re: [HACKERS] Is it time to kill support for very old servers?

2017-10-10 Thread Michael Paquier
On Wed, Oct 11, 2017 at 10:14 AM, Andres Freund wrote: > Hi, > > On 2017-10-11 10:09:34 +0900, Michael Paquier wrote: >> On Wed, Oct 11, 2017 at 9:39 AM, Andres Freund wrote: >> > On 2017-09-20 01:32:36 -0700, Andres Freund wrote: >> >> Coverage of the relevant files is a good bit higher afterwar

Re: [HACKERS] Is it time to kill support for very old servers?

2017-10-10 Thread Andres Freund
Hi, On 2017-10-11 10:09:34 +0900, Michael Paquier wrote: > On Wed, Oct 11, 2017 at 9:39 AM, Andres Freund wrote: > > On 2017-09-20 01:32:36 -0700, Andres Freund wrote: > >> Coverage of the relevant files is a good bit higher afterwards. Although > >> our libpq coverage is generally pretty damn aw

Re: [HACKERS] Is it time to kill support for very old servers?

2017-10-10 Thread Michael Paquier
On Wed, Oct 11, 2017 at 9:39 AM, Andres Freund wrote: > On 2017-09-20 01:32:36 -0700, Andres Freund wrote: >> Coverage of the relevant files is a good bit higher afterwards. Although >> our libpq coverage is generally pretty damn awful. > > Any opinions on this? Obviously this needs some cleanup,

[HACKERS] pg_regress help output

2017-10-10 Thread Joe Conway
I have been annoyed at least twice now by the lack of pg_regress command line help output for the "--bindir=" option. In passing I noted that there was no output for "--help" or "--version" options either. Any objections to the attached? It could be argued that it ought to be back-patched too, but

Re: [HACKERS] Is it time to kill support for very old servers?

2017-10-10 Thread Andres Freund
On 2017-09-20 01:32:36 -0700, Andres Freund wrote: > On 2017-09-18 02:53:03 -0700, Andres Freund wrote: > > On 2017-09-13 23:39:21 -0400, Tom Lane wrote: > > > The real problem in this area, to my mind, is that we're not testing that > > > code --- either end of it --- in any systematic way. If it

Re: [HACKERS] On markers of changed data

2017-10-10 Thread Stephen Frost
Alvaro, * Alvaro Herrera (alvhe...@alvh.no-ip.org) wrote: > Greg Stark wrote: > > > The general shape of what I would like to see is some log which lists > > where each checkpoint starts and ends and what blocks are modified > > since the previous checkpoint. Then to generate an incremental backu

Re: [HACKERS] On markers of changed data

2017-10-10 Thread Alvaro Herrera
Greg Stark wrote: > The general shape of what I would like to see is some log which lists > where each checkpoint starts and ends and what blocks are modified > since the previous checkpoint. Then to generate an incremental backup > from any point in time to the current you union all the block lis

[HACKERS] Re: Extended statistics is not working on Vars hidden under a RelabelType

2017-10-10 Thread Tomas Vondra
On 10/10/2017 05:03 AM, David Rowley wrote: > Basically, $subject is causing us not to properly find matching > extended stats in this case. > > The attached patch fixes it. > > The following test cases is an example of the misbehaviour. Note > rows=1 vs rows=98 in the Gather node. > Thanks for

Re: [HACKERS] On markers of changed data

2017-10-10 Thread Greg Stark
On 8 October 2017 at 08:52, Andrey Borodin wrote: > > 1. Any other marker would be better (It can be WAL scan during archiving, > some new LSN-based mechanics* et c.) The general shape of what I would like to see is some log which lists where each checkpoint starts and ends and what blocks are m

Re: [HACKERS] Help required to debug pg_repack breaking logical replication

2017-10-10 Thread Petr Jelinek
On 08/10/17 15:21, Craig Ringer wrote: > On 8 October 2017 at 02:37, Daniele Varrazzo > wrote: >> Hello, >> >> we have been reported, and I have experienced a couple of times, >> pg_repack breaking logical replication. >> >> - https://github.com/reorg/pg_repack/issues/135 >> - https://github.com/

[HACKERS] Re: [BUGS] 10.0: Logical replication doesn't execute BEFORE UPDATE OF trigger

2017-10-10 Thread Petr Jelinek
On 10/10/17 17:22, Aleksander Alekseev wrote: > Hi Petr, > >> let me start by saying that my view is that this is simply a >> documentation bug. Meaning that I didn't document that it does not work, >> but I also never intended it to work. Main reason is that we can't >> support the semantics of "

Re: [HACKERS] How does postgres store the join predicate for a relation in a given query

2017-10-10 Thread Nico Williams
On Tue, Oct 10, 2017 at 07:29:24PM +0530, Gourav Kumar wrote: > When you fire a query in postgresql, it will first parse the query and > create the data structures for storing various aspects of the query and > executing the query. (Like RangeTblEntry, PlannerInfo, RangeOptInfo etc.). > > I want t

[HACKERS] Re: [BUGS] 10.0: Logical replication doesn't execute BEFORE UPDATE OF trigger

2017-10-10 Thread Aleksander Alekseev
Hi Petr, > let me start by saying that my view is that this is simply a > documentation bug. Meaning that I didn't document that it does not work, > but I also never intended it to work. Main reason is that we can't > support the semantics of "UPDATE OF" correctly (see bellow). And I think > it's

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-10-10 Thread Alvaro Herrera
Robert Haas wrote: > On Fri, Oct 6, 2017 at 12:37 PM, Alvaro Herrera > wrote: > > 2. create one index for each existing partition. These would be > >identical to what would happen if you created the index directly on > >each partition, except that there is an additional dependency to the

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-10-10 Thread Alvaro Herrera
Wood, Dan wrote: > I’m unclear on what is being repro’d in 9.6. Are you getting the > duplicate rows problem or just the reindex problem? Are you testing > with asserts enabled(I’m not)? I was seeing just the reindex problem. I don't see any more dups. But I've tried to reproduce it afresh now

[HACKERS] How does postgres store the join predicate for a relation in a given query

2017-10-10 Thread Gourav Kumar
Hi all, When you fire a query in postgresql, it will first parse the query and create the data structures for storing various aspects of the query and executing the query. (Like RangeTblEntry, PlannerInfo, RangeOptInfo etc.). I want to know how does postgresql stores the join predicates of a quer

[HACKERS] Re: [BUGS] 10.0: Logical replication doesn't execute BEFORE UPDATE OF trigger

2017-10-10 Thread Petr Jelinek
On 10/10/17 09:53, Masahiko Sawada wrote: > On Tue, Oct 10, 2017 at 11:29 AM, Masahiko Sawada > wrote: >> On Mon, Oct 9, 2017 at 11:13 PM, Aleksander Alekseev >> wrote: >>> Hi hackers, >>> >>> I've found something that looks like a bug. >>> >>> Steps to reproduce >>> -- >>> >>> T

Re: [JDBC] [HACKERS] Channel binding support for SCRAM-SHA-256

2017-10-10 Thread Michael Paquier
On Tue, Sep 26, 2017 at 11:09 AM, Michael Paquier wrote: > On Mon, Sep 25, 2017 at 11:22 PM, Peter Eisentraut > wrote: >> I think the channel-binding negotiation on the client side is wrong. >> The logic in the patch is >> >> +#ifdef USE_SSL >> + if (state->ssl_in_use) >> + appendPQExpBuf

Re: [HACKERS] [POC] hash partitioning

2017-10-10 Thread amul sul
On Tue, Oct 10, 2017 at 3:42 PM, Ashutosh Bapat wrote: > On Tue, Oct 10, 2017 at 3:32 PM, amul sul wrote: > >>> +hash_part? true : key->parttypbyval[j], >>> +key->parttyplen[j]); >>> parttyplen is the length of partition key

[HACKERS] Re: [BUGS] 10.0: Logical replication doesn't execute BEFORE UPDATE OF trigger

2017-10-10 Thread Aleksander Alekseev
Hi Masahiko, > > I think the cause of this issue is that the apply worker doesn't set > > updatedCols of RangeTblEntry when applying updates. So TriggerEnabled > > always ends up with false. I'll make a patch and submit. > > > > Attached patch store the updated columns bitmap set to RangeTblEntry

[HACKERS] More stats about skipped vacuums

2017-10-10 Thread Kyotaro HORIGUCHI
Hello. Once in a while I am asked about table bloat. In most cases the cause is long lasting transactions and vacuum canceling in some cases. Whatever the case users don't have enough clues to why they have bloated tables. At the top of the annoyances list for users would be that they cannot know

Re: [HACKERS] [POC] hash partitioning

2017-10-10 Thread Ashutosh Bapat
On Tue, Oct 10, 2017 at 3:40 PM, amul sul wrote: >> >> natts represents the number of attributes, but for the hash partition bound >> we >> are not dealing with the attribute so that I have used short-form of >> dimension, >> thoughts? > > Okay, I think the dimension(dim) is also unfit here. An

Re: [HACKERS] [POC] hash partitioning

2017-10-10 Thread Ashutosh Bapat
On Tue, Oct 10, 2017 at 3:32 PM, amul sul wrote: >> +hash_part? true : key->parttypbyval[j], >> +key->parttyplen[j]); >> parttyplen is the length of partition key attribute, whereas what you want >> here >> is the length of

Re: [HACKERS] [POC] hash partitioning

2017-10-10 Thread amul sul
On Tue, Oct 10, 2017 at 3:32 PM, amul sul wrote: > On Mon, Oct 9, 2017 at 5:51 PM, Ashutosh Bapat > wrote: >> On Mon, Oct 9, 2017 at 4:44 PM, amul sul wrote: >> > > Thanks Ashutosh for your review, please find my comment inline. > >> >>> 0002 few changes in partition-wise join code to support >>

Re: [HACKERS] [POC] hash partitioning

2017-10-10 Thread amul sul
On Mon, Oct 9, 2017 at 5:51 PM, Ashutosh Bapat wrote: > On Mon, Oct 9, 2017 at 4:44 PM, amul sul wrote: > Thanks Ashutosh for your review, please find my comment inline. > >> 0002 few changes in partition-wise join code to support >> hash-partitioned table as well & regression tests. > > +s

Re: [HACKERS] Partition-wise aggregation/grouping

2017-10-10 Thread Ashutosh Bapat
On Tue, Oct 10, 2017 at 1:31 PM, David Rowley wrote: > > I don't think there's any need to invent any new GUC. You could just > divide cpu_tuple_cost by something. > > I did a quick benchmark on my laptop to see how much Append really > costs, and with the standard costs the actual cost seems to b

Re: [HACKERS] GUC for cleanup indexes threshold.

2017-10-10 Thread Pavel Golub
Hello, Darafei. You wrote: DP> The following review has been posted through the commitfest application: DP> make installcheck-world: tested, passed DP> Implements feature: tested, passed DP> Spec compliant: tested, passed DP> Documentation:tested, passed DP> We're us

Re: [HACKERS] Columnar storage support

2017-10-10 Thread Konstantin Knizhnik
Unfortunately C-Store doesn't allow to take all advantages of columnar store: you still not be able to perform vector operation.s C-Store allows to reduce size of data read from the disk because of 1. fetching only columns which are used in the query, 2. data compression. It will lead to some be

Re: [HACKERS] Partition-wise aggregation/grouping

2017-10-10 Thread David Rowley
On 10 October 2017 at 17:57, Ashutosh Bapat wrote: > Append node just returns the result of ExecProcNode(). Charging > cpu_tuple_cost may make it too expensive. In other places where we > charge cpu_tuple_cost there's some processing done to the tuple like > ExecStoreTuple() in SeqNext(). May be w

[HACKERS] Re: [BUGS] 10.0: Logical replication doesn't execute BEFORE UPDATE OF trigger

2017-10-10 Thread Masahiko Sawada
On Tue, Oct 10, 2017 at 11:29 AM, Masahiko Sawada wrote: > On Mon, Oct 9, 2017 at 11:13 PM, Aleksander Alekseev > wrote: >> Hi hackers, >> >> I've found something that looks like a bug. >> >> Steps to reproduce >> -- >> >> There are 3 instances of PostgreSQL 10.0 - inst1, inst2 an

[HACKERS] Omission in GRANT documentation

2017-10-10 Thread Laurenz Albe
grant.sgml says that the default privileges granted to PUBLIC are as follows: CONNECT and CREATE TEMP TABLE for databases; EXECUTE privilege for functions; and USAGE privilege for languages. But types also have the USAGE privilege for PUBLIC by default: test=> CREATE TYPE bug_status A