Re: Postgres, fsync, and OSs (specifically linux)

2018-06-13 Thread Thomas Munro
On Wed, May 23, 2018 at 8:02 AM, Andres Freund wrote: > [patches] Hi Andres, Obviously there is more work to be done here but the basic idea in your clone-fd-checkpointer branch as of today seems OK to me. I think Craig and I both had similar ideas (sending file descriptors that have an old

Re: Partitioning with temp tables is broken

2018-06-13 Thread Amit Langote
On 2018/06/13 21:06, David Rowley wrote: > There's also something pretty weird around the removal of the temp > relation from the partition bound. I've had cases where the session > that attached the temp table is long gone, but \d+ shows the table is > still there and I can't attach another

Re: Two round for Client Authentication

2018-06-13 Thread Marko Tiikkaja
On Thu, Jun 14, 2018 at 7:12 AM, Yinjie Lin wrote: > Currently I am reading and testing code about Client Authentication, but I > find that there are two progresses forked if I login using psql, while only > one progress is forked if using pgAdmin. > If psql finds the server asks for a

Re: Partitioning with temp tables is broken

2018-06-13 Thread Amit Langote
On 2018/06/14 11:09, Michael Paquier wrote: > On Wed, Jun 13, 2018 at 10:25:23PM +0530, amul sul wrote: >> On Wed, Jun 13, 2018, 8:34 PM Tom Lane wrote: >>> Even if you want to argue that there's a use case for these situations, >>> it seems far too late in the release cycle to be trying to fix

Re: WAL prefetch

2018-06-13 Thread Amit Kapila
On Wed, Jun 13, 2018 at 6:39 PM, Konstantin Knizhnik wrote: > There was very interesting presentation at pgconf about pg_prefaulter: > > http://www.pgcon.org/2018/schedule/events/1204.en.html > > But it is implemented in GO and using pg_waldump. > I tried to do the same but using built-on

Re: [HACKERS] Pluggable storage

2018-06-13 Thread Amit Kapila
On Thu, Jun 14, 2018 at 1:50 AM, Haribabu Kommi wrote: > > On Fri, Apr 20, 2018 at 4:44 PM Haribabu Kommi > wrote: > > VACUUM: > Not much changes are done in this apart moving the Vacuum visibility > functions as part of the > storage. But idea for the Vacuum was with each access method can

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-13 Thread Masahiko Sawada
On Wed, Jun 13, 2018 at 10:03 PM, Tomas Vondra wrote: > On 06/11/2018 11:22 AM, Masahiko Sawada wrote: >> >> On Fri, May 25, 2018 at 8:41 PM, Moon, Insung >> wrote: >>> >>> Hello Hackers, >>> >>> This propose a way to develop "Table-level" Transparent Data Encryption >>> (TDE) and Key Management

Re: Index maintenance function for BRIN doesn't check RecoveryInProgress()

2018-06-13 Thread Michael Paquier
On Thu, Jun 14, 2018 at 02:06:57AM +0900, Masahiko Sawada wrote: > On Thu, Jun 14, 2018 at 12:04 AM, Simon Riggs wrote: >> On 13 June 2018 at 15:51, Alvaro Herrera wrote: >>> I guess you could go either way ... we're just changing one unhelpful >>> error with a better one: there is no change in

Re: Partitioning with temp tables is broken

2018-06-13 Thread Michael Paquier
On Wed, Jun 13, 2018 at 10:25:23PM +0530, amul sul wrote: > On Wed, Jun 13, 2018, 8:34 PM Tom Lane wrote: >> Even if you want to argue that there's a use case for these situations, >> it seems far too late in the release cycle to be trying to fix all these >> issues. I think we need to forbid

Re: pg_config.h.win32 missing a set of flags from pg_config.h.in added in v11 development

2018-06-13 Thread Michael Paquier
On Wed, Jun 13, 2018 at 08:55:46PM -0400, Andrew Dunstan wrote: > I installed 1.1.0h and got errors you can see on the buildfarm. I've now > installed 1.0.2o and everything is good. Good thing you tested that. I have just used the LTS 1.0.2 for my tests. And there are a couple of problems here.

RE: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-13 Thread Tsunakawa, Takayuki
> From: Tomas Vondra [mailto:tomas.von...@2ndquadrant.com] > On 05/25/2018 01:41 PM, Moon, Insung wrote: > > BTW, I want to support CBC mode encryption[3]. However, I'm not sure > > how to use the IV in CBC mode for this proposal. I'd like to hear > > opinions by security engineer. > > > > I'm

Re: pg_config.h.win32 missing a set of flags from pg_config.h.in added in v11 development

2018-06-13 Thread Andrew Dunstan
On 06/12/2018 08:07 PM, Michael Paquier wrote: On Tue, Jun 12, 2018 at 04:51:59PM -0400, Andrew Dunstan wrote: On 06/12/2018 10:51 AM, Andrew Dunstan wrote: meanwhile, Bowerbird (MSVC) is on 1.0.1d, lorikeet (Cygwin64) is on 1.0.2d and jacana (Mingw) doesn't build with openssl. I will look

RE: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-13 Thread Tsunakawa, Takayuki
From: Tomas Vondra [mailto:tomas.von...@2ndquadrant.com] > Let me share some of the issues mentioned as possibly addressed by TDE > (I'm not entirely sure TDE actually solves them, I'm just saying those > were mentioned in previous discussions): FYI, our product provides TDE like Oracle and SQL

Re: commitfest 2018-07

2018-06-13 Thread Michael Paquier
On Mon, Jun 11, 2018 at 10:34:55AM -0700, Andres Freund wrote: > On 2018-06-11 11:50:55 +0900, Michael Paquier wrote: >> So, we have confirmation from the RTM that there will be a 2018-07. And >> there is visibly consensus that renaming 2018-09 to 2018-07 makes the >> most sense. Any objections

Re: Slow planning time for simple query

2018-06-13 Thread Maksim Milyutin
13.06.2018 12:40, Maksim Milyutin wrote: On 09.06.2018 22:49, Tom Lane wrote: Maksim Milyutin writes: On hot standby I faced with the similar problem. ... is planned 4.940 ms on master and *254.741* ms on standby. (I wonder though why, if you executed the same query on the master, its

Re: Logging transaction IDs for DDL.

2018-06-13 Thread Andres Freund
Hi, On 2018-06-14 00:34:54 +0200, Vik Fearing wrote: > I just noticed a problem with log_statement = 'ddl' and log_line_prefix > containing '%x'. If the statement is the first in the transaction, it > will be logged before it is executed, and more importantly, before a > transaction ID is

Logging transaction IDs for DDL.

2018-06-13 Thread Vik Fearing
I just noticed a problem with log_statement = 'ddl' and log_line_prefix containing '%x'. If the statement is the first in the transaction, it will be logged before it is executed, and more importantly, before a transaction ID is assigned. That means that %x will be 0. If the administrator has

Re: Portability concerns over pq_sendbyte?

2018-06-13 Thread Michael Paquier
On Wed, Jun 13, 2018 at 11:53:21AM -0700, Andres Freund wrote: > On 2018-06-13 14:10:37 +0900, Michael Paquier wrote: >> On Mon, Jun 11, 2018 at 02:25:44PM +0900, Michael Paquier wrote: >>> On top of that it seems to me that we'd want to rename any new >>> routines to include "uint" in their name

Re: Locking B-tree leafs immediately in exclusive mode

2018-06-13 Thread Peter Geoghegan
On Mon, Jun 11, 2018 at 9:30 AM, Alexander Korotkov wrote: > On Mon, Jun 11, 2018 at 1:06 PM Simon Riggs wrote: >> It's a good idea. How does it perform with many duplicate entries? I agree that this is a good idea. It independently occurred to me to do this. The L algorithm doesn't take a

Re: Portability concerns over pq_sendbyte?

2018-06-13 Thread Andres Freund
Hi, On 2018-06-13 22:02:13 +0100, Andrew Gierth wrote: > > "Andres" == Andres Freund writes: > > >> unsigned char x = 128; > >> pq_sendbyte(, x); > >> > >> which I believe is not well-defined since pq_sendbyte takes an int8, > >> and conversions of unrepresentable values to _signed_

Re: Portability concerns over pq_sendbyte?

2018-06-13 Thread Andrew Gierth
> "Andres" == Andres Freund writes: >> unsigned char x = 128; >> pq_sendbyte(, x); >> >> which I believe is not well-defined since pq_sendbyte takes an int8, >> and conversions of unrepresentable values to _signed_ integer types >> are (iirc) implementation-dependent. Andres> It's

Re: [HACKERS] WIP Patch: Pgbench Serialization and deadlock errors

2018-06-13 Thread Alvaro Herrera
On 2018-Jun-13, Fabien COELHO wrote: > > > > With that, is there a need for elog()? In the backend we have > > > > it because $HISTORY but there's no need for that here -- I > > > > propose to lose elog() and use only ereport everywhere. > > See commit 8a07ebb3c172 which turns some ereport into

Re: Spilling hashed SetOps and aggregates to disk

2018-06-13 Thread Jeff Davis
On Wed, 2018-06-13 at 11:50 -0300, Claudio Freire wrote: > In essence, the technique I've been using always uses a fixed-size > hash table to do partial grouping. The table is never grown, when > collisions do happen, the existing entry in the hash table is flushed > to disk and the aggregate

Re: [HACKERS] WIP Patch: Pgbench Serialization and deadlock errors

2018-06-13 Thread Fabien COELHO
Hello Marina, I suppose that this is related; because of my patch there may be a lot of such code (see v7 in [1]): - fprintf(stderr, - "malformed variable \"%s\" value: \"%s\"\n", - var->name, var->svalue); +

Re: Spilling hashed SetOps and aggregates to disk

2018-06-13 Thread Jeff Davis
On Wed, 2018-06-13 at 10:08 -0400, Robert Haas wrote: > I wasn't using the term "average" in a mathematical sense.  I suppose > I really meant "typical".  I agree that thinking about cases where > the > group size is nonuniform is a good idea, but I don't think I agree > that all uniform

Re: [HACKERS] Optional message to user when terminating/cancelling backend

2018-06-13 Thread Andres Freund
Hi, Onder, I CCed you because it seems worthwhile to ensure that the relevant Citus code could use this instead of the gross hack you and I committed... On 2018-06-13 20:54:03 +0200, Daniel Gustafsson wrote: > > On 9 Apr 2018, at 23:55, Andres Freund wrote: > > On 2017-06-20 13:01:35 -0700,

Re: [HACKERS] Optional message to user when terminating/cancelling backend

2018-06-13 Thread Daniel Gustafsson
> On 9 Apr 2018, at 23:55, Andres Freund wrote: > On 2017-06-20 13:01:35 -0700, Andres Freund wrote: >> For extensions it'd also be useful if it'd be possible to overwrite the >> error code. E.g. for citus there's a distributed deadlock detector, >> running out of process because there's no way

Re: Portability concerns over pq_sendbyte?

2018-06-13 Thread Andres Freund
On 2018-06-13 14:10:37 +0900, Michael Paquier wrote: > On Mon, Jun 11, 2018 at 02:25:44PM +0900, Michael Paquier wrote: > > On Wed, Jun 06, 2018 at 12:27:58PM -0400, Alvaro Herrera wrote: > >> Do you have an answer to this question? Does anybody else? > >> > >> (My guts tell me it'd be better to

Re: Portability concerns over pq_sendbyte?

2018-06-13 Thread Andres Freund
Hi, On 2018-05-24 18:13:23 +0100, Andrew Gierth wrote: > In PG11, pq_sendbyte got changed from taking an int parameter to taking > an int8. > > While that seems to work in general, it does mean that there are now > several places in the code that do the equivalent of: > > unsigned char x =

Re: Partitioning with temp tables is broken

2018-06-13 Thread amul sul
On Wed, Jun 13, 2018, 8:34 PM Tom Lane wrote: > Ashutosh Bapat writes: > > [ lots o' problems with $subject ] > > > But a larger question is what use such temporary partitions are? > > Should we just prohibit adding temporary partitions to a permanant > > partitioned table? We should allow

Re: POC: GROUP BY optimization

2018-06-13 Thread Teodor Sigaev
I see v9 is now calling estimate_num_groups(), so it already benefits from extended statistics. Nice! I think the algorithm is more or less the greedy option I proposed in one of the earlier messages - I don't know if it's sufficient or not, but the only other idea I have is essentially an

Re: why partition pruning doesn't work?

2018-06-13 Thread Tom Lane
David Rowley writes: > On 13 June 2018 at 16:15, Tom Lane wrote: >> It seems not to be that bad: we just need a shutdown call for the >> PartitionPruneState, and then we can remember the open relation there. >> The attached is based on David's patch from yesterday. > I've looked over this and

Re: Duplicate Item Pointers in Gin index

2018-06-13 Thread Masahiko Sawada
On Wed, Jun 13, 2018 at 10:22 PM, Alexander Korotkov wrote: > On Wed, Jun 13, 2018 at 11:40 AM Masahiko Sawada > wrote: >> >> On Wed, Jun 13, 2018 at 3:32 PM, Peter Geoghegan wrote: >> > On Tue, Jun 12, 2018 at 11:01 PM, Masahiko Sawada >> > wrote: >> >> FWIW, I've looked at this again. I

Re: Index maintenance function for BRIN doesn't check RecoveryInProgress()

2018-06-13 Thread Simon Riggs
On 13 June 2018 at 15:51, Alvaro Herrera wrote: > On 2018-Jun-13, Alexander Korotkov wrote: > >> On Wed, Jun 13, 2018 at 12:48 PM Kuntal Ghosh >> wrote: >> > On Wed, Jun 13, 2018 at 2:28 PM, Masahiko Sawada >> > wrote: >> > > Hi, >> > > >> > > Three functions: brin_summarize_new_values,

Re: Partitioning with temp tables is broken

2018-06-13 Thread Tom Lane
Ashutosh Bapat writes: > [ lots o' problems with $subject ] > But a larger question is what use such temporary partitions are? > Should we just prohibit adding temporary partitions to a permanant > partitioned table? We should allow adding temporary partitions to a > temporary partitioned table

Re: Index maintenance function for BRIN doesn't check RecoveryInProgress()

2018-06-13 Thread Alvaro Herrera
On 2018-Jun-13, Alexander Korotkov wrote: > On Wed, Jun 13, 2018 at 12:48 PM Kuntal Ghosh > wrote: > > On Wed, Jun 13, 2018 at 2:28 PM, Masahiko Sawada > > wrote: > > > Hi, > > > > > > Three functions: brin_summarize_new_values, brin_summarize_range and > > > brin_desummarize_range can be

Re: Spilling hashed SetOps and aggregates to disk

2018-06-13 Thread Claudio Freire
On Tue, Jun 5, 2018 at 5:05 AM Tomas Vondra wrote: > > On 06/05/2018 07:46 AM, Jeff Davis wrote: > > On Tue, 2018-06-05 at 07:04 +0200, Tomas Vondra wrote: > >> I expect the eviction strategy to be the primary design challenge of > >> this patch. The other bits will be mostly determined by this

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-06-13 Thread Simon Riggs
On 11 June 2018 at 17:56, Andres Freund wrote: > I don't think this is a good idea. We shouldn't continue down the path > of having running xacts not actually running xacts, but rather go back > to including everything. The case presented in the thread didn't > actually do what it claimed

Re: Spilling hashed SetOps and aggregates to disk

2018-06-13 Thread Robert Haas
On Mon, Jun 11, 2018 at 1:50 PM, Jeff Davis wrote: > I think average group size is the wrong way to look at it, because > averages are only useful for a normal distribution. The two most > interesting cases are: > > 1. Fairly uniform group size (e.g. normal dist) > 2. Skew values, power law

Re: Partitioning with temp tables is broken

2018-06-13 Thread Ashutosh Bapat
On Wed, Jun 13, 2018 at 5:36 PM, David Rowley wrote: > Hi, > > It seems temp tables with partitioned tables is not working correctly. > 9140cf8269b0c has not considered that in build_simple_rel() an > AppendRelInfo could be missing due to it having been skipped in > expand_partitioned_rtentry() >

Re: why partition pruning doesn't work?

2018-06-13 Thread Robert Haas
On Wed, Jun 13, 2018 at 12:15 AM, Tom Lane wrote: >> As for whether to change it at this point in the release cycle, I >> guess that, to me, depends on how invasive the fix is. > > It seems not to be that bad: we just need a shutdown call for the > PartitionPruneState, and then we can remember

Re: Duplicate Item Pointers in Gin index

2018-06-13 Thread Alexander Korotkov
On Wed, Jun 13, 2018 at 11:40 AM Masahiko Sawada wrote: > > On Wed, Jun 13, 2018 at 3:32 PM, Peter Geoghegan wrote: > > On Tue, Jun 12, 2018 at 11:01 PM, Masahiko Sawada > > wrote: > >> FWIW, I've looked at this again. I think that the situation Siva > >> reported in the first mail can happen

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-13 Thread Joe Conway
On 06/11/2018 05:22 AM, Masahiko Sawada wrote: > As per discussion at PGCon unconference, I think that firstly we need > to discuss what threats we want to defend database data against. Exactly. While certainly there is demand for encryption for the sake of "checking a box", different designs

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-13 Thread Tomas Vondra
Hi, On 05/25/2018 01:41 PM, Moon, Insung wrote: Hello Hackers, ... BTW, I want to support CBC mode encryption[3]. However, I'm not sure how to use the IV in CBC mode for this proposal. I'd like to hear opinions by security engineer. I'm not a cryptographer either, but this is exactly

WAL prefetch

2018-06-13 Thread Konstantin Knizhnik
There was very interesting presentation at pgconf about pg_prefaulter: http://www.pgcon.org/2018/schedule/events/1204.en.html But it is implemented in GO and using pg_waldump. I tried to do the same but using built-on Postgres WAL traverse functions. I have implemented it as extension for

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-13 Thread Tomas Vondra
On 06/11/2018 11:22 AM, Masahiko Sawada wrote: On Fri, May 25, 2018 at 8:41 PM, Moon, Insung wrote: Hello Hackers, This propose a way to develop "Table-level" Transparent Data Encryption (TDE) and Key Management Service (KMS) support in PostgreSQL. ... As per discussion at PGCon

Partitioning with temp tables is broken

2018-06-13 Thread David Rowley
Hi, It seems temp tables with partitioned tables is not working correctly. 9140cf8269b0c has not considered that in build_simple_rel() an AppendRelInfo could be missing due to it having been skipped in expand_partitioned_rtentry() Assert(cnt_parts == nparts); fails in build_simple_rel, and

Re: Index maintenance function for BRIN doesn't check RecoveryInProgress()

2018-06-13 Thread Alexander Korotkov
On Wed, Jun 13, 2018 at 12:48 PM Kuntal Ghosh wrote: > On Wed, Jun 13, 2018 at 2:28 PM, Masahiko Sawada > wrote: > > Hi, > > > > Three functions: brin_summarize_new_values, brin_summarize_range and > > brin_desummarize_range can be called during recovery as follows. > > > > =# select

Re: [HACKERS] WIP Patch: Pgbench Serialization and deadlock errors

2018-06-13 Thread Marina Polyakova
On 10-06-2018 10:38, Fabien COELHO wrote: Hello Marina, Hello! v9-0003-Pgbench-errors-use-the-ereport-macro-to-report-de.patch - a patch for the ereport() macro (this is used to report client failures that do not cause an aborts and this depends on the level of debugging). ISTM that

Re: Index maintenance function for BRIN doesn't check RecoveryInProgress()

2018-06-13 Thread Kuntal Ghosh
On Wed, Jun 13, 2018 at 2:28 PM, Masahiko Sawada wrote: > Hi, > > Three functions: brin_summarize_new_values, brin_summarize_range and > brin_desummarize_range can be called during recovery as follows. > > =# select brin_summarize_new_values('a_idx'); > ERROR: cannot acquire lock mode

Re: Slow planning time for simple query

2018-06-13 Thread Maksim Milyutin
On 09.06.2018 22:49, Tom Lane wrote: Maksim Milyutin writes: On hot standby I faced with the similar problem. ... is planned 4.940 ms on master and *254.741* ms on standby. (I wonder though why, if you executed the same query on the master, its setting of the index-entry-is-dead bits didn't

Index maintenance function for BRIN doesn't check RecoveryInProgress()

2018-06-13 Thread Masahiko Sawada
Hi, Three functions: brin_summarize_new_values, brin_summarize_range and brin_desummarize_range can be called during recovery as follows. =# select brin_summarize_new_values('a_idx'); ERROR: cannot acquire lock mode ShareUpdateExclusiveLock on database objects while recovery is in progress

Re: Duplicate Item Pointers in Gin index

2018-06-13 Thread Masahiko Sawada
On Wed, Jun 13, 2018 at 3:32 PM, Peter Geoghegan wrote: > On Tue, Jun 12, 2018 at 11:01 PM, Masahiko Sawada > wrote: >> FWIW, I've looked at this again. I think that the situation Siva >> reported in the first mail can happen before we get commit 3b2787e. >> That is, gin indexes had had a data

Re: Server crashed with dense_rank on partition table.

2018-06-13 Thread Amit Langote
Hi. On 2018/06/13 14:55, Michael Paquier wrote: > On Wed, Jun 13, 2018 at 11:08:38AM +0530, Rajkumar Raghuwanshi wrote: >> postgres=# SELECT dense_rank(b) WITHIN GROUP (ORDER BY a) FROM pagg_tab >> GROUP BY b ORDER BY 1; >> server closed the connection unexpectedly >> This probably means the

Re: Server crashed with dense_rank on partition table.

2018-06-13 Thread David Rowley
On 13 June 2018 at 17:55, Michael Paquier wrote: > On Wed, Jun 13, 2018 at 11:08:38AM +0530, Rajkumar Raghuwanshi wrote: >> postgres=# SELECT dense_rank(b) WITHIN GROUP (ORDER BY a) FROM pagg_tab >> GROUP BY b ORDER BY 1; >> server closed the connection unexpectedly >> This probably means the

Re: Duplicate Item Pointers in Gin index

2018-06-13 Thread Peter Geoghegan
On Tue, Jun 12, 2018 at 11:01 PM, Masahiko Sawada wrote: > FWIW, I've looked at this again. I think that the situation Siva > reported in the first mail can happen before we get commit 3b2787e. > That is, gin indexes had had a data corruption bug. I've reproduced > the situation with PostgreSQL

Re: Duplicate Item Pointers in Gin index

2018-06-13 Thread Masahiko Sawada
On Thu, Feb 22, 2018 at 10:26 AM, Masahiko Sawada wrote: > On Thu, Feb 22, 2018 at 8:28 AM, Peter Geoghegan wrote: >> On Wed, Feb 21, 2018 at 3:02 PM, R, Siva wrote: >>> Did you mean pin on the metapage buffer during ginInsertCleanup and not lock >>> during addition of tuples to the