Re: Tid scan improvements

2019-07-31 Thread Thomas Munro
On Thu, Aug 1, 2019 at 5:51 PM Edmund Horner wrote: > On Thu, 1 Aug 2019 at 17:47, Thomas Munro wrote: > > On Thu, Aug 1, 2019 at 5:34 PM Edmund Horner wrote: > > > Should we move to CF? We have been in the CF cycle for almost a year > > > now... > > > > Hi Edmund, > > > > No worries, that's

Re: Tid scan improvements

2019-07-31 Thread Edmund Horner
On Thu, 1 Aug 2019 at 17:47, Thomas Munro wrote: > On Thu, Aug 1, 2019 at 5:34 PM Edmund Horner wrote: > > Should we move to CF? We have been in the CF cycle for almost a year now... > > Hi Edmund, > > No worries, that's how it goes sometimes. Please move it to the next > CF if you think

Re: Tid scan improvements

2019-07-31 Thread Thomas Munro
On Thu, Aug 1, 2019 at 5:34 PM Edmund Horner wrote: > Sadly it is the end of the CF and I have not had much time to work on > this. I'll probably get some time in the next month to look at the > heapam changes. > > Should we move to CF? We have been in the CF cycle for almost a year now... Hi

Re: concerns around pg_lsn

2019-07-31 Thread Jeevan Ladhe
Hi Michael, > On further review of the first patch, I think that it could be a good > idea to apply the same safeguards within float8in_internal_opt_error. > Jeevan, what do you think? > Sure, agree, it makes sense to address float8in_internal_opt_error(), there might be more occurrences of

Re: Rearranging ALTER TABLE to avoid multi-operations bugs

2019-07-31 Thread Thomas Munro
On Tue, Jul 2, 2019 at 2:00 PM Tom Lane wrote: > movead li writes: > > I applied the 'alter-table-with-recursive-process-utility-calls-wip.patch' > > on the master(e788e849addd56007a0e75f3b5514f294a0f3bca). And > > when I test the cases, I find it works well on 'alter table t1 add column > > f2

Re: Tid scan improvements

2019-07-31 Thread Edmund Horner
On Mon, 22 Jul 2019 at 19:44, Edmund Horner wrote: > On Mon, 22 Jul 2019 at 19:25, David Rowley > > On Sat, 20 Jul 2019 at 06:21, Andres Freund wrote: > > > Yea, I was thinking of something like 2. We already have a few extra > > > types of scan nodes (bitmap heap and sample scan), it'd not be

Re: Multivariate MCV list vs. statistics target

2019-07-31 Thread Thomas Munro
On Thu, Aug 1, 2019 at 12:16 PM Jamison, Kirk wrote: > > On Saturday, July 27, 2019 7:06 AM(GMT+9), Tomas Vondra wrote: > > > On Fri, Jul 26, 2019 at 07:03:41AM +, Jamison, Kirk wrote: > > > >Overall, the patch is almost already in good shape for commit. > > > >I'll wait for the next update.

Re: Commitfest 2019-07, the first of five* for PostgreSQL 13

2019-07-31 Thread Robert Eckhardt
On Thu, Aug 1, 2019 at 12:10 AM Thomas Munro wrote: > Hi all, > > CF1 officially ends in about 8 hours, when August arrives on the > volcanic islands of Howard and Baker, according to CURRENT_TIMESTAMP > AT TIME ZONE '+12'. I'll probably mark it closed at least 8 hours > later than that

Re: refactoring - share str2*int64 functions

2019-07-31 Thread Michael Paquier
On Mon, Jul 29, 2019 at 07:04:09AM +0200, Fabien COELHO wrote: > Bonjour Michaël, > Yep, I will try for this week. Please note that for now I have marked the patch as returned with feedback as the CF is ending. -- Michael signature.asc Description: PGP signature

More refactoring for BuildIndexInfo

2019-07-31 Thread Michael Paquier
Hi all, 7cce1593 has introduced a new routine makeIndexInfo to create IndexInfo nodes. It happens that we can do a bit more refactoring as per the attached, as BuildIndexInfo can make use of this routine, removing some duplication on the way (filling in IndexInfo was still duplicated in a couple

Re: Commitfest 2019-07, the first of five* for PostgreSQL 13

2019-07-31 Thread Thomas Munro
Hi all, CF1 officially ends in about 8 hours, when August arrives on the volcanic islands of Howard and Baker, according to CURRENT_TIMESTAMP AT TIME ZONE '+12'. I'll probably mark it closed at least 8 hours later than that because I'll be asleep. Anything that is waiting on author and hasn't

Refactoring code stripping trailing \n and \r from strings

2019-07-31 Thread Michael Paquier
Hi Tom, b654714 has reworked the way we handle removal of CLRF for several code paths, and has repeated the same code patterns to do that in 8 different places. Could it make sense to refactor things as per the attached with a new routine in common/string.c? Thanks, -- Michael diff --git

Re: Global temporary tables

2019-07-31 Thread Craig Ringer
On Wed, 31 Jul 2019 at 23:05, Konstantin Knizhnik wrote: > Current Postgres implementation of temporary table causes number of > problems: > > 1. Catalog bloating: if client creates and deletes too many temporary > tables, then autovacuum get stuck on catalog. > This also upsets logical

Re: concerns around pg_lsn

2019-07-31 Thread Craig Ringer
On the topic of pg_lsn, I recently noticed that there's no operator(+)(pg_lsn,bigint) nor is there an operator(-)(pg_lsn,bigint) so you can't compute offsets easily. We don't have a cast between pg_lsn and bigint because we don't expose an unsigned bigint type in SQL, so you can't work around it

Re: [PATCH] minor bugfix for pg_basebackup (9.6 ~ )

2019-07-31 Thread Ian Barwick
On 7/27/19 6:52 AM, Alvaro Herrera wrote: On 2019-Jul-25, Michael Paquier wrote: On Wed, Jul 24, 2019 at 11:23:30AM -0400, Alvaro Herrera wrote: Heh, yesterday I revised the original patch as attached and was about to push when the bell rang. I like this one because it keeps the comment to

Re: partition routing layering in nodeModifyTable.c

2019-07-31 Thread Amit Langote
Fujita-san, Thanks for the reply and sorry I didn't wait a bit more before posting the patch. On Wed, Jul 31, 2019 at 9:04 PM Etsuro Fujita wrote: > On Wed, Jul 31, 2019 at 5:05 PM Amit Langote wrote: > > On Tue, Jul 30, 2019 at 4:20 PM Amit Langote > > wrote: > > > On Sat, Jul 20, 2019 at

Re: Possible race condition in pg_mkdir_p()?

2019-07-31 Thread Ning Yu
On Wed, Jul 31, 2019 at 2:21 PM Michael Paquier wrote: > That's my impression as well. Please note that this does not involve > an actual bug in Postgres and that this is rather invasive, so this > does not really qualify for a back-patch. No objections to simplify > the backend on HEAD though.

Re: POC: converting Lists into arrays

2019-07-31 Thread Andres Freund
Hi, On 2019-07-31 19:40:09 -0400, Tom Lane wrote: > Andres Freund writes: > > Unfortunately foreach(ListCell *lc, ...) doesn't work with the current > > definition. Which I think isn't great, because the large scopes for loop > > iteration variables imo makes the code harder to reason about. > >

Re: concerns around pg_lsn

2019-07-31 Thread Michael Paquier
On Wed, Jul 31, 2019 at 09:51:30AM +0900, Michael Paquier wrote: > I am adding Peter Eisentraut in CC as 21f428e is his commit. I think > that the first patch is a good idea, so I would be fine to apply it, > but let's see the original committer's opinion first. On further review of the first

Re: POC: converting Lists into arrays

2019-07-31 Thread Tom Lane
I wrote: > BTW, I think we could make equivalent macros in the old regime, > which would be a good thing because then it would be possible to > back-patch code using this notation. Oh, wait-a-second. I was envisioning that for (ListCell *anonymous__lc = ...) would work for that, but of

RE: Multivariate MCV list vs. statistics target

2019-07-31 Thread Jamison, Kirk
Hi, > From: Jamison, Kirk [mailto:k.jami...@jp.fujitsu.com] > Sent: Monday, July 29, 2019 10:53 AM > To: 'Tomas Vondra' > Cc: Dean Rasheed ; PostgreSQL Hackers > > Subject: RE: Multivariate MCV list vs. statistics target > > On Saturday, July 27, 2019 7:06 AM(GMT+9), Tomas Vondra wrote: > >

Re: POC: converting Lists into arrays

2019-07-31 Thread Tom Lane
I wrote: > One idea is that we could do something like > foreach_variant(identifier, list_value) > { >type *v = (type *) lfirst_variant(identifier); >... > } > where the "identifier" isn't actually a variable name but just something > we use to construct the

Re: POC: converting Lists into arrays

2019-07-31 Thread Tom Lane
Andres Freund writes: > Unfortunately foreach(ListCell *lc, ...) doesn't work with the current > definition. Which I think isn't great, because the large scopes for loop > iteration variables imo makes the code harder to reason about. Yeah, I tried to make that possible when I redid those

Re: Unused header file inclusion

2019-07-31 Thread Tom Lane
Andres Freund writes: > On 2019-07-31 16:55:31 -0400, Tom Lane wrote: >> Yeah. I seem to recall a proposal that nodes.h should contain >> >> typedef struct Foo Foo; >> >> for every node type Foo, and then the other headers would just >> fill in the structs, and we could get rid of a lot of

Re: POC: converting Lists into arrays

2019-07-31 Thread Petr Jelinek
On 01/08/2019 01:04, Andres Freund wrote: Hi, On 2019-07-31 16:00:47 -0700, Andres Freund wrote: On 2019-07-31 15:57:56 -0700, Andres Freund wrote: I also wonder if a foreach version that includes the typical (Type *) var = (Type *) lfirst(lc); or (Type *) var = castNode(Type, lfirst(lc)); or

Re: POC: converting Lists into arrays

2019-07-31 Thread Andres Freund
Hi, On 2019-07-31 16:00:47 -0700, Andres Freund wrote: > On 2019-07-31 15:57:56 -0700, Andres Freund wrote: > > I also wonder if a foreach version that includes the typical > > (Type *) var = (Type *) lfirst(lc); > > or > > (Type *) var = castNode(Type, lfirst(lc)); > > or > > OpExpr

Re: POC: converting Lists into arrays

2019-07-31 Thread Andres Freund
Hi, On 2019-07-31 15:57:56 -0700, Andres Freund wrote: > I also wonder if a foreach version that includes the typical > (Type *) var = (Type *) lfirst(lc); > or > (Type *) var = castNode(Type, lfirst(lc)); > or > OpExpr *hclause = lfirst_node(OpExpr, lc); > > would make it nicer to use

Re: POC: converting Lists into arrays

2019-07-31 Thread Andres Freund
Hi, I was just looking at the diff for a fix, which adds a "ListCell *lc;" to function scope, even though it's only needed in a pretty narrow scope. Unfortunately foreach(ListCell *lc, ...) doesn't work with the current definition. Which I think isn't great, because the large scopes for loop

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2019-07-31 Thread Andres Freund
Hi, On 2019-07-31 23:51:40 +0200, Julien Rouhaud wrote: > On Wed, Jul 31, 2019 at 10:55 AM Evgeny Efimkin > wrote: > > What reason to use pg_atomic_uint64? > > The queryid is read and written without holding any lock on the PGPROC > entry, so the pg_atomic_uint64 will guarantee that we get a

Re: Unused header file inclusion

2019-07-31 Thread Andres Freund
Hi, On 2019-07-31 16:55:31 -0400, Tom Lane wrote: > Alvaro Herrera writes: > > On 2019-Jul-31, Andres Freund wrote: > >> * I think a lot of the interlinking stems from the bad idea to use > >> typedef's everywhere. In contrast to structs they cannot be forward > >> declared portably in our

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2019-07-31 Thread Julien Rouhaud
Hello, On Wed, Jul 31, 2019 at 10:55 AM Evgeny Efimkin wrote: > > What reason to use pg_atomic_uint64? The queryid is read and written without holding any lock on the PGPROC entry, so the pg_atomic_uint64 will guarantee that we get a consistent value in pg_stat_get_activity(). Other reads

Re: pgbench - implement strict TPC-B benchmark

2019-07-31 Thread Peter Geoghegan
On Wed, Jul 31, 2019 at 2:11 PM Tom Lane wrote: > > I agree with this. When I was at EnterpriseDB, while it wasn't audited, we > > had to develop an actual TPC-B implementation because pgbench was too > > different. pgbench itself isn't that useful as a benchmark tool, imo, but > > if we have the

Re: pgbench - implement strict TPC-B benchmark

2019-07-31 Thread Tom Lane
"Jonah H. Harris" writes: > On Wed, Jul 31, 2019 at 10:10 AM Fabien COELHO wrote: >> I agree that nobody really cares about TPC-B per se. The point of this >> patch is to provide a built-in example of recent and useful pgbench >> features that match a real specification. > I agree with this.

Re: Remove HeapTuple and Buffer dependency for predicate locking functions

2019-07-31 Thread Andres Freund
Hi, On 2019-07-31 12:37:58 -0700, Ashwin Agrawal wrote: > On Wed, Jul 31, 2019 at 10:55 AM Andres Freund wrote: > > > Hi, > > > > On 2019-07-31 10:42:50 -0700, Ashwin Agrawal wrote: > > > On Tue, Jul 30, 2019 at 2:58 PM Thomas Munro > > wrote: > > > > > > > On Tue, Jun 25, 2019 at 6:02 AM

Re: Remove HeapTuple and Buffer dependency for predicate locking functions

2019-07-31 Thread Ashwin Agrawal
On Wed, Jul 31, 2019 at 12:37 PM Ashwin Agrawal wrote: > > On Wed, Jul 31, 2019 at 10:55 AM Andres Freund wrote: > >> Hi, >> >> On 2019-07-31 10:42:50 -0700, Ashwin Agrawal wrote: >> > On Tue, Jul 30, 2019 at 2:58 PM Thomas Munro >> wrote: >> > >> > > On Tue, Jun 25, 2019 at 6:02 AM Andres

Re: Unused header file inclusion

2019-07-31 Thread Tom Lane
Alvaro Herrera writes: > On 2019-Jul-31, Andres Freund wrote: >> * I think a lot of the interlinking stems from the bad idea to use >> typedef's everywhere. In contrast to structs they cannot be forward >> declared portably in our version of C. We should use a lot more struct >> forward

Re: Replication & recovery_min_apply_delay

2019-07-31 Thread Alvaro Herrera
On 2019-Jul-31, Konstantin Knizhnik wrote: > On 08.07.2019 11:05, Michael Paquier wrote: > > Please note that I have not looked at that stuff in details, but I > > find the patch proposed kind of ugly with the scan of the last segment > > using a WAL reader to find out what is the last LSN and

Re: Unused header file inclusion

2019-07-31 Thread Alvaro Herrera
On 2019-Jul-31, Andres Freund wrote: > IDK, I find the compilation times annoying. And it's gotten quite > noticably worse with all the speculative execution mitigations. Although > to some degree that's not really the fault of individual compilations, > but our buildsystem being pretty slow.

Re: block-level incremental backup

2019-07-31 Thread Robert Haas
On Wed, Jul 31, 2019 at 1:59 PM vignesh C wrote: > I feel Robert's suggestion is good. > We can probably keep one meta file for each backup with some basic information > of all the files being backed up, this metadata file will be useful in the > below case: > Table dropped before incremental

Re: [HACKERS] Cached plans and statement generalization

2019-07-31 Thread Konstantin Knizhnik
On 31.07.2019 19:56, Heikki Linnakangas wrote: On 09/07/2019 23:59, Konstantin Knizhnik wrote: Fixed patch version of the path is attached. Much of the patch and the discussion has been around the raw parsing, and guessing which literals are actually parameters that have been inlined

Re: Remove HeapTuple and Buffer dependency for predicate locking functions

2019-07-31 Thread Ashwin Agrawal
On Wed, Jul 31, 2019 at 10:55 AM Andres Freund wrote: > Hi, > > On 2019-07-31 10:42:50 -0700, Ashwin Agrawal wrote: > > On Tue, Jul 30, 2019 at 2:58 PM Thomas Munro > wrote: > > > > > On Tue, Jun 25, 2019 at 6:02 AM Andres Freund > wrote: > > > > > - CheckForSerializableConflictOut() no more

Re: block-level incremental backup

2019-07-31 Thread vignesh C
On Tue, Jul 30, 2019 at 1:58 AM Robert Haas wrote: > > On Wed, Jul 10, 2019 at 2:17 PM Anastasia Lubennikova > wrote: > > In attachments, you can find a prototype of incremental pg_basebackup, > > which consists of 2 features: > > > > 1) To perform incremental backup one should call

Re: Remove HeapTuple and Buffer dependency for predicate locking functions

2019-07-31 Thread Andres Freund
Hi, On 2019-07-31 10:42:50 -0700, Ashwin Agrawal wrote: > On Tue, Jul 30, 2019 at 2:58 PM Thomas Munro wrote: > > > On Tue, Jun 25, 2019 at 6:02 AM Andres Freund wrote: > > > > - CheckForSerializableConflictOut() no more takes HeapTuple nor > > > > buffer, instead just takes xid. Push heap

Re: Remove HeapTuple and Buffer dependency for predicate locking functions

2019-07-31 Thread Andres Freund
Hi, On 2019-07-31 09:57:58 +1200, Thomas Munro wrote: > On Tue, Jun 25, 2019 at 6:02 AM Andres Freund wrote: > > Hm. I wonder if we somehow ought to generalize the granularity scheme > > for predicate locks to not be tuple/page/relation. But even if, that's > > probably a separate patch. > >

Re: Patch for SortSupport implementation on inet/cdir

2019-07-31 Thread Peter Geoghegan
On Fri, Jul 26, 2019 at 7:25 PM Peter Geoghegan wrote: > I guess that the idea here was to prevent masking on ipv6 addresses, > though not on ipv4 addresses. Obviously we're only dealing with a > prefix with ipv6 addresses, whereas we usually have the whole raw > ipaddr with ipv4. Not sure if I'm

Re: Remove HeapTuple and Buffer dependency for predicate locking functions

2019-07-31 Thread Ashwin Agrawal
On Tue, Jul 30, 2019 at 2:58 PM Thomas Munro wrote: > On Tue, Jun 25, 2019 at 6:02 AM Andres Freund wrote: > > > - CheckForSerializableConflictOut() no more takes HeapTuple nor > > > buffer, instead just takes xid. Push heap specific parts from > > > CheckForSerializableConflictOut() into

Re: Assertion for logically decoding multi inserts into the catalog

2019-07-31 Thread Heikki Linnakangas
On 08/07/2019 22:42, Daniel Gustafsson wrote: My patch for using heap_multi_insert in the catalog [1] failed the logical decoding part of test/recovery [2]. The assertion it failed on seems to not take multi inserts into the catalog into consideration, while the main logic does. This assertion

Re: [HACKERS] Cached plans and statement generalization

2019-07-31 Thread Heikki Linnakangas
On 09/07/2019 23:59, Konstantin Knizhnik wrote: Fixed patch version of the path is attached. Much of the patch and the discussion has been around the raw parsing, and guessing which literals are actually parameters that have been inlined into the SQL text. Do we really need to do that,

Re: How to retain lesser paths at add_path()?

2019-07-31 Thread Tom Lane
Robert Haas writes: > Yeah, but I have to admit that this whole design makes me kinda > uncomfortable. Every time somebody comes up with a new figure of > merit, it increases not only the number of paths retained but also the > cost of comparing two paths to possibly reject one of them. A few >

Re: [HACKERS] [WIP] Effective storage of duplicates in B-tree index.

2019-07-31 Thread Anastasia Lubennikova
24.07.2019 4:22, Peter Geoghegan wrote: Attached is a revised version of your v2 that fixes this issue -- I'll call this v3. In general, my goal for the revision was to make sure that all of my old tests from the v12 work passed, and to make sure that amcheck can detect almost any possible

Re: TopoSort() fix

2019-07-31 Thread Robert Haas
On Tue, Jul 30, 2019 at 2:10 PM Tom Lane wrote: > Sure. But I think what we can foresee is that if there are any bugs > reachable this way, they'd be reachable and need fixing regardless. > We've already established that parallel workers can take and release locks > that their leader isn't

Re: How to retain lesser paths at add_path()?

2019-07-31 Thread Robert Haas
On Wed, Jul 31, 2019 at 11:07 AM Tom Lane wrote: > What you'd want to do for something like the above, I think, is to > have some kind of figure of merit or other special marking for paths > that will have some possible special advantage in later planning > steps. Then you can teach add_path

Re: Unused header file inclusion

2019-07-31 Thread Andres Freund
Hi, On 2019-07-31 11:23:22 -0400, Alvaro Herrera wrote: > I think removing unnecessary include lines from header files is much > more useful than from .c files. However, nowadays even I am not very > convinced that that is a very fruitful use of time, since many/most > developers use ccache

Re: partition routing layering in nodeModifyTable.c

2019-07-31 Thread Andres Freund
Hi, On 2019-07-31 21:03:58 +0900, Etsuro Fujita wrote: > I'm still not sure that it's a good idea to remove > es_result_relation_info, but if I had to say then I think the latter > would probably be better. I'm planning to rework on direct > modification to base it on upper planner pathification

Re: Unused header file inclusion

2019-07-31 Thread Alvaro Herrera
On 2019-Jul-31, vignesh C wrote: > I noticed that there are many header files being > included which need not be included. Yeah, we have tooling for this already in src/tools/pginclude. It's been used before, and it has wreaked considerable havoc; see "git log --grep pgrminclude". I think

Re: How to retain lesser paths at add_path()?

2019-07-31 Thread Tom Lane
Kohei KaiGai writes: > When we add a new path using add_path(), it checks estimated cost and > path-keys, > then it also removes dominated paths, if any. > Do we have a reasonable way to retain these "dominated" paths? Once it > is considered > lesser paths at a level, however, it may have a

Global temporary tables

2019-07-31 Thread Konstantin Knizhnik
Current Postgres implementation of temporary table causes number of problems: 1. Catalog bloating: if client creates and deletes too many temporary tables, then autovacuum get stuck on catalog. 2. Parallel queries: right now usage of temporary tables in query disables parallel plan. 3. It is

Re: [HACKERS] [WIP] Effective storage of duplicates in B-tree index.

2019-07-31 Thread Rafia Sabih
On Thu, 25 Jul 2019 at 05:49, Peter Geoghegan wrote: > > On Wed, Jul 24, 2019 at 3:06 PM Peter Geoghegan wrote: > > There seems to be a kind of "synergy" between the nbtsplitloc.c > > handling of pages that have lots of duplicates and posting list > > compression. It seems as if the former

Re: pgbench - implement strict TPC-B benchmark

2019-07-31 Thread Jonah H. Harris
On Wed, Jul 31, 2019 at 10:10 AM Fabien COELHO wrote: > > Hello Tom, > > >>> I'm also highly dubious about labeling this script "standard TPC-B", > >>> when it resolves only some of the reasons why our traditional script > >>> is not really TPC-B. That's treading on being false advertising. > >

Re: pgbench - implement strict TPC-B benchmark

2019-07-31 Thread Fabien COELHO
Hello Tom, I'm also highly dubious about labeling this script "standard TPC-B", when it resolves only some of the reasons why our traditional script is not really TPC-B. That's treading on being false advertising. IANAL, but it may not even be permissible to claim that we have implemented

Re: Replication & recovery_min_apply_delay

2019-07-31 Thread Konstantin Knizhnik
On 08.07.2019 11:05, Michael Paquier wrote: On Mon, Jul 08, 2019 at 07:56:25PM +1200, Thomas Munro wrote: On Thu, Jan 31, 2019 at 3:34 AM Alvaro Herrera wrote: On 2019-Jan-30, Konstantin Knizhnik wrote: I wonder if it can be considered as acceptable solution of the problem or there can be

Re: Unused header file inclusion

2019-07-31 Thread Tom Lane
Michael Paquier writes: > On Wed, Jul 31, 2019 at 11:55:37AM +0530, Amit Kapila wrote: >> If we can come up with some such tool, we might be able to integrate >> it with Thomas's patch tester [1] wherein it can apply the patch, >> verify if there are unnecessary includes in the patch and report

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-31 Thread Sehrope Sarkuni
On Wed, Jul 31, 2019 at 2:32 AM Masahiko Sawada wrote: > Just to confirm, we have 21 bits left for nonce in CTR? We have LSN (8 > bytes), page-number (4 bytes) and counter (11 bits) in 16 bytes nonce > space. Even though we have 21 bits left we cannot store relfilenode to > the IV. > Fields

Re: proposal - patch: psql - sort_by_size

2019-07-31 Thread Fabien COELHO
Hello Jeremy, Comments, notes? One oddity about pg_relation_size and pg_table_size is that they can be easily blocked by user activity. In fact it happens to us often in reporting environments and we have instead written different versions of them that avoid the lock contention and still

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-31 Thread Sehrope Sarkuni
On Tue, Jul 30, 2019 at 4:48 PM Bruce Momjian wrote: > I had more time to think about the complexity of adding relfilenode to > the IV. Since relfilenode is only unique within a database/tablespace, > we would need to have pg_upgrade preserve database/tablespace oids > (which I assume are the

Re: POC: Cleaning up orphaned files using undo logs

2019-07-31 Thread Amit Kapila
On Wed, Jul 31, 2019 at 10:13 AM Amit Kapila wrote: > > On Tue, Jul 30, 2019 at 5:26 PM Thomas Munro wrote: > > > > Hi Amit > > > > I've been testing some undo worker workloads (more on that soon), > > > > One small point, there is one small bug in the error queues which is > that the element

Re: POC: Cleaning up orphaned files using undo logs

2019-07-31 Thread Amit Kapila
On Wed, Jul 24, 2019 at 10:00 AM Dilip Kumar wrote: > > On Mon, Jul 22, 2019 at 3:51 PM Dilip Kumar wrote: > > > Please find my review comments for > 0013-Allow-foreground-transactions-to-perform-undo-action > > + /* initialize undo record locations for the transaction */ > + for (i = 0; i <

Re: proposal - patch: psql - sort_by_size

2019-07-31 Thread Jeremy Finzel
On Fri, Jun 28, 2019 at 10:13 AM Pavel Stehule wrote: > Hi > > I returned to possibility to sort output of \d* and \l by size. There was > more a experiments in this area, but without success. Last patch was > example of over engineering, and now, I try to implement this feature > simply how it

Re: proposal - patch: psql - sort_by_size

2019-07-31 Thread Rafia Sabih
On Mon, 15 Jul 2019 at 06:12, Pavel Stehule wrote: > > Hi > > pá 12. 7. 2019 v 15:10 odesílatel Fabien COELHO napsal: >> >> >> Hello Pavel, >> >> > rebased patch attached >> >> I prefer patches with a number rather than a date, if possible. For one >> thing, there may be several updates in one

Re: Problem with default partition pruning

2019-07-31 Thread Alvaro Herrera
On 2019-Jul-31, Amit Langote wrote: > Hello, > > I noticed that the patch is still marked as "Waiting on Author" ever > since Shawn set it that way on June 17. Since Hosoya-san posted > updated patches on June 27, the status should've been changed to > "Needs Review". Or maybe "Ready for

Re: partition routing layering in nodeModifyTable.c

2019-07-31 Thread Etsuro Fujita
On Wed, Jul 31, 2019 at 5:05 PM Amit Langote wrote: > On Tue, Jul 30, 2019 at 4:20 PM Amit Langote wrote: > > On Sat, Jul 20, 2019 at 1:52 AM Andres Freund wrote: > > > IOW, we should just change the direct modify calls to get the relevant > > > ResultRelationInfo or something in that vein

Re: concerns around pg_lsn

2019-07-31 Thread Jeevan Ladhe
On Tue, Jul 30, 2019 at 6:06 PM Robert Haas wrote: > On Tue, Jul 30, 2019 at 4:52 AM Jeevan Ladhe > wrote: > > My only concern was something that we internally treat as invalid, why do > > we allow, that as a valid value for that type. While I am not trying to > > reinvent the wheel here, I am

Re: how to run encoding-dependent tests by default

2019-07-31 Thread Peter Eisentraut
On 2019-07-29 16:47, Tom Lane wrote: >> (The two tests create the same schema name, so they cannot be run in >> parallel. I opted against changing that here, since it would blow up >> the patch and increase the diff between the two tests.) > > This does create one tiny nit, which is that the

Re: idea: log_statement_sample_rate - bottom limit for sampling

2019-07-31 Thread Adrien Nayrat
On 7/28/19 12:19 AM, Tomas Vondra wrote: > Hi, > > I've started reviewing this patch, thinking that maybe I could get it > committed as it's marked as RFC. In general I agree with having this > fuature, but I think we need to rethink the GUC because the current > approach is just confusing. > >

Re: Implementing Incremental View Maintenance

2019-07-31 Thread Yugo Nagata
Hi, Attached is the latest patch for supporting min and max aggregate functions. When new tuples are inserted into base tables, if new values are small (for min) or large (for max), matview just have to be updated with these new values. Otherwise, old values just remains. However, in the

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2019-07-31 Thread Evgeny Efimkin
What reason to use pg_atomic_uint64? In docs: occured - > occurred

Re: Built-in connection pooler

2019-07-31 Thread Konstantin Knizhnik
On 26.07.2019 19:20, Ryan Lambert wrote: > PgPRO EE version of connection pooler has "idle_pool_worker_timeout" > parameter which allows to terminate idle workers. +1 I have implemented idle_pool_worker_timeout. Also I added _idle_clients and n_idle_backends fields to proxy statistic

Re: Problem with default partition pruning

2019-07-31 Thread Amit Langote
Hello, I noticed that the patch is still marked as "Waiting on Author" ever since Shawn set it that way on June 17. Since Hosoya-san posted updated patches on June 27, the status should've been changed to "Needs Review". Or maybe "Ready for Committer", because the last time I looked, at least

Re: Parallel grouping sets

2019-07-31 Thread Richard Guo
On Tue, Jul 30, 2019 at 11:05 PM Tomas Vondra wrote: > On Tue, Jul 30, 2019 at 03:50:32PM +0800, Richard Guo wrote: > >On Wed, Jun 12, 2019 at 10:58 AM Richard Guo wrote: > > > >> Hi all, > >> > >> Paul and I have been hacking recently to implement parallel grouping > >> sets, and here we have

Re: partition routing layering in nodeModifyTable.c

2019-07-31 Thread Amit Langote
On Tue, Jul 30, 2019 at 4:20 PM Amit Langote wrote: > On Sat, Jul 20, 2019 at 1:52 AM Andres Freund wrote: > > > The first one (0001) deals with reducing the core executor's reliance > > > on es_result_relation_info to access the currently active result > > > relation, in favor of receiving it

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-31 Thread Masahiko Sawada
On Wed, Jul 31, 2019 at 3:29 PM Masahiko Sawada wrote: > > > For WAL encryption, before flushing WAL we encrypt whole 8k WAL page > and then write only the encrypted data of the new WAL record using > pg_pwrite() rather than write whole encrypted page. So each time we > encrypt 8k WAL page we

Re: [HACKERS] Restricting maximum keep segments by repslots

2019-07-31 Thread Kyotaro Horiguchi
At Tue, 30 Jul 2019 21:30:45 +0900 (Tokyo Standard Time), Kyotaro Horiguchi wrote in <20190730.213045.221405075.horikyota@gmail.com> > I attach the revised patch. I'll repost the polished version > sooner. This is the revised patch. - Status criteria has been changed. "streaming" :

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-31 Thread Masahiko Sawada
On Tue, Jul 30, 2019 at 10:45 PM Sehrope Sarkuni wrote: > > On Tue, Jul 30, 2019 at 8:16 AM Masahiko Sawada wrote: >> >> On Mon, Jul 29, 2019 at 8:18 PM Sehrope Sarkuni wrote: >> > >> > On Mon, Jul 29, 2019 at 6:42 AM Masahiko Sawada >> > wrote: >> > > > An argument could be made to push that

Re: minor fixes after pgindent prototype fixes

2019-07-31 Thread Andres Freund
On 2019-07-28 00:09:51 -0400, Tom Lane wrote: > Andres Freund writes: > > a few prototypes look odd. It appears to be cases where previously the > > odd indentation was put to some use, by indenting parameters less: > > ... > > but now that looks odd: > > extern void DefineCustomBoolVariable( > >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-31 Thread Masahiko Sawada
On Wed, Jul 31, 2019 at 5:48 AM Bruce Momjian wrote: > > On Tue, Jul 30, 2019 at 10:14:14AM -0400, Sehrope Sarkuni wrote: > > > In general it's fine to use the same IV with different keys. Only > > reuse > > of Key > > > + IV is a problem and the entire set of possible counter values

Re: Unused header file inclusion

2019-07-31 Thread Andres Freund
Hi, On 2019-07-31 11:19:08 +0530, vignesh C wrote: > I noticed that there are many header files being > included which need not be included. > I have tried this in a few files and found the > compilation and regression to be working. > I have attached the patch for the files that > I tried. > I

Re: Unused header file inclusion

2019-07-31 Thread Michael Paquier
On Wed, Jul 31, 2019 at 11:55:37AM +0530, Amit Kapila wrote: > If we can come up with some such tool, we might be able to integrate > it with Thomas's patch tester [1] wherein it can apply the patch, > verify if there are unnecessary includes in the patch and report the > same. > > [1] -

Re: Unused header file inclusion

2019-07-31 Thread vignesh C
On Wed, Jul 31, 2019 at 11:55 AM Amit Kapila wrote: > > On Wed, Jul 31, 2019 at 11:31 AM vignesh C wrote: > > > > On Wed, Jul 31, 2019 at 11:26 AM Michael Paquier > > wrote: > > > > > > On Wed, Jul 31, 2019 at 11:19:08AM +0530, vignesh C wrote: > > > > I noticed that there are many header

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-31 Thread Masahiko Sawada
On Mon, Jul 29, 2019 at 10:44 PM Bruce Momjian wrote: > > On Mon, Jul 29, 2019 at 08:43:06PM +0900, Masahiko Sawada wrote: > > > I am thinking of writing some Assert() code that checks that all buffers > > > using a single LSN are from the same relation (and therefore different > > > page

Re: Possible race condition in pg_mkdir_p()?

2019-07-31 Thread Michael Paquier
On Wed, Jul 31, 2019 at 02:02:03PM +0800, Ning Yu wrote: > Yes, in current postgres source code there are several wrappers of > mkdir() that do similar jobs. If we could have a safe mkdir_p() > implementation then we could use it directly in all these wrappers, that > could save a lot of

Re: Possible race condition in pg_mkdir_p()?

2019-07-31 Thread Ning Yu
On Wed, Jul 31, 2019 at 1:31 PM Michael Paquier wrote: > > On Tue, Jul 30, 2019 at 10:19:45PM -0700, Andres Freund wrote: > > I don't really have a problem fixing this case if we think it's > > useful. But I'm a bit bothered by all the "fixes" being submitted that > > don't matter for PG itself.

Re: Unused header file inclusion

2019-07-31 Thread vignesh C
On Wed, Jul 31, 2019 at 11:26 AM Michael Paquier wrote: > > On Wed, Jul 31, 2019 at 11:19:08AM +0530, vignesh C wrote: > > I noticed that there are many header files being included which need > > not be included. I have tried this in a few files and found the > > compilation and regression to be