Re: Creating foreign key on partitioned table is too slow

2019-10-30 Thread Tom Lane
David Rowley writes: > In Ottawa this year, Andres and I briefly talked about the possibility > of making a series of changes to how equalfuncs.c works. The idea was > to make it easy by using some pre-processor magic to allow us to > create another version of equalfuncs which would let us have

Re: tableam vs. TOAST

2019-10-30 Thread Prabhat Sahu
On Wed, Oct 30, 2019 at 9:46 PM Robert Haas wrote: > On Wed, Oct 30, 2019 at 3:49 AM Prabhat Sahu < > prabhat.s...@enterprisedb.com> wrote: > >> While testing the Toast patch(PG+v7 patch) I found below server crash. >> System configuration: >> VCPUs: 4, RAM: 8GB, Storage: 320GB >> >> This issue

Re: [BUG] Partition creation fails after dropping a column and adding a partial index

2019-10-30 Thread Michael Paquier
On Tue, Oct 29, 2019 at 01:16:58PM +0900, Michael Paquier wrote: > Yes, something looks wrong with that. I have not looked at it in > details yet though. I'll see about that tomorrow. So.. When building the attribute map for a cloned index (with either LIKE during the transformation or for

Re: Allow CREATE OR REPLACE VIEW to rename the columns

2019-10-30 Thread Tom Lane
Fujii Masao writes: > Currently CREATE OR REPLACE VIEW command fails if the column names > are changed. That is, I believe, intentional. It's an effective aid to catching mistakes in view redefinitions, such as misaligning the new set of columns relative to the old. That's particularly

Re: Allow cluster_name in log_line_prefix

2019-10-30 Thread Tatsuo Ishii
> Hi folks > > I was recently surprised to notice that log_line_prefix doesn't support a > cluster_name placeholder. I suggest adding one. If I don't hear objections > I'll send a patch. I think it'd be a good thing for users. Best regards, -- Tatsuo Ishii SRA OSS, Inc. Japan English:

Re: Add SQL function to show total block numbers in the relation

2019-10-30 Thread Michael Paquier
On Wed, Oct 30, 2019 at 10:09:47AM -0400, Tom Lane wrote: > btkimurayuzk writes: >> I propose new simple sql query, which shows total block numbers in the >> relation. >> ... >> Of cource, we can know this value such as >> select (pg_relation_size('t') / >>

Re: Allow cluster_name in log_line_prefix

2019-10-30 Thread Thomas Munro
On Mon, Oct 28, 2019 at 3:33 PM Craig Ringer wrote: > I was recently surprised to notice that log_line_prefix doesn't support a > cluster_name placeholder. I suggest adding one. If I don't hear objections > I'll send a patch. +1

Allow CREATE OR REPLACE VIEW to rename the columns

2019-10-30 Thread Fujii Masao
Hi, Currently CREATE OR REPLACE VIEW command fails if the column names are changed. For example, =# CREATE VIEW test AS SELECT 0 AS a; =# CREATE OR REPLACE VIEW test AS SELECT 0 AS x; ERROR: cannot change name of view column "a" to "x" I'd like to propose the attached patch that

Re: Problem with synchronous replication

2019-10-30 Thread Michael Paquier
On Wed, Oct 30, 2019 at 05:43:04PM +0900, Kyotaro Horiguchi wrote: > At Wed, 30 Oct 2019 17:21:17 +0900, Fujii Masao wrote > in >> This change causes every ending backends to always take the exclusive lock >> even when it's not in SyncRep queue. This may be problematic, for example, >> when

Re: Problem with synchronous replication

2019-10-30 Thread Michael Paquier
On Wed, Oct 30, 2019 at 12:34:28PM +0900, Kyotaro Horiguchi wrote: > If we do that strictly, other functions like > SyncRepGetOldestSyncRecPtr need the same Assert()s. I think static > functions don't need Assert() and caution in their comments would be > enough. Perhaps. I'd rather be careful

Re: Creating foreign key on partitioned table is too slow

2019-10-30 Thread Andres Freund
Hi, On 2019-10-31 11:19:05 +1300, David Rowley wrote: > In Ottawa this year, Andres and I briefly talked about the possibility > of making a series of changes to how equalfuncs.c works. The idea was > to make it easy by using some pre-processor magic to allow us to > create another version of

Re: Zedstore - compressed in-core columnar storage

2019-10-30 Thread Taylor Vesely
Alex Wang and I have been doing some performance analysis of the most recent version of the zedstore branch, and have some interesting statistics to share. We specifically focused on TPC-DS query 2, because it plays to what should be the strength of zedstore- namely it does a full table scan of

Re: Creating foreign key on partitioned table is too slow

2019-10-30 Thread David Rowley
On Thu, 31 Oct 2019 at 07:30, Tomas Vondra wrote: > > On Thu, Oct 24, 2019 at 04:28:38PM -0700, Andres Freund wrote: > >Hi, > > > >On 2019-10-23 05:59:01 +, kato-...@fujitsu.com wrote: > >> To benchmark with tpcb model, I tried to create a foreign key in the > >> partitioned history table,

Re: Thoughts on nbtree with logical/varwidth table identifiers, v12 on-disk representation

2019-10-30 Thread Peter Geoghegan
On Wed, Oct 30, 2019 at 1:25 PM Andres Freund wrote: > I assume you mean that the index would dynamically recognize when it > needs the wider tids ("for the higher portion")? If so, yea, that makes > sense to me. Would that need to encode the 6/8byteness of a tid on a > per-element basis? Or are

Re: v12.0: ERROR: could not find pathkey item to sort

2019-10-30 Thread David Rowley
On Thu, 31 Oct 2019 at 05:09, Tom Lane wrote: > David --- much of the complexity here comes from the addition of > the eclass_indexes infrastructure, so do you have any thoughts? Hindsight makes me think I should have mentioned in the comment for eclass_indexes that it's only used for simple

Re: Thoughts on nbtree with logical/varwidth table identifiers, v12 on-disk representation

2019-10-30 Thread Andres Freund
Hi, On 2019-10-30 12:37:50 -0700, Peter Geoghegan wrote: > On Wed, Oct 30, 2019 at 12:03 PM Andres Freund wrote: > > I'd much rather not entrench this further, even leaving global indexes > > aside. The 4 byte block number is a significant limitation for heap > > tables too, and we should lift

pgstat.c has brittle response to transient problems

2019-10-30 Thread Tom Lane
While fooling with the NetBSD-vs-libpython issue noted in a nearby thread, I observed that the core regression tests sometimes hang up in the "stats" test on this platform (NetBSD 8.1/amd64). Investigation found that the stats collector process was sometimes exiting like this: 2019-10-29

Re: MarkBufferDirtyHint() and LSN update

2019-10-30 Thread Tomas Vondra
On Wed, Oct 30, 2019 at 02:44:18PM +0100, Antonin Houska wrote: Please consider this scenario (race conditions): 1. FlushBuffer() has written the buffer but hasn't yet managed to clear the BM_DIRTY flag (however BM_JUST_DIRTIED could be cleared by now). 2. Another backend modified a hint bit

Re: Thoughts on nbtree with logical/varwidth table identifiers, v12 on-disk representation

2019-10-30 Thread Peter Geoghegan
On Wed, Oct 30, 2019 at 12:03 PM Andres Freund wrote: > I'd much rather not entrench this further, even leaving global indexes > aside. The 4 byte block number is a significant limitation for heap > tables too, and we should lift that at some point not too far away. > Then there's also other AMs

Re: Parallel leader process info in EXPLAIN

2019-10-30 Thread Tomas Vondra
On Wed, Oct 30, 2019 at 10:39:04AM -0700, Peter Geoghegan wrote: On Wed, Oct 30, 2019 at 9:24 AM Melanie Plageman wrote: Checked out the patches a bit and noticed that the tuplesort instrumentation uses spaceUsed and I saw this comment in tuplesort_get_stats() might it be worth trying out

Re: Thoughts on nbtree with logical/varwidth table identifiers, v12 on-disk representation

2019-10-30 Thread Andres Freund
Hi, On 2019-10-30 11:33:21 -0700, Peter Geoghegan wrote: > On Mon, Apr 22, 2019 at 9:35 AM Andres Freund wrote: > > On 2019-04-21 17:46:09 -0700, Peter Geoghegan wrote: > > > Andres has suggested that I work on teaching nbtree to accommodate > > > variable-width, logical table identifiers, such

Re: Thoughts on nbtree with logical/varwidth table identifiers, v12 on-disk representation

2019-10-30 Thread Peter Geoghegan
On Mon, Apr 22, 2019 at 9:35 AM Andres Freund wrote: > On 2019-04-21 17:46:09 -0700, Peter Geoghegan wrote: > > Andres has suggested that I work on teaching nbtree to accommodate > > variable-width, logical table identifiers, such as those required for > > indirect indexes, or clustered indexes,

Re: Creating foreign key on partitioned table is too slow

2019-10-30 Thread Tomas Vondra
On Thu, Oct 24, 2019 at 04:28:38PM -0700, Andres Freund wrote: Hi, On 2019-10-23 05:59:01 +, kato-...@fujitsu.com wrote: To benchmark with tpcb model, I tried to create a foreign key in the partitioned history table, but backend process killed by OOM. the number of partitions is 8192. I

Re: Parallel leader process info in EXPLAIN

2019-10-30 Thread Peter Geoghegan
On Wed, Oct 30, 2019 at 9:24 AM Melanie Plageman wrote: > Checked out the patches a bit and noticed that the tuplesort > instrumentation uses spaceUsed and I saw this comment in > tuplesort_get_stats() > might it be worth trying out the memory accounting API >

Re: PL/Python fails on new NetBSD/PPC 8.0 install

2019-10-30 Thread Tom Lane
I wrote: > Thomas Munro writes: >> From a quick look at the relevant trees, isn't the problem here that >> cpython thinks it can reserve pthread_t value -1 (or rather, that >> number cast to unsigned long, which is the type it uses for its own >> thread IDs): > Yeah, this. I shall now go rant

Re: Proposal: Global Index

2019-10-30 Thread Andres Freund
Hi, On 2019-10-30 13:05:57 -0400, Tom Lane wrote: > Peter Geoghegan writes: > > On Wed, Oct 30, 2019 at 9:23 AM Tom Lane wrote: > >> Well, the *effects* of the feature seem desirable, but that doesn't > >> mean that we want an implementation that actually has a shared index. > >> As soon as you

Re: [Proposal] Add accumulated statistics

2019-10-30 Thread Pavel Stehule
út 15. 1. 2019 v 2:14 odesílatel Tsunakawa, Takayuki < tsunakawa.ta...@jp.fujitsu.com> napsal: > From: Pavel Stehule [mailto:pavel.steh...@gmail.com] > > the cumulated lock statistics maybe doesn't help with debugging - but it > > is very good indicator of database (in production usage) health. >

Re: Proposal: Global Index

2019-10-30 Thread Tom Lane
Peter Geoghegan writes: > On Wed, Oct 30, 2019 at 9:23 AM Tom Lane wrote: >> Well, the *effects* of the feature seem desirable, but that doesn't >> mean that we want an implementation that actually has a shared index. >> As soon as you do that, you've thrown away most of the benefits of >>

Re: Proposal: Global Index

2019-10-30 Thread Peter Geoghegan
On Wed, Oct 30, 2019 at 9:23 AM Tom Lane wrote: > Well, the *effects* of the feature seem desirable, but that doesn't > mean that we want an implementation that actually has a shared index. > As soon as you do that, you've thrown away most of the benefits of > having a partitioned data structure

Re: Parallel leader process info in EXPLAIN

2019-10-30 Thread Melanie Plageman
On Wed, Oct 23, 2019 at 12:30 AM Thomas Munro wrote: > > While working on some slides explaining EXPLAIN, I couldn't resist the > urge to add the missing $SUBJECT. The attached 0001 patch gives the > following: > > Gather ... time=0.146..33.077 rows=1 loops=1) > Workers Planned: 2 >

Re: Proposal: Global Index

2019-10-30 Thread Tom Lane
Robert Haas writes: > On Wed, Oct 30, 2019 at 10:13 AM Tom Lane wrote: >> I believe that the current design of partitioning is explicitly intended >> to avoid the need for such a construct. It'd be absolutely disastrous >> to have such a thing from many standpoints, including the breadth of >>

Re: tableam vs. TOAST

2019-10-30 Thread Robert Haas
On Wed, Oct 30, 2019 at 3:49 AM Prabhat Sahu wrote: > While testing the Toast patch(PG+v7 patch) I found below server crash. > System configuration: > VCPUs: 4, RAM: 8GB, Storage: 320GB > > This issue is not frequently reproducible, we need to repeat the same > testcase multiple times. > I

Re: Proposal: Global Index

2019-10-30 Thread Robert Haas
On Wed, Oct 30, 2019 at 10:13 AM Tom Lane wrote: > I believe that the current design of partitioning is explicitly intended > to avoid the need for such a construct. It'd be absolutely disastrous > to have such a thing from many standpoints, including the breadth of > locking needed to work with

Re: v12.0: ERROR: could not find pathkey item to sort

2019-10-30 Thread Tom Lane
Amit Langote writes: > Attached updated patches. [ looks at that... ] I seriously, seriously dislike what you did in build_join_rel, ie adding the new joinrel to the global data structures before it's fully filled in. That's inevitably going to bite us on the ass someday, and you couldn't even

Re: PL/Python fails on new NetBSD/PPC 8.0 install

2019-10-30 Thread Tom Lane
Thomas Munro writes: > On Wed, Oct 30, 2019 at 9:25 AM Tom Lane wrote: >> What I'm inclined to do is go file a bug report saying that this >> behavior contradicts both POSIX and NetBSD's own man page, and >> see what they say about that. So I went and filed that bug,

Re: WIP/PoC for parallel backup

2019-10-30 Thread Asif Rehman
On Mon, Oct 28, 2019 at 8:29 PM Robert Haas wrote: > On Mon, Oct 28, 2019 at 10:03 AM Asif Rehman > wrote: > > I have updated the patch to include the changes suggested by Jeevan. > This patch also implements the thread workers instead of > > processes and fetches a single file at a time. The

Re: Proposal: Global Index

2019-10-30 Thread Tom Lane
Ibrar Ahmed writes: > A global index by very definition is a single index on the parent table > that maps to many > underlying table partitions. I believe that the current design of partitioning is explicitly intended to avoid the need for such a construct. It'd be absolutely disastrous to have

Re: Add SQL function to show total block numbers in the relation

2019-10-30 Thread Tom Lane
btkimurayuzk writes: > I propose new simple sql query, which shows total block numbers in the > relation. > ... > Of cource, we can know this value such as > select (pg_relation_size('t') / > current_setting('block_size')::bigint)::int; I don't really see why the existing solution isn't

Re: Binary support for pgoutput plugin

2019-10-30 Thread Dave Cramer
On Sun, 27 Oct 2019 at 11:00, Dmitry Dolgov <9erthali...@gmail.com> wrote: > > On Mon, Jun 17, 2019 at 10:29:26AM -0400, Dave Cramer wrote: > > > Which is what I have done. Thanks > > > > > > I've attached both patches for comments. > > > I still have to add documentation. > > > > Additional

Remove HAVE_LONG_LONG_INT

2019-10-30 Thread Peter Eisentraut
HAVE_LONG_LONG_INT is now implied by the requirement for C99, so the separate Autoconf check can be removed. The uses are almost all in ecpg code, and AFAICT the check was originally added specifically for ecpg. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL

MarkBufferDirtyHint() and LSN update

2019-10-30 Thread Antonin Houska
Please consider this scenario (race conditions): 1. FlushBuffer() has written the buffer but hasn't yet managed to clear the BM_DIRTY flag (however BM_JUST_DIRTIED could be cleared by now). 2. Another backend modified a hint bit and called MarkBufferDirtyHint(). 3. In MarkBufferDirtyHint(), if

Re: Problem with synchronous replication

2019-10-30 Thread Michael Paquier
On Wed, Oct 30, 2019 at 05:21:17PM +0900, Fujii Masao wrote: > This change causes every ending backends to always take the exclusive lock > even when it's not in SyncRep queue. This may be problematic, for example, > when terminating multiple backends at the same time? If yes, > it might be better

Re: Unix-domain socket support on Windows

2019-10-30 Thread Peter Eisentraut
To move this topic a long, I'll submit some preparatory patches in a committable order. First is the patch to deal with getpeereid() that was already included in the previous patch series. This is just some refactoring that reduces the difference between Windows and other platforms and

Re: Remove one use of IDENT_USERNAME_MAX

2019-10-30 Thread Peter Eisentraut
On 2019-10-29 15:34, Tom Lane wrote: Peter Eisentraut writes: On 2019-10-28 14:45, Tom Lane wrote: Kyotaro Horiguchi writes: In think one of the reasons for the coding is the fact that *pw is described to be placed in the static area, which can be overwritten by succeeding calls to getpw*()

Re: pgbench - extend initialization phase control

2019-10-30 Thread Fujii Masao
On Mon, Oct 28, 2019 at 10:36 PM Fabien COELHO wrote: > > > Hello Masao-san, > > >> Maybe. If you cannot check, you can only guess. Probably it should be > >> small, but the current version does not allow to check whether it is so. > > > > Could you elaborate what you actually want to measure the

Re: v12.0: ERROR: could not find pathkey item to sort

2019-10-30 Thread Amit Langote
Thanks for taking a look and sorry about the delay in replying. On Fri, Oct 25, 2019 at 1:51 AM Tom Lane wrote: > Amit Langote writes: > > On Mon, Oct 14, 2019 at 11:54 PM Tom Lane wrote: > >> In view of the proposed patches being dependent on some other > >> 13-only changes, I wonder if we

Re: [HACKERS] Block level parallel vacuum

2019-10-30 Thread Masahiko Sawada
On Mon, Oct 28, 2019 at 3:50 PM Amit Kapila wrote: > > On Sun, Oct 27, 2019 at 12:52 PM Dilip Kumar wrote: > > > > On Fri, Oct 25, 2019 at 9:19 PM Masahiko Sawada > > wrote: > > > > > > > > I haven't yet read the new set of the patch. But, I have noticed one > > thing. That we are getting

Re: Problem with synchronous replication

2019-10-30 Thread Kyotaro Horiguchi
Hello. At Wed, 30 Oct 2019 17:21:17 +0900, Fujii Masao wrote in > This change causes every ending backends to always take the exclusive lock > even when it's not in SyncRep queue. This may be problematic, for example, > when terminating multiple backends at the same time? If yes, > it might be

Proposal: Global Index

2019-10-30 Thread Ibrar Ahmed
A global index by very definition is a single index on the parent table that maps to many underlying table partitions. The parent table itself does not have any underlying storage, so it must, therefore, retrieve the data satisfying index constraints from the underlying tables. In very crude

Re: Problem with synchronous replication

2019-10-30 Thread Fujii Masao
On Wed, Oct 30, 2019 at 4:16 PM lingce.ldm wrote: > > On Oct 29, 2019, at 18:50, Kyotaro Horiguchi wrote: > > > Hello. > > At Fri, 25 Oct 2019 15:18:34 +0800, "Dongming Liu" > wrote in > > > Hi, > > I recently discovered two possible bugs about synchronous replication. > > 1.

Re: Join Correlation Name

2019-10-30 Thread Fabien COELHO
Bonjour Vik, Is quoting the spec good enough? SQL:2016 Part 2 Foundation Section 7.10 : Ah, this is the one information I did not have when reviewing Peter's patch. ::=     USING[ AS ] ::=     I think possibly what the spec says (and that neither my patch nor Peter's

Re: tableam vs. TOAST

2019-10-30 Thread Prabhat Sahu
Hi All, While testing the Toast patch(PG+v7 patch) I found below server crash. System configuration: VCPUs: 4, RAM: 8GB, Storage: 320GB This issue is not frequently reproducible, we need to repeat the same testcase multiple times. CREATE OR REPLACE FUNCTION toast_chunks_cnt_func(p1 IN text)

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-10-30 Thread Dilip Kumar
On Wed, Oct 30, 2019 at 9:38 AM vignesh C wrote: > I have noticed one more problem in the logic of setting the logical decoding work mem from the create subscription command. Suppose in subscription command we don't give the work mem then it sends the garbage value to the walsender and the

Add SQL function to show total block numbers in the relation

2019-10-30 Thread btkimurayuzk
Hello, I propose new simple sql query, which shows total block numbers in the relation. I now reviewing this patch (https://commitfest.postgresql.org/25/2211/) and I think, it is usefull for knowing how many blocks there are in the relation to determine whether we use VACUUM RESUME or not.

Re: RFC: split OBJS lines to one object per line

2019-10-30 Thread Michael Paquier
On Tue, Oct 29, 2019 at 11:32:09PM -0700, Andres Freund wrote: > Cool. Any opinion on whether to got for > > OBJS = \ > dest.o \ > fastpath.o \ > ... > > or > > OBJS = dest.o \ > fastpath.o \ > ... > > I'm mildly inclined to go for the former. FWIW, I am more used to the

Re: Problem with synchronous replication

2019-10-30 Thread lingce . ldm
On Oct 30, 2019, at 09:45, Michael Paquier mailto:mich...@paquier.xyz>> wrote: > > On Tue, Oct 29, 2019 at 07:50:01PM +0900, Kyotaro Horiguchi wrote: >> At Fri, 25 Oct 2019 15:18:34 +0800, "Dongming Liu" >> wrote in >>> I recently discovered two possible bugs about synchronous replication. >>>

Re: Problem with synchronous replication

2019-10-30 Thread lingce . ldm
On Oct 29, 2019, at 18:50, Kyotaro Horiguchi mailto:horikyota@gmail.com>> wrote: > > Hello. > > At Fri, 25 Oct 2019 15:18:34 +0800, "Dongming Liu" > mailto:lingce@alibaba-inc.com>> wrote in >> >> Hi, >> >> I recently discovered two possible bugs about synchronous replication. >> >>

Re: RFC: split OBJS lines to one object per line

2019-10-30 Thread Tom Lane
Andres Freund writes: > On 2019-10-29 16:31:11 -0400, Tom Lane wrote: >> We did something similar not too long ago in configure.in (bfa6c5a0c), >> and it seems to have helped. +1 > Cool. Any opinion on whether to got for ... Not here. regards, tom lane

Re: RFC: split OBJS lines to one object per line

2019-10-30 Thread Andres Freund
Hi, On 2019-10-29 16:31:11 -0400, Tom Lane wrote: > Andres Freund writes: > > one of the most frequent conflicts I see is that two patches add files > > to OBJS (or one of its other spellings), and there are conflicts because > > another file has been added. > > ... > > Now, obviously these

Re: [HACKERS] Block level parallel vacuum

2019-10-30 Thread Dilip Kumar
On Tue, Oct 29, 2019 at 3:11 PM Dilip Kumar wrote: > > On Tue, Oct 29, 2019 at 1:59 PM Masahiko Sawada wrote: > > > > On Tue, Oct 29, 2019 at 4:06 PM Masahiko Sawada > > wrote: > > > > > > On Mon, Oct 28, 2019 at 2:13 PM Dilip Kumar wrote: > > > > > > > > On Thu, Oct 24, 2019 at 4:33 PM Dilip

Re: pg_waldump erroneously outputs newline for FPWs, and another minor bug

2019-10-30 Thread Andres Freund
Hi, On 2019-10-29 16:33:41 -0700, Andres Freund wrote: > Hi, > > When using -b, --bkp-details pg_waldump outputs an unnecessary newline > for blocks that contain an FPW. > > In --bkp-details block references are output on their own lines, like: > > rmgr: SPGist len (rec/tot): 4348/