Re: About to add WAL write/fsync statistics to pg_stat_wal view

2020-12-07 Thread Li Japin
Hi, > On Dec 8, 2020, at 1:06 PM, Masahiro Ikeda wrote: > > Hi, > > I propose to add wal write/fsync statistics to pg_stat_wal view. > It's useful not only for developing/improving source code related to WAL > but also for users to detect workload changes, HW failure, and so on. > > I

Printing page request trace from buffer manager

2020-12-07 Thread Irodotos Terpizis
Hello, A description of what you are trying to achieve and what results you expect.: I am a student and I am new in PSQL. I am working on a research project and an initial step is to trace the page request of the buffer manager. I need to know which page was evicted from the buffer and which

Re: Blocking I/O, async I/O and io_uring

2020-12-07 Thread Andres Freund
Hi, On 2020-12-08 04:24:44 +, tsunakawa.ta...@fujitsu.com wrote: > I'm looking forward to this from the async+direct I/O, since the > throughput of some write-heavy workload decreased by half or more > during checkpointing (due to fsync?) Depends on why that is. The most common, I think,

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-12-07 Thread tsunakawa.ta...@fujitsu.com
From: Jamison, Kirk/ジャミソン カーク > Because one of the rel's cached value was false, it forced the > full-scan path for TRUNCATE. > Is there a possible workaround for this? Hmm, the other two relfilenodes are for the TOAST table and index of the target table. I think the INSERT didn't access those

Re: PG vs LLVM 12 on seawasp, next round

2020-12-07 Thread Andres Freund
Hi, On 2020-12-07 19:38:19 -0800, Andres Freund wrote: > On 2020-12-01 12:08:10 -0800, Andres Freund wrote: > > On 2020-12-01 21:04:44 +0100, Fabien COELHO wrote: > > > Andres investigated a few days ago, managed to reproduce the issue > > > locally, > > > and has one line patch. I'm unsure if

Re: Blocking I/O, async I/O and io_uring

2020-12-07 Thread Andres Freund
Hi, On 2020-12-08 13:01:38 +0800, Craig Ringer wrote: > Have you done much bpf / systemtap / perf based work on measurement and > tracing of latencies etc? If not that's something I'd be keen to help with. > I've mostly been using systemtap so far but I'm trying to pivot over to > bpf. Not much

Re: Single transaction in the tablesync worker?

2020-12-07 Thread Peter Smith
On Mon, Dec 7, 2020 at 7:49 PM Amit Kapila wrote: > As the slot of apply worker is created before all the tablesync > workers it should never miss any LSN which tablesync workers would > have processed. Also, the table sync workers should not process any > xact if the apply worker has not

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-12-07 Thread k.jami...@fujitsu.com
On Tuesday, December 8, 2020 2:35 PM, Amit Kapila wrote: > On Tue, Dec 8, 2020 at 10:41 AM Kyotaro Horiguchi > wrote: > > > > At Tue, 8 Dec 2020 08:08:25 +0530, Amit Kapila > > wrote in > > > On Tue, Dec 8, 2020 at 7:24 AM Kyotaro Horiguchi > > > wrote: > > > > We drop > > > > buffers for the

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-12-07 Thread Amit Kapila
On Tue, Dec 8, 2020 at 10:41 AM Kyotaro Horiguchi wrote: > > At Tue, 8 Dec 2020 08:08:25 +0530, Amit Kapila > wrote in > > On Tue, Dec 8, 2020 at 7:24 AM Kyotaro Horiguchi > > wrote: > > > We drop > > > buffers for the old relfilenode on truncation anyway. > > > > > > What I did is: > > > > >

Re: [Proposal] Global temporary tables

2020-12-07 Thread Pavel Stehule
Hi čt 26. 11. 2020 v 12:46 odesílatel 曾文旌 napsal: > This is the latest patch for feature GTT(v38). > Everybody, if you have any questions, please let me know. > please, co you send a rebased patch. It is broken again Regards Pavel > > Wenjing > > >

Re: Single transaction in the tablesync worker?

2020-12-07 Thread Amit Kapila
On Mon, Dec 7, 2020 at 2:21 PM Amit Kapila wrote: > > On Mon, Dec 7, 2020 at 9:21 AM Amit Kapila wrote: > > > > On Mon, Dec 7, 2020 at 6:20 AM Craig Ringer > > wrote: > > > > > > > >> > > >> I am not sure why but it seems acceptable to original authors that the > > >> data of transactions are

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-12-07 Thread Kyotaro Horiguchi
At Tue, 8 Dec 2020 08:08:25 +0530, Amit Kapila wrote in > On Tue, Dec 8, 2020 at 7:24 AM Kyotaro Horiguchi > wrote: > > We drop > > buffers for the old relfilenode on truncation anyway. > > > > What I did is: > > > > a: Create a physical replication pair. > > b: On the master, create a table.

About to add WAL write/fsync statistics to pg_stat_wal view

2020-12-07 Thread Masahiro Ikeda
Hi, I propose to add wal write/fsync statistics to pg_stat_wal view. It's useful not only for developing/improving source code related to WAL but also for users to detect workload changes, HW failure, and so on. I introduce "track_wal_io_timing" parameter and provide the following information

Re: Blocking I/O, async I/O and io_uring

2020-12-07 Thread Craig Ringer
On Tue, 8 Dec 2020 at 12:02, Andres Freund wrote: > Hi, > > On 2020-12-08 10:55:37 +0800, Craig Ringer wrote: > > A new kernel API called io_uring has recently come to my attention. I > > assume some of you (Andres?) have been following it for a while. > > Yea, I've spent a *lot* of time working

Re: pg_rewind race condition just after promotion

2020-12-07 Thread Kyotaro Horiguchi
At Mon, 7 Dec 2020 20:13:25 +0200, Heikki Linnakangas wrote in > There's a race condition between the checkpoint at promotion and > pg_rewind. When a server is promoted, the startup process writes an > end-of-recovery checkpoint that includes the new TLI, and the server > is immediate opened

Re: On login trigger: take three

2020-12-07 Thread Pavel Stehule
út 8. 12. 2020 v 1:17 odesílatel Greg Nancarrow napsal: > On Fri, Dec 4, 2020 at 9:05 PM Konstantin Knizhnik > wrote: > > > > As far as I understand Pavel concern was about the case when superuser > > defines wrong login trigger which prevents login to the system > > all user including himself.

RE: Blocking I/O, async I/O and io_uring

2020-12-07 Thread tsunakawa.ta...@fujitsu.com
From: Andres Freund > Especially with direct IO > checkpointing can be a lot faster *and* less impactful on the "regular" > load. I'm looking forward to this from the async+direct I/O, since the throughput of some write-heavy workload decreased by half or more during checkpointing (due to

Re: Blocking I/O, async I/O and io_uring

2020-12-07 Thread Andres Freund
Hi, On 2020-12-08 10:55:37 +0800, Craig Ringer wrote: > A new kernel API called io_uring has recently come to my attention. I > assume some of you (Andres?) have been following it for a while. Yea, I've spent a *lot* of time working on AIO support, utilizing io_uring. Recently Thomas also joined

Re: Huge memory consumption on partitioned table with FKs

2020-12-07 Thread Amit Langote
On Tue, Dec 8, 2020 at 12:04 PM Kyotaro Horiguchi wrote: > At Tue, 8 Dec 2020 01:16:00 +0900, Amit Langote > wrote in > > Hi Alvaro, > > > > On Mon, Dec 7, 2020 at 23:48 Alvaro Herrera wrote: > > > > > On 2020-Dec-07, Amit Langote wrote: > > > > > > > On Fri, Dec 4, 2020 at 12:05 PM Kyotaro

Re: Blocking I/O, async I/O and io_uring

2020-12-07 Thread Andreas Karlsson
On 12/8/20 3:55 AM, Craig Ringer wrote: A new kernel API called io_uring has recently come to my attention. I assume some of you (Andres?) have been following it for a while. Andres did a talk on this at FOSDEM PGDay earlier this year. You can see his slides below, but since they are from

Re: upcoming API changes for LLVM 12

2020-12-07 Thread Andres Freund
Hi, On 2020-11-09 20:13:43 -0800, Andres Freund wrote: > I pushed the change to master. If that doesn't show any problems, I'll > backpatch in a week or so. Seawasp runs only on master, so it should > satisfy the buildfarm at least. It was a bit longer than a week, but I finally have done so...

Re: PG vs LLVM 12 on seawasp, next round

2020-12-07 Thread Andres Freund
Hi, On 2020-12-01 12:08:10 -0800, Andres Freund wrote: > On 2020-12-01 21:04:44 +0100, Fabien COELHO wrote: > > Andres investigated a few days ago, managed to reproduce the issue locally, > > and has one line patch. I'm unsure if it should be prevently back-patched, > > though. > > I see no

Re: vac_update_datfrozenxid will raise "wrong tuple length" if pg_database tuple contains toast attribute.

2020-12-07 Thread Michael Paquier
On Mon, Nov 30, 2020 at 04:37:14PM -0800, Ashwin Agrawal wrote: > I am thinking if there is some way to assert this aspect, but seems no way. > So, yes, having at least a comment is good for now. Yeah, I have been thinking about this one without coming to a clear idea, but that would be nice. >

Re: Blocking I/O, async I/O and io_uring

2020-12-07 Thread Thomas Munro
On Tue, Dec 8, 2020 at 3:56 PM Craig Ringer wrote: > I thought I'd start the discussion on this and see where we can go with it. > What incremental steps can be done to move us toward parallelisable I/O > without having to redesign everything? > > I'm thinking that redo is probably a good first

RE: [bug fix] ALTER TABLE SET LOGGED/UNLOGGED on a partitioned table does nothing silently

2020-12-07 Thread tsunakawa.ta...@fujitsu.com
From: Alvaro Herrera > Does "ALTER TABLE ONLY parent" work correctly? Namely, do not affect > existing partitions, but cause future partitions to acquire the new > setting. Yes, it works correctly in the sense that ALTER TABLE ONLY on a partitioned table does nothing because it has no storage

RE: Stronger safeguard for archive recovery not to miss data

2020-12-07 Thread osumi.takami...@fujitsu.com
Hi On Thursday, November 26, 2020 4:29 PM Kyotaro Horiguchi wrote: > At Thu, 26 Nov 2020 07:18:39 +, "osumi.takami...@fujitsu.com" > wrote in > > The attached patch is intended to prevent a scenario that archive > > recovery hits WALs which come from wal_level=minimal and the server > >

Re: Huge memory consumption on partitioned table with FKs

2020-12-07 Thread Kyotaro Horiguchi
At Tue, 8 Dec 2020 01:16:00 +0900, Amit Langote wrote in > Hi Alvaro, > > On Mon, Dec 7, 2020 at 23:48 Alvaro Herrera wrote: > > > On 2020-Dec-07, Amit Langote wrote: > > > > > On Fri, Dec 4, 2020 at 12:05 PM Kyotaro Horiguchi > > > wrote: > > > > > Also, the comment that was in

Re: Blocking I/O, async I/O and io_uring

2020-12-07 Thread Craig Ringer
References to get things started: * https://lwn.net/Articles/810414/ * https://unixism.net/loti/what_is_io_uring.html * https://blogs.oracle.com/linux/an-introduction-to-the-io_uring-asynchronous-io-framework * https://thenewstack.io/how-io_uring-and-ebpf-will-revolutionize-programming-in-linux/

Blocking I/O, async I/O and io_uring

2020-12-07 Thread Craig Ringer
Hi all A new kernel API called io_uring has recently come to my attention. I assume some of you (Andres?) have been following it for a while. io_uring appears to offer a way to make system calls including reads, writes, fsync()s, and more in a non-blocking, batched and pipelined manner, with or

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-12-07 Thread Zhihong Yu
>> just removing the logic that has the soft upper limit and just have it do cache evictions after each allocation after the cache first fills Yeah - having one fewer limit would simplify the code. Cheers On Mon, Dec 7, 2020 at 5:27 PM David Rowley wrote: > On Mon, 7 Dec 2020 at 14:25,

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-12-07 Thread Amit Kapila
On Tue, Dec 8, 2020 at 7:24 AM Kyotaro Horiguchi wrote: > > I'm out of it more than usual.. > > At Tue, 08 Dec 2020 09:45:53 +0900 (JST), Kyotaro Horiguchi > wrote in > > At Mon, 7 Dec 2020 17:18:31 +0530, Amit Kapila > > wrote in > > > On Mon, Dec 7, 2020 at 12:32 PM k.jami...@fujitsu.com >

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-12-07 Thread Kyotaro Horiguchi
I'm out of it more than usual.. At Tue, 08 Dec 2020 09:45:53 +0900 (JST), Kyotaro Horiguchi wrote in > At Mon, 7 Dec 2020 17:18:31 +0530, Amit Kapila > wrote in > > On Mon, Dec 7, 2020 at 12:32 PM k.jami...@fujitsu.com > > wrote: > > > > > > On Friday, December 4, 2020 8:27 PM, Amit Kapila

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-12-07 Thread Amit Kapila
On Tue, Dec 8, 2020 at 6:23 AM Kyotaro Horiguchi wrote: > > At Tue, 08 Dec 2020 09:45:53 +0900 (JST), Kyotaro Horiguchi > wrote in > > At Mon, 7 Dec 2020 17:18:31 +0530, Amit Kapila > > wrote in > > > Hmm, how is it possible if Insert is done before Truncate? The insert > > > should happen in

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-12-07 Thread David Rowley
On Mon, 7 Dec 2020 at 14:25, Zhihong Yu wrote: > > > + /* Make a guess at a good size when we're not given a valid size. */ > > + if (size == 0) > > + size = 1024; > > > > Should the default size be logged ? > > > I'm not too sure if I know what you mean here. Should it be a power of >

RE: [bug fix] ALTER TABLE SET LOGGED/UNLOGGED on a partitioned table does nothing silently

2020-12-07 Thread tsunakawa.ta...@fujitsu.com
From: Bharath Rupireddy > Do you mean to say that if we detach all the partitions(assuming they > are all unlogged) then the parent table(assuming logged) gets changed > to unlogged? Does it happen on master? Am I missing something here? No, the parent remains logged in that case both on master

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-12-07 Thread Kyotaro Horiguchi
At Tue, 08 Dec 2020 09:45:53 +0900 (JST), Kyotaro Horiguchi wrote in > At Mon, 7 Dec 2020 17:18:31 +0530, Amit Kapila > wrote in > > Hmm, how is it possible if Insert is done before Truncate? The insert > > should happen in old RelFileNode only. I have verified by adding a > > break-in

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-12-07 Thread Kyotaro Horiguchi
At Mon, 7 Dec 2020 17:18:31 +0530, Amit Kapila wrote in > On Mon, Dec 7, 2020 at 12:32 PM k.jami...@fujitsu.com > wrote: > > > > On Friday, December 4, 2020 8:27 PM, Amit Kapila > > wrote: > > Hi, > > I have reported before that it is not always the case that the "cached" > > flag of > >

Re: On login trigger: take three

2020-12-07 Thread Greg Nancarrow
On Fri, Dec 4, 2020 at 9:05 PM Konstantin Knizhnik wrote: > > As far as I understand Pavel concern was about the case when superuser > defines wrong login trigger which prevents login to the system > all user including himself. Right now solution of this problem is to > include "options='-c

Re: range_agg

2020-12-07 Thread Alvaro Herrera
On 2020-Dec-08, Alexander Korotkov wrote: > I also found a problem in multirange types naming logic. Consider the > following example. > > create type a_multirange AS (x float, y float); > create type a as range(subtype=text, collation="C"); > create table tbl (x __a_multirange); > drop type

Re: range_agg

2020-12-07 Thread Alexander Korotkov
On Mon, Nov 30, 2020 at 11:39 PM Alexander Korotkov wrote: > On Mon, Nov 30, 2020 at 10:35 PM Alexander Korotkov > wrote: > > On Sun, Nov 29, 2020 at 11:53 PM Paul A Jungwirth > > wrote: > > > > > > On Sun, Nov 29, 2020 at 11:43 AM Alexander Korotkov > > > wrote: > > > > Thank you. Could you

Re: Consider parallel for lateral subqueries with limit

2020-12-07 Thread James Coleman
Brian's email didn't keep the relevant headers, and so didn't show up as a reply, so I've pasted it here and am replying in this original thread. You can find the original at [1]. On Sun, Dec 6, 2020 at 7:34 PM Brian Davis wrote: > > > Note that near the end of grouping planner we have a similar

Re: Add header support to text format and matching feature

2020-12-07 Thread Rémi Lapeyre
Hi, here’s a rebased version of the patch. Best regards, Rémi v7-0001-Add-header-support-to-COPY-TO-text-format.patch Description: Binary data v7-0002-Add-header-matching-mode-to-COPY-FROM.patch Description: Binary data > On 21 Oct 2020, at 19:49, Daniel Verite wrote: > > Rémi

Re: Commitfest statistics

2020-12-07 Thread Alvaro Herrera
On 2020-Dec-07, Tom Lane wrote: > Anastasia Lubennikova writes: > > Firstly, we use it to track patches that we want to see in the nearest > > releases and concentrate our efforts on. And current CFM guideline [1] > > reflects this idea. It suggests, that after the commitfest closure date >

Re: Minor documentation error regarding streaming replication protocol

2020-12-07 Thread Jeff Davis
On Thu, 2020-12-03 at 13:14 -0500, Bruce Momjian wrote: > How do we mandate that? Just mention it in the docs and C comments? Can't we just throw an error in pg_create_restore_point() if any high bits are set? And perhaps error or Assert at a later point in case we change the code in such a way

Re: [HACKERS] [PATCH] Generic type subscripting

2020-12-07 Thread Andres Freund
Hi, On 2020-12-07 17:25:41 -0500, Tom Lane wrote: > Fair enough. It wasn't entirely clear to me whether it'd be kosher to > write > EEO_CASE(EEOP_SBSREF_OLD) > EEO_CASE(EEOP_SBSREF_ASSIGN) > EEO_CASE(EEOP_SBSREF_FETCH) > { >

Re: [HACKERS] [PATCH] Generic type subscripting

2020-12-07 Thread Tom Lane
Andres Freund writes: > On 2020-12-07 16:32:32 -0500, Tom Lane wrote: >> What did you think of the idea of merging EEOP_SBSREF_OLD / ASSIGN / FETCH >> into a single step type distinguished only by the callback function? > I don't have a strong opinion on this. I guess find it a bit easier to >

Re: [HACKERS] [PATCH] Generic type subscripting

2020-12-07 Thread Andres Freund
Hi, On 2020-12-07 16:32:32 -0500, Tom Lane wrote: > Andres Freund writes: > > I think it'd be a better to rely on the backend's definition of > > ExecEvalBoolSubroutine etc. For the functions implementing expression > > steps I've found that far easier to work with over time (because you can > >

Re: Commitfest statistics

2020-12-07 Thread Tom Lane
Anastasia Lubennikova writes: > I want to share some stats and thoughts about CF. First, thanks again for managing this CF! > The first is a graph with the numbers of committed, moved, returned, and > rejected CF patches over time - [cf_items_status.png]. Credits to Dmitry > Dolgov for

Re: Add table access method as an option to pgbench

2020-12-07 Thread David Zhang
Again, thanks a lot for the feedback. On 2020-12-02 12:03 p.m., Fabien COELHO wrote: Hello David, Some feedback about v4. It looks that the option is *silently* ignored when creating a partitionned table, which currently results in an error, and not passed to partitions, which would accept

Re: [HACKERS] [PATCH] Generic type subscripting

2020-12-07 Thread Tom Lane
Andres Freund writes: > The TypeParamBool stuff here is ok. Basically LLVM uses a '1bit' integer > to represent booleans in the IR. But when it comes to storing such a > value in memory, it uses 1 byte, for obvious reasons. Hence the two > types. Cool, thanks for taking a look. > I think it'd

Re: [HACKERS] [PATCH] Generic type subscripting

2020-12-07 Thread Andres Freund
Hi, On 2020-12-07 14:08:35 -0500, Tom Lane wrote: > 1. I'm still wondering if TypeParamBool is the right thing to pass to > LLVMFunctionType() to describe a function-returning-bool. It does > seem to work on x64_64 and aarch64, for what that's worth. > -

small cleanup in unicode_norm.c

2020-12-07 Thread John Naylor
We've had get_canonical_class() for a while as a backend-only function. There is some ad-hoc code elsewhere that implements the same logic in a couple places, so it makes sense for all sites to use this function instead, as in the attached. -- John Naylor EnterpriseDB:

Re: Change default of checkpoint_completion_target

2020-12-07 Thread Stephen Frost
Greetings, * Peter Eisentraut (peter.eisentr...@enterprisedb.com) wrote: > On 2020-12-07 18:53, Stephen Frost wrote: > >* Michael Paquier (mich...@paquier.xyz) wrote: > >>On Sun, Dec 06, 2020 at 10:03:08AM -0500, Stephen Frost wrote: > >>>* Alvaro Herrera (alvhe...@alvh.no-ip.org) wrote: >

Re: [HACKERS] [PATCH] Generic type subscripting

2020-12-07 Thread Tom Lane
I decided that the way to get this moved forward is to ignore the jsonb parts for the moment and focus on getting the core feature into committable shape. It's possible that the lack of a concrete use-case other than arrays will cause us to miss a detail or two, but if so we can fix it later, I

Re: Change default of checkpoint_completion_target

2020-12-07 Thread Peter Eisentraut
On 2020-12-07 18:53, Stephen Frost wrote: * Michael Paquier (mich...@paquier.xyz) wrote: On Sun, Dec 06, 2020 at 10:03:08AM -0500, Stephen Frost wrote: * Alvaro Herrera (alvhe...@alvh.no-ip.org) wrote: You keep making this statement, and I don't necessarily disagree, but if that is the case,

Re: Yet another fast GiST build

2020-12-07 Thread Peter Geoghegan
On Mon, Dec 7, 2020 at 2:05 AM Andrey Borodin wrote: > Here's version with tests and docs. I still have no idea how to print some > useful information about tuples keys. I suggest calling BuildIndexValueDescription() from your own custom debug instrumentation code. -- Peter Geoghegan

pg_rewind race condition just after promotion

2020-12-07 Thread Heikki Linnakangas
There's a race condition between the checkpoint at promotion and pg_rewind. When a server is promoted, the startup process writes an end-of-recovery checkpoint that includes the new TLI, and the server is immediate opened for business. The startup process requests the checkpointer process to

Change default of checkpoint_completion_target

2020-12-07 Thread Stephen Frost
Greetings, * Michael Paquier (mich...@paquier.xyz) wrote: > On Sun, Dec 06, 2020 at 10:03:08AM -0500, Stephen Frost wrote: > > * Alvaro Herrera (alvhe...@alvh.no-ip.org) wrote: > >> You keep making this statement, and I don't necessarily disagree, but if > >> that is the case, please explain why

Re: POC: Better infrastructure for automated testing of concurrency issues

2020-12-07 Thread Alexander Korotkov
Hi! On Mon, Dec 7, 2020 at 9:10 AM Craig Ringer wrote: > On Wed, 25 Nov 2020 at 22:11, Alexander Korotkov wrote: >> PostgreSQL is a complex multi-process system, and we are periodically faced >> with complicated concurrency issues. While the postgres community does a >> great job on

Re: Huge memory consumption on partitioned table with FKs

2020-12-07 Thread Amit Langote
Hi Alvaro, On Mon, Dec 7, 2020 at 23:48 Alvaro Herrera wrote: > On 2020-Dec-07, Amit Langote wrote: > > > On Fri, Dec 4, 2020 at 12:05 PM Kyotaro Horiguchi > > wrote: > > > > Also, the comment that was in RI_ConstraintInfo now appears in > > > > RI_ConstraintParam, and the new struct

Re: Additional improvements to extended statistics

2020-12-07 Thread Dean Rasheed
On Wed, 2 Dec 2020 at 15:51, Dean Rasheed wrote: > > The sort of queries I had in mind were things like this: > > WHERE (a = 1 AND b = 1) OR (a = 2 AND b = 2) > > However, the new code doesn't apply the extended stats directly using > clauselist_selectivity_or() for this kind of query because

Re: PoC/WIP: Extended statistics on expressions

2020-12-07 Thread Dean Rasheed
On Mon, 7 Dec 2020 at 14:15, Tomas Vondra wrote: > > On 12/7/20 10:56 AM, Dean Rasheed wrote: > > it might actually be > > neater to have separate documented syntaxes for single- and > > multi-column statistics: > > > > CREATE STATISTICS [ IF NOT EXISTS ] statistics_name > > ON (expression)

Re: [bug fix] ALTER TABLE SET LOGGED/UNLOGGED on a partitioned table does nothing silently

2020-12-07 Thread Alvaro Herrera
Does "ALTER TABLE ONLY parent" work correctly? Namely, do not affect existing partitions, but cause future partitions to acquire the new setting. This sounds very much related to previous discussion on REPLICA IDENTITY not propagating to partitions; see

Re: Parallel Inserts in CREATE TABLE AS

2020-12-07 Thread Dilip Kumar
On Mon, Dec 7, 2020 at 7:04 PM Bharath Rupireddy wrote: > > On Mon, Dec 7, 2020 at 5:25 PM Amit Kapila wrote: > > > > On Mon, Dec 7, 2020 at 4:20 PM Bharath Rupireddy > > wrote: > > > > > > On Mon, Dec 7, 2020 at 4:04 PM Amit Kapila > > > wrote: > > > > > > > > What is the need of checking

Re: Huge memory consumption on partitioned table with FKs

2020-12-07 Thread Alvaro Herrera
On 2020-Dec-07, Amit Langote wrote: > On Fri, Dec 4, 2020 at 12:05 PM Kyotaro Horiguchi > wrote: > > > Also, the comment that was in RI_ConstraintInfo now appears in > > > RI_ConstraintParam, and the new struct (RI_ConstraintInfo) is now > > > undocumented. What is the relationship between

Re: PoC/WIP: Extended statistics on expressions

2020-12-07 Thread Tomas Vondra
On 12/7/20 10:56 AM, Dean Rasheed wrote: > On Thu, 3 Dec 2020 at 15:23, Tomas Vondra > wrote: >> >> Attached is a patch series rebased on top of 25a9e54d2d. > > After reading this thread and [1], I think I prefer the name > "standard" rather than "expressions", because it is meant to

Re: Huge memory consumption on partitioned table with FKs

2020-12-07 Thread Amit Langote
On Fri, Dec 4, 2020 at 12:05 PM Kyotaro Horiguchi wrote: > > Also, the comment that was in RI_ConstraintInfo now appears in > > RI_ConstraintParam, and the new struct (RI_ConstraintInfo) is now > > undocumented. What is the relationship between those two structs? I > > see that they have

PoC Refactor AM analyse API

2020-12-07 Thread Смирнов Денис
Hello all!I suggest a refactoring of analyze AM API as it is too much heap specific at the moment. The problem was inspired by Greenplum’s analyze improvement for append-optimized row and column AM with variable size compressed blocks.Currently we do analyze in two steps.1. Sample fix size blocks

Re: Parallel Inserts in CREATE TABLE AS

2020-12-07 Thread Bharath Rupireddy
On Mon, Dec 7, 2020 at 5:25 PM Amit Kapila wrote: > > On Mon, Dec 7, 2020 at 4:20 PM Bharath Rupireddy > wrote: > > > > On Mon, Dec 7, 2020 at 4:04 PM Amit Kapila wrote: > > > > > > What is the need of checking query_level when 'isForCTAS' is set only > > > when Gather is a top-node? > > > > >

Re: Refactor MD5 implementations and switch to EVP for OpenSSL

2020-12-07 Thread Daniel Gustafsson
> On 4 Dec 2020, at 08:05, Michael Paquier wrote: > > On Tue, Nov 10, 2020 at 01:28:09PM +0900, Michael Paquier wrote: >> The CF bot has been complaining on Windows and this issue is fixed in >> the attached. A refresh of src/tools/msvc for pgcrypto was just >> missing. > > Now that HEAD has

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-12-07 Thread Ashutosh Bapat
On Sun, Dec 6, 2020 at 9:09 AM Andy Fan wrote: >> >> I have not been following this thread closely enough to understand >> why we need a new "UniqueKeys" data structure at all. > > > Currently the UniqueKey is defined as a List of Expr, rather than > EquivalenceClasses. > A complete discussion

initscan for MVCC snapshot

2020-12-07 Thread Andy Fan
Hi: I see initscan calls RelationGetwNumberOfBlocks every time and rescan calls initscan as well. In my system, RelationGetNumberOfBlocks is expensive (the reason doesn't deserve a talk.. ), so in a nest loop + Bitmap heap scan case, the impact will be huge. The comments of initscan are below.

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-12-07 Thread Andy Fan
On Mon, Dec 7, 2020 at 4:16 PM Jesper Pedersen wrote: > Hi, > > On 12/5/20 10:38 PM, Andy Fan wrote: > > Currently the UniqueKey is defined as a List of Expr, rather than > > EquivalenceClasses. > > A complete discussion until now can be found at [1] (The messages I > replied > > to also > >

Re: Parallel Inserts in CREATE TABLE AS

2020-12-07 Thread Amit Kapila
On Mon, Dec 7, 2020 at 4:20 PM Bharath Rupireddy wrote: > > On Mon, Dec 7, 2020 at 4:04 PM Amit Kapila wrote: > > > > What is the need of checking query_level when 'isForCTAS' is set only > > when Gather is a top-node? > > > > isForCTAS is getting set before pg_plan_query() which is being used

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-12-07 Thread Amit Kapila
On Mon, Dec 7, 2020 at 12:32 PM k.jami...@fujitsu.com wrote: > > On Friday, December 4, 2020 8:27 PM, Amit Kapila > wrote: > > On Fri, Nov 27, 2020 at 11:36 AM Kyotaro Horiguchi > > wrote: > > > > > > At Fri, 27 Nov 2020 02:19:57 +, "k.jami...@fujitsu.com" > > > wrote in > > > > > From:

Re: [bug fix] ALTER TABLE SET LOGGED/UNLOGGED on a partitioned table does nothing silently

2020-12-07 Thread Bharath Rupireddy
On Mon, Dec 7, 2020 at 11:36 AM tsunakawa.ta...@fujitsu.com wrote: > > From: Bharath Rupireddy > > IMHO, we should also change the parent table. Say, I have 2 local > > partitions for a logged table, then I alter that table to > > unlogged(with your patch, parent table doesn't become unlogged

Re: Parallel Inserts in CREATE TABLE AS

2020-12-07 Thread Amit Kapila
On Mon, Dec 7, 2020 at 3:44 PM Bharath Rupireddy wrote: > > On Mon, Dec 7, 2020 at 2:55 PM Amit Kapila wrote: > > > > On Mon, Dec 7, 2020 at 11:32 AM Hou, Zhijie > > wrote: > > > > > > + if (!(root->parse->isForCTAS && > > > + root->query_level == 1)) > > > +

Re: Add primary keys to system catalogs

2020-12-07 Thread Peter Eisentraut
On 2020-10-03 08:40, Peter Eisentraut wrote: Since we have ADD PRIMARY KEY USING INDEX, we can declare a primary key for an existing index. So this doesn't have to affect the low-level early bootstrapping. The attached patch adds those commands manually. Another option might be to have the

Re: Parallel Inserts in CREATE TABLE AS

2020-12-07 Thread Bharath Rupireddy
On Mon, Dec 7, 2020 at 2:55 PM Amit Kapila wrote: > > On Mon, Dec 7, 2020 at 11:32 AM Hou, Zhijie wrote: > > > > Hi > > > > + /* > > +* Flag to let the planner know that the SELECT query is for CTAS. > > This is > > +* used to calculate the tuple transfer cost from workers

Re: RFC: Deadlock detector hooks for victim selection and edge injection

2020-12-07 Thread Amit Kapila
On Mon, Dec 7, 2020 at 9:25 AM Craig Ringer wrote: > > Hi folks > > Now that we're well on track for streaming logical decoding, it's becoming > more practical to look at parallel logical apply. > > The support for this in pglogical3 benefits from a deadlock detector hook. It > was added in the

get_constraint_index() and conindid

2020-12-07 Thread Peter Eisentraut
get_constraint_index() does its work by going through pg_depend. It was added before pg_constraint.conindid was added, and some callers are still not changed. Are there reasons for that? Probably not. The attached patch changes get_constraint_index() to an lsyscache-style lookup instead.

Re: Yet another fast GiST build

2020-12-07 Thread Andrey Borodin
> 28 сент. 2020 г., в 13:12, Heikki Linnakangas написал(а): > > I wrote a couple of 'pageinspect' function to inspect GiST pages for this. > See attached. > <0001-Add-functions-to-pageinspect-to-inspect-GiST-indexes.patch> Here's version with tests and docs. I still have no idea how to print

Re: PoC/WIP: Extended statistics on expressions

2020-12-07 Thread Dean Rasheed
On Thu, 3 Dec 2020 at 15:23, Tomas Vondra wrote: > > Attached is a patch series rebased on top of 25a9e54d2d. After reading this thread and [1], I think I prefer the name "standard" rather than "expressions", because it is meant to describe the kind of statistics being built rather than what

RE: Parallel copy

2020-12-07 Thread Hou, Zhijie
> > 4. > > A suggestion for CacheLineInfo. > > > > It use appendBinaryStringXXX to store the line in memory. > > appendBinaryStringXXX will double the str memory when there is no enough > spaces. > > > > How about call enlargeStringInfo in advance, if we already know the whole > line size? > > It

Re: Parallel Inserts in CREATE TABLE AS

2020-12-07 Thread Amit Kapila
On Mon, Dec 7, 2020 at 11:32 AM Hou, Zhijie wrote: > > Hi > > + /* > +* Flag to let the planner know that the SELECT query is for CTAS. > This is > +* used to calculate the tuple transfer cost from workers to gather > node(in > +* case parallelism kicks in for the

Re: Logical archiving

2020-12-07 Thread Amit Kapila
On Mon, Dec 7, 2020 at 8:35 AM Craig Ringer wrote: > > Reply follows inline. I addressed your last point first, so it's out of order. > > On Fri, 4 Dec 2020 at 15:33, Andrey Borodin wrote > > We'd also need to separate the existing apply worker into a "receiver" and > "apply/writer" part, so

Re: Single transaction in the tablesync worker?

2020-12-07 Thread Amit Kapila
On Mon, Dec 7, 2020 at 9:21 AM Amit Kapila wrote: > > On Mon, Dec 7, 2020 at 6:20 AM Craig Ringer > wrote: > > > > >> > >> I am not sure why but it seems acceptable to original authors that the > >> data of transactions are visibly partially during the initial > >> synchronization phase for a

Added missing copy related data structures to typedefs.list

2020-12-07 Thread vignesh C
Hi, Added missing copy related data structures to typedefs.list, these data structures were added while copy files were split during the recent commit. I found this while running pgindent for parallel copy patches. The Attached patch has the changes for the same. Thoughts? Regards, Vignesh

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-12-07 Thread Jesper Pedersen
Hi, On 12/5/20 10:38 PM, Andy Fan wrote: Currently the UniqueKey is defined as a List of Expr, rather than EquivalenceClasses. A complete discussion until now can be found at [1] (The messages I replied to also care a lot and the information is completed). This patch has stopped at this place