Re: Why are wait events not reported even though it reads/writes a timeline history file?

2020-04-30 Thread Fujii Masao
On 2020/05/01 10:07, Masahiro Ikeda wrote: On 2020-05-01 00:25, Fujii Masao wrote: On 2020/04/28 17:42, Masahiro Ikeda wrote: On 2020-04-28 15:09, Michael Paquier wrote: On Tue, Apr 28, 2020 at 02:49:00PM +0900, Fujii Masao wrote: Isn't it safer to report the wait event during fgets()

Re: SLRU statistics

2020-04-30 Thread Fujii Masao
On 2020/05/01 3:19, Tomas Vondra wrote: On Fri, May 01, 2020 at 03:02:59AM +0900, Fujii Masao wrote: On 2020/04/02 9:41, Tomas Vondra wrote: Hi, I've pushed this after some minor cleanup and improvements. +static char *slru_names[] = {"async", "clog", "commit_timestamp", +  

Re: Binary search in ScalarArrayOpExpr for OR'd constant arrays

2020-04-30 Thread James Coleman
On Tue, Apr 28, 2020 at 8:25 AM Tomas Vondra wrote: ... > Any particular reasons to pick dynahash over simplehash? ISTM we're > using simplehash elsewhere in the executor (grouping, tidbitmap, ...), > while there are not many places using dynahash for simple short-lived > hash tables. Of course,

Comment simplehash/dynahash trade-offs

2020-04-30 Thread James Coleman
In another thread [1] I'd mused that "there might be some value in a README or comments addition that would be a guide to what the various hash implementations are useful for...so that we have something to make the code base a bit more discoverable." I'd solicited feedback from Andres (as the

Re: do {} while (0) nitpick

2020-04-30 Thread Bruce Momjian
On Thu, Apr 30, 2020 at 09:51:10PM -0400, Tom Lane wrote: > John Naylor writes: > > As I understand it, the point of having "do {} while (0)" in a > > multi-statement macro is to turn it into a simple statement. > > Right. > > > As such, > > ending with a semicolon in both the macro definition

Re: do {} while (0) nitpick

2020-04-30 Thread Tom Lane
John Naylor writes: > As I understand it, the point of having "do {} while (0)" in a > multi-statement macro is to turn it into a simple statement. Right. > As such, > ending with a semicolon in both the macro definition and the > invocation will turn it back into multiple statements, creating

do {} while (0) nitpick

2020-04-30 Thread John Naylor
Hi, As I understand it, the point of having "do {} while (0)" in a multi-statement macro is to turn it into a simple statement. As such, ending with a semicolon in both the macro definition and the invocation will turn it back into multiple statements, creating confusion if someone were to invoke

Re: Why are wait events not reported even though it reads/writes a timeline history file?

2020-04-30 Thread Masahiro Ikeda
On 2020-05-01 00:25, Fujii Masao wrote: On 2020/04/28 17:42, Masahiro Ikeda wrote: On 2020-04-28 15:09, Michael Paquier wrote: On Tue, Apr 28, 2020 at 02:49:00PM +0900, Fujii Masao wrote: Isn't it safer to report the wait event during fgets() rather than putting those calls around the whole

Re: Raw device on PostgreSQL

2020-04-30 Thread Jonah H. Harris
On Wed, Apr 29, 2020 at 8:34 PM Jonah H. Harris wrote: > On Tue, Apr 28, 2020 at 8:10 AM Andreas Karlsson > wrote: > >> To get the performance benefits from using raw devices I think you would >> want to add support for asynchronous IO to PostgreSQL rather than >> implementing your own layer to

Re: design for parallel backup

2020-04-30 Thread Robert Haas
On Thu, Apr 30, 2020 at 3:52 PM Andres Freund wrote: > Why 8kb? That's smaller than what we currently do in pg_basebackup, > afaictl, and you're actually going to be bottlenecked by syscall > overhead at that point (unless you disable / don't have the whole intel > security mitigation stuff). I

Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options

2020-04-30 Thread Krasiyan Andreev
Thank you very much for feedback and yes, that is very useful SQL syntax. Maybe you miss my previous answer, but you are right, that patch is currently dead, because some important design questions must be discussed here, before patch rewriting. I have dropped support of from first/last for

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2020-04-30 Thread Thomas Munro
On Fri, May 1, 2020 at 2:30 AM Melanie Plageman wrote: > I made a few edits to the message and threw it into a draft patch (on > top of master, of course). I didn't want to junk up peoples' inboxes, so > I didn't start a separate thread, but, it will be pretty hard to > collaboratively edit the

Re: design for parallel backup

2020-04-30 Thread Andres Freund
Hi, On 2020-04-30 14:50:34 -0400, Robert Haas wrote: > On Mon, Apr 20, 2020 at 4:19 PM Andres Freund wrote: > > One question I have not really seen answered well: > > > > Why do we want parallelism here. Or to be more precise: What do we hope > > to accelerate by making what part of creating a

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2020-04-30 Thread Alvaro Herrera
On 2020-Apr-30, Melanie Plageman wrote: > On Tue, Apr 28, 2020 at 11:50 PM Heikki Linnakangas wrote: > > I haven't looked at the patch in detail, but thanks [...] > Thanks for taking a look, Heikki! Hmm. We don't have Heikki's message in the archives. In fact, the last message from Heikki

Re: Bug with subqueries in recursive CTEs?

2020-04-30 Thread Laurenz Albe
On Thu, 2020-04-30 at 04:37 +0100, Andrew Gierth wrote: > "Laurenz" == Laurenz Albe writes: > > Laurenz> I played with a silly example and got a result that surprises > Laurenz> me: > > Laurenz> WITH RECURSIVE fib AS ( > Laurenz> SELECT n, "fibₙ" > Laurenz> FROM (VALUES

Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options

2020-04-30 Thread Stephen Frost
Greetings, This seems to have died out, and that's pretty unfortunate because this is awfully useful SQL standard syntax that people look for and wish we had. * Andrew Gierth (and...@tao11.riddles.org.uk) wrote: > So I've tried to rough out a decision tree for the various options on > how this

Re: design for parallel backup

2020-04-30 Thread Robert Haas
On Mon, Apr 20, 2020 at 4:19 PM Andres Freund wrote: > One question I have not really seen answered well: > > Why do we want parallelism here. Or to be more precise: What do we hope > to accelerate by making what part of creating a base backup > parallel. There's several potential bottlenecks,

Re: SLRU statistics

2020-04-30 Thread Fujii Masao
On 2020/04/02 9:41, Tomas Vondra wrote: Hi, I've pushed this after some minor cleanup and improvements. +static char *slru_names[] = {"async", "clog", "commit_timestamp", + "multixact_offset", "multixact_member", +

Re: Proposing WITH ITERATIVE

2020-04-30 Thread Jeff Davis
On Tue, 2020-04-28 at 11:57 -0400, Jonah H. Harris wrote: > Yeah, in that specific case, one of the other implementations seems > to carry the counters along in the executor itself. But, as not all > uses of this functionality are iteration-count-based, I think that's > a little limiting. Using a

Re: Why are wait events not reported even though it reads/writes a timeline history file?

2020-04-30 Thread Fujii Masao
On 2020/04/28 17:42, Masahiro Ikeda wrote: On 2020-04-28 15:09, Michael Paquier wrote: On Tue, Apr 28, 2020 at 02:49:00PM +0900, Fujii Masao wrote: Isn't it safer to report the wait event during fgets() rather than putting those calls around the whole loop, like other code does? For

Re: +(pg_lsn, int8) and -(pg_lsn, int8) operators

2020-04-30 Thread Fujii Masao
On 2020/04/28 15:03, Michael Paquier wrote: On Tue, Apr 28, 2020 at 12:56:19PM +0900, Fujii Masao wrote: Yes. Attached is the updated version of the patch, which introduces +(pg_lsn, numeric) and -(pg_lsn, numeric) operators. To implement them, I added also numeric_pg_lsn() function that

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2020-04-30 Thread Melanie Plageman
On Tue, Apr 28, 2020 at 11:50 PM Heikki Linnakangas wrote: > On 29/04/2020 05:03, Melanie Plageman wrote: > > I've attached a patch which should address some of the previous feedback > > about code complexity. Two of my co-workers and I wrote what is > > essentially a new prototype of the idea.

Back-patch is necessary? Re: Don't try fetching future segment of a TLI.

2020-04-30 Thread Fujii Masao
On 2020/04/08 1:49, Fujii Masao wrote: On 2020/04/07 20:21, David Steele wrote: On 4/7/20 3:48 AM, Kyotaro Horiguchi wrote: At Tue, 7 Apr 2020 12:15:00 +0900, Fujii Masao wrote in This doesn't seem a bug, so I'm thinking to merge this to next *major* version release, i.e., v13. Not a

Re: WIP/PoC for parallel backup

2020-04-30 Thread Amit Kapila
On Thu, Apr 30, 2020 at 4:15 PM Amit Kapila wrote: > > On Wed, Apr 29, 2020 at 6:11 PM Suraj Kharage > wrote: > > > > Hi, > > > > We at EnterpriseDB did some performance testing around this parallel backup > > to check how this is beneficial and below are the results. In this testing, > > we

Postgres Windows build system doesn't work with python installed in Program Files

2020-04-30 Thread Victor Wagner
Collegues, Accidently I've come over minor bug in the Mkvcbuild.pm. It happens, that it doesn't tolerate spaces in the $config->{python} path, because it want to call python in order to find out version, prefix and so on, and doesn't properly quote command. Fix is very simple, see attach.

Re: Transactions involving multiple postgres foreign servers, take 2

2020-04-30 Thread Masahiko Sawada
On Tue, 28 Apr 2020 at 19:37, Muhammad Usama wrote: > > > > On Wed, Apr 8, 2020 at 11:16 AM Masahiko Sawada > wrote: >> >> On Fri, 27 Mar 2020 at 22:06, Muhammad Usama wrote: >> > >> > Hi Sawada San, >> > >> > I have been further reviewing and testing the transaction involving >> > multiple

Re: Optimization for hot standby XLOG_STANDBY_LOCK redo

2020-04-30 Thread Amit Kapila
On Thu, Apr 30, 2020 at 4:07 PM 邱宇航 wrote: > > I noticed that in hot standby, XLOG_STANDBY_LOCK redo is sometimes block by > another query, and all the rest redo is blocked by this lock getting > operation, which is not good and often happed in my database, so the hot > standby will be left

Re: WIP/PoC for parallel backup

2020-04-30 Thread Amit Kapila
On Wed, Apr 29, 2020 at 6:11 PM Suraj Kharage wrote: > > Hi, > > We at EnterpriseDB did some performance testing around this parallel backup > to check how this is beneficial and below are the results. In this testing, > we run the backup - > 1) Without Asif’s patch > 2) With Asif’s patch and

Optimization for hot standby XLOG_STANDBY_LOCK redo

2020-04-30 Thread 邱宇航
I noticed that in hot standby, XLOG_STANDBY_LOCK redo is sometimes block by another query, and all the rest redo is blocked by this lock getting operation, which is not good and often happed in my database, so the hot standby will be left behind and master will store a lot of WAL which can’t be

Re: WIP/PoC for parallel backup

2020-04-30 Thread Sumanta Mukherjee
Hi, Would it be possible to put in the absolute numbers of the perf so that it is easier to understand the amount of improvement with and without the patch and different loads and workers. I am also unsure why the swapper is taking such a huge percentage of the absolute time in the base run of

Re: WAL usage calculation patch

2020-04-30 Thread Julien Rouhaud
On Thu, Apr 30, 2020 at 9:18 AM Julien Rouhaud wrote: > > On Thu, Apr 30, 2020 at 5:05 AM Amit Kapila wrote: > > > > On Tue, Apr 28, 2020 at 7:38 AM Amit Kapila wrote: > > > > > > On Mon, Apr 27, 2020 at 1:22 PM Julien Rouhaud wrote: > > > > > > > > > > > I agree with that definition. I can

Re: Improve errors when setting incorrect bounds for SSL protocols

2020-04-30 Thread Daniel Gustafsson
> On 30 Apr 2020, at 01:14, Michael Paquier wrote: > > On Wed, Apr 29, 2020 at 01:57:49PM +0200, Daniel Gustafsson wrote: >> Working in the TLS corners of the backend, I found while re-reviewing and >> re-testing for the release that this patch actually was a small, but vital, >> brick shy of a

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2020-04-30 Thread David Kimura
On Wed, Apr 29, 2020 at 4:39 PM Melanie Plageman wrote: > > In addition to many assorted TODOs in the code, there are a few major > projects left: > - Batch 0 falling back > - Stripe barrier deadlock > - Performance improvements and testing > Batch 0 never spills. That behavior is an artifact

Re: Fixes for two separate bugs in nbtree VACUUM's page deletion

2020-04-30 Thread Masahiko Sawada
On Thu, 30 Apr 2020 at 07:29, Peter Geoghegan wrote: > > On Tue, Apr 28, 2020 at 12:21 AM Masahiko Sawada > wrote: > > For the first fix it seems better to push down the logic to the page > > deletion code as your 0001 patch does so. The following change changes > > the page deletion code so

Re: WAL usage calculation patch

2020-04-30 Thread Julien Rouhaud
On Thu, Apr 30, 2020 at 5:05 AM Amit Kapila wrote: > > On Tue, Apr 28, 2020 at 7:38 AM Amit Kapila wrote: > > > > On Mon, Apr 27, 2020 at 1:22 PM Julien Rouhaud wrote: > > > > > > > > I agree with that definition. I can send a cleanup patch if there's > > > no objection. > > > > > > > Okay,

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-04-30 Thread Amit Kapila
On Wed, Apr 29, 2020 at 3:19 PM Dilip Kumar wrote: > > On Wed, Apr 29, 2020 at 2:56 PM Dilip Kumar wrote: > > > > On Tue, Apr 28, 2020 at 3:55 PM Dilip Kumar wrote: > > > > > > On Tue, Apr 28, 2020 at 3:11 PM Amit Kapila > > > wrote: > > > > > > > > On Mon, Apr 27, 2020 at 4:05 PM Dilip Kumar

Re: Autovacuum on partitioned table (autoanalyze)

2020-04-30 Thread Amit Langote
On Sat, Apr 25, 2020 at 11:13 PM Justin Pryzby wrote: > > On Wed, Mar 18, 2020 at 11:30:39AM -0500, Justin Pryzby wrote: > > In the past, I think there's was talk that maybe someone would invent a > > clever > > way to dynamically combine all the partitions' statistics, so analyzing the > >

Re: Fixes for two separate bugs in nbtree VACUUM's page deletion

2020-04-30 Thread Masahiko Sawada
On Wed, 29 Apr 2020 at 01:17, Peter Geoghegan wrote: > > On Tue, Apr 28, 2020 at 12:21 AM Masahiko Sawada > wrote: > > I agree with both patches. > > Thanks for the review. > > > For the first fix it seems better to push down the logic to the page > > deletion code as your 0001 patch does so.

Re: Proposing WITH ITERATIVE

2020-04-30 Thread Fabien COELHO
Hello, more random thoughts about syntax, semantics, and keeping it relational. While I'm not a huge fan of it, one of the other databases implementing this functionality does so using the syntax: WITH ITERATIVE R AS '(' R0 ITERATE Ri UNTIL N (ITERATIONS | UPDATES) ')' Qf Where N in