Re: [HACKERS] WAL logging problem in 9.4.3?

2019-05-24 Thread Noah Misch
On Mon, May 20, 2019 at 03:54:30PM +0900, Kyotaro HORIGUCHI wrote: > Following this direction, the attached PoC works *at least for* > the wal_optimization TAP tests, but doing pending flush not in > smgr but in relcache. This task, syncing files created in the current transaction, is not the kind

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-05-23 Thread Kyotaro HORIGUCHI
Attached is a new version. At Tue, 21 May 2019 21:29:48 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20190521.212948.34357392.horiguchi.kyot...@lab.ntt.co.jp> > At Mon, 20 May 2019 15:54:30 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI > wrote in > <20190520.155430.215084510.ho

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-05-21 Thread Kyotaro HORIGUCHI
Hello. At Mon, 20 May 2019 15:54:30 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20190520.155430.215084510.horiguchi.kyot...@lab.ntt.co.jp> > > I suspect the design in the https://postgr.es/m/559fa0ba.3080...@iki.fi last > > paragraph will be simpler, not more complex. In the imple

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-05-19 Thread Kyotaro HORIGUCHI
Hello. At Thu, 16 May 2019 23:50:50 -0700, Noah Misch wrote in <20190517065050.ga1298...@rfd.leadboat.com> > On Tue, May 14, 2019 at 01:59:10PM +0900, Kyotaro HORIGUCHI wrote: > > At Sun, 12 May 2019 17:37:05 -0700, Noah Misch wrote in > > <20190513003705.ga1202...@rfd.leadboat.com> > > > On S

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-05-16 Thread Noah Misch
On Tue, May 14, 2019 at 01:59:10PM +0900, Kyotaro HORIGUCHI wrote: > At Sun, 12 May 2019 17:37:05 -0700, Noah Misch wrote in > <20190513003705.ga1202...@rfd.leadboat.com> > > On Sun, Mar 31, 2019 at 03:31:58PM -0700, Noah Misch wrote: > > > On Sun, Mar 10, 2019 at 07:27:08PM -0700, Noah Misch wro

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-05-13 Thread Kyotaro HORIGUCHI
Hello. At Sun, 12 May 2019 17:37:05 -0700, Noah Misch wrote in <20190513003705.ga1202...@rfd.leadboat.com> > On Sun, Mar 31, 2019 at 03:31:58PM -0700, Noah Misch wrote: > > On Sun, Mar 10, 2019 at 07:27:08PM -0700, Noah Misch wrote: > > > I also liked the design in the https://postgr.es/m/559fa0

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-05-12 Thread Noah Misch
On Sun, Mar 31, 2019 at 03:31:58PM -0700, Noah Misch wrote: > On Sun, Mar 10, 2019 at 07:27:08PM -0700, Noah Misch wrote: > > I also liked the design in the https://postgr.es/m/559fa0ba.3080...@iki.fi > > last paragraph, and I suspect it would have been no harder to back-patch. I > > wonder if it

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-04-04 Thread Kyotaro HORIGUCHI
At Thu, 4 Apr 2019 10:52:59 -0400, Robert Haas wrote in > On Wed, Apr 3, 2019 at 10:03 PM Kyotaro HORIGUCHI > wrote: > > > * Insert log record, using delete or insert instead of update log > > > * when only one of the two buffers needs WAL-logging. If this were a > > > * HOT-update, redoing the

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-04-04 Thread Robert Haas
On Wed, Apr 3, 2019 at 10:03 PM Kyotaro HORIGUCHI wrote: > > * Insert log record, using delete or insert instead of update log > > * when only one of the two buffers needs WAL-logging. If this were a > > * HOT-update, redoing the WAL record would result in a broken > > * hot-chain. However, that n

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-04-03 Thread Kyotaro HORIGUCHI
Thank you for looking this. At Wed, 3 Apr 2019 10:16:02 -0400, Robert Haas wrote in > On Tue, Apr 2, 2019 at 6:54 AM Kyotaro HORIGUCHI > wrote: > > > By using DELETE and INSERT records to implement an UPDATE, you lose the > > > ctid > > > chain and infomask bits that were present before crash

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-04-03 Thread Robert Haas
On Tue, Apr 2, 2019 at 6:54 AM Kyotaro HORIGUCHI wrote: > > By using DELETE and INSERT records to implement an UPDATE, you lose the ctid > > chain and infomask bits that were present before crash recovery. If that's > > okay in these circumstances, please write a comment explaining why. > > Sound

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-04-02 Thread Kyotaro HORIGUCHI
Thank you for reviewing. At Sun, 31 Mar 2019 15:31:58 -0700, Noah Misch wrote in <20190331223158.gb891...@rfd.leadboat.com> > On Sun, Mar 10, 2019 at 07:27:08PM -0700, Noah Misch wrote: > > On Mon, Mar 04, 2019 at 12:24:48PM +0900, Kyotaro HORIGUCHI wrote: > > > +/* > > > + * RelationRemovePendi

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-03-31 Thread Noah Misch
On Sun, Mar 10, 2019 at 07:27:08PM -0700, Noah Misch wrote: > I also liked the design in the https://postgr.es/m/559fa0ba.3080...@iki.fi > last paragraph, and I suspect it would have been no harder to back-patch. I > wonder if it would have been simpler and better, but I'm not asking anyone to > i

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-03-26 Thread Kyotaro HORIGUCHI
Hello. I revised the patch I think addressing all your comments. Differences from v7 patch are: v9-0001: - Renamed the script from 016_ to 017_. - Added some additional tests. v9-0002: - Fixed _bt_blwritepage(). It is re-modified by v9-0007. v9-0003: New patch. - Refactors out xlog s

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-03-25 Thread Kyotaro HORIGUCHI
Hello. This is a revised version. At Wed, 20 Mar 2019 22:48:35 -0700, Noah Misch wrote in <20190321054835.gb3842...@rfd.leadboat.com> > On Wed, Mar 20, 2019 at 05:17:54PM +0900, Kyotaro HORIGUCHI wrote: > > At Sun, 10 Mar 2019 19:27:08 -0700, Noah Misch wrote in > > <20190311022708.ga2189...@r

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-03-20 Thread Noah Misch
On Wed, Mar 20, 2019 at 05:17:54PM +0900, Kyotaro HORIGUCHI wrote: > At Sun, 10 Mar 2019 19:27:08 -0700, Noah Misch wrote in > <20190311022708.ga2189...@rfd.leadboat.com> > > On Mon, Mar 04, 2019 at 12:24:48PM +0900, Kyotaro HORIGUCHI wrote: > > > +/* > > > + * Sync to disk any relations that we

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-03-20 Thread Kyotaro HORIGUCHI
Thank you for reviewing! At Sun, 10 Mar 2019 19:27:08 -0700, Noah Misch wrote in <20190311022708.ga2189...@rfd.leadboat.com> > This has been waiting for a review since October, so I reviewed it. The code > comment at PendingRelSync summarizes the design well, and I like that design. It is Mich

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-03-10 Thread Noah Misch
This has been waiting for a review since October, so I reviewed it. The code comment at PendingRelSync summarizes the design well, and I like that design. I also liked the design in the https://postgr.es/m/559fa0ba.3080...@iki.fi last paragraph, and I suspect it would have been no harder to back-p

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-03-03 Thread Kyotaro HORIGUCHI
Rebased. No commit hit this but I fixed one space error. regards. -- Kyotaro Horiguchi NTT Open Source Software Center >From d048aedbee48a1a0d91ae6e009b7a7903f272720 Mon Sep 17 00:00:00 2001 From: Kyotaro Horiguchi Date: Thu, 11 Oct 2018 10:03:21 +0900 Subject: [PATCH 1/4] TAP test for copy-tr

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-01-29 Thread Kyotaro HORIGUCHI
Rebased. regards. -- Kyotaro Horiguchi NTT Open Source Software Center >From d5f2b47b6ba191d0ad1673f9bd9c5851d91a1b59 Mon Sep 17 00:00:00 2001 From: Kyotaro Horiguchi Date: Thu, 11 Oct 2018 10:03:21 +0900 Subject: [PATCH 1/4] TAP test for copy-truncation optimization. --- src/test/recovery/t/

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-12-20 Thread Kyotaro HORIGUCHI
Hello. At Fri, 30 Nov 2018 18:27:05 +0100, Dmitry Dolgov <9erthali...@gmail.com> wrote in > > On Wed, Nov 14, 2018 at 4:48 AM Kyotaro HORIGUCHI > > wrote: > > > > 0004 was shot by e9edc1ba0b. Rebased to the current HEAD. > > Successfully built and passeed all regression/recovery tests > > incl

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-11-30 Thread Dmitry Dolgov
> On Wed, Nov 14, 2018 at 4:48 AM Kyotaro HORIGUCHI > wrote: > > 0004 was shot by e9edc1ba0b. Rebased to the current HEAD. > Successfully built and passeed all regression/recovery tests > including additional recovery/t/016_wal_optimize.pl. Thank you for working on this patch. Unfortunately, cfb

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-11-13 Thread Kyotaro HORIGUCHI
Hello. At Thu, 11 Oct 2018 17:04:53 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20181011.170453.123148806.horiguchi.kyot...@lab.ntt.co.jp> > At Thu, 11 Oct 2018 13:42:35 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI > wrote in > <20181011.134235.218062184.horiguchi.kyot...@lab.

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-10-11 Thread Kyotaro HORIGUCHI
At Thu, 11 Oct 2018 13:42:35 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20181011.134235.218062184.horiguchi.kyot...@lab.ntt.co.jp> > Hello. > > At Fri, 27 Jul 2018 15:26:24 -0400, Andrew Dunstan > wrote in > > > > > > > On 07/18/2018 10:58 AM, Heikki Linnakangas wrote: > > >

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-10-10 Thread Kyotaro HORIGUCHI
Hello. At Fri, 27 Jul 2018 15:26:24 -0400, Andrew Dunstan wrote in > > > On 07/18/2018 10:58 AM, Heikki Linnakangas wrote: > > On 18/07/18 16:29, Robert Haas wrote: > >> On Wed, Jul 18, 2018 at 9:06 AM, Michael Paquier > >> wrote: > What's wrong with the approach proposed in > htt

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-07-27 Thread Andrew Dunstan
On 07/18/2018 10:58 AM, Heikki Linnakangas wrote: On 18/07/18 16:29, Robert Haas wrote: On Wed, Jul 18, 2018 at 9:06 AM, Michael Paquier wrote: What's wrong with the approach proposed in http://postgr.es/m/55afc302.1060...@iki.fi ? For back-branches that's very invasive so that seems risk

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-07-25 Thread Kyotaro HORIGUCHI
Hello. At Wed, 25 Jul 2018 23:08:33 +0900, Michael Paquier wrote in <20180725140833.gc6...@paquier.xyz> > On Wed, Jul 18, 2018 at 05:58:16PM +0300, Heikki Linnakangas wrote: > > I'd suggest that we continue based on the patch that Kyotaro posted at > > https://www.postgresql.org/message-id/20180

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-07-25 Thread Michael Paquier
On Wed, Jul 18, 2018 at 05:58:16PM +0300, Heikki Linnakangas wrote: > I'd suggest that we continue based on the patch that Kyotaro posted at > https://www.postgresql.org/message-id/20180330.100646.86008470.horiguchi.kyotaro%40lab.ntt.co.jp. Whatever happens here, perhaps one way to move on would b

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-07-18 Thread Heikki Linnakangas
On 18/07/18 16:29, Robert Haas wrote: On Wed, Jul 18, 2018 at 9:06 AM, Michael Paquier wrote: What's wrong with the approach proposed in http://postgr.es/m/55afc302.1060...@iki.fi ? For back-branches that's very invasive so that seems risky to me particularly seeing the low number of complain

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-07-18 Thread Robert Haas
On Wed, Jul 18, 2018 at 9:06 AM, Michael Paquier wrote: >> What's wrong with the approach proposed in >> http://postgr.es/m/55afc302.1060...@iki.fi ? > > For back-branches that's very invasive so that seems risky to me > particularly seeing the low number of complaints on the matter. Hmm. I think

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-07-18 Thread Michael Paquier
On Wed, Jul 18, 2018 at 06:42:10AM -0400, Robert Haas wrote: > On Tue, Jul 17, 2018 at 8:28 AM, Andrew Dunstan > wrote: >> Well, we'd be getting rid of it because of a danger of data loss which we >> can't otherwise mitigate. Maybe it does need to be backpatched, even if we >> haven't had complain

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-07-18 Thread Robert Haas
On Tue, Jul 17, 2018 at 8:28 AM, Andrew Dunstan wrote: > Well, we'd be getting rid of it because of a danger of data loss which we > can't otherwise mitigate. Maybe it does need to be backpatched, even if we > haven't had complaints. What's wrong with the approach proposed in http://postgr.es/m/5

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-07-17 Thread Andrew Dunstan
On 07/16/2018 08:01 PM, Michael Paquier wrote: I doubt as well that we'd be able to catch all the holes as well as the conditions where the optimization could be run safely are rather basically impossible to catch beforehand. I'd like to vote for getting rid of this optimization for COPY, t

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-07-17 Thread Kyotaro HORIGUCHI
Hello. At Mon, 16 Jul 2018 16:14:09 -0400, Alvaro Herrera wrote in <20180716201409.2qfcneo4qkdwjvpv@alvherre.pgsql> > On 2018-Jul-12, Heikki Linnakangas wrote: > > > > > Thanks for the pointer. My tap test has been covering two out of > > > > the three scenarios you have in your script. I hav

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-07-16 Thread Michael Paquier
On Mon, Jul 16, 2018 at 09:41:51PM +0300, Heikki Linnakangas wrote: > On 16 July 2018 21:38:39 EEST, Robert Haas wrote: >>On Thu, Jul 12, 2018 at 10:12 AM, Heikki Linnakangas >>wrote: >>> Doesn't have to be a trigger, could be a CHECK constraint, datatype >>input >>> function, etc. Admittedly, ha

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-07-16 Thread Alvaro Herrera
On 2018-Jul-12, Heikki Linnakangas wrote: > > > Thanks for the pointer. My tap test has been covering two out of > > > the three scenarios you have in your script. I have been able to > > > convert the extra as the attached, and I have added as well an > > > extra test with TRUNCATE triggers. S

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-07-16 Thread Heikki Linnakangas
On 16 July 2018 21:38:39 EEST, Robert Haas wrote: >On Thu, Jul 12, 2018 at 10:12 AM, Heikki Linnakangas >wrote: >> Doesn't have to be a trigger, could be a CHECK constraint, datatype >input >> function, etc. Admittedly, having a datatype input function that >inserts to >> the table is worth a "hu

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-07-16 Thread Robert Haas
On Thu, Jul 12, 2018 at 10:12 AM, Heikki Linnakangas wrote: > Doesn't have to be a trigger, could be a CHECK constraint, datatype input > function, etc. Admittedly, having a datatype input function that inserts to > the table is worth a "huh?", but I'm feeling very confident that we can > catch al

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-07-12 Thread Michael Paquier
On Thu, Jul 12, 2018 at 05:12:21PM +0300, Heikki Linnakangas wrote: > Doesn't have to be a trigger, could be a CHECK constraint, datatype input > function, etc. Admittedly, having a datatype input function that inserts to > the table is worth a "huh?", but I'm feeling very confident that we can > c

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-07-12 Thread Heikki Linnakangas
On 12/07/18 16:51, Andrew Dunstan wrote: On 07/10/2018 11:32 PM, Michael Paquier wrote: On Tue, Jul 10, 2018 at 05:35:58PM +0300, Heikki Linnakangas wrote: Thanks for picking this up! (I hope this gets through the email filters this time, sending a shell script seems to be difficult. I also

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-07-12 Thread Andrew Dunstan
On 07/10/2018 11:32 PM, Michael Paquier wrote: On Tue, Jul 10, 2018 at 05:35:58PM +0300, Heikki Linnakangas wrote: Thanks for picking this up! (I hope this gets through the email filters this time, sending a shell script seems to be difficult. I also trimmed the CC list, if that helps.) On

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-07-10 Thread Michael Paquier
On Tue, Jul 10, 2018 at 05:35:58PM +0300, Heikki Linnakangas wrote: > Thanks for picking this up! > > (I hope this gets through the email filters this time, sending a shell > script seems to be difficult. I also trimmed the CC list, if that helps.) > > On 04/07/18 07:59, Michael Paquier wrote: >>

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-07-10 Thread Heikki Linnakangas
Thanks for picking this up! (I hope this gets through the email filters this time, sending a shell script seems to be difficult. I also trimmed the CC list, if that helps.) On 04/07/18 07:59, Michael Paquier wrote: Hence I propose the patch attached which disables the TRUNCATE and COPY optimi

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-07-05 Thread Michael Paquier
On Wed, Jul 04, 2018 at 07:55:53AM -0400, Andrew Dunstan wrote: > Many thanks for working on this. No problem. Thanks for the lookup. > +1 for these changes, even though the TRUNCATE fix looks perverse. If > anyone wants to propose further optimizations in this area this would > at least give us

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-07-04 Thread Andrew Dunstan
On Wed, Jul 4, 2018 at 12:59 AM, Michael Paquier wrote: > On Fri, Mar 30, 2018 at 10:06:46AM +0900, Kyotaro HORIGUCHI wrote: >> Hello. I found that c203d6cf81 hit this and this is the rebased >> version on the current master. > > Okay, as this is visibly the oldest item in this commit fest, Andre

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-07-03 Thread Michael Paquier
On Fri, Mar 30, 2018 at 10:06:46AM +0900, Kyotaro HORIGUCHI wrote: > Hello. I found that c203d6cf81 hit this and this is the rebased > version on the current master. Okay, as this is visibly the oldest item in this commit fest, Andrew has asked me to look at a solution which would allow us to def

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-03-29 Thread Kyotaro HORIGUCHI
Hello. I found that c203d6cf81 hit this and this is the rebased version on the current master. regards, -- Kyotaro Horiguchi NTT Open Source Software Center >From 3dac5baf787dc949cfb22a698a0d72b6eb48e75e Mon Sep 17 00:00:00 2001 From: Kyotaro Horiguchi Date: Tue, 12 Sep 2017 13:01:33 +0900 Sub

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-01-11 Thread Kyotaro HORIGUCHI
Hello, At Thu, 4 Jan 2018 23:10:40 -0500, Stephen Frost wrote in <20180105041040.gi2...@tamriel.snowman.net> > > The attached patch applies on the current HEAD and passes all > > recovery tests. > > This is currently marked as 'waiting on author' in the CF app, but it > sounds like it should be

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-01-04 Thread Stephen Frost
Greetings, * Kyotaro HORIGUCHI (horiguchi.kyot...@lab.ntt.co.jp) wrote: > At Tue, 28 Nov 2017 10:36:39 +0900, Michael Paquier > wrote in > > > On Thu, Sep 14, 2017 at 3:34 PM, Kyotaro HORIGUCHI > > wrote: > > > At Wed, 13 Sep 2017 17:42:39 +0900 (Tokyo Standard Time), Kyotaro > > > HORIGUCHI

Re: [HACKERS] WAL logging problem in 9.4.3?

2017-12-11 Thread Kyotaro HORIGUCHI
At Tue, 28 Nov 2017 10:36:39 +0900, Michael Paquier wrote in > On Thu, Sep 14, 2017 at 3:34 PM, Kyotaro HORIGUCHI > wrote: > > At Wed, 13 Sep 2017 17:42:39 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI > > wrote in > > <20170913.174239.25978735.horiguchi.kyot...@lab.ntt.co.jp> > >> filterdi

Re: [HACKERS] WAL logging problem in 9.4.3?

2017-11-27 Thread Michael Paquier
On Thu, Sep 14, 2017 at 3:34 PM, Kyotaro HORIGUCHI wrote: > At Wed, 13 Sep 2017 17:42:39 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI > wrote in > <20170913.174239.25978735.horiguchi.kyot...@lab.ntt.co.jp> >> filterdiff seems to did something wrong.. > > # to did... > > The patch is broken by

<    1   2