Re: [HACKERS] WAL logging problem in 9.4.3?

2020-04-06 Thread Noah Misch
On Mon, Apr 06, 2020 at 09:46:31AM +0900, Kyotaro Horiguchi wrote: > At Sat, 4 Apr 2020 15:32:12 -0700, Noah Misch wrote in > > On Sat, Apr 04, 2020 at 06:24:34PM -0400, Tom Lane wrote: > > > Shouldn't the CF entry get closed? > > > > Once the buildfarm is clean for a day, sure. The buildfarm

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-04-05 Thread Kyotaro Horiguchi
At Sat, 4 Apr 2020 15:32:12 -0700, Noah Misch wrote in > On Sat, Apr 04, 2020 at 06:24:34PM -0400, Tom Lane wrote: > > Shouldn't the CF entry get closed? > > Once the buildfarm is clean for a day, sure. The buildfarm has already > revealed a missing perl2host call. Thank you for (re-)

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-04-04 Thread Noah Misch
On Sat, Apr 04, 2020 at 06:24:34PM -0400, Tom Lane wrote: > Shouldn't the CF entry get closed? Once the buildfarm is clean for a day, sure. The buildfarm has already revealed a missing perl2host call.

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-04-04 Thread Tom Lane
Robert Haas writes: > On Wed, Apr 1, 2020 at 11:51 PM Noah Misch wrote: >> I've translated the non-vote comments into estimated votes of -0.3, -0.6, >> -0.4, +0.5, and -0.3. Hence, I revoke the plan to back-patch. > FWIW, I also think that it would be better not to back-patch. FWIW, I also

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-04-02 Thread Robert Haas
On Wed, Apr 1, 2020 at 11:51 PM Noah Misch wrote: > I've translated the non-vote comments into estimated votes of -0.3, -0.6, > -0.4, +0.5, and -0.3. Hence, I revoke the plan to back-patch. FWIW, I also think that it would be better not to back-patch. The risk of back-patching is that this will

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-04-01 Thread Noah Misch
On Mon, Mar 30, 2020 at 11:37:57PM -0700, Andres Freund wrote: > On 2020-03-30 23:28:54 -0700, Noah Misch wrote: > > On Mon, Mar 30, 2020 at 04:43:00PM +0900, Michael Paquier wrote: > > > On Sun, Mar 29, 2020 at 09:41:01PM -0700, Noah Misch wrote: > > > > I think attached v41nm is ready for

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-03-31 Thread Andres Freund
Hi, On 2020-03-30 23:28:54 -0700, Noah Misch wrote: > On Mon, Mar 30, 2020 at 04:43:00PM +0900, Michael Paquier wrote: > > On Sun, Mar 29, 2020 at 09:41:01PM -0700, Noah Misch wrote: > > > I think attached v41nm is ready for commit. Would anyone like to vote > > > against > > > back-patching

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-03-31 Thread Noah Misch
On Mon, Mar 30, 2020 at 04:43:00PM +0900, Michael Paquier wrote: > On Sun, Mar 29, 2020 at 09:41:01PM -0700, Noah Misch wrote: > > I think attached v41nm is ready for commit. Would anyone like to vote > > against > > back-patching this? It's hard to justify lack of back-patch for a data-loss >

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-03-30 Thread Michael Paquier
On Sun, Mar 29, 2020 at 09:41:01PM -0700, Noah Misch wrote: > I think attached v41nm is ready for commit. Would anyone like to vote against > back-patching this? It's hard to justify lack of back-patch for a data-loss > bug, but this is atypically invasive. (I'm repeating the question, since

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-03-30 Thread Kyotaro Horiguchi
At Sun, 29 Mar 2020 23:08:27 -0700, Noah Misch wrote in > On Mon, Mar 30, 2020 at 02:56:11PM +0900, Kyotaro Horiguchi wrote: > > At Sun, 29 Mar 2020 21:41:01 -0700, Noah Misch wrote in > > > Since pendingSyncHash is always NULL under XLogIsNeeded(), I also removed > > > some > > >

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-03-30 Thread Noah Misch
On Mon, Mar 30, 2020 at 02:56:11PM +0900, Kyotaro Horiguchi wrote: > At Sun, 29 Mar 2020 21:41:01 -0700, Noah Misch wrote in > > Since pendingSyncHash is always NULL under XLogIsNeeded(), I also removed > > some > > XLogIsNeeded() tests that immediately preceded !pendingSyncHash tests. > >

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-03-29 Thread Kyotaro Horiguchi
At Sun, 29 Mar 2020 21:41:01 -0700, Noah Misch wrote in > I think attached v41nm is ready for commit. Would anyone like to vote against > back-patching this? It's hard to justify lack of back-patch for a data-loss > bug, but this is atypically invasive. (I'm repeating the question, since some

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-03-23 Thread Kyotaro Horiguchi
Thanks for the labour on this. At Sat, 21 Mar 2020 15:49:20 -0700, Noah Misch wrote in > On Sat, Mar 21, 2020 at 12:01:27PM -0700, Noah Misch wrote: > > Pushed, after adding a missing "break" to gist_identify() and tweaking two .. > The proximate cause is the RelFileNodeSkippingWAL() call that

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-03-21 Thread Noah Misch
On Sat, Mar 21, 2020 at 12:01:27PM -0700, Noah Misch wrote: > Pushed, after adding a missing "break" to gist_identify() and tweaking two > more comments. However, a diverse minority of buildfarm members are failing > like this, in most branches: > > Mar 21 13:16:37 # Failed test 'wal_level =

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-03-21 Thread Bruce Momjian
Wow, this thread started in 2015. :-O Date: Fri, 3 Jul 2015 00:05:24 +0200 --- On Sat, Mar 21, 2020 at 12:01:27PM -0700, Noah Misch wrote: > On Sun, Mar 15, 2020 at 08:46:47PM -0700, Noah Misch wrote: > > On Wed,

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-03-21 Thread Noah Misch
On Sun, Mar 15, 2020 at 08:46:47PM -0700, Noah Misch wrote: > On Wed, Mar 04, 2020 at 04:29:19PM +0900, Kyotaro Horiguchi wrote: > > The attached is back-patches from 9.5 through master. > > Thanks. I've made some edits. I'll plan to push the attached patches on > Friday or Saturday. Pushed,

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-03-03 Thread Kyotaro Horiguchi
Some fixes.. At Wed, 04 Mar 2020 16:29:19 +0900 (JST), Kyotaro Horiguchi wrote in > At first I fixed several ssues in 018_wal_optimize.pl: > > - TRUNCATE INSERT, TRUNCATE INSERT PREPARE > > It wrongly passes if finally we see the value only from the first > INSERT. I changed it so that it

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-03-01 Thread Kyotaro Horiguchi
At Sun, 1 Mar 2020 11:56:32 -0800, Noah Misch wrote in > On Thu, Feb 27, 2020 at 04:00:24PM +0900, Kyotaro Horiguchi wrote: > > It sounds somewhat obscure. > > I see. I won't use that. Thanks. > > Coulnd't we enumetate examples? And if we > > could use pg_relation_filenode, I think we can

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-03-01 Thread Noah Misch
On Thu, Feb 27, 2020 at 04:00:24PM +0900, Kyotaro Horiguchi wrote: > At Tue, 25 Feb 2020 21:36:12 -0800, Noah Misch wrote in > > On Tue, Feb 25, 2020 at 10:01:51AM +0900, Kyotaro Horiguchi wrote: > > > At Sat, 22 Feb 2020 21:12:20 -0800, Noah Misch wrote > > > in > > > > On Fri, Feb 21, 2020

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-02-26 Thread Kyotaro Horiguchi
At Tue, 25 Feb 2020 21:36:12 -0800, Noah Misch wrote in > On Tue, Feb 25, 2020 at 10:01:51AM +0900, Kyotaro Horiguchi wrote: > > At Sat, 22 Feb 2020 21:12:20 -0800, Noah Misch wrote in > > > On Fri, Feb 21, 2020 at 04:49:59PM +0900, Kyotaro Horiguchi wrote: > > I aggree that the new #ifdef can

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-02-25 Thread Noah Misch
On Tue, Feb 25, 2020 at 10:01:51AM +0900, Kyotaro Horiguchi wrote: > At Sat, 22 Feb 2020 21:12:20 -0800, Noah Misch wrote in > > On Fri, Feb 21, 2020 at 04:49:59PM +0900, Kyotaro Horiguchi wrote: > > > At Wed, 19 Feb 2020 17:29:08 +0900 (JST), Kyotaro Horiguchi > > > wrote in > > > > At Tue,

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-02-24 Thread Kyotaro Horiguchi
At Sat, 22 Feb 2020 21:12:20 -0800, Noah Misch wrote in > On Fri, Feb 21, 2020 at 04:49:59PM +0900, Kyotaro Horiguchi wrote: > > At Wed, 19 Feb 2020 17:29:08 +0900 (JST), Kyotaro Horiguchi > > wrote in > > > At Tue, 18 Feb 2020 23:44:52 -0800, Noah Misch wrote > > > in > > > > - When

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-02-22 Thread Noah Misch
On Fri, Feb 21, 2020 at 04:49:59PM +0900, Kyotaro Horiguchi wrote: > At Wed, 19 Feb 2020 17:29:08 +0900 (JST), Kyotaro Horiguchi > wrote in > > At Tue, 18 Feb 2020 23:44:52 -0800, Noah Misch wrote in > > > - When reusing an index build, instead of storing the dropped relid in the > > >

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-02-20 Thread Kyotaro Horiguchi
Hello. I looked through the latest patch. At Wed, 19 Feb 2020 17:29:08 +0900 (JST), Kyotaro Horiguchi wrote in > At Tue, 18 Feb 2020 23:44:52 -0800, Noah Misch wrote in > > - When reusing an index build, instead of storing the dropped relid in the > > IndexStmt and opening the dropped

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-02-19 Thread Kyotaro Horiguchi
Sorry, just one fix. (omitting some typos, though..) At Wed, 19 Feb 2020 17:29:08 +0900 (JST), Kyotaro Horiguchi wrote in > At Tue, 18 Feb 2020 23:44:52 -0800, Noah Misch wrote in > > I think attached v35nm is ready for commit to master. Would anyone like to > > talk me out of back-patching

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-02-19 Thread Kyotaro Horiguchi
At Tue, 18 Feb 2020 23:44:52 -0800, Noah Misch wrote in > I think attached v35nm is ready for commit to master. Would anyone like to > talk me out of back-patching this? I would not enjoy back-patching it, but > it's hard to justify lack of back-patch for a data-loss bug. > > Notable changes

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-02-18 Thread Noah Misch
I think attached v35nm is ready for commit to master. Would anyone like to talk me out of back-patching this? I would not enjoy back-patching it, but it's hard to justify lack of back-patch for a data-loss bug. Notable changes since v34: - Separate a few freestanding fixes into their own

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-02-18 Thread Kyotaro Horiguchi
Oops. I played on a wrong branch and got stuck in slow build on Windows... At Tue, 18 Feb 2020 00:53:37 -0800, Noah Misch wrote in > On Tue, Feb 18, 2020 at 03:56:15PM +1300, Thomas Munro wrote: > > CREATE TYPE priv_testtype1 AS (a int, b text); > > +ERROR: relation 24844 deleted while still in

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-02-18 Thread Noah Misch
On Tue, Feb 18, 2020 at 03:56:15PM +1300, Thomas Munro wrote: > CREATE TYPE priv_testtype1 AS (a int, b text); > +ERROR: relation 24844 deleted while still in use > REVOKE USAGE ON TYPE priv_testtype1 FROM PUBLIC; > > https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.79923 >

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-02-17 Thread Thomas Munro
On Mon, Jan 27, 2020 at 11:30 PM Kyotaro Horiguchi wrote: > Hello, this is rebased then addressed version. Hi, I haven't followed this thread but I just noticed this strange looking failure: CREATE TYPE priv_testtype1 AS (a int, b text); +ERROR: relation 24844 deleted while still in use REVOKE

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-01-27 Thread Kyotaro Horiguchi
Hello, this is rebased then addressed version. - Now valid rd_firstRelfilenodeSubid causes drop-pending of relcache as well as rd_createSubid. The oblivion in the last example no longer happens. - Revert the (really) useless change of AssertPendingSyncs_RelationCache. - Fix several

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-01-26 Thread Kyotaro Horiguchi
By the way, the previous version looks somewhat different from what I thought I posted.. At Sun, 26 Jan 2020 20:57:00 -0800, Noah Misch wrote in > On Mon, Jan 27, 2020 at 01:44:13PM +0900, Kyotaro Horiguchi wrote: > > > The purpose of this loop is to create relcache entries for rels locked in

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-01-26 Thread Noah Misch
On Mon, Jan 27, 2020 at 01:44:13PM +0900, Kyotaro Horiguchi wrote: > At Sun, 26 Jan 2020 20:22:01 -0800, Noah Misch wrote in > > Diffing the two latest versions of one patch: > > > --- v32-0002-Fix-the-defect-1.patch 2020-01-18 14:32:47.499129940 > > > -0800 > > > +++

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-01-26 Thread Kyotaro Horiguchi
Thanks! At Sun, 26 Jan 2020 20:22:01 -0800, Noah Misch wrote in > Diffing the two latest versions of one patch: > > --- v32-0002-Fix-the-defect-1.patch 2020-01-18 14:32:47.499129940 -0800 > > +++ v33-0002-Fix-the-defect-1.patch 2020-01-26 16:23:52.846391035 -0800 > > +@@ -2978,8 +3054,8 @@

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-01-26 Thread Noah Misch
Diffing the two latest versions of one patch: > --- v32-0002-Fix-the-defect-1.patch 2020-01-18 14:32:47.499129940 -0800 > +++ v33-0002-Fix-the-defect-1.patch 2020-01-26 16:23:52.846391035 -0800 > +@@ -2978,8 +3054,8 @@ AssertPendingSyncs_RelationCache(void) > +

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-01-21 Thread Kyotaro Horiguchi
Thank you for the comment. At Sat, 18 Jan 2020 19:51:39 -0800, Noah Misch wrote in > On Tue, Jan 14, 2020 at 07:35:22PM +0900, Kyotaro Horiguchi wrote: > > At Thu, 26 Dec 2019 12:46:39 +0900 (JST), Kyotaro Horiguchi > > wrote in > > > At Wed, 25 Dec 2019 16:15:21 -0800, Noah Misch wrote >

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-01-18 Thread Noah Misch
On Tue, Jan 14, 2020 at 07:35:22PM +0900, Kyotaro Horiguchi wrote: > At Thu, 26 Dec 2019 12:46:39 +0900 (JST), Kyotaro Horiguchi > wrote in > > At Wed, 25 Dec 2019 16:15:21 -0800, Noah Misch wrote in > > > === Defect 1: Forgets to skip WAL after SAVEPOINT; DROP TABLE; ROLLBACK TO > > > > > >

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-01-15 Thread Kyotaro Horiguchi
All the known defects are fixed. At Wed, 25 Dec 2019 16:15:21 -0800, Noah Misch wrote in > === Defect 3: storage.c checks size decrease of MAIN_FORKNUM only > > storage.c tracks only MAIN_FORKNUM in pendingsync->max_truncated. Is it > possible for MAIN_FORKNUM to have a net size increase

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-01-15 Thread Kyotaro Horiguchi
Hello. I added a fix for the defect 2. At Wed, 25 Dec 2019 16:15:21 -0800, Noah Misch wrote in > === Defect 2: Forgets to skip WAL due to oversimplification in heap_create() > > In ALTER TABLE cases where TryReuseIndex() avoided an index rebuild, we need > to transfer WAL-skipped state to the

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-01-14 Thread Kyotaro Horiguchi
Hello, this is a fix for the defect 1 of 3. At Thu, 26 Dec 2019 12:46:39 +0900 (JST), Kyotaro Horiguchi wrote in > Thank you for the findings. > > At Wed, 25 Dec 2019 16:15:21 -0800, Noah Misch wrote in > > By improving AssertPendingSyncs_RelationCache() and by testing with > >

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-12-26 Thread Kyotaro Horiguchi
Hello, Noah. At Wed, 25 Dec 2019 20:22:04 -0800, Noah Misch wrote in > On Thu, Dec 26, 2019 at 12:46:39PM +0900, Kyotaro Horiguchi wrote: > > At Wed, 25 Dec 2019 16:15:21 -0800, Noah Misch wrote in > > > Skip AssertPendingSyncs_RelationCache() at abort, like v24nm did. > > > Making > > >

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-12-25 Thread Kyotaro Horiguchi
At Thu, 26 Dec 2019 12:46:39 +0900 (JST), Kyotaro Horiguchi wrote in > > - Reverted most post-v24nm changes to swap_relation_files(). Under > > "-DRELCACHE_FORCE_RELEASE", relcache.c quickly discards the > > rel1->rd_node.relNode update. Clearing rel2->rd_createSubid is not right > > if

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-12-25 Thread Noah Misch
On Thu, Dec 26, 2019 at 12:46:39PM +0900, Kyotaro Horiguchi wrote: > At Wed, 25 Dec 2019 16:15:21 -0800, Noah Misch wrote in > > Skip AssertPendingSyncs_RelationCache() at abort, like v24nm did. Making > > that work no matter what does ereport(ERROR) would be tricky and > > low-value. > >

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-12-25 Thread Kyotaro Horiguchi
Thank you for the findings. At Wed, 25 Dec 2019 16:15:21 -0800, Noah Misch wrote in > By improving AssertPendingSyncs_RelationCache() and by testing with > -DRELCACHE_FORCE_RELEASE, I now know of three defects in the attached v30nm. > Would you fix these? I'd like to do that, please give me

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-12-25 Thread Noah Misch
By improving AssertPendingSyncs_RelationCache() and by testing with -DRELCACHE_FORCE_RELEASE, I now know of three defects in the attached v30nm. Would you fix these? === Defect 1: Forgets to skip WAL after SAVEPOINT; DROP TABLE; ROLLBACK TO A test in transactions.sql now fails in

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-12-25 Thread Kyotaro Horiguchi
At Tue, 24 Dec 2019 16:35:35 +0900 (JST), Kyotaro Horiguchi wrote in > I rebased the patch and changed the default value for the GUC variable > wal_skip_threshold to 4096 kilobytes in config.sgml, storage.c and > guc.c. 4096kB is choosed as it is the nice round number of 500 pages * > 8kB =

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-12-23 Thread Kyotaro Horiguchi
At Tue, 10 Dec 2019 16:59:25 +0900 (JST), Kyotaro Horiguchi wrote in > shared_buffers=1GB/wal_buffers=16MB(defalut). pgbench -s 20 uses 256MB > of storage so all of them can be loaded on shared memory. > > The attached graph shows larger benefit in TPS drop and latency > increase for HDD. The

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-12-09 Thread Robert Haas
On Mon, Dec 9, 2019 at 4:04 AM Kyotaro Horiguchi wrote: > Yeah, only 0.5GB of shared_buffers makes the default value of > wal_buffers reach to the heaven. I think I can take numbers on that > condition. (I doubt that it's meaningful if I increase only > wal_buffers manually.) Heaven seems a bit

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-12-09 Thread Kyotaro Horiguchi
Hello. At Sun, 8 Dec 2019 10:09:51 -0800, Noah Misch wrote in > I reviewed your latest code, and it's nearly complete. mdimmedsync() syncs > only "active segments" (as defined in md.c), but smgrDoPendingSyncs() must > sync active and inactive segments. This matters when mdtruncate() truncated

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-12-08 Thread Noah Misch
I reviewed your latest code, and it's nearly complete. mdimmedsync() syncs only "active segments" (as defined in md.c), but smgrDoPendingSyncs() must sync active and inactive segments. This matters when mdtruncate() truncated the relation after the last checkpoint, causing active segments to

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-12-03 Thread Kyotaro Horiguchi
Hello. At Thu, 28 Nov 2019 17:23:19 -0500, Noah Misch wrote in > On Thu, Nov 28, 2019 at 09:35:08PM +0900, Kyotaro Horiguchi wrote: > > I measured the performance with the latest patch set. > > > > > 1. Determine $DDL_COUNT, a number of DDL transactions that take about one > > >minute when

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-11-28 Thread Noah Misch
On Thu, Nov 28, 2019 at 09:35:08PM +0900, Kyotaro Horiguchi wrote: > I measured the performance with the latest patch set. > > > 1. Determine $DDL_COUNT, a number of DDL transactions that take about one > >minute when done via syncs. > > 2. Start "pgbench -rP1 --progress-timestamp -T180 -c10

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-11-28 Thread Kyotaro Horiguchi
I measured the performance with the latest patch set. > 1. Determine $DDL_COUNT, a number of DDL transactions that take about one >minute when done via syncs. > 2. Start "pgbench -rP1 --progress-timestamp -T180 -c10 -j10". > 3. Wait 10s. > 4. Start one DDL backend that runs $DDL_COUNT

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-11-28 Thread Kyotaro Horiguchi
At Tue, 26 Nov 2019 21:37:52 +0900 (JST), Kyotaro Horiguchi Is is not fully checked. I didn't merged and mesured performance yet, > but I post the status-quo patch for now. It was actually inconsistency caused by swap_relation_files. 1. rd_createSubid of relcache for r2 is not turned off.

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-11-26 Thread Kyotaro Horiguchi
At Sun, 24 Nov 2019 22:08:39 -0500, Noah Misch wrote in > On Mon, Nov 25, 2019 at 11:08:54AM +0900, Kyotaro Horiguchi wrote: > > At Sat, 23 Nov 2019 16:21:36 -0500, Noah Misch wrote in > > > I noticed an additional defect: > > > > > > BEGIN; > > > CREATE TABLE t (c) AS SELECT 1; > > >

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-11-25 Thread Noah Misch
On Mon, Nov 25, 2019 at 03:58:14PM -0500, Robert Haas wrote: > On Sat, Nov 23, 2019 at 4:21 PM Noah Misch wrote: > > I noticed an additional defect: > > > > BEGIN; > > CREATE TABLE t (c) AS SELECT 1; > > CHECKPOINT; -- write and fsync the table's one page > > TRUNCATE t; -- no WAL > > COMMIT; --

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-11-25 Thread Robert Haas
On Sat, Nov 23, 2019 at 4:21 PM Noah Misch wrote: > I noticed an additional defect: > > BEGIN; > CREATE TABLE t (c) AS SELECT 1; > CHECKPOINT; -- write and fsync the table's one page > TRUNCATE t; -- no WAL > COMMIT; -- no FPI, just the commit record > > If we crash after the COMMIT and before

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-11-24 Thread Noah Misch
On Mon, Nov 25, 2019 at 11:08:54AM +0900, Kyotaro Horiguchi wrote: > At Sat, 23 Nov 2019 16:21:36 -0500, Noah Misch wrote in > > This benchmark procedure may help: > > > > 1. Determine $DDL_COUNT, a number of DDL transactions that take about one > >minute when done via syncs. > > 2. Start

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-11-24 Thread Kyotaro Horiguchi
At Sat, 23 Nov 2019 16:21:36 -0500, Noah Misch wrote in > On Wed, Nov 20, 2019 at 03:05:46PM +0900, Kyotaro Horiguchi wrote: > > By the way, before finalize this, I'd like to share the result of a > > brief benchmarking. > > What non-default settings did you use? Please give the output of this

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-11-24 Thread Michael Paquier
On Sat, Nov 23, 2019 at 11:35:09AM -0500, Noah Misch wrote: > That longstanding optimization is too useful to remove, but likely not useful > enough to add today if we didn't already have it. The initial-data-load use > case remains plausible. I can also imagine using wal_level=minimal for data

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-11-23 Thread Noah Misch
On Wed, Nov 20, 2019 at 03:05:46PM +0900, Kyotaro Horiguchi wrote: > By the way, before finalize this, I'd like to share the result of a > brief benchmarking. What non-default settings did you use? Please give the output of this or a similar command: select name, setting from pg_settings

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-11-23 Thread Noah Misch
On Fri, Nov 22, 2019 at 01:21:31PM +0100, Peter Eisentraut wrote: > On 2019-11-05 22:16, Robert Haas wrote: > >First, I'd like to restate my understanding of the problem just to see > >whether I've got the right idea and whether we're all on the same > >page. When wal_level=minimal, we sometimes

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-11-22 Thread Peter Eisentraut
On 2019-11-05 22:16, Robert Haas wrote: First, I'd like to restate my understanding of the problem just to see whether I've got the right idea and whether we're all on the same page. When wal_level=minimal, we sometimes try to skip WAL logging on newly-created relations in favor of fsync-ing the

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-11-21 Thread Kyotaro Horiguchi
At Thu, 21 Nov 2019 16:01:07 +0900 (JST), Kyotaro Horiguchi wrote in > > For deleting relfilenodes, smgrDoPendingDeletes() collects a list for > > smgrdounlinkall() to pass to DropRelFileNodesAllBuffers(), which is > > sophisticated about optimizing the shared buffers scan. Commit 279628a > >

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-11-20 Thread Kyotaro Horiguchi
Wow.. This is embarrassing.. *^^*. At Thu, 21 Nov 2019 16:01:07 +0900 (JST), Kyotaro Horiguchi wrote in > I should have replied this first. > > At Sun, 17 Nov 2019 20:54:34 -0800, Noah Misch wrote in > > On Tue, Nov 05, 2019 at 02:53:35PM -0800, Noah Misch wrote: > > > I started pre-commit

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-11-20 Thread Kyotaro Horiguchi
I should have replied this first. At Sun, 17 Nov 2019 20:54:34 -0800, Noah Misch wrote in > On Tue, Nov 05, 2019 at 02:53:35PM -0800, Noah Misch wrote: > > I started pre-commit editing on 2019-10-28, and comment+README updates have > > been the largest part of that. I'll check my edits against

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-11-20 Thread Kyotaro Horiguchi
At Sun, 17 Nov 2019 20:54:34 -0800, Noah Misch wrote in > On Tue, Nov 05, 2019 at 02:53:35PM -0800, Noah Misch wrote: > > I started pre-commit editing on 2019-10-28, and comment+README updates have > > been the largest part of that. I'll check my edits against the things you > > list here, and

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-11-19 Thread Kyotaro Horiguchi
I'm in the benchmarking week.. Thanks for reviewing!. At Sun, 17 Nov 2019 20:54:34 -0800, Noah Misch wrote in > On Tue, Nov 05, 2019 at 02:53:35PM -0800, Noah Misch wrote: > > I started pre-commit editing on 2019-10-28, and comment+README updates have > > been the largest part of that. I'll

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-11-17 Thread Noah Misch
On Tue, Nov 05, 2019 at 02:53:35PM -0800, Noah Misch wrote: > I started pre-commit editing on 2019-10-28, and comment+README updates have > been the largest part of that. I'll check my edits against the things you > list here, and I'll share on-list before committing. I've now marked the CF >

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-11-06 Thread Kyotaro Horiguchi
Thank you for looking this. At Tue, 5 Nov 2019 16:16:14 -0500, Robert Haas wrote in > On Fri, Oct 25, 2019 at 9:21 AM Kyotaro Horiguchi > wrote: > > This is the fixed verison v22. > First, I'd like to restate my understanding of the problem just to see .. > Second, for anyone who is not

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-11-05 Thread Noah Misch
On Tue, Nov 05, 2019 at 04:16:14PM -0500, Robert Haas wrote: > On Fri, Oct 25, 2019 at 9:21 AM Kyotaro Horiguchi > wrote: > > This is the fixed verison v22. > > I'd like to offer a few thoughts on this thread and on these patches, > which is now more than 4 years old and more than 150 messages

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-11-05 Thread Robert Haas
On Fri, Oct 25, 2019 at 9:21 AM Kyotaro Horiguchi wrote: > This is the fixed verison v22. I'd like to offer a few thoughts on this thread and on these patches, which is now more than 4 years old and more than 150 messages in length. First, I'd like to restate my understanding of the problem

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-10-25 Thread Kyotaro Horiguchi
On Fri, Oct 25, 2019 at 1:13 PM Kyotaro Horiguchi wrote: > Hello. Thanks for the comment. > > # Sorry in advance for possilbe breaking the thread. > > > MarkBufferDirtyHint() writes WAL even when rd_firstRelfilenodeSubid or > > rd_createSubid is set; see attached test case. It needs to skip WAL

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-10-24 Thread Kyotaro Horiguchi
Ugh! 2019年10月25日(金) 13:13 Kyotaro Horiguchi : > that. Instead, In the attached, MarkBufferDirtyHint() asks storage.c > for sync-pending state of the relfilenode for the buffer. In the > attached patch (0003) > regards. > It's wrong that it also skips chnging flags. I"ll fix it soon -- Kyotaro

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-10-24 Thread Kyotaro Horiguchi
Hello. Thanks for the comment. # Sorry in advance for possilbe breaking the thread. > MarkBufferDirtyHint() writes WAL even when rd_firstRelfilenodeSubid or > rd_createSubid is set; see attached test case. It needs to skip WAL whenever > RelationNeedsWAL() returns false. Thanks for pointing

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-09-10 Thread Noah Misch
[Casual readers with opinions on GUC naming: consider skipping to the end.] MarkBufferDirtyHint() writes WAL even when rd_firstRelfilenodeSubid or rd_createSubid is set; see attached test case. It needs to skip WAL whenever RelationNeedsWAL() returns false. On Tue, Aug 27, 2019 at 03:49:32PM

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-09-02 Thread Noah Misch
On Mon, Sep 02, 2019 at 05:15:00PM -0400, Alvaro Herrera wrote: > I have updated this patch's status to "needs review", since v20 has not > received any comments yet. > > Noah, you're listed as committer for this patch. Are you still on the > hook for getting it done during the v13 timeframe?

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-09-02 Thread Alvaro Herrera
I have updated this patch's status to "needs review", since v20 has not received any comments yet. Noah, you're listed as committer for this patch. Are you still on the hook for getting it done during the v13 timeframe? -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-08-28 Thread Kyotaro Horiguchi
Hello, Noah. At Tue, 27 Aug 2019 15:49:32 +0900 (Tokyo Standard Time), Kyotaro Horiguchi wrote in <20190827.154932.250364935.horikyota@gmail.com> > I'm not sure whether the knob shows apparent performance gain and > whether we can offer the criteria to identify the proper > value. But I'll

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-08-27 Thread Kyotaro Horiguchi
Hello. At Sun, 25 Aug 2019 22:08:43 -0700, Noah Misch wrote in <20190826050843.gb3153...@rfd.leadboat.com> noah> On Thu, Aug 22, 2019 at 09:06:06PM +0900, Kyotaro Horiguchi wrote: noah> > At Mon, 19 Aug 2019 23:03:14 -0700, Noah Misch wrote in <20190820060314.ga3086...@rfd.leadboat.com> > > >

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-08-27 Thread Kyotaro Horiguchi
At Tue, 27 Aug 2019 15:49:32 +0900 (Tokyo Standard Time), Kyotaro Horiguchi wrote in <20190827.154932.250364935.horikyota@gmail.com> > 128GB shared buffers contain 16M buffers. On my > perhaps-Windows-Vista-era box, such loop takes 15ms. (Since it > has only 6GB, the test is ignoring the

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-08-25 Thread Noah Misch
On Thu, Aug 22, 2019 at 09:06:06PM +0900, Kyotaro Horiguchi wrote: > At Mon, 19 Aug 2019 23:03:14 -0700, Noah Misch wrote in > <20190820060314.ga3086...@rfd.leadboat.com> > > On Mon, Aug 19, 2019 at 06:59:59PM +0900, Kyotaro Horiguchi wrote: > > > At Sat, 17 Aug 2019 20:52:30 -0700, Noah Misch

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-08-22 Thread Kyotaro Horiguchi
Hello. At Mon, 19 Aug 2019 23:03:14 -0700, Noah Misch wrote in <20190820060314.ga3086...@rfd.leadboat.com> > On Mon, Aug 19, 2019 at 06:59:59PM +0900, Kyotaro Horiguchi wrote: > > At Sat, 17 Aug 2019 20:52:30 -0700, Noah Misch wrote in > > <20190818035230.gb3021...@rfd.leadboat.com> > > > For

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-08-21 Thread Noah Misch
On Wed, Aug 21, 2019 at 04:32:38PM +0900, Kyotaro Horiguchi wrote: > At Mon, 19 Aug 2019 18:59:59 +0900 (Tokyo Standard Time), Kyotaro Horiguchi > wrote in > <20190819.185959.118543656.horikyota@gmail.com> > > At Sat, 17 Aug 2019 20:52:30 -0700, Noah Misch wrote in > >

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-08-21 Thread Kyotaro Horiguchi
Hello. New version is attached. At Mon, 19 Aug 2019 18:59:59 +0900 (Tokyo Standard Time), Kyotaro Horiguchi wrote in <20190819.185959.118543656.horikyota@gmail.com> > Thank you for taking time. > > At Sat, 17 Aug 2019 20:52:30 -0700, Noah Misch wrote in >

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-08-20 Thread Kyotaro Horiguchi
Hello. At Mon, 19 Aug 2019 18:59:59 +0900 (Tokyo Standard Time), Kyotaro Horiguchi wrote in <20190819.185959.118543656.horikyota@gmail.com> > > The comment material being deleted is still correct, so don't delete it. > > Moreover, the code managing rd_firstRelfilenodeSubid has a similar

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-08-20 Thread Noah Misch
On Mon, Aug 19, 2019 at 06:59:59PM +0900, Kyotaro Horiguchi wrote: > At Sat, 17 Aug 2019 20:52:30 -0700, Noah Misch wrote in > <20190818035230.gb3021...@rfd.leadboat.com> > > For two-phase commit, PrepareTransaction() needs to execute pending syncs. > > Now TwoPhaseFileHeader has two new

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-08-19 Thread Kyotaro Horiguchi
Thank you for taking time. At Sat, 17 Aug 2019 20:52:30 -0700, Noah Misch wrote in <20190818035230.gb3021...@rfd.leadboat.com> > For two-phase commit, PrepareTransaction() needs to execute pending syncs. Now TwoPhaseFileHeader has two new members for (commit-time) pending syncs. Pending-syncs

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-08-17 Thread Noah Misch
For two-phase commit, PrepareTransaction() needs to execute pending syncs. On Thu, Jul 25, 2019 at 10:39:36AM +0900, Kyotaro Horiguchi wrote: > --- a/src/backend/access/heap/heapam_handler.c > +++ b/src/backend/access/heap/heapam_handler.c > @@ -715,12 +702,6 @@

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-08-02 Thread Kyotaro Horiguchi
Hello. At Fri, 2 Aug 2019 11:35:06 +1200, Thomas Munro wrote in > On Sat, Jul 27, 2019 at 6:26 PM Noah Misch wrote: > > On Thu, Jul 25, 2019 at 10:39:36AM +0900, Kyotaro Horiguchi wrote: > > > No substantial change have been made by this rebasing. > > > > Thanks. I'll likely review this on

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-08-01 Thread Thomas Munro
On Sat, Jul 27, 2019 at 6:26 PM Noah Misch wrote: > On Thu, Jul 25, 2019 at 10:39:36AM +0900, Kyotaro Horiguchi wrote: > > No substantial change have been made by this rebasing. > > Thanks. I'll likely review this on 2019-08-20. If someone opts to review it > earlier, I welcome that. Cool.

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-07-27 Thread Noah Misch
On Thu, Jul 25, 2019 at 10:39:36AM +0900, Kyotaro Horiguchi wrote: > No substantial change have been made by this rebasing. Thanks. I'll likely review this on 2019-08-20. If someone opts to review it earlier, I welcome that.

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-07-24 Thread Kyotaro Horiguchi
I found that CF-bot complaining on this. Seems that some comment fixes by the recent 21039555cd are the cause. No substantial change have been made by this rebasing. regards. On Fri, Jul 12, 2019 at 5:37 PM Kyotaro Horiguchi wrote: > > At Fri, 12 Jul 2019 17:30:41 +0900 (Tokyo Standard Time),

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-07-12 Thread Kyotaro Horiguchi
At Fri, 12 Jul 2019 17:30:41 +0900 (Tokyo Standard Time), Kyotaro Horiguchi wrote in <20190712.173041.236938840.horikyota@gmail.com> > The v16 seems no longer works so I'll send further rebased version. It's just by renaming of TestLib::real_dir to perl2host. This is rebased version v17.

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-07-12 Thread Kyotaro Horiguchi
Many message seem lost during moving to new environmet.. I'm digging the archive but coudn't find the message for v15.. At Thu, 11 Jul 2019 18:03:35 -0700, Noah Misch wrote in <20190712010335.gb1610...@rfd.leadboat.com> > On Wed, Jul 10, 2019 at 01:19:14PM +0900, Kyotaro Horiguchi wrote: > >

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-07-11 Thread Noah Misch
On Wed, Jul 10, 2019 at 01:19:14PM +0900, Kyotaro Horiguchi wrote: > Hello. Rebased the patch to master(bd56cd75d2). It looks like you did more than just a rebase, because this v16 no longer modifies many files that v14 did modify. (That's probably good, since you had pending review comments.)

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-07-09 Thread Kyotaro Horiguchi
Hello. Rebased the patch to master(bd56cd75d2). regards. -- Kyotaro Horiguchi NTT Open Source Software Center >From ac52e2c1c56a96c1745149ff4220a3a116d6c811 Mon Sep 17 00:00:00 2001 From: Kyotaro Horiguchi Date: Thu, 11 Oct 2018 10:03:21 +0900 Subject: [PATCH 1/3] TAP test for copy-truncation

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-06-28 Thread Amit Kapila
On Tue, May 28, 2019 at 4:33 AM Noah Misch wrote: > > On Mon, May 27, 2019 at 02:08:26PM +0900, Kyotaro HORIGUCHI wrote: > > At Fri, 24 May 2019 19:33:32 -0700, Noah Misch wrote in > > <20190525023332.ge1624...@rfd.leadboat.com> > > > On Mon, May 20, 2019 at 03:54:30PM +0900, Kyotaro HORIGUCHI

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-05-27 Thread Noah Misch
On Mon, May 27, 2019 at 02:08:26PM +0900, Kyotaro HORIGUCHI wrote: > At Fri, 24 May 2019 19:33:32 -0700, Noah Misch wrote in > <20190525023332.ge1624...@rfd.leadboat.com> > > On Mon, May 20, 2019 at 03:54:30PM +0900, Kyotaro HORIGUCHI wrote: > > > Following this direction, the attached PoC works

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-05-26 Thread Kyotaro HORIGUCHI
Thanks for the comment! At Fri, 24 May 2019 19:33:32 -0700, Noah Misch wrote in <20190525023332.ge1624...@rfd.leadboat.com> > On Mon, May 20, 2019 at 03:54:30PM +0900, Kyotaro HORIGUCHI wrote: > > Following this direction, the attached PoC works *at least for* > > the wal_optimization TAP

  1   2   >