Re: 001_rep_changes.pl stalls

2020-04-16 Thread Noah Misch
On Mon, Apr 13, 2020 at 09:45:16PM -0400, Tom Lane wrote: > Noah Misch writes: > > This seems to have made the following race condition easier to hit: > > https://www.postgresql.org/message-id/flat/20200206074552.GB3326097%40rfd.leadboat.com > > https://www.postgresq

Re: cleaning perl code

2020-04-15 Thread Noah Misch
On Wed, Apr 15, 2020 at 03:43:36PM -0400, Andrew Dunstan wrote: > On 4/14/20 4:44 PM, Alvaro Herrera wrote: > > On 2020-Apr-14, Andrew Dunstan wrote: > >> One of the things that's a bit sad is that perlcritic doesn't generally > >> let you apply policies to a given set of files or files matching

Re: Race condition in SyncRepGetSyncStandbysPriority

2020-04-14 Thread Noah Misch
On Tue, Apr 14, 2020 at 04:32:40PM -0400, Tom Lane wrote: > I wrote: > > It doesn't seem to me to be that hard to implement the desired > > semantics for synchronous_standby_names with inconsistent info. > > In FIRST mode you basically just need to take the N smallest > > priorities you see in the

Re: 001_rep_changes.pl stalls

2020-04-13 Thread Noah Misch
On Sun, Apr 05, 2020 at 11:36:49PM -0700, Noah Misch wrote: > Executive summary: the "MyWalSnd->write < sentPtr" in WalSndWaitForWal() is > important for promptly updating pg_stat_replication. When caught up, we > should impose that logic before every sleep. The

Re: cleaning perl code

2020-04-12 Thread Noah Misch
On Sat, Apr 11, 2020 at 12:13:08PM -0400, Andrew Dunstan wrote: > --- a/src/tools/msvc/Project.pm > +++ b/src/tools/msvc/Project.pm > @@ -420,13 +420,10 @@ sub read_file > { > my $filename = shift; > my $F; > - my $t = $/; > - > - undef $/; > + local $/ = undef; >

Re: cleaning perl code

2020-04-12 Thread Noah Misch
On Sat, Apr 11, 2020 at 11:14:52AM -0400, Tom Lane wrote: > Noah Misch writes: > > In summary, among those warnings, I see non-negative value in "Code before > > warnings are enabled" only. While we're changing this, I propose removing > > Subroutines::RequireFin

Re: cleaning perl code

2020-04-10 Thread Noah Misch
On Thu, Apr 09, 2020 at 11:44:11AM -0400, Andrew Dunstan wrote: > 39 Always unpack @_ first Requiring a "my @args = @_" does not improve this code: sub CreateSolution { ... if ($visualStudioVersion eq '12.00') { return new VS2013Solution(@_);

Re: Spurious "apparent wraparound" via SimpleLruTruncate() rounding

2020-04-06 Thread Noah Misch
On Mon, Apr 06, 2020 at 02:46:09PM -0400, Tom Lane wrote: > Noah Misch writes: > > On Wed, Mar 25, 2020 at 04:42:31PM -0400, Tom Lane wrote: > >> So I think what we're actually trying to accomplish here is to > >> ensure that instead of deleting up to half of the SLRU s

Optimizing RelationFindReplTupleSeq() for CLOBBER_CACHE_ALWAYS

2020-04-06 Thread Noah Misch
er 1800 polls separated by 0.1s sleeps, however long that takes.) I had guessed that commit 1c7a0b3 greatly accelerated this test case, but it gave about a 4% improvement under CLOBBER_CACHE_ALWAYS. Author: Noah Misch Commit: Noah Misch Optimize RelationFindReplTupleSeq() for CLOBBER

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-04-06 Thread Noah Misch
On Mon, Apr 06, 2020 at 09:46:31AM +0900, Kyotaro Horiguchi wrote: > At Sat, 4 Apr 2020 15:32:12 -0700, Noah Misch wrote in > > On Sat, Apr 04, 2020 at 06:24:34PM -0400, Tom Lane wrote: > > > Shouldn't the CF entry get closed? > > > > Once the buildfarm is clean f

001_rep_changes.pl stalls

2020-04-06 Thread Noah Misch
happen in WalSndWaitForWal(). Attached. I don't know whether this is important outside of testing scenarios. I lean against back-patching, but I will back-patch if someone thinks this qualifies as a performance bug. Thanks, nm [1] I spot-checked only my animals, since I wanted to experiment on an

Re: where should I stick that backup?

2020-04-05 Thread Noah Misch
On Fri, Apr 03, 2020 at 10:19:21AM -0400, Robert Haas wrote: > What I'm thinking about is: suppose we add an option to pg_basebackup > with a name like --pipe-output. This would be mutually exclusive with > -D, but would work at least with -Ft and maybe also with -Fp. The > argument to

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-04-04 Thread Noah Misch
On Sat, Apr 04, 2020 at 06:24:34PM -0400, Tom Lane wrote: > Shouldn't the CF entry get closed? Once the buildfarm is clean for a day, sure. The buildfarm has already revealed a missing perl2host call.

Re: Autovacuum vs vac_update_datfrozenxid() vs ?

2020-04-01 Thread Noah Misch
On Mon, Mar 23, 2020 at 04:50:36PM -0700, Andres Freund wrote: > I think there's also another (even larger?) race in > vac_update_datfrozenxid(): Unless I miss something, two backends can > concurrently run through the scan in vac_update_datfrozenxid() for two > different tables in the same

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-04-01 Thread Noah Misch
On Mon, Mar 30, 2020 at 11:37:57PM -0700, Andres Freund wrote: > On 2020-03-30 23:28:54 -0700, Noah Misch wrote: > > On Mon, Mar 30, 2020 at 04:43:00PM +0900, Michael Paquier wrote: > > > On Sun, Mar 29, 2020 at 09:41:01PM -0700, Noah Misch wrote: > > > > I think atta

Re: backup manifests

2020-03-31 Thread Noah Misch
On Tue, Mar 31, 2020 at 03:50:34PM -0700, Andres Freund wrote: > On 2020-03-31 14:10:34 -0400, Robert Haas wrote: > > +/* > > + * Attempt to parse the WAL files required to restore from backup using > > + * pg_waldump. > > + */ > > +static void > > +parse_required_wal(validator_context *context,

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-03-31 Thread Noah Misch
On Mon, Mar 30, 2020 at 04:43:00PM +0900, Michael Paquier wrote: > On Sun, Mar 29, 2020 at 09:41:01PM -0700, Noah Misch wrote: > > I think attached v41nm is ready for commit. Would anyone like to vote > > against > > back-patching this? It's hard to justify lack of back-

Re: backup manifests

2020-03-30 Thread Noah Misch
On Mon, Mar 30, 2020 at 12:16:31PM -0700, Andres Freund wrote: > On 2020-03-30 15:04:55 -0400, Robert Haas wrote: > > I guess I'd like to be clear here that I have no fundamental > > disagreement with taking this tool in any direction that people would > > like it to go. For me it's just a

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-03-30 Thread Noah Misch
On Mon, Mar 30, 2020 at 02:56:11PM +0900, Kyotaro Horiguchi wrote: > At Sun, 29 Mar 2020 21:41:01 -0700, Noah Misch wrote in > > Since pendingSyncHash is always NULL under XLogIsNeeded(), I also removed > > some > > XLogIsNeeded() tests that immediately preceded !

Re: backup manifests

2020-03-30 Thread Noah Misch
On Sun, Mar 29, 2020 at 08:42:35PM -0400, Robert Haas wrote: > On Sat, Mar 28, 2020 at 11:40 PM Noah Misch wrote: > > I think this functionality doesn't belong in its own program. If you > > suspect > > pg_basebackup or pg_restore will eventually gain the ability to

Re: Spurious "apparent wraparound" via SimpleLruTruncate() rounding

2020-03-29 Thread Noah Misch
On Wed, Mar 25, 2020 at 04:42:31PM -0400, Tom Lane wrote: > Noah Misch writes: > > On Thu, Mar 19, 2020 at 06:04:52PM -0400, Tom Lane wrote: > >> 3. It feels like the proposed test of cutoff position against both > >> ends of a segment is a roundabout way of fixing

Re: backup manifests

2020-03-28 Thread Noah Misch
On Fri, Mar 27, 2020 at 01:53:54PM -0400, Robert Haas wrote: > - Replace a doc paragraph about the advantages and disadvantages of > CRC-32C with one by Stephen Frost, with a slightly change by me that I > thought made it sound more grammatical. Defaulting to CRC-32C seems prudent to me: - As

Re: Spurious "apparent wraparound" via SimpleLruTruncate() rounding

2020-03-22 Thread Noah Misch
On Thu, Mar 19, 2020 at 06:04:52PM -0400, Tom Lane wrote: > Yeah, this patch has been waiting in the queue for way too long :-(. Thanks for reviewing. > I spent some time studying this, and I have a few comments/questions: > > 1. It seems like this discussion is conflating two different issues.

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-03-21 Thread Noah Misch
On Sat, Mar 21, 2020 at 12:01:27PM -0700, Noah Misch wrote: > Pushed, after adding a missing "break" to gist_identify() and tweaking two > more comments. However, a diverse minority of buildfarm members are failing > like this, in most branches: > > Mar 21 13:16:37 #

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-03-21 Thread Noah Misch
On Sun, Mar 15, 2020 at 08:46:47PM -0700, Noah Misch wrote: > On Wed, Mar 04, 2020 at 04:29:19PM +0900, Kyotaro Horiguchi wrote: > > The attached is back-patches from 9.5 through master. > > Thanks. I've made some edits. I'll plan to push the attached patches on > Friday or

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-03-01 Thread Noah Misch
On Thu, Feb 27, 2020 at 04:00:24PM +0900, Kyotaro Horiguchi wrote: > At Tue, 25 Feb 2020 21:36:12 -0800, Noah Misch wrote in > > On Tue, Feb 25, 2020 at 10:01:51AM +0900, Kyotaro Horiguchi wrote: > > > At Sat, 22 Feb 2020 21:12:20 -0800, Noah Misch wrote > > > in &

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-02-25 Thread Noah Misch
On Tue, Feb 25, 2020 at 10:01:51AM +0900, Kyotaro Horiguchi wrote: > At Sat, 22 Feb 2020 21:12:20 -0800, Noah Misch wrote in > > On Fri, Feb 21, 2020 at 04:49:59PM +0900, Kyotaro Horiguchi wrote: > > > At Wed, 19 Feb 2020 17:29:08 +0900 (JST), Kyotaro Horiguchi > > >

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-02-22 Thread Noah Misch
On Fri, Feb 21, 2020 at 04:49:59PM +0900, Kyotaro Horiguchi wrote: > At Wed, 19 Feb 2020 17:29:08 +0900 (JST), Kyotaro Horiguchi > wrote in > > At Tue, 18 Feb 2020 23:44:52 -0800, Noah Misch wrote in > > > - When reusing an index build, instead of stor

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-02-18 Thread Noah Misch
uture-proof to me. - When reusing an index build, instead of storing the dropped relid in the IndexStmt and opening the dropped relcache entry in ATExecAddIndex(), store the subid fields in the IndexStmt. This is less code, and I felt RelationIdGetRelationCache() invited misuse. Author: N

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-02-18 Thread Noah Misch
On Tue, Feb 18, 2020 at 03:56:15PM +1300, Thomas Munro wrote: > CREATE TYPE priv_testtype1 AS (a int, b text); > +ERROR: relation 24844 deleted while still in use > REVOKE USAGE ON TYPE priv_testtype1 FROM PUBLIC; > > https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.79923 >

SyncRepGetSyncStandbysPriority() vs. SIGHUP

2020-02-05 Thread Noah Misch
Buildfarm runs have triggered the assertion at the end of SyncRepGetSyncStandbysPriority(): sysname │ snapshot │branch │ bfurl

TestLib condition for deleting temporary directories

2020-02-02 Thread Noah Misch
Forking thread "logical decoding : exceeded maxAllocatedDescs for .spill files" for this side issue: On Wed, Jan 08, 2020 at 09:37:04PM -0800, Noah Misch wrote: > v10 > deletes PostgresNode base directories at the end of this test file, despite > the failur

Re: SimpleLruTruncate() mutual exclusion

2020-01-31 Thread Noah Misch
On Thu, Jan 30, 2020 at 04:34:33PM +0100, Dmitry Dolgov wrote: > > On Sun, Nov 17, 2019 at 10:14:26PM -0800, Noah Misch wrote: > > > I'm probably missing something, so just wanted to clarify. Do I > > > understand correctly, that thread [1] and this one are independent,

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-01-26 Thread Noah Misch
On Mon, Jan 27, 2020 at 01:44:13PM +0900, Kyotaro Horiguchi wrote: > At Sun, 26 Jan 2020 20:22:01 -0800, Noah Misch wrote in > > Diffing the two latest versions of one patch: > > > --- v32-0002-Fix-the-defect-1.patch 2020-01-18 14:32:47.499129940 > > > -

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-01-26 Thread Noah Misch
Diffing the two latest versions of one patch: > --- v32-0002-Fix-the-defect-1.patch 2020-01-18 14:32:47.499129940 -0800 > +++ v33-0002-Fix-the-defect-1.patch 2020-01-26 16:23:52.846391035 -0800 > +@@ -2978,8 +3054,8 @@ AssertPendingSyncs_RelationCache(void) > +

Re: Strange coding in _mdfd_openseg()

2020-01-25 Thread Noah Misch
On Thu, Apr 04, 2019 at 12:15:52PM +0900, Kyotaro HORIGUCHI wrote: > At Wed, 3 Apr 2019 13:47:46 -0700, Andres Freund wrote > in <20190403204746.2yumq7c2mirmo...@alap3.anarazel.de> > > On 2019-04-04 09:24:49 +1300, Thomas Munro wrote: > > > On Wed, Apr 3, 2019 at 5:34 PM Kyotaro HORIGUCHI > > >

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-01-18 Thread Noah Misch
On Tue, Jan 14, 2020 at 07:35:22PM +0900, Kyotaro Horiguchi wrote: > At Thu, 26 Dec 2019 12:46:39 +0900 (JST), Kyotaro Horiguchi > wrote in > > At Wed, 25 Dec 2019 16:15:21 -0800, Noah Misch wrote in > > > === Defect 1: Forgets to skip WAL after SAVEPOINT;

Re: Allow relocatable extension to use @extschema@?

2020-01-18 Thread Noah Misch
On Thu, Dec 05, 2019 at 09:52:50AM -0500, Robert Haas wrote: > On Tue, Dec 3, 2019 at 10:18 AM Tom Lane wrote: > > Alexander Korotkov writes: > > > But nevertheless should we allow relocatable extension to use > > > @extschema@. Any thoughts? > > > > No. The reasoning in the comment still

Re: Consolidate 'unique array values' logic into a reusable function?

2020-01-12 Thread Noah Misch
On Wed, Jan 08, 2020 at 02:49:48PM +1300, Thomas Munro wrote: > On Sun, Dec 29, 2019 at 8:02 PM Noah Misch wrote: > > ==00:00:00:28.322 1527557== Source and destination overlap in > > memcpy(0x1000104, 0x1000104, 4) > > ==00:00:00:28.322 1527557==at 0x4C2E74

Re: Windows UTF-8, non-ICU collation trouble

2020-01-11 Thread Noah Misch
On Wed, Dec 11, 2019 at 01:54:47PM +1300, Thomas Munro wrote: > On Tue, Dec 10, 2019 at 10:29 PM Noah Misch wrote: > > This does suggest some set of CompareString* parameters is free from the > > problem. If that's right, we could offer collations based on that. (I'm > >

Re: logical decoding : exceeded maxAllocatedDescs for .spill files

2020-01-09 Thread Noah Misch
On Thu, Jan 09, 2020 at 12:45:41AM -0500, Tom Lane wrote: > Noah Misch writes: > > Even so, a web search for "extend_brk" led to the answer. By default, > > 32-bit > > AIX binaries get only 256M of RAM for stack and sbrk. The new regression > > test >

Re: logical decoding : exceeded maxAllocatedDescs for .spill files

2020-01-08 Thread Noah Misch
On Wed, Jan 08, 2020 at 02:50:53PM +0530, Amit Khandekar wrote: > On Sun, 5 Jan 2020 at 00:21, Noah Misch wrote: > > The buildfarm client can capture stack traces, but it currently doesn't do > > so > > for TAP test suites (search the client code for get_stack_trace). If

Re: sidewinder has one failure

2020-01-04 Thread Noah Misch
On Sat, Jan 04, 2020 at 06:56:48AM +0530, Amit Kapila wrote: > On Sat, Jan 4, 2020 at 6:19 AM Tom Lane wrote: > > =?UTF-8?Q?Mikael_Kjellstr=c3=b6m?= writes: > > > I tried starting it from cron and then I got: > > > max_safe_fds = 981, usable_fds = 1000, already_open = 9 > > > > Oh! There we

Re: Spurious "apparent wraparound" via SimpleLruTruncate() rounding

2020-01-04 Thread Noah Misch
On Sun, Jan 05, 2020 at 01:33:55AM +0100, Tomas Vondra wrote: > On Wed, Feb 13, 2019 at 11:26:23PM -0800, Noah Misch wrote: > >a. Fix the rounding in SimpleLruTruncate(). (The patch I posted upthread is > > wrong; I will correct it in a separate message.) > > > >b.

Re: logical decoding : exceeded maxAllocatedDescs for .spill files

2020-01-04 Thread Noah Misch
On Fri, Jan 03, 2020 at 02:20:09PM +0530, Amit Khandekar wrote: > On Fri, 3 Jan 2020 at 10:19, Amit Kapila wrote: > > On Fri, Jan 3, 2020 at 8:29 AM Amit Kapila wrote: > >> I see one failure in REL_10_STABLE [1] which seems to be due to this > >> commit: > > > > I tried this test on my CentOs

Re: mdclose() does not cope w/ FileClose() failure

2020-01-01 Thread Noah Misch
On Wed, Dec 25, 2019 at 10:39:32AM +0900, Kyotaro Horiguchi wrote: > At Tue, 24 Dec 2019 11:57:39 -0800, Noah Misch wrote in > > On Mon, Dec 23, 2019 at 07:41:49PM +0900, Kyotaro Horiguchi wrote: > > > If we regard repalloc as far faster than FileOpen/FileClose or we car

Re: Consolidate 'unique array values' logic into a reusable function?

2019-12-28 Thread Noah Misch
On Mon, Nov 04, 2019 at 12:02:21PM +1300, Thomas Munro wrote: > Rebased. I'm planning to commit this soon. In each installcheck-parallel run under valgrind-3.14.0, I now see ~1200 reports like this: ==00:00:00:28.322 1527557== Source and destination overlap in memcpy(0x1000104, 0x1000104, 4)

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-12-25 Thread Noah Misch
On Thu, Dec 26, 2019 at 12:46:39PM +0900, Kyotaro Horiguchi wrote: > At Wed, 25 Dec 2019 16:15:21 -0800, Noah Misch wrote in > > Skip AssertPendingSyncs_RelationCache() at abort, like v24nm did. Making > > that work no matter what does ereport(ERROR) would be tricky and

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-12-25 Thread Noah Misch
By improving AssertPendingSyncs_RelationCache() and by testing with -DRELCACHE_FORCE_RELEASE, I now know of three defects in the attached v30nm. Would you fix these? === Defect 1: Forgets to skip WAL after SAVEPOINT; DROP TABLE; ROLLBACK TO A test in transactions.sql now fails in

Re: mdclose() does not cope w/ FileClose() failure

2019-12-24 Thread Noah Misch
On Mon, Dec 23, 2019 at 07:41:49PM +0900, Kyotaro Horiguchi wrote: > At Sun, 22 Dec 2019 12:21:00 -0800, Noah Misch wrote in > > On Sun, Dec 22, 2019 at 01:19:30AM -0800, Noah Misch wrote: > > > An alternative would be to call > > > _fdvec_resize() after every FileClos

Re: mdclose() does not cope w/ FileClose() failure

2019-12-22 Thread Noah Misch
On Mon, Dec 23, 2019 at 09:33:29AM +1300, Thomas Munro wrote: > On Sun, Dec 22, 2019 at 10:19 PM Noah Misch wrote: > > Assert(segno == reln->md_num_open_segs[forknum]); > > _fdvec_resize(reln, forknum, segno + 1); > > Oh yeah, I spotted that part

Re: mdclose() does not cope w/ FileClose() failure

2019-12-22 Thread Noah Misch
On Sun, Dec 22, 2019 at 01:19:30AM -0800, Noah Misch wrote: > I am inclined to fix this by decrementing md_num_open_segs before modifying > md_seg_fds (second attachment). That leaked memory, since _fdvec_resize() assumes md_num_open_segs is also the allocated array length. The alter

mdclose() does not cope w/ FileClose() failure

2019-12-22 Thread Noah Misch
Forking thread "WAL logging problem in 9.4.3?" for this tangent: On Mon, Dec 09, 2019 at 06:04:06PM +0900, Kyotaro Horiguchi wrote: > I don't understand why mdclose checks for (v->mdfd_vfd >= 0) of open > segment but anyway mdimmedsync is believing that that won't happen and > I follow the

Re: Windows UTF-8, non-ICU collation trouble

2019-12-10 Thread Noah Misch
On Tue, Dec 10, 2019 at 03:41:15PM +1300, Thomas Munro wrote: > On Fri, Dec 6, 2019 at 8:33 PM Noah Misch wrote: > > On Fri, Dec 06, 2019 at 07:56:08PM +1300, Thomas Munro wrote: > > > On Fri, Dec 6, 2019 at 7:34 PM Noah Misch wrote: > > > > We use system UTF-1

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-12-08 Thread Noah Misch
e segments; that's fine to do in other smgrimmedsync() callers, even though they operate on relations that can't have inactive segments. On Tue, Dec 03, 2019 at 08:51:46PM +0900, Kyotaro Horiguchi wrote: > At Thu, 28 Nov 2019 17:23:19 -0500, Noah Misch wrote in > > On Thu, Nov 28, 2019

Re: libpq sslpassword parameter and callback function

2019-12-05 Thread Noah Misch
On Fri, Nov 29, 2019 at 09:27:02AM -0500, Andrew Dunstan wrote: > On 11/28/19 10:25 PM, Greg Nancarrow wrote: > > 3) Documentation for the "PQsslpassword" function should be added to the > > libpq "33.2 Connection Status Functions" section. > I plan to commit this tomorrow. The PQsslpassword()

Re: Windows UTF-8, non-ICU collation trouble

2019-12-05 Thread Noah Misch
On Fri, Dec 06, 2019 at 07:56:08PM +1300, Thomas Munro wrote: > On Fri, Dec 6, 2019 at 7:34 PM Noah Misch wrote: > > We use system UTF-16 collation to implement UTF-8 collation on Windows. The > > PostgreSQL security team received a report, from Timothy Kuun, that this >

Windows UTF-8, non-ICU collation trouble

2019-12-05 Thread Noah Misch
We use system UTF-16 collation to implement UTF-8 collation on Windows. The PostgreSQL security team received a report, from Timothy Kuun, that this collation does not uphold the "symmetric law" and "transitive law" that we require for btree operator classes. The attached test program

Recovery vs. RelationTruncate(); skipFsync vs. unlogged rels

2019-11-29 Thread Noah Misch
While reviewing the patches in http://postgr.es/m/20191126.213752.2132434859202124793.horikyota@gmail.com I noticed three related problems. The defects motivating that other thread are specific to wal_level=minimal. The related problems are not specific to any wal_level, hence my starting a

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-11-28 Thread Noah Misch
On Thu, Nov 28, 2019 at 09:35:08PM +0900, Kyotaro Horiguchi wrote: > I measured the performance with the latest patch set. > > > 1. Determine $DDL_COUNT, a number of DDL transactions that take about one > >minute when done via syncs. > > 2. Start "pgbench -rP1 --progress-timestamp -T180 -c10

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-11-25 Thread Noah Misch
On Mon, Nov 25, 2019 at 03:58:14PM -0500, Robert Haas wrote: > On Sat, Nov 23, 2019 at 4:21 PM Noah Misch wrote: > > I noticed an additional defect: > > > > BEGIN; > > CREATE TABLE t (c) AS SELECT 1; > > CHECKPOINT; -- write and fsync the table's one page >

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-11-24 Thread Noah Misch
On Mon, Nov 25, 2019 at 11:08:54AM +0900, Kyotaro Horiguchi wrote: > At Sat, 23 Nov 2019 16:21:36 -0500, Noah Misch wrote in > > This benchmark procedure may help: > > > > 1. Determine $DDL_COUNT, a number of DDL transactions that take about one > >minute when d

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-11-23 Thread Noah Misch
siveLock, in which case this has no side effects. On Thu, Nov 21, 2019 at 04:01:07PM +0900, Kyotaro Horiguchi wrote: > At Sun, 17 Nov 2019 20:54:34 -0800, Noah Misch wrote in > > === Defect 1: gistGetFakeLSN() > > > > When I modified pg_regress.c to use wal_level=m

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-11-23 Thread Noah Misch
On Fri, Nov 22, 2019 at 01:21:31PM +0100, Peter Eisentraut wrote: > On 2019-11-05 22:16, Robert Haas wrote: > >First, I'd like to restate my understanding of the problem just to see > >whether I've got the right idea and whether we're all on the same > >page. When wal_level=minimal, we sometimes

Re: SimpleLruTruncate() mutual exclusion

2019-11-17 Thread Noah Misch
On Sun, Nov 17, 2019 at 12:56:52PM +0100, Dmitry Dolgov wrote: > > On Wed, Jul 31, 2019 at 11:51:17PM -0700, Noah Misch wrote: > > > Also, if I understand the data-loss hazard properly, it's what you > > > said in the other thread: the latest_page_number could advance

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-11-17 Thread Noah Misch
On Tue, Nov 05, 2019 at 02:53:35PM -0800, Noah Misch wrote: > I started pre-commit editing on 2019-10-28, and comment+README updates have > been the largest part of that. I'll check my edits against the things you > list here, and I'll share on-list before committing. I've now mark

Re: SimpleLruTruncate() mutual exclusion

2019-11-15 Thread Noah Misch
On Mon, Nov 04, 2019 at 03:43:09PM -0800, Noah Misch wrote: > On Mon, Nov 04, 2019 at 03:26:35PM +1300, Thomas Munro wrote: > > On Thu, Aug 1, 2019 at 6:51 PM Noah Misch wrote: > > > vac_truncate_clog() instance 1 starts, considers segment ABCD eligible to > > > unl

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-11-05 Thread Noah Misch
On Tue, Nov 05, 2019 at 04:16:14PM -0500, Robert Haas wrote: > On Fri, Oct 25, 2019 at 9:21 AM Kyotaro Horiguchi > wrote: > > This is the fixed verison v22. > > I'd like to offer a few thoughts on this thread and on these patches, > which is now more than 4 years old and more than 150 messages

Re: SimpleLruTruncate() mutual exclusion

2019-11-04 Thread Noah Misch
On Mon, Nov 04, 2019 at 03:26:35PM +1300, Thomas Munro wrote: > On Thu, Aug 1, 2019 at 6:51 PM Noah Misch wrote: > > vac_truncate_clog() instance 1 starts, considers segment ABCD eligible to > > unlink > > vac_truncate_clog() instance 2 starts, considers segment ABCD el

ProcArrayGroupClearXid() compare-exchange style

2019-10-14 Thread Noah Misch
ProcArrayGroupClearXid() has this: while (true) { nextidx = pg_atomic_read_u32(>procArrayGroupFirst); ... if (pg_atomic_compare_exchange_u32(>procArrayGroupFirst,

Re: [HACKERS] Deadlock in XLogInsert at AIX

2019-10-12 Thread Noah Misch
On Wed, Oct 09, 2019 at 01:15:29PM -0400, Tom Lane wrote: > Noah Misch writes: > > On Mon, Oct 07, 2019 at 03:06:35PM -0400, Tom Lane wrote: > >> This still fails on Apple's compilers. ... > > > Thanks for testing. That error boils down to "need to use some

Re: [HACKERS] Deadlock in XLogInsert at AIX

2019-10-09 Thread Noah Misch
On Mon, Oct 07, 2019 at 03:06:35PM -0400, Tom Lane wrote: > Noah Misch writes: > > [ fetch-add-gcc-xlc-unify-v2.patch ] > > This still fails on Apple's compilers. The first failure I get is > > ccache gcc -std=gnu99 -Wall -Wmissing-prototypes -Wpointer-arith > -Wdec

Re: expressive test macros (was: Report test_atomic_ops() failures consistently, via macros)

2019-10-07 Thread Noah Misch
On Mon, Oct 07, 2019 at 09:56:20PM +0200, Peter Eisentraut wrote: > On 2019-10-06 04:20, Noah Misch wrote: > > elog(ERROR, \ > > "%s yielded %u, expected %s (%u) in file \"%s\" line %u", \ > > #result_expr, result, #expected_expr, expected, __

Re: expressive test macros (was: Report test_atomic_ops() failures consistently, via macros)

2019-10-05 Thread Noah Misch
On Sat, Oct 05, 2019 at 12:07:29PM -0700, Andres Freund wrote: > +#define EXPECT_EQ_U32(result_expr, expected_expr) \ > + do { \ > + uint32 result = (result_expr); \ > + uint32 expected = (expected_expr); \ > + if (result != expected) \ > + elog(ERROR, \ >

Re: [HACKERS] Deadlock in XLogInsert at AIX

2019-10-05 Thread Noah Misch
On Sat, Aug 31, 2019 at 03:30:26PM -0700, Noah Misch wrote: > On Sat, Aug 31, 2019 at 02:27:55PM -0400, Tom Lane wrote: > > Noah Misch writes: > > > Done. fetch-add-variable-test-v1.patch just adds tests for non-constant > > > addends and 16-bit edge cases. Today's i

Re: pgsql: Use data directory inode number, not port, to select SysV resour

2019-09-13 Thread Noah Misch
On Sun, Sep 08, 2019 at 05:54:12PM -0400, Andrew Dunstan wrote: > I'm going to disable this test (src/test/recovery/t/017_shm.pl) on > Windows on the back branches too unless there's a violent objection. The > reason is that the script runs "postgres --single" and that fails on > Windows when run

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-09-10 Thread Noah Misch
+0900, Kyotaro Horiguchi wrote: > At Sun, 25 Aug 2019 22:08:43 -0700, Noah Misch wrote in > <20190826050843.gb3153...@rfd.leadboat.com> > > Consider a one-page relfilenode. Doing all the things you list for a single > > page may be cheaper than locking millions of b

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-09-02 Thread Noah Misch
On Mon, Sep 02, 2019 at 05:15:00PM -0400, Alvaro Herrera wrote: > I have updated this patch's status to "needs review", since v20 has not > received any comments yet. > > Noah, you're listed as committer for this patch. Are you still on the > hook for getting it done during the v13 timeframe?

Re: [HACKERS] Deadlock in XLogInsert at AIX

2019-08-31 Thread Noah Misch
On Sat, Aug 31, 2019 at 02:27:55PM -0400, Tom Lane wrote: > Noah Misch writes: > > Done. fetch-add-variable-test-v1.patch just adds tests for non-constant > > addends and 16-bit edge cases. Today's implementation handles those, > > PostgreSQL doesn't use them, and I mig

Re: [HACKERS] Deadlock in XLogInsert at AIX

2019-08-31 Thread Noah Misch
On Wed, Jan 17, 2018 at 12:36:31AM -0800, Noah Misch wrote: > On Tue, Jan 16, 2018 at 08:50:24AM -0800, Andres Freund wrote: > > On 2018-01-16 16:12:11 +0900, Michael Paquier wrote: > > > On Fri, Feb 03, 2017 at 12:26:50AM +, Noah Misch wrote: > > > > Since this

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-08-25 Thread Noah Misch
On Thu, Aug 22, 2019 at 09:06:06PM +0900, Kyotaro Horiguchi wrote: > At Mon, 19 Aug 2019 23:03:14 -0700, Noah Misch wrote in > <20190820060314.ga3086...@rfd.leadboat.com> > > On Mon, Aug 19, 2019 at 06:59:59PM +0900, Kyotaro Horiguchi wrote: > > > At Sat, 17 Aug 2019

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-08-21 Thread Noah Misch
On Wed, Aug 21, 2019 at 04:32:38PM +0900, Kyotaro Horiguchi wrote: > At Mon, 19 Aug 2019 18:59:59 +0900 (Tokyo Standard Time), Kyotaro Horiguchi > wrote in > <20190819.185959.118543656.horikyota@gmail.com> > > At Sat, 17 Aug 2019 20:52:30 -0700, Noah Misch wrote in

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-08-20 Thread Noah Misch
On Mon, Aug 19, 2019 at 06:59:59PM +0900, Kyotaro Horiguchi wrote: > At Sat, 17 Aug 2019 20:52:30 -0700, Noah Misch wrote in > <20190818035230.gb3021...@rfd.leadboat.com> > > For two-phase commit, PrepareTransaction() needs to execute pending syncs. > > Now TwoPha

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-08-17 Thread Noah Misch
For two-phase commit, PrepareTransaction() needs to execute pending syncs. On Thu, Jul 25, 2019 at 10:39:36AM +0900, Kyotaro Horiguchi wrote: > --- a/src/backend/access/heap/heapam_handler.c > +++ b/src/backend/access/heap/heapam_handler.c > @@ -715,12 +702,6 @@

Re: SimpleLruTruncate() mutual exclusion

2019-08-01 Thread Noah Misch
On Mon, Jul 29, 2019 at 12:58:17PM -0400, Tom Lane wrote: > > On Sun, Feb 17, 2019 at 11:31:03PM -0800, Noah Misch wrote: > >>> b. Arrange so only one backend runs vac_truncate_clog() at a time. Other > >>> than > >>> AsyncCtl, every SL

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-07-27 Thread Noah Misch
On Thu, Jul 25, 2019 at 10:39:36AM +0900, Kyotaro Horiguchi wrote: > No substantial change have been made by this rebasing. Thanks. I'll likely review this on 2019-08-20. If someone opts to review it earlier, I welcome that.

Re: Spurious "apparent wraparound" via SimpleLruTruncate() rounding

2019-07-24 Thread Noah Misch
On Wed, Jul 24, 2019 at 05:27:18PM +0900, Kyotaro Horiguchi wrote: > Sorry in advance for link-breaking message forced by gmail.. Using the archives page "Resend email" link avoids that. > https://www.postgresql.org/message-id/flat/20190202083822.gc32...@gust.leadboat.com > > > 1. The result of

Re: Race conditions with TAP test for syncrep

2019-07-23 Thread Noah Misch
On Tue, Jun 18, 2019 at 09:59:08AM +0900, Michael Paquier wrote: > On Mon, Jun 17, 2019 at 10:50:39AM -0400, Alvaro Herrera wrote: > > I think this should be note() rather than print(), or maybe diag(). (I > > see that we have a couple of other cases which use print() in the tap > > tests, which

Re: [RFC] Removing "magic" oids

2019-07-20 Thread Noah Misch
On Sat, Jul 20, 2019 at 12:56:34PM -0700, Andres Freund wrote: > On 2019-07-20 11:21:52 -0700, Noah Misch wrote: > > On Fri, Jul 19, 2019 at 10:12:57AM -0700, Andres Freund wrote: > > > On 2019-07-07 10:00:35 -0700, Noah Misch wrote: > > > > +# Test concurrent OID ge

Re: [RFC] Removing "magic" oids

2019-07-20 Thread Noah Misch
On Fri, Jul 19, 2019 at 10:12:57AM -0700, Andres Freund wrote: > On 2019-07-07 10:00:35 -0700, Noah Misch wrote: > > +# Test concurrent OID generation via pg_enum_oid_index. This indirectly > > +# exercises LWLock and spinlock concurrency. > > +my $labels = join ',', map

Re: [Patch] Mingw: Fix import library extension, build actual static libraries

2019-07-11 Thread Noah Misch
On Tue, Jul 09, 2019 at 08:26:52AM -0400, Andrew Dunstan wrote: > On 4/16/19 1:22 AM, Noah Misch wrote: > > On Thu, Mar 07, 2019 at 03:23:50PM +0100, Sandro Mani wrote: > >> Related, no actual static libraries are produced alongside the respective > >> dlls. The attac

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-07-11 Thread Noah Misch
On Wed, Jul 10, 2019 at 01:19:14PM +0900, Kyotaro Horiguchi wrote: > Hello. Rebased the patch to master(bd56cd75d2). It looks like you did more than just a rebase, because this v16 no longer modifies many files that v14 did modify. (That's probably good, since you had pending review comments.)

Re: [RFC] Removing "magic" oids

2019-07-07 Thread Noah Misch
On Tue, Nov 20, 2018 at 01:20:04AM -0800, Andres Freund wrote: > On 2018-11-14 21:02:41 -0800, Andres Freund wrote: > > On 2018-11-15 04:57:28 +0000, Noah Misch wrote: > > > On Wed, Nov 14, 2018 at 12:01:52AM -0800, Andres Freund wrote: > > > > - one pgbench tes

Re: Fix runtime errors from -fsanitize=undefined

2019-07-05 Thread Noah Misch
On Fri, Jul 05, 2019 at 06:14:31PM +0200, Peter Eisentraut wrote: > On 2019-07-05 01:33, Noah Misch wrote: > > I just saw this proposal. The undefined behavior in question is strictly > > academic. These changes do remove the need for new users to discover > > -fno-sanit

Re: Fix runtime errors from -fsanitize=undefined

2019-07-04 Thread Noah Misch
On Mon, Jun 03, 2019 at 09:21:48PM +0200, Peter Eisentraut wrote: > After many years of trying, it seems the -fsanitize=undefined checking > in gcc is now working somewhat reliably. Attached is a patch that fixes > all errors of the kind > > runtime error: null pointer passed as argument N,

Re: SimpleLruTruncate() mutual exclusion

2019-06-28 Thread Noah Misch
On Sun, Feb 17, 2019 at 11:31:03PM -0800, Noah Misch wrote: > I'm forking this thread from > https://postgr.es/m/flat/20190202083822.gc32...@gust.leadboat.com, which > reported a race condition involving the "apparent wraparound" safety check in > SimpleLruTruncate(): >

Re: fix psql \conninfo & \connect when using hostaddr

2019-06-24 Thread Noah Misch
On Mon, Jun 24, 2019 at 08:52:00AM -0400, Alvaro Herrera wrote: > On 2019-Jun-24, Michael Paquier wrote: > > On Fri, Jun 14, 2019 at 06:31:52PM -0400, Alvaro Herrera wrote: > > > BTW I tested this manually and it seemed to work as intended, ie., if I > > > change /etc/hosts for the hostname I am

Re: $host_cpu -> $target_cpu in configure?

2019-06-16 Thread Noah Misch
On Sun, Jun 16, 2019 at 12:56:52PM -0400, Tom Lane wrote: > There are a few places in configure and the makefiles that are looking > at $host_cpu to decide what to do. As far as I can tell, almost all of > them are wrong and should be looking at $target_cpu instead. (The > lack of complaints

Re: Draft back-branch release notes are up for review

2019-06-15 Thread Noah Misch
On Sat, Jun 15, 2019 at 06:05:00PM -0400, Tom Lane wrote: > Noah Misch writes: > > On Sat, Jun 15, 2019 at 02:11:41PM -0700, Peter Geoghegan wrote: > >> I agree that this isn't terribly significant in general. Your proposed > >> wording seems better than what w

Re: Draft back-branch release notes are up for review

2019-06-15 Thread Noah Misch
On Sat, Jun 15, 2019 at 02:11:41PM -0700, Peter Geoghegan wrote: > On Sat, Jun 15, 2019 at 1:39 PM Noah Misch wrote: > > To me, this text implies a cautious DBA should amcheck every index. Reading > > the thread[1], I no longer think that. It's enough to monitor that VACUUM &g

<    2   3   4   5   6   7   8   9   >