On Mon, Feb 08, 2016 at 10:55:24PM -0500, Tom Lane wrote:
> Noah Misch <n...@leadboat.com> writes:
> > On Mon, Feb 08, 2016 at 02:15:48PM -0500, Tom Lane wrote:
> >> We've seen variants
> >> on this theme on half a dozen machines just in the past week --- and it
> >> seems to mostly happen in 9.5 and HEAD, which is fishy.
> > It has been affecting only the four AIX animals, which do share hardware.
> > (Back in 2015 and once in 2016-01, it did affect axolotl and shearwater.)
> Certainly your AIX critters have shown this a bunch, but here's another
> current example:
> http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=axolotl&dt=2016-02-08%2014%3A49%3A23

Oops; I did not consider Monday's results before asserting that.

> > That's reasonable.  If you would like higher-fidelity data, I can run loops 
> > of
> > "pg_ctl -w start; make installcheck; pg_ctl -t900 -w stop", and I could run
> > that for HEAD and 9.2 simultaneously.  A day of logs from that should show
> > clearly if HEAD is systematically worse than 9.2.
> That sounds like a fine plan, please do it.

Log files:


While I didn't study those logs in detail, a few things jumped out.  Since
9.2, we've raised the default shared_buffers from 32MB to 128MB, and we've
replaced checkpoint_segments=3 with max_wal_size=1GB.  Both changes encourage
bulkier checkpoints.  The 9.2 test runs get one xlog-driven checkpoint before
the shutdown checkpoint, while HEAD gets one time-driven checkpoint.  Also,
the HEAD suite just tests more things.  Here's pg_stat_bgwriter afterward:

checkpoints_timed     | 156
checkpoints_req       | 799
checkpoint_write_time | 16035847
checkpoint_sync_time  | 6555396
buffers_checkpoint    | 744131
buffers_clean         | 0
maxwritten_clean      | 0
buffers_backend       | 3023444
buffers_backend_fsync | 0
buffers_alloc         | 1777010
stats_reset           | 2016-02-08 21:04:24.499607-08

checkpoints_timed     | 39
checkpoints_req       | 1369
checkpoint_write_time | 14875776
checkpoint_sync_time  | 8397536
buffers_checkpoint    | 396272
buffers_clean         | 466392
maxwritten_clean      | 1336
buffers_backend       | 1961531
buffers_backend_fsync | 0
buffers_alloc         | 1681324
stats_reset           | 2016-02-08 21:09:21.925487-08

Most notable there is the lack of bgwriter help in HEAD.  The clusters had
initdb-default configuration apart from these additions:

log_line_prefix = '%p %m '
logging_collector = on
log_autovacuum_min_duration = 0
log_checkpoints = on
log_lock_waits = on
log_temp_files = 128kB

I should have added fsync=off, too.  Notice how the AIX animals failed left
and right today, likely thanks to contention from these runs.

On Tue, Feb 09, 2016 at 02:10:50PM -0500, Tom Lane wrote:
> (1) Slow file system, specifically slow unlink, is the core of the
> problem.  (I wonder if the AIX critters are using an NFS filesystem?)

The buildfarm files lie on 600 GiB SAS disks.  I suspect metadata operations,
like unlink(), bottleneck on the jfs2 journal.

On Tue, Feb 09, 2016 at 03:05:01PM -0500, Tom Lane wrote:
> I'm now in favor of applying the PGCTLTIMEOUT patch Noah proposed, and
> *removing* the two existing hacks in run_build.pl that try to force -t 120.
> The only real argument I can see against that approach is that we'd have
> to back-patch the PGCTLTIMEOUT patch to all active branches if we want
> to stop the buildfarm failures.  We don't usually back-patch feature
> additions.  On the other hand, this wouldn't be the first time we've
> back-patched something on grounds of helping the buildfarm, so I find
> that argument pretty weak.

If I were a purist against back-patching features, I might name the variable
PGINTERNAL_TEST_PGCTLTIMEOUT and not document it.  Meh to that.  I'll plan to
commit the original tomorrow.

Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:

Reply via email to