Re: [COMMITTERS] pgsql: Don't vacuum all-frozen pages.

2016-03-10 Thread Michael Paquier 
On Fri, Mar 11, 2016 at 2:08 AM, Amit Langote
 wrote:
> On 2016/03/11 8:19, Peter Geoghegan wrote:
>> On Thu, Mar 10, 2016 at 1:22 PM, Andres Freund  wrote:
>>> Yeha!
>>
>> Fantastic effort, particularly from Masahiko. Well done.
>
> +1!

Yuhu.
-- 
Michael


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [COMMITTERS] pgsql: Checkpoint sorting and balancing.

2016-03-10 Thread Michael Paquier 
On Fri, Mar 11, 2016 at 2:29 AM, Andres Freund  wrote:
> Checkpoint sorting and balancing.

+1.
-- 
Michael


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Simplify GetLockNameFromTagType.

2016-03-10 Thread Robert Haas 
Simplify GetLockNameFromTagType.

The old code is wrong, because it returns a pointer to an automatic
variable.  And it's also more clever than we really need to be
considering that the case it's worrying about should never happen.

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/a414d96ad2b001cddf10e3dedb0bde96d0af6a0b

Modified Files
--
src/backend/storage/lmgr/lmgr.c | 16 +++-
1 file changed, 3 insertions(+), 13 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [COMMITTERS] pgsql: Checkpoint sorting and balancing.

2016-03-10 Thread Peter Eisentraut 
On 3/10/16 8:52 PM, Andres Freund wrote:
> Hrmpf.  I tried to fix the issue; could you quickly verify?

looks good



-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [COMMITTERS] pgsql: Provide much better wait information in pg_stat_activity.

2016-03-10 Thread Peter Eisentraut 
On 3/10/16 7:32 PM, Andres Freund wrote:
> On 2016-03-10 18:55:47 +, Robert Haas wrote:
>> Provide much better wait information in pg_stat_activity.
>>
>> When a process is waiting for a heavyweight lock, we will now indicate
>> the type of heavyweight lock for which it is waiting.  Also, you can
>> now see when a process is waiting for a lightweight lock - in which
>> case we will indicate the individual lock name or the tranche, as
>> appropriate - or for a buffer pin.
> 
> My compiler quite validly complains:
> 
> /home/andres/src/postgresql/src/backend/storage/lmgr/lmgr.c: In function 
> ‘GetLockNameFromTagType’:
> /home/andres/src/postgresql/src/backend/storage/lmgr/lmgr.c:1018:9: warning: 
> function may return address of local variable [-Wreturn-local-addr]
>   return locktypename;
>  ^
> /home/andres/src/postgresql/src/backend/storage/lmgr/lmgr.c:1007:8: note: 
> declared here
>   char  tnbuf[32];
> ^
> In file included from 
> /home/andres/src/postgresql/src/backend/commands/dbcommands.c:20:0:

Needs a "static", it seems.



-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [COMMITTERS] pgsql: Checkpoint sorting and balancing.

2016-03-10 Thread Andres Freund 
On 2016-03-10 20:47:10 -0500, Peter Eisentraut wrote:
> On 3/10/16 8:29 PM, Andres Freund wrote:
> > Checkpoint sorting and balancing.
>
> This patch appears to have introduced a change that makes the build
> with dtrace fail:
>
> In file included from ../../../../src/include/pg_trace.h:15:0,
>  from bufmgr.c:42:
> bufmgr.c: In function 'BufferSync':
> bufmgr.c:1939:59: error: 'num_to_write' undeclared (first use in this 
> function)
>   TRACE_POSTGRESQL_BUFFER_SYNC_DONE(NBuffers, num_written, num_to_write);
>^
> ../../../../src/include/utils/probes.h:93:83: note: in definition of macro 
> 'TRACE_POSTGRESQL_BUFFER_SYNC_DONE'
>   __dtrace_probe$postgresql$buffer__sync__done$v1$696e74$696e74$696e74(arg0, 
> arg1, arg2); \
>   
>  ^~~~
> bufmgr.c:1939:59: note: each undeclared identifier is reported only once for 
> each function it appears in
>   TRACE_POSTGRESQL_BUFFER_SYNC_DONE(NBuffers, num_written, num_to_write);
>^
> ../../../../src/include/utils/probes.h:93:83: note: in definition of macro 
> 'TRACE_POSTGRESQL_BUFFER_SYNC_DONE'
>   __dtrace_probe$postgresql$buffer__sync__done$v1$696e74$696e74$696e74(arg0, 
> arg1, arg2); \
>   
>  ^~~~

Hrmpf.  I tried to fix the issue; could you quickly verify?

These probes are a bit annoying; and I seriously doubt anybody actually
uses them. But obviously we should keep them working as long as we have
them.

Thanks!

Andres


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Blindly try to fix dtrace enabled builds, broken in 9cd00c45.

2016-03-10 Thread Andres Freund 
Blindly try to fix dtrace enabled builds, broken in 9cd00c45.

Reported-By: Peter Eisentraut
Discussion: 56e2239e.1050...@gmx.net

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/c94f0c29cecc7944a14aa645c8a97a7250bf146b

Modified Files
--
src/backend/storage/buffer/bufmgr.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [COMMITTERS] pgsql: Checkpoint sorting and balancing.

2016-03-10 Thread Peter Eisentraut 
On 3/10/16 8:29 PM, Andres Freund wrote:
> Checkpoint sorting and balancing.

This patch appears to have introduced a change that makes the build
with dtrace fail:

In file included from ../../../../src/include/pg_trace.h:15:0,
 from bufmgr.c:42:
bufmgr.c: In function 'BufferSync':
bufmgr.c:1939:59: error: 'num_to_write' undeclared (first use in this function)
  TRACE_POSTGRESQL_BUFFER_SYNC_DONE(NBuffers, num_written, num_to_write);
   ^
../../../../src/include/utils/probes.h:93:83: note: in definition of macro 
'TRACE_POSTGRESQL_BUFFER_SYNC_DONE'
  __dtrace_probe$postgresql$buffer__sync__done$v1$696e74$696e74$696e74(arg0, 
arg1, arg2); \

   ^~~~
bufmgr.c:1939:59: note: each undeclared identifier is reported only once for 
each function it appears in
  TRACE_POSTGRESQL_BUFFER_SYNC_DONE(NBuffers, num_written, num_to_write);
   ^
../../../../src/include/utils/probes.h:93:83: note: in definition of macro 
'TRACE_POSTGRESQL_BUFFER_SYNC_DONE'
  __dtrace_probe$postgresql$buffer__sync__done$v1$696e74$696e74$696e74(arg0, 
arg1, arg2); \

   ^~~~



-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Checkpoint sorting and balancing.

2016-03-10 Thread Andres Freund 
Checkpoint sorting and balancing.

Up to now checkpoints were written in the order they're in the
BufferDescriptors. That's nearly random in a lot of cases, which
performs badly on rotating media, but even on SSDs it causes slowdowns.

To avoid that, sort checkpoints before writing them out. We currently
sort by tablespace, relfilenode, fork and block number.

One of the major reasons that previously wasn't done, was fear of
imbalance between tablespaces. To address that balance writes between
tablespaces.

The other prime concern was that the relatively large allocation to sort
the buffers in might fail, preventing checkpoints from happening. Thus
pre-allocate the required memory in shared memory, at server startup.

This particularly makes it more efficient to have checkpoint flushing
enabled, because that'll often result in a lot of writes that can be
coalesced into one flush.

Discussion: alpine.DEB.2.10.150601132.28433@sto
Author: Fabien Coelho and Andres Freund

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/9cd00c457e6a1ebb984167ac556a9961812a683c

Modified Files
--
src/backend/storage/buffer/README |   5 -
src/backend/storage/buffer/buf_init.c |  22 ++-
src/backend/storage/buffer/bufmgr.c   | 274 +-
src/include/storage/buf_internals.h   |  18 +++
src/tools/pgindent/typedefs.list  |   2 +
5 files changed, 277 insertions(+), 44 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Allow to trigger kernel writeback after a configurable number of

2016-03-10 Thread Andres Freund 
Allow to trigger kernel writeback after a configurable number of writes.

Currently writes to the main data files of postgres all go through the
OS page cache. This means that some operating systems can end up
collecting a large number of dirty buffers in their respective page
caches.  When these dirty buffers are flushed to storage rapidly, be it
because of fsync(), timeouts, or dirty ratios, latency for other reads
and writes can increase massively.  This is the primary reason for
regular massive stalls observed in real world scenarios and artificial
benchmarks; on rotating disks stalls on the order of hundreds of seconds
have been observed.

On linux it is possible to control this by reducing the global dirty
limits significantly, reducing the above problem. But global
configuration is rather problematic because it'll affect other
applications; also PostgreSQL itself doesn't always generally want this
behavior, e.g. for temporary files it's undesirable.

Several operating systems allow some control over the kernel page
cache. Linux has sync_file_range(2), several posix systems have msync(2)
and posix_fadvise(2). sync_file_range(2) is preferable because it
requires no special setup, whereas msync() requires the to-be-flushed
range to be mmap'ed. For the purpose of flushing dirty data
posix_fadvise(2) is the worst alternative, as flushing dirty data is
just a side-effect of POSIX_FADV_DONTNEED, which also removes the pages
from the page cache.  Thus the feature is enabled by default only on
linux, but can be enabled on all systems that have any of the above
APIs.

While desirable and likely possible this patch does not contain an
implementation for windows.

With the infrastructure added, writes made via checkpointer, bgwriter
and normal user backends can be flushed after a configurable number of
writes. Each of these sources of writes controlled by a separate GUC,
checkpointer_flush_after, bgwriter_flush_after and backend_flush_after
respectively; they're separate because the number of flushes that are
good are separate, and because the performance considerations of
controlled flushing for each of these are different.

A later patch will add checkpoint sorting - after that flushes from the
ckeckpoint will almost always be desirable. Bgwriter flushes are most of
the time going to be random, which are slow on lots of storage hardware.
Flushing in backends works well if the storage and bgwriter can keep up,
but if not it can have negative consequences.  This patch is likely to
have negative performance consequences without checkpoint sorting, but
unfortunately so has sorting without flush control.

Discussion: alpine.DEB.2.10.150601132.28433@sto
Author: Fabien Coelho and Andres Freund

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/428b1d6b29ca599c5700d4bc4f4ce4c5880369bf

Modified Files
--
doc/src/sgml/config.sgml  |  87 +++
doc/src/sgml/wal.sgml |  11 ++
src/backend/postmaster/bgwriter.c |   8 +-
src/backend/storage/buffer/buf_init.c |   5 +
src/backend/storage/buffer/bufmgr.c   | 193 +-
src/backend/storage/file/copydir.c|   4 +-
src/backend/storage/file/fd.c | 157 +++
src/backend/storage/smgr/md.c |  50 +
src/backend/storage/smgr/smgr.c   |  19 +++-
src/backend/utils/misc/guc.c  |  36 +++
src/include/storage/buf_internals.h   |  31 +-
src/include/storage/bufmgr.h  |  22 +++-
src/include/storage/fd.h  |   3 +-
src/include/storage/smgr.h|   4 +
src/tools/pgindent/typedefs.list  |   2 +
15 files changed, 601 insertions(+), 31 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [COMMITTERS] pgsql: Don't vacuum all-frozen pages.

2016-03-10 Thread Amit Langote 
On 2016/03/11 8:19, Peter Geoghegan wrote:
> On Thu, Mar 10, 2016 at 1:22 PM, Andres Freund  wrote:
>> Yeha!
> 
> Fantastic effort, particularly from Masahiko. Well done.

+1!

- Amit




-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [COMMITTERS] pgsql: Provide much better wait information in pg_stat_activity.

2016-03-10 Thread Andres Freund 
On 2016-03-10 18:55:47 +, Robert Haas wrote:
> Provide much better wait information in pg_stat_activity.
> 
> When a process is waiting for a heavyweight lock, we will now indicate
> the type of heavyweight lock for which it is waiting.  Also, you can
> now see when a process is waiting for a lightweight lock - in which
> case we will indicate the individual lock name or the tranche, as
> appropriate - or for a buffer pin.

My compiler quite validly complains:

/home/andres/src/postgresql/src/backend/storage/lmgr/lmgr.c: In function 
‘GetLockNameFromTagType’:
/home/andres/src/postgresql/src/backend/storage/lmgr/lmgr.c:1018:9: warning: 
function may return address of local variable [-Wreturn-local-addr]
  return locktypename;
 ^
/home/andres/src/postgresql/src/backend/storage/lmgr/lmgr.c:1007:8: note: 
declared here
  char  tnbuf[32];
^
In file included from 
/home/andres/src/postgresql/src/backend/commands/dbcommands.c:20:0:


Andres


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [COMMITTERS] pgsql: Don't vacuum all-frozen pages.

2016-03-10 Thread Peter Geoghegan 
On Thu, Mar 10, 2016 at 1:22 PM, Andres Freund  wrote:
> Yeha!

Fantastic effort, particularly from Masahiko. Well done.


-- 
Peter Geoghegan


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Give pull_var_clause() reject/recurse/return behavior for Window

2016-03-10 Thread Tom Lane 
Give pull_var_clause() reject/recurse/return behavior for WindowFuncs too.

All along, this function should have treated WindowFuncs in a manner
similar to Aggrefs, ie with an option whether or not to recurse into them.
By not considering the case, it was always recursing, which is OK for most
callers (although I suspect that the case in prepare_sort_from_pathkeys
might represent a bug).  But now we need return-without-recursing behavior
as well.  There are also more than a few callers that should never see a
WindowFunc, and now we'll get some error checking on that.

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/c82c92b111b7b636e80f8a432de10c62011b35b6

Modified Files
--
src/backend/optimizer/path/equivclass.c  |  1 +
src/backend/optimizer/plan/createplan.c  |  4 +++-
src/backend/optimizer/plan/initsplan.c   |  5 -
src/backend/optimizer/plan/planner.c |  9 ++---
src/backend/optimizer/prep/preptlist.c   |  1 +
src/backend/optimizer/util/placeholder.c |  2 ++
src/backend/optimizer/util/var.c | 25 +
src/backend/utils/adt/selfuncs.c |  1 +
src/include/optimizer/var.h  |  6 --
9 files changed, 47 insertions(+), 7 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [COMMITTERS] pgsql: Don't vacuum all-frozen pages.

2016-03-10 Thread Andres Freund 
On 2016-03-10 21:16:37 +, Robert Haas wrote:
> Don't vacuum all-frozen pages.

Yeha!


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Don't vacuum all-frozen pages.

2016-03-10 Thread Robert Haas 
Don't vacuum all-frozen pages.

Commit a892234f830e832110f63fc0a2afce2fb21d1584 gave us enough
infrastructure to avoid vacuuming pages where every tuple on the
page is already frozen.  So, replace the notion of a scan_all or
whole-table vacuum with the less onerous notion of an "aggressive"
vacuum, which will pages that are all-visible, but still skip those
that are all-frozen.

This should greatly reduce the cost of anti-wraparound vacuuming
on large clusters where the majority of data is never touched
between one cycle and the next, because we'll no longer have to
read all of those pages only to find out that we don't need to
do anything with them.

Patch by me, reviewed by Masahiko Sawada.

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/fd31cd265138019d9b5fe53043670898bc9f

Modified Files
--
doc/src/sgml/config.sgml  |  20 ++--
doc/src/sgml/maintenance.sgml |  86 +---
src/backend/commands/vacuumlazy.c | 209 --
3 files changed, 196 insertions(+), 119 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Refactor pull_var_clause's API to make it less tedious to extend

2016-03-10 Thread Tom Lane 
Refactor pull_var_clause's API to make it less tedious to extend.

In commit 1d97c19a0f748e94 and later c1d9579dd8bf3c92, we extended
pull_var_clause's API by adding enum-type arguments.  That's sort of a pain
to maintain, though, because it means every time we add a new behavior we
must touch every last one of the call sites, even if there's a reasonable
default behavior that most of them could use.  Let's switch over to using a
bitmask of flags, instead; that seems more maintainable and might save a
nanosecond or two as well.  This commit changes no behavior in itself,
though I'm going to follow it up with one that does add a new behavior.

In passing, remove flatten_tlist(), which has not been used since 9.1
and would otherwise need the same API changes.

Removing these enums means that optimizer/tlist.h no longer needs to
depend on optimizer/var.h.  Changing that caused a number of C files to
need addition of #include "optimizer/var.h" (probably we can thank old
runs of pgrminclude for that); but on balance it seems like a good change
anyway.

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/364a9f47ab363250f62dd2c381c4da435283725a

Modified Files
--
contrib/postgres_fdw/deparse.c|   1 -
src/backend/catalog/heap.c|   8 +--
src/backend/commands/trigger.c|   4 +-
src/backend/optimizer/path/allpaths.c |   4 +-
src/backend/optimizer/path/equivclass.c   |   2 +-
src/backend/optimizer/plan/analyzejoins.c |   1 +
src/backend/optimizer/plan/createplan.c   |   2 +-
src/backend/optimizer/plan/initsplan.c|   6 +-
src/backend/optimizer/plan/planner.c  |   5 +-
src/backend/optimizer/plan/setrefs.c  |  15 +++--
src/backend/optimizer/prep/prepjointree.c |   1 +
src/backend/optimizer/prep/preptlist.c|   3 +-
src/backend/optimizer/util/placeholder.c  |   4 +-
src/backend/optimizer/util/tlist.c|  28 
src/backend/optimizer/util/var.c  | 108 --
src/backend/parser/parse_agg.c|   1 +
src/backend/parser/parse_clause.c |   1 +
src/backend/utils/adt/selfuncs.c  |   2 +-
src/include/optimizer/tlist.h |   4 +-
src/include/optimizer/var.h   |  22 +++---
20 files changed, 94 insertions(+), 128 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [COMMITTERS] pgsql: Provide much better wait information in pg_stat_activity.

2016-03-10 Thread Pavel Stehule 
Hi

2016-03-10 19:55 GMT+01:00 Robert Haas :

> Provide much better wait information in pg_stat_activity.
>
> When a process is waiting for a heavyweight lock, we will now indicate
> the type of heavyweight lock for which it is waiting.  Also, you can
> now see when a process is waiting for a lightweight lock - in which
> case we will indicate the individual lock name or the tranche, as
> appropriate - or for a buffer pin.
>
> Amit Kapila, Ildus Kurbangaliev, reviewed by me.  Lots of helpful
> discussion and suggestions by many others, including Alexander
> Korotkov, Vladimir Borodin, and many others.
>
> Branch
> --
> master
>
>
I am trying to test this feature, and there I see not actual data. Maybe
this behave is not related to this patch:

create table foo(a int);
insert into foo values(10);

session one:

begin; select * from foo for update;

session two:

begin; select * from foo for update;
session two is waiting

session one:
select * from pg_stat_activity -- I don't see correct information about
session two

session two:
rollback; begin; select * from foo where a = 10 for update;
session two is waiting again

session one:
select * from pg_stat_activity; -- The content is not changed

So content of pg_stat_activity is not correct in holder lock session.
Independent third session see valid content of pg_stat_activity.

Hypothesis: the pg_stat_activity is not refreshed under opened transaction?

Regards

Pavel

[COMMITTERS] pgsql: Rework wait for AccessExclusiveLocks on Hot Standby

2016-03-10 Thread Simon Riggs 
Rework wait for AccessExclusiveLocks on Hot Standby

Earlier version committed in 9.0 caused spurious waits in some cases.
New infrastructure for lock waits in 9.3 used to correct and improve this.

Jeff Janes based upon a proposal by Simon Riggs, who also reviewed
Additional review comments from Amit Kapila

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/37c54863cf71a4a1126d21db8eb68974bef34374

Modified Files
--
src/backend/postmaster/startup.c  |  1 +
src/backend/storage/ipc/standby.c | 94 +--
src/backend/storage/lmgr/proc.c   | 74 ++
src/include/storage/standby.h |  3 ++
src/include/utils/timeout.h   |  1 +
5 files changed, 112 insertions(+), 61 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Provide much better wait information in pg_stat_activity.

2016-03-10 Thread Robert Haas 
Provide much better wait information in pg_stat_activity.

When a process is waiting for a heavyweight lock, we will now indicate
the type of heavyweight lock for which it is waiting.  Also, you can
now see when a process is waiting for a lightweight lock - in which
case we will indicate the individual lock name or the tranche, as
appropriate - or for a buffer pin.

Amit Kapila, Ildus Kurbangaliev, reviewed by me.  Lots of helpful
discussion and suggestions by many others, including Alexander
Korotkov, Vladimir Borodin, and many others.

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/53be0b1add7064ca5db3cd884302dfc3268d884e

Modified Files
--
doc/src/sgml/monitoring.sgml  | 414 +-
src/backend/access/transam/xact.c |  13 +-
src/backend/bootstrap/bootstrap.c |   2 +
src/backend/catalog/system_views.sql  |   3 +-
src/backend/postmaster/bgwriter.c |   3 +
src/backend/postmaster/checkpointer.c |   1 +
src/backend/postmaster/pgstat.c   | 116 +++---
src/backend/postmaster/walwriter.c|   2 +
src/backend/replication/walsender.c   |   2 +
src/backend/storage/buffer/bufmgr.c   |   5 +
src/backend/storage/lmgr/lmgr.c   |  23 ++
src/backend/storage/lmgr/lock.c   |   6 +-
src/backend/storage/lmgr/lwlock.c |  67 +-
src/backend/storage/lmgr/proc.c   |   3 +
src/backend/utils/adt/lockfuncs.c |   2 +-
src/backend/utils/adt/pgstatfuncs.c   | 126 +++
src/include/catalog/catversion.h  |   2 +-
src/include/catalog/pg_proc.h |   8 +-
src/include/pgstat.h  |  81 ++-
src/include/storage/lmgr.h|   2 +
src/include/storage/lock.h|   2 +
src/include/storage/lwlock.h  |   2 +
src/include/storage/proc.h|   2 +
src/test/regress/expected/rules.out   |   9 +-
24 files changed, 794 insertions(+), 102 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [COMMITTERS] pgsql: ltree: Zero padding bytes when allocating memory for externally

2016-03-10 Thread Tom Lane 
Andres Freund  writes:
> On 2016-03-10 10:57:59 -0500, Tom Lane wrote:
>>  sysname  |  snapshot   | count 
>> --+-+---
>> okapi | 2012-05-10 06:15:02 |   138
>> okapi | 2012-05-20 00:15:02 |   174
>> dromedary | 2014-03-28 00:13:52 | 4
>> dromedary | 2014-05-07 06:23:09 | 2
>> dromedary | 2016-03-08 23:27:02 | 2
>> 
>> So yeah, we've seen it before, but it sure looks like a system-specific
>> glitch.  dromedary is a pretty old machine (2006 Mac mini), so I would not
>> be surprised if it dropped a bit now and then.

> dromedary's indeed look rare enough. But okapi had a lot of those.

But note those were a lot of the same failures in a single buildfarm run.

BTW, this search would have found both the client-side report in
regression.diffs and the postmaster-log report of the same error, so
all of the counts above should be divided by two to identify how many
glitches actually occurred in a given run.

okapi clearly saw some time-extended condition that caused the failure to
occur for most regression-test sessions within a run.  But it hasn't
reappeared since 2012, so my money is on some short-lived kernel bug.
(okapi is running gentoo ... nuf said.)

I don't know what's happening on dromedary, but it's clearly pretty
darn rare.  Given the lack of comparable failures on other Mac buildfarm
members, I'm willing to write it off as a flaky memory bit or some such.

regards, tom lane


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [COMMITTERS] pgsql: ltree: Zero padding bytes when allocating memory for externally

2016-03-10 Thread Andres Freund 
On 2016-03-10 10:57:59 -0500, Tom Lane wrote:
> I wrote:
> > Andres Freund  writes:
> >> This is marked as the originating commit for a failure on dromedary:
> >> http://pgbuildfarm.org/cgi-bin/show_log.pl?nm=dromedary=2016-03-08%2023%3A27%3A02
> >> the origin of the failure is:
> >> FATAL:  no PostgreSQL user name specified in startup packet
> >> while in the 'comments' tests, which looks fairly independent of this 
> >> commit.
> 
> >> Tom, did you happen to change config on that machine or anything?
> 
> > No, haven't touched it in awhile.  Looks like just a glitch maybe?
> 
> I had a vague feeling that this looked familiar, and it occurred to me
> to scan the buildfarm database for occurrences of that error message.
> Quite a few hours later, I have these results:

Heh, I tried the same, but limited my search too much timewise; to limit
the cost of the query...

>   sysname  |  snapshot   | count 
> ---+-+---
>  okapi | 2012-05-10 06:15:02 |   138
>  okapi | 2012-05-20 00:15:02 |   174
>  dromedary | 2014-03-28 00:13:52 | 4
>  dromedary | 2014-05-07 06:23:09 | 2
>  dromedary | 2016-03-08 23:27:02 | 2
> 
> So yeah, we've seen it before, but it sure looks like a system-specific
> glitch.  dromedary is a pretty old machine (2006 Mac mini), so I would not
> be surprised if it dropped a bit now and then.

dromedary's indeed look rare enough. But okapi had a lot of those. But
I'm also not going to care about failures from 2012...

Thanks for checking!

Andres


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Document BRIN a bit more thoroughly

2016-03-10 Thread Alvaro Herrera 
Document BRIN a bit more thoroughly

The chapter "Interfacing Extensions To Indexes" and CREATE OPERATOR
CLASS reference page were missed when BRIN was added.  We document
all our other index access methods there, so make sure BRIN complies.

Author: Álvaro Herrera
Reported-By: Julien Rouhaud, Tom Lane
Reviewed-By: Emre Hasegeli
Discussion: https://www.postgresql.org/message-id/56CF604E.9000303%40dalibo.com
Backpatch: 9.5, where BRIN was introduced

Branch
--
REL9_5_STABLE

Details
---
http://git.postgresql.org/pg/commitdiff/c84c87ccc0e219a3f263aea9ec80068022ca1cc0

Modified Files
--
doc/src/sgml/ref/create_opclass.sgml |   4 +-
doc/src/sgml/xindex.sgml | 106 ++-
2 files changed, 106 insertions(+), 4 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Document BRIN a bit more thoroughly

2016-03-10 Thread Alvaro Herrera 
Document BRIN a bit more thoroughly

The chapter "Interfacing Extensions To Indexes" and CREATE OPERATOR
CLASS reference page were missed when BRIN was added.  We document
all our other index access methods there, so make sure BRIN complies.

Author: Álvaro Herrera
Reported-By: Julien Rouhaud, Tom Lane
Reviewed-By: Emre Hasegeli
Discussion: https://www.postgresql.org/message-id/56CF604E.9000303%40dalibo.com
Backpatch: 9.5, where BRIN was introduced

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/a3a8309d450f7c4d1b743e84ba54ef5f7877d7be

Modified Files
--
doc/src/sgml/ref/create_opclass.sgml |   4 +-
doc/src/sgml/xindex.sgml | 106 ++-
2 files changed, 106 insertions(+), 4 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [COMMITTERS] pgsql: ltree: Zero padding bytes when allocating memory for externally

2016-03-10 Thread Tom Lane 
I wrote:
> Andres Freund  writes:
>> This is marked as the originating commit for a failure on dromedary:
>> http://pgbuildfarm.org/cgi-bin/show_log.pl?nm=dromedary=2016-03-08%2023%3A27%3A02
>> the origin of the failure is:
>> FATAL:  no PostgreSQL user name specified in startup packet
>> while in the 'comments' tests, which looks fairly independent of this commit.

>> Tom, did you happen to change config on that machine or anything?

> No, haven't touched it in awhile.  Looks like just a glitch maybe?

I had a vague feeling that this looked familiar, and it occurred to me
to scan the buildfarm database for occurrences of that error message.
Quite a few hours later, I have these results:

  sysname  |  snapshot   | count 
---+-+---
 okapi | 2012-05-10 06:15:02 |   138
 okapi | 2012-05-20 00:15:02 |   174
 dromedary | 2014-03-28 00:13:52 | 4
 dromedary | 2014-05-07 06:23:09 | 2
 dromedary | 2016-03-08 23:27:02 | 2

So yeah, we've seen it before, but it sure looks like a system-specific
glitch.  dromedary is a pretty old machine (2006 Mac mini), so I would not
be surprised if it dropped a bit now and then.

regards, tom lane


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Avoid crash on old Windows with AVX2-capable CPU for VS2013 buil

2016-03-10 Thread Magnus Hagander 
Avoid crash on old Windows with AVX2-capable CPU for VS2013 builds

The Visual Studio 2013 CRT generates invalid code when it makes a 64-bit
build that is later used on a CPU that supports AVX2 instructions using a
version of Windows before 7SP1/2008R2SP1.

Detect this combination, and in those cases turn off the generation of
FMA3, per recommendation from the Visual Studio team.

The bug is actually in the CRT shipping with Visual Studio 2013, but
Microsoft have stated they're only fixing it in newer major versions.
The fix is therefor conditioned specifically on being built with this
version of Visual Studio, and not previous or later versions.

Author: Christian Ullrich

Branch
--
REL9_4_STABLE

Details
---
http://git.postgresql.org/pg/commitdiff/8857488b64c34c8084a14ebefbd667fd5285d1a8

Modified Files
--
src/backend/main/main.c | 21 +
1 file changed, 21 insertions(+)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Avoid crash on old Windows with AVX2-capable CPU for VS2013 buil

2016-03-10 Thread Magnus Hagander 
Avoid crash on old Windows with AVX2-capable CPU for VS2013 builds

The Visual Studio 2013 CRT generates invalid code when it makes a 64-bit
build that is later used on a CPU that supports AVX2 instructions using a
version of Windows before 7SP1/2008R2SP1.

Detect this combination, and in those cases turn off the generation of
FMA3, per recommendation from the Visual Studio team.

The bug is actually in the CRT shipping with Visual Studio 2013, but
Microsoft have stated they're only fixing it in newer major versions.
The fix is therefor conditioned specifically on being built with this
version of Visual Studio, and not previous or later versions.

Author: Christian Ullrich

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/9d90388247e093cd9b3ead79954df2ac18bfeb66

Modified Files
--
src/backend/main/main.c | 21 +
1 file changed, 21 insertions(+)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Avoid crash on old Windows with AVX2-capable CPU for VS2013 buil

2016-03-10 Thread Magnus Hagander 
Avoid crash on old Windows with AVX2-capable CPU for VS2013 builds

The Visual Studio 2013 CRT generates invalid code when it makes a 64-bit
build that is later used on a CPU that supports AVX2 instructions using a
version of Windows before 7SP1/2008R2SP1.

Detect this combination, and in those cases turn off the generation of
FMA3, per recommendation from the Visual Studio team.

The bug is actually in the CRT shipping with Visual Studio 2013, but
Microsoft have stated they're only fixing it in newer major versions.
The fix is therefor conditioned specifically on being built with this
version of Visual Studio, and not previous or later versions.

Author: Christian Ullrich

Branch
--
REL9_1_STABLE

Details
---
http://git.postgresql.org/pg/commitdiff/3e6385e27105a95ccc00ae475a37f83d431fc730

Modified Files
--
src/backend/main/main.c | 21 +
1 file changed, 21 insertions(+)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Avoid crash on old Windows with AVX2-capable CPU for VS2013 buil

2016-03-10 Thread Magnus Hagander 
Avoid crash on old Windows with AVX2-capable CPU for VS2013 builds

The Visual Studio 2013 CRT generates invalid code when it makes a 64-bit
build that is later used on a CPU that supports AVX2 instructions using a
version of Windows before 7SP1/2008R2SP1.

Detect this combination, and in those cases turn off the generation of
FMA3, per recommendation from the Visual Studio team.

The bug is actually in the CRT shipping with Visual Studio 2013, but
Microsoft have stated they're only fixing it in newer major versions.
The fix is therefor conditioned specifically on being built with this
version of Visual Studio, and not previous or later versions.

Author: Christian Ullrich

Branch
--
REL9_3_STABLE

Details
---
http://git.postgresql.org/pg/commitdiff/270d8a12e57d995b333bdd0a0936c5dc6e5aa1e4

Modified Files
--
src/backend/main/main.c | 21 +
1 file changed, 21 insertions(+)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Avoid crash on old Windows with AVX2-capable CPU for VS2013 buil

2016-03-10 Thread Magnus Hagander 
Avoid crash on old Windows with AVX2-capable CPU for VS2013 builds

The Visual Studio 2013 CRT generates invalid code when it makes a 64-bit
build that is later used on a CPU that supports AVX2 instructions using a
version of Windows before 7SP1/2008R2SP1.

Detect this combination, and in those cases turn off the generation of
FMA3, per recommendation from the Visual Studio team.

The bug is actually in the CRT shipping with Visual Studio 2013, but
Microsoft have stated they're only fixing it in newer major versions.
The fix is therefor conditioned specifically on being built with this
version of Visual Studio, and not previous or later versions.

Author: Christian Ullrich

Branch
--
REL9_2_STABLE

Details
---
http://git.postgresql.org/pg/commitdiff/78b597808108a9837b7cc3660eec4ddb9471ab2f

Modified Files
--
src/backend/main/main.c | 21 +
1 file changed, 21 insertions(+)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Avoid crash on old Windows with AVX2-capable CPU for VS2013 buil

2016-03-10 Thread Magnus Hagander 
Avoid crash on old Windows with AVX2-capable CPU for VS2013 builds

The Visual Studio 2013 CRT generates invalid code when it makes a 64-bit
build that is later used on a CPU that supports AVX2 instructions using a
version of Windows before 7SP1/2008R2SP1.

Detect this combination, and in those cases turn off the generation of
FMA3, per recommendation from the Visual Studio team.

The bug is actually in the CRT shipping with Visual Studio 2013, but
Microsoft have stated they're only fixing it in newer major versions.
The fix is therefor conditioned specifically on being built with this
version of Visual Studio, and not previous or later versions.

Author: Christian Ullrich

Branch
--
REL9_5_STABLE

Details
---
http://git.postgresql.org/pg/commitdiff/ad9a15d3f156dd42f6ed5e47fff1fa77e974c351

Modified Files
--
src/backend/main/main.c | 21 +
1 file changed, 21 insertions(+)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Reduce size of two phase file header

2016-03-10 Thread Simon Riggs 
Reduce size of two phase file header

Previously 2PC header was fixed at 200 bytes, which in most cases wasted
WAL space for a workload using 2PC heavily.

Pavan Deolasee, reviewed by Petr Jelinek

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/e0694cf9c74455fb87beb0f4d66b4c8f5f53715e

Modified Files
--
src/backend/access/transam/twophase.c | 16 
1 file changed, 12 insertions(+), 4 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Reduce lock level for altering fillfactor

2016-03-10 Thread Simon Riggs 
Reduce lock level for altering fillfactor

Fabrízio de Royes Mello and Simon Riggs

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/fcb4bfddb6ff46e4495b923987ed6a89bd256174

Modified Files
--
doc/src/sgml/ref/alter_table.sgml |  2 +-
src/backend/access/common/reloptions.c| 10 +-
src/test/regress/expected/alter_table.out | 24 
3 files changed, 18 insertions(+), 18 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Code review for b6fb6471f6afaf649e52f38269fd8c5c60647669.

2016-03-10 Thread Robert Haas 
Code review for b6fb6471f6afaf649e52f38269fd8c5c60647669.

Reports by Tomas Vondra, Vinayak Pokale, and Aleksander Alekseev.
Patch by Amit Langote.

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/090b287fc59e7a44da8c3e0823eecdc8ea4522f2

Modified Files
--
src/backend/postmaster/pgstat.c | 7 ---
src/backend/utils/adt/pgstatfuncs.c | 2 +-
2 files changed, 5 insertions(+), 4 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers