Re: [HACKERS] Function and view to retrieve WAL receiver status

[Michael Paquier]

On Fri, Jan 8, 2016 at 4:38 AM, Alvaro Herrera  wrote:
> Michael Paquier wrote:
>> On Thu, Jan 7, 2016 at 1:57 PM, Haribabu Kommi  
>> wrote:
>> > On Wed, Jan 6, 2016 at 8:00 PM, Michael Paquier
>> >  wrote:
>> >> On Wed, Jan 6, 2016 at 3:04 PM, Michael Paquier
>> >>  wrote:
>> >>> Attached is an updated patch.
>> >>
>> >> Forgot to update rules.out...
>> >
>> > Thanks for the update. Patch looks good to me.
>> > I marked it as ready for committer.
>>
>> Thanks!
>
> Messed around with it, couldn't find any fault, pushed.

Thanks!
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Function and view to retrieve WAL receiver status

[Michael Paquier]

On Wed, Jan 6, 2016 at 3:04 PM, Michael Paquier
 wrote:
> Attached is an updated patch.

Forgot to update rules.out...
-- 
Michael
From 4bc33d1497c302b8669b1f1d9d43f2f806029693 Mon Sep 17 00:00:00 2001
From: Michael Paquier 
Date: Fri, 18 Dec 2015 22:44:21 +0900
Subject: [PATCH] Add system view and function to report WAL receiver activity

---
 doc/src/sgml/monitoring.sgml  |  91 
 src/backend/catalog/system_views.sql  |  16 
 src/backend/replication/walreceiver.c | 154 ++
 src/include/catalog/pg_proc.h |   2 +
 src/include/replication/walreceiver.h |   2 +
 src/test/regress/expected/rules.out   |  13 +++
 6 files changed, 278 insertions(+)

diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index c503636..85459d0 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -301,6 +301,14 @@ postgres   27093  0.0  0.0  30096  2752 ?Ss   11:34   0:00 postgres: ser
  
 
  
+  pg_stat_wal_receiverpg_stat_wal_receiver
+  Only one row, showing statistics about the WAL receiver from
+   that receiver's connected server.
+   See  for details.
+  
+ 
+
+ 
   pg_stat_sslpg_stat_ssl
   One row per connection (regular and replication), showing information about
SSL used on this connection.
@@ -833,6 +841,89 @@ postgres   27093  0.0  0.0  30096  2752 ?Ss   11:34   0:00 postgres: ser
listed; no information is available about downstream standby servers.
   
 
+  
+   pg_stat_wal_receiver View
+   
+
+
+  Column
+  Type
+  Description
+ 
+
+
+   
+
+ pid
+ integer
+ Process ID of the WAL receiver process
+
+
+ status
+ text
+ Activity status of the WAL receiver process
+
+
+ receive_start_lsn
+ pg_lsn
+ First transaction log position used when WAL receiver is
+  started
+
+
+ receive_start_tli
+ integer
+ First timeline number used when WAL receiver is started
+
+
+ received_lsn
+ pg_lsn
+ Last transaction log position already received and flushed to
+  disk, the initial value of this field being the first log position used
+  when WAL receiver is started
+
+
+ received_tli
+ integer
+ Timeline number of last transaction log position received and
+  flushed to disk, the initial value of this field being the timeline
+  number of the first log position used when WAL receiver is started
+ 
+
+
+ last_msg_send_time
+ timestamp with time zone
+ Send time of last message received from origin WAL sender
+
+
+ last_msg_receipt_time
+ timestamp with time zone
+ Receipt time of last message received from origin WAL sender
+
+
+ latest_end_lsn
+ pg_lsn
+ Last transaction log position reported to origin WAL sender
+
+
+ latest_end_time
+ timestamp with time zone
+ Time of last transaction log position reported to origin WAL sender
+
+
+ slot_name
+ text
+ Replication slot name used by this WAL receiver
+
+   
+   
+  
+
+  
+   The pg_stat_wal_receiver view will contain only
+   one row, showing statistics about the WAL receiver from that receiver's
+   connected server.
+  
+
   
pg_stat_ssl View

diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 2052afd..506a884 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -662,6 +662,22 @@ CREATE VIEW pg_stat_replication AS
 WHERE S.usesysid = U.oid AND
 S.pid = W.pid;
 
+CREATE VIEW pg_stat_wal_receiver AS
+SELECT
+s.pid,
+s.status,
+s.receive_start_lsn,
+s.receive_start_tli,
+s.received_lsn,
+s.received_tli,
+s.last_msg_send_time,
+s.last_msg_receipt_time,
+s.latest_end_lsn,
+s.latest_end_time,
+s.slot_name
+FROM pg_stat_get_wal_receiver() s
+WHERE s.pid IS NOT NULL;
+
 CREATE VIEW pg_stat_ssl AS
 SELECT
 S.pid,
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index 81f1529..7b36e02 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -46,9 +46,12 @@
 #include 
 #include 
 
+#include "access/htup_details.h"
 #include "access/timeline.h"
 #include "access/transam.h"
 #include "access/xlog_internal.h"
+#include "catalog/pg_type.h"
+#include "funcapi.h"
 #include "libpq/pqformat.h"
 #include "libpq/pqsignal.h"
 #include "miscadmin.h"
@@ -57,7 +60,9 @@
 #include "storage/ipc.h"
 #include "storage/pmsignal.h"
 #include "storage/procarray.h"
+#include "utils/builtins.h"
 #include "utils/guc.h"
+#include "utils/pg_lsn.h"
 #include 

Re: [HACKERS] Function and view to retrieve WAL receiver status

[Haribabu Kommi]

On Wed, Jan 6, 2016 at 8:00 PM, Michael Paquier
 wrote:
> On Wed, Jan 6, 2016 at 3:04 PM, Michael Paquier
>  wrote:
>> Attached is an updated patch.
>
> Forgot to update rules.out...

Thanks for the update. Patch looks good to me.
I marked it as ready for committer.

Regards,
Hari Babu
Fujitsu Australia


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Function and view to retrieve WAL receiver status

[Michael Paquier]

On Wed, Jan 6, 2016 at 8:14 AM, Haribabu Kommi  wrote:
> Following are my observations on the latest patch.

Thanks for your review.

> + If no WAL receiver is present on the server queried,
> +   a single tuple filled with NULL values is returned instead.
> +  
>
> The above documentation change is not required if we change the system
> view.

Affirmative.

> +s.received_up_to_lsn,
>
> The column name can be changed as "received_lsn" similar to "received_tli".
> up_to may not be required.
>
> + XLogRecPtr received_up_lsn;
> + TimeLineID received_up_tli;
>
> same as like above comment.

Indeed, let's make the variable names more simple and consistent by
removing this _up_ portion everywhere.

> + /* lock? */
>
> I find out that walrcv data is updated only under mutex. it is better
> to take that mutex to provide a consistent snapshot data to user.

The lock is taken, the comment is just incorrect:
+/* lock? */
+SpinLockAcquire(>mutex);
[...]
+SpinLockRelease(>mutex);

I also found out that the description of those fields was not clear
enough actually: received_tli and received _lsn are related to what
has been received *and* flushed to disk, with an initial value being
their start equivalent. This deserves a clear description with all
those things addressed.

Attached is an updated patch.
-- 
Michael
From d924cb2f4f8594208ea6127b1135a213c48e0b89 Mon Sep 17 00:00:00 2001
From: Michael Paquier 
Date: Fri, 18 Dec 2015 22:44:21 +0900
Subject: [PATCH] Add system view and function to report WAL receiver activity

---
 doc/src/sgml/monitoring.sgml  |  91 
 src/backend/catalog/system_views.sql  |  16 
 src/backend/replication/walreceiver.c | 154 ++
 src/include/catalog/pg_proc.h |   2 +
 src/include/replication/walreceiver.h |   2 +
 src/test/regress/expected/rules.out   |  12 +++
 6 files changed, 277 insertions(+)

diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index c503636..85459d0 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -301,6 +301,14 @@ postgres   27093  0.0  0.0  30096  2752 ?Ss   11:34   0:00 postgres: ser
  
 
  
+  pg_stat_wal_receiverpg_stat_wal_receiver
+  Only one row, showing statistics about the WAL receiver from
+   that receiver's connected server.
+   See  for details.
+  
+ 
+
+ 
   pg_stat_sslpg_stat_ssl
   One row per connection (regular and replication), showing information about
SSL used on this connection.
@@ -833,6 +841,89 @@ postgres   27093  0.0  0.0  30096  2752 ?Ss   11:34   0:00 postgres: ser
listed; no information is available about downstream standby servers.
   
 
+  
+   pg_stat_wal_receiver View
+   
+
+
+  Column
+  Type
+  Description
+ 
+
+
+   
+
+ pid
+ integer
+ Process ID of the WAL receiver process
+
+
+ status
+ text
+ Activity status of the WAL receiver process
+
+
+ receive_start_lsn
+ pg_lsn
+ First transaction log position used when WAL receiver is
+  started
+
+
+ receive_start_tli
+ integer
+ First timeline number used when WAL receiver is started
+
+
+ received_lsn
+ pg_lsn
+ Last transaction log position already received and flushed to
+  disk, the initial value of this field being the first log position used
+  when WAL receiver is started
+
+
+ received_tli
+ integer
+ Timeline number of last transaction log position received and
+  flushed to disk, the initial value of this field being the timeline
+  number of the first log position used when WAL receiver is started
+ 
+
+
+ last_msg_send_time
+ timestamp with time zone
+ Send time of last message received from origin WAL sender
+
+
+ last_msg_receipt_time
+ timestamp with time zone
+ Receipt time of last message received from origin WAL sender
+
+
+ latest_end_lsn
+ pg_lsn
+ Last transaction log position reported to origin WAL sender
+
+
+ latest_end_time
+ timestamp with time zone
+ Time of last transaction log position reported to origin WAL sender
+
+
+ slot_name
+ text
+ Replication slot name used by this WAL receiver
+
+   
+   
+  
+
+  
+   The pg_stat_wal_receiver view will contain only
+   one row, showing statistics about the WAL receiver from that receiver's
+   connected server.
+  
+
   
pg_stat_ssl View

diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 2052afd..506a884 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -662,6 +662,22 @@ CREATE VIEW pg_stat_replication AS
 WHERE S.usesysid = U.oid AND
 S.pid = W.pid;
 
+CREATE VIEW 

Re: [HACKERS] Function and view to retrieve WAL receiver status

[Haribabu Kommi]

On Sat, Dec 19, 2015 at 12:54 AM, Michael Paquier
 wrote:
> On Fri, Dec 18, 2015 at 8:39 AM, Robert Haas  wrote:
>> On Mon, Dec 14, 2015 at 7:23 PM, Michael Paquier
>>  wrote:
>>> On Tue, Dec 15, 2015 at 5:27 AM, Gurjeet Singh wrote:
 The function, maybe. But emitting an all-nulls row from a view seems
 counter-intuitive, at least when looking at it in context of relational
 database.
>>>
>>> OK, noted. Any other opinions?
>>
>> I wouldn't bother with the view.  If we're going to do it, I'd say
>> just provide the function and let people SELECT * from it if they want
>> to.
>
> OK, I took some time to write a patch for that as attached, added in
> the next CF here:
> https://commitfest.postgresql.org/8/447/
> I am fine switching to an SRF depending on other opinions of people
> here, it just seems like an overkill knowing the uniqueness of the WAL
> sender in a server.
>
> I have finished with a function and a system view, this came up more
> in line with the existing things like pg_stat_archiver, and this makes
> as well the documentation clearer, at least that was my feeling when
> hacking that.

I also feel showing NULL values may not be good, when there is
no walreceiver. Instead of SRF function to avoid showing NULL vallues
how about adding "WHERE s.pid IS NOT NULL" to the system view.
pid value cannot be NULL, until unless there is no walreceiver.


Regards,
Hari Babu
Fujitsu Australia


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Function and view to retrieve WAL receiver status

[Michael Paquier]

On Tue, Jan 5, 2016 at 7:49 PM, Haribabu Kommi  wrote:
> On Sat, Dec 19, 2015 at 12:54 AM, Michael Paquier
>  wrote:
>> On Fri, Dec 18, 2015 at 8:39 AM, Robert Haas  wrote:
>>> On Mon, Dec 14, 2015 at 7:23 PM, Michael Paquier
>>>  wrote:
 On Tue, Dec 15, 2015 at 5:27 AM, Gurjeet Singh wrote:
> The function, maybe. But emitting an all-nulls row from a view seems
> counter-intuitive, at least when looking at it in context of relational
> database.

 OK, noted. Any other opinions?
>>>
>>> I wouldn't bother with the view.  If we're going to do it, I'd say
>>> just provide the function and let people SELECT * from it if they want
>>> to.
>>
>> OK, I took some time to write a patch for that as attached, added in
>> the next CF here:
>> https://commitfest.postgresql.org/8/447/
>> I am fine switching to an SRF depending on other opinions of people
>> here, it just seems like an overkill knowing the uniqueness of the WAL
>> sender in a server.
>>
>> I have finished with a function and a system view, this came up more
>> in line with the existing things like pg_stat_archiver, and this makes
>> as well the documentation clearer, at least that was my feeling when
>> hacking that.
>
> I also feel showing NULL values may not be good, when there is
> no walreceiver. Instead of SRF function to avoid showing NULL vallues
> how about adding "WHERE s.pid IS NOT NULL" to the system view.
> pid value cannot be NULL, until unless there is no walreceiver.

Yeah, I would not mind switching it to that. A couple of other stat
catalog views do it as well.
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Function and view to retrieve WAL receiver status

[Haribabu Kommi]

On Tue, Jan 5, 2016 at 10:24 PM, Michael Paquier
 wrote:
> On Tue, Jan 5, 2016 at 7:49 PM, Haribabu Kommi  
> wrote:
>> On Sat, Dec 19, 2015 at 12:54 AM, Michael Paquier
>>  wrote:
>>> On Fri, Dec 18, 2015 at 8:39 AM, Robert Haas  wrote:
 On Mon, Dec 14, 2015 at 7:23 PM, Michael Paquier
  wrote:
> On Tue, Dec 15, 2015 at 5:27 AM, Gurjeet Singh wrote:
>> The function, maybe. But emitting an all-nulls row from a view seems
>> counter-intuitive, at least when looking at it in context of relational
>> database.
>
> OK, noted. Any other opinions?

 I wouldn't bother with the view.  If we're going to do it, I'd say
 just provide the function and let people SELECT * from it if they want
 to.
>>>
>>> OK, I took some time to write a patch for that as attached, added in
>>> the next CF here:
>>> https://commitfest.postgresql.org/8/447/
>>> I am fine switching to an SRF depending on other opinions of people
>>> here, it just seems like an overkill knowing the uniqueness of the WAL
>>> sender in a server.
>>>
>>> I have finished with a function and a system view, this came up more
>>> in line with the existing things like pg_stat_archiver, and this makes
>>> as well the documentation clearer, at least that was my feeling when
>>> hacking that.
>>
>> I also feel showing NULL values may not be good, when there is
>> no walreceiver. Instead of SRF function to avoid showing NULL vallues
>> how about adding "WHERE s.pid IS NOT NULL" to the system view.
>> pid value cannot be NULL, until unless there is no walreceiver.
>
> Yeah, I would not mind switching it to that. A couple of other stat
> catalog views do it as well.

Following are my observations on the latest patch.

+ If no WAL receiver is present on the server queried,
+   a single tuple filled with NULL values is returned instead.
+  

The above documentation change is not required if we change the system
view.

+s.received_up_to_lsn,

The column name can be changed as "received_lsn" similar to "received_tli".
up_to may not be required.

+ XLogRecPtr received_up_lsn;
+ TimeLineID received_up_tli;

same as like above comment.

+ /* lock? */

I find out that walrcv data is updated only under mutex. it is better
to take that
mutex to provide a consistent snapshot data to user.


Regards,
Hari Babu
Fujitsu Australia


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Function and view to retrieve WAL receiver status

[Michael Paquier]

On Fri, Dec 18, 2015 at 8:39 AM, Robert Haas  wrote:
> On Mon, Dec 14, 2015 at 7:23 PM, Michael Paquier
>  wrote:
>> On Tue, Dec 15, 2015 at 5:27 AM, Gurjeet Singh wrote:
>>> The function, maybe. But emitting an all-nulls row from a view seems
>>> counter-intuitive, at least when looking at it in context of relational
>>> database.
>>
>> OK, noted. Any other opinions?
>
> I wouldn't bother with the view.  If we're going to do it, I'd say
> just provide the function and let people SELECT * from it if they want
> to.

OK, I took some time to write a patch for that as attached, added in
the next CF here:
https://commitfest.postgresql.org/8/447/
I am fine switching to an SRF depending on other opinions of people
here, it just seems like an overkill knowing the uniqueness of the WAL
sender in a server.

I have finished with a function and a system view, this came up more
in line with the existing things like pg_stat_archiver, and this makes
as well the documentation clearer, at least that was my feeling when
hacking that.
Regards,
-- 
Michael


0001-Add-system-view-and-function-to-report-WAL-receiver-.patch
Description: binary/octet-stream

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Function and view to retrieve WAL receiver status

[Robert Haas]

On Mon, Dec 14, 2015 at 7:23 PM, Michael Paquier
 wrote:
> On Tue, Dec 15, 2015 at 5:27 AM, Gurjeet Singh wrote:
>> The function, maybe. But emitting an all-nulls row from a view seems
>> counter-intuitive, at least when looking at it in context of relational
>> database.
>
> OK, noted. Any other opinions?

I wouldn't bother with the view.  If we're going to do it, I'd say
just provide the function and let people SELECT * from it if they want
to.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Function and view to retrieve WAL receiver status

[Michael Paquier]

On Tue, Dec 15, 2015 at 5:27 AM, Gurjeet Singh wrote:
> The function, maybe. But emitting an all-nulls row from a view seems
> counter-intuitive, at least when looking at it in context of relational
> database.

OK, noted. Any other opinions?
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Function and view to retrieve WAL receiver status

[Gurjeet Singh]

On Sun, Dec 13, 2015 at 10:15 PM, Michael Paquier  wrote:

> On Mon, Dec 14, 2015 at 3:09 PM, Gurjeet Singh 
> wrote:
> > On Dec 13, 2015 9:56 PM, "Michael Paquier" 
> > wrote:
> >> If the node has no WAL receiver active, a tuple with NULL values is
> >> returned instead.
> >
> > IMO, in the absence of a WAL receiver the SRF (and the view) should not
> > return any rows.
>
> The whole point is to not use a SRF in this case: there is always at
> most one WAL receiver.
>

The function, maybe. But emitting an all-nulls row from a view seems
counter-intuitive, at least when looking at it in context of relational
database.

-- 
Gurjeet Singh http://gurjeet.singh.im/

[HACKERS] Function and view to retrieve WAL receiver status

[Michael Paquier]

Hi all,

Currently there is no equivalent of pg_stat_get_wal_senders for the
WAL receiver on a node, and it seems that it would be useful to have
an SQL representation of what is in shared memory should a WAL
receiver be active without going through the ps display for example.
So, any opinion about having in core a function called
pg_stat_get_wal_receiver that returns a single tuple that translates
the data WalRcvData?
We could bundle on top of it a system view, say called
pg_stat_wal_receiver, with this layer:
View "public.pg_stat_wal_receiver"
Column|Type|Modifiers
pid|integer|
status|text|
receive_start_lsn|pg_lsn|
receive_start_tli|integer|
received_up_to_lsn|pg_lsn|
received_tli|integer|
latest_chunk_start_lsn|pg_lsn|
last_msg_send_time|timestamp with time zone|
last_msg_receipt_time|timestamp with time zone|
latest_end_lsn|pg_lsn|
latest_end_time|timestamp with time zone|
slot_name|text|

If the node has no WAL receiver active, a tuple with NULL values is
returned instead.
Thoughts?
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Function and view to retrieve WAL receiver status

[Gurjeet Singh]

On Dec 13, 2015 9:56 PM, "Michael Paquier" 
wrote:

>
> If the node has no WAL receiver active, a tuple with NULL values is
> returned instead.

IMO, in the absence of a WAL receiver the SRF (and the view) should not
return any rows.

Re: [HACKERS] Function and view to retrieve WAL receiver status

[Michael Paquier]

On Mon, Dec 14, 2015 at 3:09 PM, Gurjeet Singh  wrote:
> On Dec 13, 2015 9:56 PM, "Michael Paquier" 
> wrote:
>> If the node has no WAL receiver active, a tuple with NULL values is
>> returned instead.
>
> IMO, in the absence of a WAL receiver the SRF (and the view) should not
> return any rows.

The whole point is to not use a SRF in this case: there is always at
most one WAL receiver.
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers