Re: [HACKERS] Support for pg_receivexlog --post-segment command

[David Steele]

On 1/6/17 10:15 AM, Feike Steenbergen wrote:


On 6 January 2017 at 15:42, Magnus Hagander > wrote:


Is there actual value in providing both %p and %f? It's not like it's

really hard to do, but since the path will be specified on the same
commandline, you could just put it in the command?

As %f can be determined from %p I don't mind that much. However, having
a single static --whatever command may be very useful for configuration
management or backup tools that want to use static commands.

The other reason why I'd offer both is to have some uniformity with
archive_command, possibly allowing some reuse of code.


Agreed.  Currently pgBackRest only accepts %p for archive_command 
because it already knows where $PGDATA is.  Of course I can change that, 
but I think it makes sense to keep command options uniform anyway.


--
-David
da...@pgmasters.net


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Support for pg_receivexlog --post-segment command

[Feike Steenbergen]

On 6 January 2017 at 15:42, Magnus Hagander  wrote:

> Is there actual value in providing both %p and %f? It's not like it's
really hard to do, but since the path will be specified on the same
commandline, you could just put it in the command?

As %f can be determined from %p I don't mind that much. However, having a
single static --whatever command may be very useful for configuration
management or backup tools that want to use static commands.

The other reason why I'd offer both is to have some uniformity with
archive_command, possibly allowing some reuse of code.

Re: [HACKERS] Support for pg_receivexlog --post-segment command

[Magnus Hagander]

On Fri, Jan 6, 2017 at 2:55 PM, David Steele  wrote:

> On 1/6/17 8:49 AM, Feike Steenbergen wrote:
>
>>
>> On Fri, Jan 6, 2017 at 2:30 PM, David Steele > > wrote:
>>
>>> For my part I still prefer an actual command to be executed so it will
>>>
>> start/restart the archiver if it is not already running or died.  This
>> reduces the number of processes that I need to ensure are running.
>>
>>>
>>> If the consensus is that a signal is better then I'll make that work.
>>>
>> I will say this raises the bar on what is required to write a good
>> archive command and we already know it is quite a difficult task.
>>
>> On 6 January 2017 at 14:37, Magnus Hagander > > wrote:
>>
>>> I like the idea of a command as well, for flexibility. If you want a
>>>
>> signal, you can write a trivial command that sends the signal... Maximum
>> flexibility, as long as we don't create a lot of caveats for users.
>>
>> Agreed, I think it is also easier to understand the mechanism (instead
>> of a signal), and would allow for some reuse of already existing scripts.
>>
>> If we do use a full command (vs a signal), I propose we do also offer
>> the %p and %f placeholders for the command.
>>
>
> Agreed.  It shouldn't be that hard and could be very useful.  If nothing
> else it will eliminate the need to configure path to the pg_receivexlog
> queue in the archiver.


Is there actual value in providing both %p and %f? It's not like it's
really hard to do, but since the path will be specified on the same
commandline, you could just put it in the command?

E.g.
pg_receivexlog -D /blah/foo --whatever /some/where/myscript %p
if you want the path in it could just be written
pg_receivexlog -D /blah/foo --whatever /some/where/myscript /blah/foo/%f

And you have that path in the *same place* already? (unless you want to
teach the script about it, in which case you want just %f anyway)

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Re: [HACKERS] Support for pg_receivexlog --post-segment command

[David Steele]

On 1/6/17 8:49 AM, Feike Steenbergen wrote:


On Fri, Jan 6, 2017 at 2:30 PM, David Steele > wrote:

For my part I still prefer an actual command to be executed so it will

start/restart the archiver if it is not already running or died.  This
reduces the number of processes that I need to ensure are running.


If the consensus is that a signal is better then I'll make that work.

I will say this raises the bar on what is required to write a good
archive command and we already know it is quite a difficult task.

On 6 January 2017 at 14:37, Magnus Hagander > wrote:

I like the idea of a command as well, for flexibility. If you want a

signal, you can write a trivial command that sends the signal... Maximum
flexibility, as long as we don't create a lot of caveats for users.

Agreed, I think it is also easier to understand the mechanism (instead
of a signal), and would allow for some reuse of already existing scripts.

If we do use a full command (vs a signal), I propose we do also offer
the %p and %f placeholders for the command.


Agreed.  It shouldn't be that hard and could be very useful.  If nothing 
else it will eliminate the need to configure path to the pg_receivexlog 
queue in the archiver.


--
-David
da...@pgmasters.net


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Support for pg_receivexlog --post-segment command

[Feike Steenbergen]

On Fri, Jan 6, 2017 at 2:30 PM, David Steele  wrote:
> For my part I still prefer an actual command to be executed so it will
start/restart the archiver if it is not already running or died.  This
reduces the number of processes that I need to ensure are running.
>
> If the consensus is that a signal is better then I'll make that work.  I
will say this raises the bar on what is required to write a good archive
command and we already know it is quite a difficult task.

On 6 January 2017 at 14:37, Magnus Hagander  wrote:
> I like the idea of a command as well, for flexibility. If you want a
signal, you can write a trivial command that sends the signal... Maximum
flexibility, as long as we don't create a lot of caveats for users.

Agreed, I think it is also easier to understand the mechanism (instead of a
signal), and would allow for some reuse of already existing scripts.

If we do use a full command (vs a signal), I propose we do also offer the
%p and %f placeholders for the command.

Re: [HACKERS] Support for pg_receivexlog --post-segment command

[Magnus Hagander]

On Fri, Jan 6, 2017 at 2:30 PM, David Steele  wrote:

> On 1/6/17 8:09 AM, Feike Steenbergen wrote:
>
>> On 6 January 2017 at 13:50, Magnus Hagander > > wrote:
>>
>>> I think we're better off clearly documenting that we don't care about
>>>
>> it. And basically let the external command be responsible for that part.
>>
>> So for example, your typical backup manager would listen to this
>>>
>> signal or whatever to react quickly. But it would *also* have some sort
>> of fallback. For example, whenever it's triggered it also checks if
>> there are any previous segments it missed (this would also cover the
>> startup sequence).
>>
>
> I'm fine with the backup manager doing all the work of keeping track of
> what has been compressed, moved to archive, etc.  No need to reinvent the
> wheel here.
>
> For my part I still prefer an actual command to be executed so it will
> start/restart the archiver if it is not already running or died.  This
> reduces the number of processes that I need to ensure are running.
>
> If the consensus is that a signal is better then I'll make that work.  I
> will say this raises the bar on what is required to write a good archive
> command and we already know it is quite a difficult task.


I like the idea of a command as well, for flexibility. If you want a
signal, you can write a trivial command that sends the signal... Maximum
flexibility, as long as we don't create a lot of caveats for users.

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Re: [HACKERS] Support for pg_receivexlog --post-segment command

[David Steele]

On 1/6/17 8:09 AM, Feike Steenbergen wrote:

On 6 January 2017 at 13:50, Magnus Hagander > wrote:

I think we're better off clearly documenting that we don't care about

it. And basically let the external command be responsible for that part.


So for example, your typical backup manager would listen to this

signal or whatever to react quickly. But it would *also* have some sort
of fallback. For example, whenever it's triggered it also checks if
there are any previous segments it missed (this would also cover the
startup sequence).


I'm fine with the backup manager doing all the work of keeping track of 
what has been compressed, moved to archive, etc.  No need to reinvent 
the wheel here.


For my part I still prefer an actual command to be executed so it will 
start/restart the archiver if it is not already running or died.  This 
reduces the number of processes that I need to ensure are running.


If the consensus is that a signal is better then I'll make that work.  I 
will say this raises the bar on what is required to write a good archive 
command and we already know it is quite a difficult task.



For me this works fine. I just want to ensure that if there is any work
to be done, my backup manager will do the work quickly. My
implementation might be very simply a process that checks every n
seconds or when signalled.


Since we never actually remove anything (unlike archive_command which

has the integration with wal segment rotation), I think this can be done
perfectly safe.


Looking at the usecases where you have been doing it, are there any

where this would not work?

This would work for all usecases I've come across.


Agreed.

--
-David
da...@pgmasters.net


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Support for pg_receivexlog --post-segment command

[Feike Steenbergen]

On 6 January 2017 at 13:50, Magnus Hagander  wrote:
> I think we're better off clearly documenting that we don't care about it.
And basically let the external command be responsible for that part.

> So for example, your typical backup manager would listen to this signal
or whatever to react quickly. But it would *also* have some sort of
fallback. For example, whenever it's triggered it also checks if there are
any previous segments it missed (this would also cover the startup
sequence).

For me this works fine. I just want to ensure that if there is any work to
be done, my backup manager will do the work quickly. My implementation
might be very simply a process that checks every n seconds or when
signalled.

> Since we never actually remove anything (unlike archive_command which has
the integration with wal segment rotation), I think this can be done
perfectly safe.
>
> Looking at the usecases where you have been doing it, are there any where
this would not work?

This would work for all usecases I've come across.

Re: [HACKERS] Support for pg_receivexlog --post-segment command

[Magnus Hagander]

On Fri, Jan 6, 2017 at 1:45 PM, Feike Steenbergen <
feikesteenber...@gmail.com> wrote:

> Hi all,
>
> When reading through "Support for pg_receivexlog --format=plain|tar"[1], I
> came across a notion from Magnus Hagander that has crossed my mind a few
> times as well in the past years. As the feature proposed here is not
> directly related to that thread, I thought it best to start a new thread to
> discuss.
>
> > I have been talking to David about it a couple of times, and he agreed
> that it'd be useful to have a post-segment command. We haven't discussed it
> in much detail though. I'll add him to direct-cc here to see if he has any
> further input :)
>
> I'm coming across a few usecases where this would seem very useful. I'm
> looking to push finished segments to some api as soon as possible. Having
> the post-segment command would allow me to get there. I've tried the
> following approaches, none of them are very satisfying however:
>
> - periodic checking of new files (cron)
> - using inotify
> - tailing verbose pg_receivexlog output to see when a segment was switched
>

Yeah, these are all kind of ugly. Or platform specific. Or both.

You can make them work, but they feel hackish.


> > It could be that the best idea is to just notify some other process of
> what's happening. But making it an external command would give that a lot
> of flexibility. Of course, we need to be careful not to put ourselves back
> in the position we are in with archive_command, in that it's very difficult
> to write a good one.
>
> A signal for  would be good enough for my use case, I don't necessarily
> need all the bookkeeping to ensure the post-segment command was finished
> successfully. However I can see people expecting similar behaviour for the
> post-segment command as for the archive_command. If we would use an
> external command, does this also imply that we need some bookkeeping around
> which segments are ready and done, similar to what is done on the server in
> the archive_status directory?
>

I'd really like to avoid having to re-implement that functionality in
pg_receivexlog. Given the experiences we have with how difficult it is to
get right in archive_command...

I think we're better off clearly documenting that we don't care about it.
And basically let the external command be responsible for that part.

So for example, your typical backup manager would listen to this signal or
whatever to react quickly. But it would *also* have some sort of fallback.
For example, whenever it's triggered it also checks if there are any
previous segments it missed (this would also cover the startup sequence).

Since we never actually remove anything (unlike archive_command which has
the integration with wal segment rotation), I think this can be done
perfectly safe.

Looking at the usecases where you have been doing it, are there any where
this would not work?

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

[HACKERS] Support for pg_receivexlog --post-segment command

[Feike Steenbergen]

Hi all,

When reading through "Support for pg_receivexlog --format=plain|tar"[1], I
came across a notion from Magnus Hagander that has crossed my mind a few
times as well in the past years. As the feature proposed here is not
directly related to that thread, I thought it best to start a new thread to
discuss.

> I have been talking to David about it a couple of times, and he agreed
that it'd be useful to have a post-segment command. We haven't discussed it
in much detail though. I'll add him to direct-cc here to see if he has any
further input :)

I'm coming across a few usecases where this would seem very useful. I'm
looking to push finished segments to some api as soon as possible. Having
the post-segment command would allow me to get there. I've tried the
following approaches, none of them are very satisfying however:

- periodic checking of new files (cron)
- using inotify
- tailing verbose pg_receivexlog output to see when a segment was switched

> It could be that the best idea is to just notify some other process of
what's happening. But making it an external command would give that a lot
of flexibility. Of course, we need to be careful not to put ourselves back
in the position we are in with archive_command, in that it's very difficult
to write a good one.

A signal for  would be good enough for my use case, I don't necessarily
need all the bookkeeping to ensure the post-segment command was finished
successfully. However I can see people expecting similar behaviour for the
post-segment command as for the archive_command. If we would use an
external command, does this also imply that we need some bookkeeping around
which segments are ready and done, similar to what is done on the server in
the archive_status directory?

Thanks,

Feike

[1]
https://www.postgresql.org/message-id/CAB7nPqTEVXjtH+fehcCbP791H71cfLN_p9rrd-h=ymjfshz...@mail.gmail.com