Re: [HACKERS] Streaming replication and WAL archive interactions

2015-05-13 Thread Heikki Linnakangas
On 05/13/2015 04:29 PM, Robert Haas wrote: On Wed, May 13, 2015 at 8:53 AM, Heikki Linnakangas wrote: Our manual says that archive_command should refuse to overwrite an existing file. But to work-around the double-archival problem, where the same file is archived twice, it would be even better

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-05-13 Thread Robert Haas
On Wed, May 13, 2015 at 8:53 AM, Heikki Linnakangas wrote: > Our manual says that archive_command should refuse to overwrite an existing > file. But to work-around the double-archival problem, where the same file is > archived twice, it would be even better if it would simply return success if > t

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-05-13 Thread Heikki Linnakangas
On 05/13/2015 03:36 PM, Robert Haas wrote: On Mon, May 11, 2015 at 12:00 PM, Heikki Linnakangas wrote: And here is a new version of the patch. I kept the approach of using pgstat, but it now only polls pgstat every 10 seconds, and doesn't block to wait for updated stats. It's not entirely a n

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-05-13 Thread Robert Haas
On Mon, May 11, 2015 at 12:00 PM, Heikki Linnakangas wrote: > And here is a new version of the patch. I kept the approach of using pgstat, > but it now only polls pgstat every 10 seconds, and doesn't block to wait for > updated stats. It's not entirely a new problem, but this error message has go

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-05-11 Thread Heikki Linnakangas
On 05/08/2015 04:21 PM, Heikki Linnakangas wrote: On 04/22/2015 10:07 AM, Michael Paquier wrote: On Wed, Apr 22, 2015 at 3:38 PM, Heikki Linnakangas wrote: I feel that the best approach is to archive the last, partial segment, but with the .partial suffix. I don't see any plausible real-wold s

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-05-08 Thread Heikki Linnakangas
On 04/22/2015 10:07 AM, Michael Paquier wrote: On Wed, Apr 22, 2015 at 3:38 PM, Heikki Linnakangas wrote: I feel that the best approach is to archive the last, partial segment, but with the .partial suffix. I don't see any plausible real-wold setup where the current behavior would be better. I

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-23 Thread Heikki Linnakangas
On 04/22/2015 11:58 PM, Robert Haas wrote: On Wed, Apr 22, 2015 at 3:34 PM, Heikki Linnakangas wrote: On 04/22/2015 10:21 PM, Robert Haas wrote: On Wed, Apr 22, 2015 at 3:01 PM, Heikki Linnakangas wrote: For example, imagine that perform point-in-time recovery to WAL position 0/1237E568, on

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-22 Thread Robert Haas
On Wed, Apr 22, 2015 at 3:34 PM, Heikki Linnakangas wrote: > On 04/22/2015 10:21 PM, Robert Haas wrote: >> On Wed, Apr 22, 2015 at 3:01 PM, Heikki Linnakangas >> wrote: >>> For example, imagine that perform point-in-time recovery to WAL position >>> 0/1237E568, on timeline 1. That falls within se

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-22 Thread Heikki Linnakangas
On 04/22/2015 10:21 PM, Robert Haas wrote: On Wed, Apr 22, 2015 at 3:01 PM, Heikki Linnakangas wrote: For example, imagine that perform point-in-time recovery to WAL position 0/1237E568, on timeline 1. That falls within segment 00010012. Then we end recovery, and switch to timel

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-22 Thread Robert Haas
On Wed, Apr 22, 2015 at 3:01 PM, Heikki Linnakangas wrote: > On 04/22/2015 09:30 PM, Robert Haas wrote: >> On Wed, Apr 22, 2015 at 2:17 AM, Heikki Linnakangas >> wrote: >>> >>> Note that it's a bit complicated to set up that scenario today. Archiving >>> is >>> never enabled in recovery mode, so

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-22 Thread Heikki Linnakangas
On 04/22/2015 09:30 PM, Robert Haas wrote: On Wed, Apr 22, 2015 at 2:17 AM, Heikki Linnakangas wrote: Note that it's a bit complicated to set up that scenario today. Archiving is never enabled in recovery mode, so you'll need to use a custom cron job or something to maintain the archive that C

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-22 Thread Robert Haas
On Tue, Apr 21, 2015 at 8:30 PM, Michael Paquier wrote: > This .partial segment renaming is something that we > should let the archive_command manage with its internal logic. This strikes me as equivalent to saying "we don't know how to make this work right, but maybe our users will know". That

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-22 Thread Robert Haas
On Wed, Apr 22, 2015 at 2:17 AM, Heikki Linnakangas wrote: > Note that it's a bit complicated to set up that scenario today. Archiving is > never enabled in recovery mode, so you'll need to use a custom cron job or > something to maintain the archive that C uses. The files will not > automatically

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-22 Thread Michael Paquier
On Wed, Apr 22, 2015 at 3:38 PM, Heikki Linnakangas wrote: > On 04/22/2015 03:30 AM, Michael Paquier wrote: >> >> This is going to change a behavior that people are used to for a >> couple of releases. I would not mind having this patch do >> "archive_mode = on during recovery" => archive only seg

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-21 Thread Heikki Linnakangas
On 04/22/2015 03:30 AM, Michael Paquier wrote: This is going to change a behavior that people are used to for a couple of releases. I would not mind having this patch do "archive_mode = on during recovery" => archive only segments generated by this node + the last partial segment on the old timel

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-21 Thread Heikki Linnakangas
On 04/22/2015 12:42 AM, Robert Haas wrote: On Tue, Apr 21, 2015 at 6:55 AM, Heikki Linnakangas wrote: On 04/21/2015 12:04 PM, Michael Paquier wrote: On Tue, Apr 21, 2015 at 4:38 PM, Heikki Linnakangas wrote: Note that even though we don't archive the partial last segment on the previous time

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-21 Thread Michael Paquier
On Wed, Apr 22, 2015 at 6:42 AM, Robert Haas wrote: > > On Tue, Apr 21, 2015 at 6:55 AM, Heikki Linnakangas wrote: > > On 04/21/2015 12:04 PM, Michael Paquier wrote: > >> On Tue, Apr 21, 2015 at 4:38 PM, Heikki Linnakangas > >> wrote: > >>> Note that even though we don't archive the partial last

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-21 Thread Robert Haas
On Tue, Apr 21, 2015 at 6:55 AM, Heikki Linnakangas wrote: > On 04/21/2015 12:04 PM, Michael Paquier wrote: >> On Tue, Apr 21, 2015 at 4:38 PM, Heikki Linnakangas >> wrote: >>> Note that even though we don't archive the partial last segment on the >>> previous timeline, the same WAL is copied to

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-21 Thread Heikki Linnakangas
On 04/21/2015 12:04 PM, Michael Paquier wrote: On Tue, Apr 21, 2015 at 4:38 PM, Heikki Linnakangas wrote: Note that even though we don't archive the partial last segment on the previous timeline, the same WAL is copied to the first segment on the new timeline. So the WAL isn't lost. But if t

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-21 Thread Michael Paquier
On Tue, Apr 21, 2015 at 4:38 PM, Heikki Linnakangas wrote: > On 04/21/2015 09:53 AM, Michael Paquier wrote: > >> On Thu, Apr 16, 2015 at 8:57 PM, Heikki Linnakangas wrote: >> >>> Oh, hang on, that's not necessarily true. On promotion, the standby >>> >> archives >> >>> the last, partial WAL segme

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-21 Thread Heikki Linnakangas
On 04/21/2015 09:53 AM, Michael Paquier wrote: On Thu, Apr 16, 2015 at 8:57 PM, Heikki Linnakangas wrote: Oh, hang on, that's not necessarily true. On promotion, the standby archives the last, partial WAL segment from the old timeline. That's just wrong (http://www.postgresql.org/message-id/52

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-20 Thread Michael Paquier
On Thu, Apr 16, 2015 at 8:57 PM, Heikki Linnakangas wrote: > Oh, hang on, that's not necessarily true. On promotion, the standby archives > the last, partial WAL segment from the old timeline. That's just wrong > (http://www.postgresql.org/message-id/52fcd37c.3070...@vmware.com), and in > fact I so

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-16 Thread Heikki Linnakangas
On 03/01/2015 12:36 AM, Venkata Balaji N wrote: Patch did get applied successfully to the latest master. Can you please rebase. Here you go. On 01/31/2015 03:07 PM, Andres Freund wrote: On 2014-12-19 22:56:40 +0200, Heikki Linnakangas wrote: This add two new archive_modes, 'shared' and 'alwa

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-02-28 Thread Venkata Balaji N
> > > Here's a first cut at this. It includes the changes from your > standby_wal_archiving_v1.patch, so you get that behaviour if you set > archive_mode='always', and the new behaviour I wanted with > archive_mode='shared'. I wrote it on top of the other patch I posted > recently to not archive bo

[HACKERS] Re: [HACKERS] Streaming replication and WAL archive interactions

2015-02-11 Thread Миша Тюрин
>  This should be a very common setup in the field, so how are  people doing it >in practice? One of possible workaround with archive and streaming was to use pg_receivexlog from standby to copy/save WALs to archive. but with pg_receivexlog was also issue with fsync. [ master ] -- streaming 

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-01-31 Thread Andres Freund
Hi, On 2014-12-19 22:56:40 +0200, Heikki Linnakangas wrote: > This add two new archive_modes, 'shared' and 'always', to indicate whether > the WAL archive is shared between the primary and standby, or not. In > shared mode, the standby tracks which files have been archived by the > primary. The st

Re: [HACKERS] Streaming replication and WAL archive interactions

2014-12-19 Thread Heikki Linnakangas
On 12/18/2014 12:32 PM, Fujii Masao wrote: On Wed, Dec 17, 2014 at 4:11 AM, Heikki Linnakangas wrote: On 12/16/2014 10:24 AM, Borodin Vladimir wrote: 12 дек. 2014 г., в 16:46, Heikki Linnakangas написал(а): There have been a few threads on the behavior of WAL archiving, after a standby ser

Re: [HACKERS] Streaming replication and WAL archive interactions

2014-12-18 Thread Fujii Masao
On Wed, Dec 17, 2014 at 4:11 AM, Heikki Linnakangas wrote: > On 12/16/2014 10:24 AM, Borodin Vladimir wrote: >> >> 12 дек. 2014 г., в 16:46, Heikki Linnakangas >> написал(а): >> >>> There have been a few threads on the behavior of WAL archiving, >>> after a standby server is promoted [1] [2]. In

Re: [HACKERS] Streaming replication and WAL archive interactions

2014-12-16 Thread Heikki Linnakangas
On 12/16/2014 10:24 AM, Borodin Vladimir wrote: 12 дек. 2014 г., в 16:46, Heikki Linnakangas написал(а): There have been a few threads on the behavior of WAL archiving, after a standby server is promoted [1] [2]. In short, it doesn't work as you might expect. The standby will start archiving a

Re: [HACKERS] Streaming replication and WAL archive interactions

2014-12-16 Thread Borodin Vladimir
12 дек. 2014 г., в 16:46, Heikki Linnakangas написал(а): > There have been a few threads on the behavior of WAL archiving, after a > standby server is promoted [1] [2]. In short, it doesn't work as you might > expect. The standby will start archiving after it's promoted, but it will not > ar

[HACKERS] Streaming replication and WAL archive interactions

2014-12-12 Thread Heikki Linnakangas
There have been a few threads on the behavior of WAL archiving, after a standby server is promoted [1] [2]. In short, it doesn't work as you might expect. The standby will start archiving after it's promoted, but it will not archive files that were replicated from the old master via streaming r