Re: [HACKERS] Streaming replication and WAL archive interactions

2015-05-13 Thread Robert Haas
On Mon, May 11, 2015 at 12:00 PM, Heikki Linnakangas hlinn...@iki.fi wrote: And here is a new version of the patch. I kept the approach of using pgstat, but it now only polls pgstat every 10 seconds, and doesn't block to wait for updated stats. It's not entirely a new problem, but this error

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-05-13 Thread Robert Haas
On Wed, May 13, 2015 at 8:53 AM, Heikki Linnakangas hlinn...@iki.fi wrote: Our manual says that archive_command should refuse to overwrite an existing file. But to work-around the double-archival problem, where the same file is archived twice, it would be even better if it would simply return

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-05-13 Thread Heikki Linnakangas
On 05/13/2015 03:36 PM, Robert Haas wrote: On Mon, May 11, 2015 at 12:00 PM, Heikki Linnakangas hlinn...@iki.fi wrote: And here is a new version of the patch. I kept the approach of using pgstat, but it now only polls pgstat every 10 seconds, and doesn't block to wait for updated stats. It's

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-05-13 Thread Heikki Linnakangas
On 05/13/2015 04:29 PM, Robert Haas wrote: On Wed, May 13, 2015 at 8:53 AM, Heikki Linnakangas hlinn...@iki.fi wrote: Our manual says that archive_command should refuse to overwrite an existing file. But to work-around the double-archival problem, where the same file is archived twice, it would

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-05-11 Thread Heikki Linnakangas
On 05/08/2015 04:21 PM, Heikki Linnakangas wrote: On 04/22/2015 10:07 AM, Michael Paquier wrote: On Wed, Apr 22, 2015 at 3:38 PM, Heikki Linnakangas hlinn...@iki.fi wrote: I feel that the best approach is to archive the last, partial segment, but with the .partial suffix. I don't see any

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-05-08 Thread Heikki Linnakangas
On 04/22/2015 10:07 AM, Michael Paquier wrote: On Wed, Apr 22, 2015 at 3:38 PM, Heikki Linnakangas hlinn...@iki.fi wrote: I feel that the best approach is to archive the last, partial segment, but with the .partial suffix. I don't see any plausible real-wold setup where the current behavior

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-23 Thread Heikki Linnakangas
On 04/22/2015 11:58 PM, Robert Haas wrote: On Wed, Apr 22, 2015 at 3:34 PM, Heikki Linnakangas hlinn...@iki.fi wrote: On 04/22/2015 10:21 PM, Robert Haas wrote: On Wed, Apr 22, 2015 at 3:01 PM, Heikki Linnakangas hlinn...@iki.fi wrote: For example, imagine that perform point-in-time recovery

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-22 Thread Heikki Linnakangas
On 04/22/2015 09:30 PM, Robert Haas wrote: On Wed, Apr 22, 2015 at 2:17 AM, Heikki Linnakangas hlinn...@iki.fi wrote: Note that it's a bit complicated to set up that scenario today. Archiving is never enabled in recovery mode, so you'll need to use a custom cron job or something to maintain the

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-22 Thread Robert Haas
On Wed, Apr 22, 2015 at 2:17 AM, Heikki Linnakangas hlinn...@iki.fi wrote: Note that it's a bit complicated to set up that scenario today. Archiving is never enabled in recovery mode, so you'll need to use a custom cron job or something to maintain the archive that C uses. The files will not

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-22 Thread Robert Haas
On Tue, Apr 21, 2015 at 8:30 PM, Michael Paquier michael.paqu...@gmail.com wrote: This .partial segment renaming is something that we should let the archive_command manage with its internal logic. This strikes me as equivalent to saying we don't know how to make this work right, but maybe our

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-22 Thread Heikki Linnakangas
On 04/22/2015 10:21 PM, Robert Haas wrote: On Wed, Apr 22, 2015 at 3:01 PM, Heikki Linnakangas hlinn...@iki.fi wrote: For example, imagine that perform point-in-time recovery to WAL position 0/1237E568, on timeline 1. That falls within segment 00010012. Then we end recovery, and

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-22 Thread Robert Haas
On Wed, Apr 22, 2015 at 3:01 PM, Heikki Linnakangas hlinn...@iki.fi wrote: On 04/22/2015 09:30 PM, Robert Haas wrote: On Wed, Apr 22, 2015 at 2:17 AM, Heikki Linnakangas hlinn...@iki.fi wrote: Note that it's a bit complicated to set up that scenario today. Archiving is never enabled in

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-22 Thread Robert Haas
On Wed, Apr 22, 2015 at 3:34 PM, Heikki Linnakangas hlinn...@iki.fi wrote: On 04/22/2015 10:21 PM, Robert Haas wrote: On Wed, Apr 22, 2015 at 3:01 PM, Heikki Linnakangas hlinn...@iki.fi wrote: For example, imagine that perform point-in-time recovery to WAL position 0/1237E568, on timeline 1.

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-21 Thread Michael Paquier
On Thu, Apr 16, 2015 at 8:57 PM, Heikki Linnakangas wrote: Oh, hang on, that's not necessarily true. On promotion, the standby archives the last, partial WAL segment from the old timeline. That's just wrong (http://www.postgresql.org/message-id/52fcd37c.3070...@vmware.com), and in fact I

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-21 Thread Michael Paquier
On Tue, Apr 21, 2015 at 4:38 PM, Heikki Linnakangas hlinn...@iki.fi wrote: On 04/21/2015 09:53 AM, Michael Paquier wrote: On Thu, Apr 16, 2015 at 8:57 PM, Heikki Linnakangas wrote: Oh, hang on, that's not necessarily true. On promotion, the standby archives the last, partial WAL segment

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-21 Thread Heikki Linnakangas
On 04/21/2015 09:53 AM, Michael Paquier wrote: On Thu, Apr 16, 2015 at 8:57 PM, Heikki Linnakangas wrote: Oh, hang on, that's not necessarily true. On promotion, the standby archives the last, partial WAL segment from the old timeline. That's just wrong

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-21 Thread Heikki Linnakangas
On 04/21/2015 12:04 PM, Michael Paquier wrote: On Tue, Apr 21, 2015 at 4:38 PM, Heikki Linnakangas hlinn...@iki.fi wrote: Note that even though we don't archive the partial last segment on the previous timeline, the same WAL is copied to the first segment on the new timeline. So the WAL isn't

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-21 Thread Robert Haas
On Tue, Apr 21, 2015 at 6:55 AM, Heikki Linnakangas hlinn...@iki.fi wrote: On 04/21/2015 12:04 PM, Michael Paquier wrote: On Tue, Apr 21, 2015 at 4:38 PM, Heikki Linnakangas hlinn...@iki.fi wrote: Note that even though we don't archive the partial last segment on the previous timeline, the

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-16 Thread Heikki Linnakangas
On 03/01/2015 12:36 AM, Venkata Balaji N wrote: Patch did get applied successfully to the latest master. Can you please rebase. Here you go. On 01/31/2015 03:07 PM, Andres Freund wrote: On 2014-12-19 22:56:40 +0200, Heikki Linnakangas wrote: This add two new archive_modes, 'shared' and

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-02-28 Thread Venkata Balaji N
Here's a first cut at this. It includes the changes from your standby_wal_archiving_v1.patch, so you get that behaviour if you set archive_mode='always', and the new behaviour I wanted with archive_mode='shared'. I wrote it on top of the other patch I posted recently to not archive bogus

[HACKERS] Re: [HACKERS] Streaming replication and WAL archive interactions

2015-02-11 Thread Миша Тюрин
  This should be a very common setup in the field, so how are  people doing it in practice? One of possible workaround with archive and streaming was to use pg_receivexlog from standby to copy/save WALs to archive. but with pg_receivexlog was also issue with fsync. [ master ] -- streaming 

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-01-31 Thread Andres Freund
Hi, On 2014-12-19 22:56:40 +0200, Heikki Linnakangas wrote: This add two new archive_modes, 'shared' and 'always', to indicate whether the WAL archive is shared between the primary and standby, or not. In shared mode, the standby tracks which files have been archived by the primary. The

Re: [HACKERS] Streaming replication and WAL archive interactions

2014-12-19 Thread Heikki Linnakangas
On 12/18/2014 12:32 PM, Fujii Masao wrote: On Wed, Dec 17, 2014 at 4:11 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 12/16/2014 10:24 AM, Borodin Vladimir wrote: 12 дек. 2014 г., в 16:46, Heikki Linnakangas hlinnakan...@vmware.com написал(а): There have been a few threads on the

Re: [HACKERS] Streaming replication and WAL archive interactions

2014-12-18 Thread Fujii Masao
On Wed, Dec 17, 2014 at 4:11 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 12/16/2014 10:24 AM, Borodin Vladimir wrote: 12 дек. 2014 г., в 16:46, Heikki Linnakangas hlinnakan...@vmware.com написал(а): There have been a few threads on the behavior of WAL archiving, after a standby

Re: [HACKERS] Streaming replication and WAL archive interactions

2014-12-16 Thread Borodin Vladimir
12 дек. 2014 г., в 16:46, Heikki Linnakangas hlinnakan...@vmware.com написал(а): There have been a few threads on the behavior of WAL archiving, after a standby server is promoted [1] [2]. In short, it doesn't work as you might expect. The standby will start archiving after it's promoted,

Re: [HACKERS] Streaming replication and WAL archive interactions

2014-12-16 Thread Heikki Linnakangas
On 12/16/2014 10:24 AM, Borodin Vladimir wrote: 12 дек. 2014 г., в 16:46, Heikki Linnakangas hlinnakan...@vmware.com написал(а): There have been a few threads on the behavior of WAL archiving, after a standby server is promoted [1] [2]. In short, it doesn't work as you might expect. The