Re: [HACKERS] Time-Delayed Standbys

2013-12-14 Thread Andres Freund
On 2013-12-13 13:44:30 +, Simon Riggs wrote: On 13 December 2013 13:22, Andres Freund and...@2ndquadrant.com wrote: On 2013-12-13 13:09:13 +, Simon Riggs wrote: On 13 December 2013 11:58, Andres Freund and...@2ndquadrant.com wrote: I removed it because it was after the pause. I'll

Re: [HACKERS] Time-Delayed Standbys

2013-12-13 Thread Simon Riggs
On 12 December 2013 21:58, Fabrízio de Royes Mello fabriziome...@gmail.com wrote: On Thu, Dec 12, 2013 at 3:42 PM, Fabrízio de Royes Mello fabriziome...@gmail.com wrote: On Thu, Dec 12, 2013 at 3:39 PM, Simon Riggs si...@2ndquadrant.com wrote: On 12 December 2013 15:19, Simon Riggs

Re: [HACKERS] Time-Delayed Standbys

2013-12-13 Thread Andres Freund
On 2013-12-13 11:56:47 +, Simon Riggs wrote: On 12 December 2013 21:58, Fabrízio de Royes Mello fabriziome...@gmail.com wrote: Reviewing the committed patch I noted that the CheckForStandbyTrigger() after the delay was removed. If we promote the standby during the delay and don't

Re: [HACKERS] Time-Delayed Standbys

2013-12-13 Thread Simon Riggs
On 13 December 2013 11:58, Andres Freund and...@2ndquadrant.com wrote: On 2013-12-13 11:56:47 +, Simon Riggs wrote: On 12 December 2013 21:58, Fabrízio de Royes Mello fabriziome...@gmail.com wrote: Reviewing the committed patch I noted that the CheckForStandbyTrigger() after the delay

Re: [HACKERS] Time-Delayed Standbys

2013-12-13 Thread Andres Freund
On 2013-12-13 13:09:13 +, Simon Riggs wrote: On 13 December 2013 11:58, Andres Freund and...@2ndquadrant.com wrote: On 2013-12-13 11:56:47 +, Simon Riggs wrote: On 12 December 2013 21:58, Fabrízio de Royes Mello fabriziome...@gmail.com wrote: Reviewing the committed patch I noted

Re: [HACKERS] Time-Delayed Standbys

2013-12-13 Thread Simon Riggs
On 13 December 2013 13:22, Andres Freund and...@2ndquadrant.com wrote: On 2013-12-13 13:09:13 +, Simon Riggs wrote: On 13 December 2013 11:58, Andres Freund and...@2ndquadrant.com wrote: On 2013-12-13 11:56:47 +, Simon Riggs wrote: On 12 December 2013 21:58, Fabrízio de Royes Mello

Re: [HACKERS] Time-Delayed Standbys

2013-12-13 Thread Fabrízio de Royes Mello
On Fri, Dec 13, 2013 at 11:44 AM, Simon Riggs si...@2ndquadrant.com wrote: On 13 December 2013 13:22, Andres Freund and...@2ndquadrant.com wrote: On 2013-12-13 13:09:13 +, Simon Riggs wrote: On 13 December 2013 11:58, Andres Freund and...@2ndquadrant.com wrote: On 2013-12-13 11:56:47

Re: [HACKERS] Time-Delayed Standbys

2013-12-12 Thread KONDO Mitsumasa
(2013/12/12 7:23), Fabrízio de Royes Mello wrote: On Wed, Dec 11, 2013 at 7:47 PM, Andres Freund and...@2ndquadrant.com * hot_standby=off: Makes delay useable with wal_level=archive (and thus a lower WAL volume) * standby_mode=off: Configurations that use tools like pg_standby and

Re: [HACKERS] Time-Delayed Standbys

2013-12-12 Thread Simon Riggs
On 12 December 2013 08:19, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: (2013/12/12 7:23), Fabrízio de Royes Mello wrote: On Wed, Dec 11, 2013 at 7:47 PM, Andres Freund and...@2ndquadrant.com * hot_standby=off: Makes delay useable with wal_level=archive (and thus a lower WAL

Re: [HACKERS] Time-Delayed Standbys

2013-12-12 Thread Simon Riggs
On 9 December 2013 10:54, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: (2013/12/09 19:35), Pavel Stehule wrote: 2013/12/9 KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp mailto:kondo.mitsum...@lab.ntt.co.jp Hi Fabrízio, I test your v4 patch, and send your review comments.

Re: [HACKERS] Time-Delayed Standbys

2013-12-12 Thread KONDO Mitsumasa
(2013/12/12 18:09), Simon Riggs wrote: On 9 December 2013 10:54, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: (2013/12/09 19:35), Pavel Stehule wrote: 2013/12/9 KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp mailto:kondo.mitsum...@lab.ntt.co.jp Hi Fabrízio, I test your

Re: [HACKERS] Time-Delayed Standbys

2013-12-12 Thread Simon Riggs
On 12 December 2013 10:42, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: I agree with your request here, but I don't think negative values are the right way to implement that, at least it would not be very usable. I think that my proposal is the easiest and simplist way to solve this

Re: [HACKERS] Time-Delayed Standbys

2013-12-12 Thread Andres Freund
On 2013-12-12 09:09:21 +, Simon Riggs wrote: * Add functionality (I propose) We can set negative number at min_standby_apply_delay. I think that this feature is for world wide replication situation. For example, master server is in Japan and slave server is in San

Re: [HACKERS] Time-Delayed Standbys

2013-12-12 Thread Simon Riggs
On 12 December 2013 11:05, Andres Freund and...@2ndquadrant.com wrote: My suggestion would be to add the TZ to the checkpoint record. This way all users of WAL can see the TZ of the master and act accordingly. I'll do a separate patch for that. Intuitively I'd say that might be useful - but

Re: [HACKERS] Time-Delayed Standbys

2013-12-12 Thread Mitsumasa KONDO
2013/12/12 Simon Riggs si...@2ndquadrant.com On 12 December 2013 10:42, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: I agree with your request here, but I don't think negative values are the right way to implement that, at least it would not be very usable. I think that my

Re: [HACKERS] Time-Delayed Standbys

2013-12-12 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On 12 December 2013 11:05, Andres Freund and...@2ndquadrant.com wrote: My suggestion would be to add the TZ to the checkpoint record. This way all users of WAL can see the TZ of the master and act accordingly. I'll do a separate patch for that.

Re: [HACKERS] Time-Delayed Standbys

2013-12-12 Thread Robert Haas
On Thu, Dec 12, 2013 at 9:52 AM, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: On 12 December 2013 11:05, Andres Freund and...@2ndquadrant.com wrote: My suggestion would be to add the TZ to the checkpoint record. This way all users of WAL can see the TZ of the

Re: [HACKERS] Time-Delayed Standbys

2013-12-12 Thread Simon Riggs
On 12 December 2013 15:03, Robert Haas robertmh...@gmail.com wrote: On Thu, Dec 12, 2013 at 9:52 AM, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: On 12 December 2013 11:05, Andres Freund and...@2ndquadrant.com wrote: My suggestion would be to add the TZ to the

Re: [HACKERS] Time-Delayed Standbys

2013-12-12 Thread Simon Riggs
On 12 December 2013 15:19, Simon Riggs si...@2ndquadrant.com wrote: Don't panic guys! I meant UTC offset only. And yes, it may not be needed, will check. Checked, all non-UTC TZ offsets work without further effort here. -- Simon Riggs http://www.2ndQuadrant.com/

Re: [HACKERS] Time-Delayed Standbys

2013-12-12 Thread Fabrízio de Royes Mello
On Thu, Dec 12, 2013 at 3:39 PM, Simon Riggs si...@2ndquadrant.com wrote: On 12 December 2013 15:19, Simon Riggs si...@2ndquadrant.com wrote: Don't panic guys! I meant UTC offset only. And yes, it may not be needed, will check. Checked, all non-UTC TZ offsets work without further effort

Re: [HACKERS] Time-Delayed Standbys

2013-12-12 Thread Fabrízio de Royes Mello
On Thu, Dec 12, 2013 at 3:42 PM, Fabrízio de Royes Mello fabriziome...@gmail.com wrote: On Thu, Dec 12, 2013 at 3:39 PM, Simon Riggs si...@2ndquadrant.com wrote: On 12 December 2013 15:19, Simon Riggs si...@2ndquadrant.com wrote: Don't panic guys! I meant UTC offset only. And yes, it

Re: [HACKERS] Time-Delayed Standbys

2013-12-11 Thread Simon Riggs
On 11 December 2013 06:36, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: I think this feature will be used in a lot of scenarios in which PITR is currently used. We have to judge which is better, we get something potential or to protect stupid. And we had better to wait author's

Re: [HACKERS] Time-Delayed Standbys

2013-12-11 Thread Fabrízio de Royes Mello
On Wed, Dec 11, 2013 at 6:27 AM, Simon Riggs si...@2ndquadrant.com wrote: On 11 December 2013 06:36, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: I think this feature will be used in a lot of scenarios in which PITR is currently used. We have to judge which is better, we get

Re: [HACKERS] Time-Delayed Standbys

2013-12-11 Thread Andres Freund
On 2013-12-11 16:37:54 -0200, Fabrízio de Royes Mello wrote: On Wed, Dec 11, 2013 at 6:27 AM, Simon Riggs si...@2ndquadrant.com wrote: I think this feature will be used in a lot of scenarios in which PITR is currently used. We have to judge which is better, we get something potential

Re: [HACKERS] Time-Delayed Standbys

2013-12-11 Thread Fabrízio de Royes Mello
On Wed, Dec 11, 2013 at 7:47 PM, Andres Freund and...@2ndquadrant.com wrote: I don't think that position has any merit, sorry: Think about the way this stuff gets setup. The user creates a new basebackup (pg_basebackup, manual pg_start/stop_backup, shutdown primary). Then he creates a

Re: [HACKERS] Time-Delayed Standbys

2013-12-10 Thread Andres Freund
On 2013-12-10 13:26:27 +0900, KONDO Mitsumasa wrote: (2013/12/09 20:29), Andres Freund wrote: On 2013-12-09 19:51:01 +0900, KONDO Mitsumasa wrote: Add my comment. We have to consider three situations. 1. PITR 2. replication standby 3. replication standby with restore_command I think

Re: [HACKERS] Time-Delayed Standbys

2013-12-10 Thread KONDO Mitsumasa
(2013/12/10 18:38), Andres Freund wrote: master PITR? What's that? All PITR is based on recovery.conf and thus not really a master? master PITR is PITR with standby_mode = off. It's just recovery from basebackup. They have difference between master PITR and standby that the former will be

Re: [HACKERS] Time-Delayed Standbys

2013-12-09 Thread KONDO Mitsumasa
Hi Fabrízio, I test your v4 patch, and send your review comments. * Fix typo 49 -# commited transactions from the master, specify a recovery time delay. 49 +# committed transactions from the master, specify a recovery time delay. * Fix white space 177 - if (secs = 0 microsecs

Re: [HACKERS] Time-Delayed Standbys

2013-12-09 Thread Pavel Stehule
2013/12/9 KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp Hi Fabrízio, I test your v4 patch, and send your review comments. * Fix typo 49 -# commited transactions from the master, specify a recovery time delay. 49 +# committed transactions from the master, specify a recovery time delay.

Re: [HACKERS] Time-Delayed Standbys

2013-12-09 Thread KONDO Mitsumasa
(2013/12/09 19:36), KONDO Mitsumasa wrote: * Problem 1 I read your wittened document. There is PITR has not affected. However, when I run PITR with min_standby_apply_delay=300, it cannot start server. The log is under following. [mitsu-ko@localhost postgresql]$ bin/pg_ctl -D data2 start

Re: [HACKERS] Time-Delayed Standbys

2013-12-09 Thread KONDO Mitsumasa
(2013/12/09 19:35), Pavel Stehule wrote: 2013/12/9 KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp mailto:kondo.mitsum...@lab.ntt.co.jp Hi Fabrízio, I test your v4 patch, and send your review comments. * Fix typo 49 -# commited transactions from the master, specify a

Re: [HACKERS] Time-Delayed Standbys

2013-12-09 Thread Andres Freund
On 2013-12-09 19:51:01 +0900, KONDO Mitsumasa wrote: Add my comment. We have to consider three situations. 1. PITR 2. replication standby 3. replication standby with restore_command I think this patch cannot delay in 1 situation. Why? Greetings, Andres Freund -- Andres Freund

Re: [HACKERS] Time-Delayed Standbys

2013-12-09 Thread Craig Ringer
On 12/04/2013 02:46 AM, Robert Haas wrote: Thanks for your review Christian... So, I proposed this patch previously and I still think it's a good idea, but it got voted down on the grounds that it didn't deal with clock drift. I view that as insufficient reason to reject the feature, but

Re: [HACKERS] Time-Delayed Standbys

2013-12-09 Thread Greg Stark
On 9 Dec 2013 12:16, Craig Ringer cr...@2ndquadrant.com wrote: The only way to deal with clock drift that isn't fragile in the face of variable latency, etc, is to basically re-implement (S)NTP in order to find out what the clock difference with the remote is. There's actually an entirely

Re: [HACKERS] Time-Delayed Standbys

2013-12-09 Thread KONDO Mitsumasa
(2013/12/09 20:29), Andres Freund wrote: On 2013-12-09 19:51:01 +0900, KONDO Mitsumasa wrote: Add my comment. We have to consider three situations. 1. PITR 2. replication standby 3. replication standby with restore_command I think this patch cannot delay in 1 situation. Why? I have three

Re: [HACKERS] Time-Delayed Standbys

2013-12-06 Thread Robert Haas
On Thu, Dec 5, 2013 at 11:07 PM, Fabrízio de Royes Mello fabriziome...@gmail.com wrote: On Tue, Dec 3, 2013 at 5:33 PM, Simon Riggs si...@2ndquadrant.com wrote: - compute recoveryUntilDelayTime in XLOG_XACT_COMMIT and XLOG_XACT_COMMIT_COMPACT checks Why just those? Why not aborts and

Re: [HACKERS] Time-Delayed Standbys

2013-12-06 Thread Fabrízio de Royes Mello
On Fri, Dec 6, 2013 at 1:36 PM, Robert Haas robertmh...@gmail.com wrote: On Thu, Dec 5, 2013 at 11:07 PM, Fabrízio de Royes Mello fabriziome...@gmail.com wrote: On Tue, Dec 3, 2013 at 5:33 PM, Simon Riggs si...@2ndquadrant.com wrote: - compute recoveryUntilDelayTime in XLOG_XACT_COMMIT

Re: [HACKERS] Time-Delayed Standbys

2013-12-05 Thread Magnus Hagander
On Thu, Dec 5, 2013 at 1:45 AM, Simon Riggs si...@2ndquadrant.com wrote: On 3 December 2013 18:46, Robert Haas robertmh...@gmail.com wrote: On Tue, Dec 3, 2013 at 12:36 PM, Fabrízio de Royes Mello fabriziome...@gmail.com wrote: On Tue, Dec 3, 2013 at 2:33 PM, Christian Kruse

Re: [HACKERS] Time-Delayed Standbys

2013-12-05 Thread Simon Riggs
On 5 December 2013 08:51, Magnus Hagander mag...@hagander.net wrote: Not recalling the older thread, but it seems the breaks on clock drift, I think we can fairly easily make that situation good enough. Just have IDENTIFY_SYSTEM return the current timestamp on the master, and refuse to start

Re: [HACKERS] Time-Delayed Standbys

2013-12-05 Thread Fabrízio de Royes Mello
On Thu, Dec 5, 2013 at 7:57 AM, Simon Riggs si...@2ndquadrant.com wrote: On 5 December 2013 08:51, Magnus Hagander mag...@hagander.net wrote: Not recalling the older thread, but it seems the breaks on clock drift, I think we can fairly easily make that situation good enough. Just have

Re: [HACKERS] Time-Delayed Standbys

2013-12-05 Thread Fabrízio de Royes Mello
On Tue, Dec 3, 2013 at 5:33 PM, Simon Riggs si...@2ndquadrant.com wrote: - compute recoveryUntilDelayTime in XLOG_XACT_COMMIT and XLOG_XACT_COMMIT_COMPACT checks Why just those? Why not aborts and restore points also? I think make no sense execute the delay after aborts and/or restore

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Andres Freund
On 2013-12-04 11:13:58 +0900, KONDO Mitsumasa wrote: 4) Start the slave and connect to it using psql and in another session I can see all archive recovery log Hmm... I had thought my mistake in reading your email, but it reproduce again. When I sat small recovery_time_delay(=3), it might

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Christian Kruse
Hi, On 04/12/13 11:13, KONDO Mitsumasa wrote: 1) Clusters - build master - build slave and attach to the master using SR and config recovery_time_delay to 1min. 2) Stop de Slave 3) Run some transactions on the master using pgbench to generate a lot of archives 4) Start the slave

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Mitsumasa KONDO
2013/12/4 Andres Freund and...@2ndquadrant.com On 2013-12-04 11:13:58 +0900, KONDO Mitsumasa wrote: 4) Start the slave and connect to it using psql and in another session I can see all archive recovery log Hmm... I had thought my mistake in reading your email, but it reproduce again.

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Andres Freund
On 2013-12-04 22:47:47 +0900, Mitsumasa KONDO wrote: 2013/12/4 Andres Freund and...@2ndquadrant.com When it happened, psql cannot connect standby server at all. I think this behavior is not good. It should only delay recovery position and can seen old delay table data. That doesn't sound like

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Mitsumasa KONDO
2013/12/4 Christian Kruse christ...@2ndquadrant.com You created a master node and a hot standby with 300 delay. Then you stopped the standby, did the pgbench and startet the hot standby again. It did not get in line with the master. Is this correct? No. First, I start master, and execute

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Andres Freund
Hi, On 2013-12-03 19:33:16 +, Simon Riggs wrote: - compute recoveryUntilDelayTime in XLOG_XACT_COMMIT and XLOG_XACT_COMMIT_COMPACT checks Why just those? Why not aborts and restore points also? What would the advantage of waiting on anything but commits be? If it's not a commit, the

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Mitsumasa KONDO
2013/12/4 Andres Freund and...@2ndquadrant.com On 2013-12-04 22:47:47 +0900, Mitsumasa KONDO wrote: 2013/12/4 Andres Freund and...@2ndquadrant.com When it happened, psql cannot connect standby server at all. I think this behavior is not good. It should only delay recovery position and

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: So, I proposed this patch previously and I still think it's a good idea, but it got voted down on the grounds that it didn't deal with clock drift.  I view that as insufficient reason to reject the feature, but others disagreed.  Unless some of those

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Christian Kruse
Hi, On 04/12/13 07:22, Kevin Grittner wrote: There are many things that a system admin can get wrong.  Failing to supply this feature because the sysadmin might not be running ntpd (or equivalent) correctly seems to me to be like not having the software do fsync because the sysadmin might not

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Peter Eisentraut
src/backend/access/transam/xlog.c:5889: trailing whitespace. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Simon Riggs
On 3 December 2013 18:46, Robert Haas robertmh...@gmail.com wrote: On Tue, Dec 3, 2013 at 12:36 PM, Fabrízio de Royes Mello fabriziome...@gmail.com wrote: On Tue, Dec 3, 2013 at 2:33 PM, Christian Kruse christ...@2ndquadrant.com wrote: Hi Fabrizio, looks good to me. I did some testing on

Re: [HACKERS] Time-Delayed Standbys

2013-12-03 Thread Christian Kruse
Hi Fabrizio, looks good to me. I did some testing on 9.2.4, 9.2.5 and HEAD. It applies and compiles w/o errors or warnings. I set up a master and two hot standbys replicating from the master, one with 5 minutes delay and one without delay. After that I created a new database and generated some

Re: [HACKERS] Time-Delayed Standbys

2013-12-03 Thread Fabrízio de Royes Mello
On Tue, Dec 3, 2013 at 2:33 PM, Christian Kruse christ...@2ndquadrant.comwrote: Hi Fabrizio, looks good to me. I did some testing on 9.2.4, 9.2.5 and HEAD. It applies and compiles w/o errors or warnings. I set up a master and two hot standbys replicating from the master, one with 5 minutes

Re: [HACKERS] Time-Delayed Standbys

2013-12-03 Thread Robert Haas
On Tue, Dec 3, 2013 at 12:36 PM, Fabrízio de Royes Mello fabriziome...@gmail.com wrote: On Tue, Dec 3, 2013 at 2:33 PM, Christian Kruse christ...@2ndquadrant.com wrote: Hi Fabrizio, looks good to me. I did some testing on 9.2.4, 9.2.5 and HEAD. It applies and compiles w/o errors or

Re: [HACKERS] Time-Delayed Standbys

2013-12-03 Thread Joshua D. Drake
On 12/03/2013 10:46 AM, Robert Haas wrote: On Tue, Dec 3, 2013 at 12:36 PM, Fabrízio de Royes Mello fabriziome...@gmail.com wrote: On Tue, Dec 3, 2013 at 2:33 PM, Christian Kruse christ...@2ndquadrant.com wrote: Hi Fabrizio, looks good to me. I did some testing on 9.2.4, 9.2.5 and HEAD. It

Re: [HACKERS] Time-Delayed Standbys

2013-12-03 Thread Andres Freund
On 2013-12-03 13:46:28 -0500, Robert Haas wrote: On Tue, Dec 3, 2013 at 12:36 PM, Fabrízio de Royes Mello fabriziome...@gmail.com wrote: On Tue, Dec 3, 2013 at 2:33 PM, Christian Kruse christ...@2ndquadrant.com wrote: Hi Fabrizio, looks good to me. I did some testing on 9.2.4, 9.2.5

Re: [HACKERS] Time-Delayed Standbys

2013-12-03 Thread Simon Riggs
On 18 October 2013 19:03, Fabrízio de Royes Mello fabriziome...@gmail.com wrote: The attached patch is a continuation of Robert's work [1]. Reviewing v2... I made some changes: - use of Latches instead of pg_usleep, so we don't have to wakeup regularly. OK - call

Re: [HACKERS] Time-Delayed Standbys

2013-12-03 Thread KONDO Mitsumasa
(2013/11/30 5:34), Fabrízio de Royes Mello wrote: On Fri, Nov 29, 2013 at 5:49 AM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp mailto:kondo.mitsum...@lab.ntt.co.jp wrote: * Problem1 Your patch does not code recovery.conf.sample about recovery_time_delay. Please add it. Fixed. OK. It

Re: [HACKERS] Time-Delayed Standbys

2013-12-03 Thread KONDO Mitsumasa
(2013/12/04 4:00), Andres Freund wrote: On 2013-12-03 13:46:28 -0500, Robert Haas wrote: On Tue, Dec 3, 2013 at 12:36 PM, Fabrízio de Royes Mello fabriziome...@gmail.com wrote: On Tue, Dec 3, 2013 at 2:33 PM, Christian Kruse christ...@2ndquadrant.com wrote: Hi Fabrizio, looks good to me. I

Re: [HACKERS] Time-Delayed Standbys

2013-11-29 Thread Fabrízio de Royes Mello
On Fri, Nov 29, 2013 at 5:49 AM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: Hi Royes, I'm sorry for my late review... No problem... I feel potential of your patch in PG replication function, and it might be something useful for all people. I check your patch and have some

Re: [HACKERS] Time-Delayed Standbys

2013-11-28 Thread KONDO Mitsumasa
Hi Royes, I'm sorry for my late review... I feel potential of your patch in PG replication function, and it might be something useful for all people. I check your patch and have some comment for improvement. I haven't executed detail of unexpected sutuation yet. But I think that under

Re: [HACKERS] time-delayed standbys

2011-07-02 Thread Simon Riggs
On Thu, Jun 30, 2011 at 6:25 PM, Robert Haas robertmh...@gmail.com wrote: I think the time problems are more complex than said. The patch relies upon transaction completion times, but not all WAL records have a time attached to them. Plus you only used commits anyway, not sure why. For the

Re: [HACKERS] time-delayed standbys

2011-06-30 Thread Simon Riggs
On Thu, Jun 30, 2011 at 2:56 AM, Robert Haas robertmh...@gmail.com wrote: On Wed, Jun 29, 2011 at 9:54 PM, Josh Berkus j...@agliodbs.com wrote: I am not sure exactly how walreceiver handles it if the disk is full. I assume it craps out and eventually retries, so probably what will happen is

Re: [HACKERS] time-delayed standbys

2011-06-30 Thread Simon Riggs
On Wed, Jun 29, 2011 at 7:11 PM, Robert Haas robertmh...@gmail.com wrote: I don't really see how that's any different from what happens now.  If (for whatever reason) the master is generating WAL faster than a streaming standby can replay it, then the excess WAL is going to pile up someplace,

Re: [HACKERS] time-delayed standbys

2011-06-30 Thread Josh Berkus
On 6/30/11 2:00 AM, Simon Riggs wrote: Manual (or scripted) intervention is always necessary if you reach disk 100% full. Wow, that's a pretty crappy failure mode... but I don't think we need to fix it just on account of this patch. It would be nice to fix, of course. How is that

Re: [HACKERS] time-delayed standbys

2011-06-30 Thread Robert Haas
On Thu, Jun 30, 2011 at 1:00 PM, Josh Berkus j...@agliodbs.com wrote: On 6/30/11 2:00 AM, Simon Riggs wrote: Manual (or scripted) intervention is always necessary if you reach disk 100% full. Wow, that's a pretty crappy failure mode... but I don't think we need to fix it just on account

Re: [HACKERS] time-delayed standbys

2011-06-30 Thread Robert Haas
On Thu, Jun 30, 2011 at 6:45 AM, Simon Riggs si...@2ndquadrant.com wrote: The only way to control this is with a time delay that can be changed while the server is running. A recovery.conf parameter doesn't allow that, so another way is preferable. True. We've talked about making the

Re: [HACKERS] time-delayed standbys

2011-06-30 Thread Josh Berkus
On 6/30/11 10:25 AM, Robert Haas wrote: So I think keeping it defined it terms of time is the right way forward, even though the need for external time synchronization is, certainly, not ideal. Actually, when we last had the argument about time synchronization, Kevin Grittner (I believe)

Re: [HACKERS] time-delayed standbys

2011-06-30 Thread Kevin Grittner
Josh Berkus j...@agliodbs.com wrote: when we last had the argument about time synchronization, Kevin Grittner (I believe) pointed out that unsynchronized replication servers have an assortment of other issues ... like any read query involving now(). I don't remember making that point,

Re: [HACKERS] time-delayed standbys

2011-06-30 Thread Josh Berkus
Kevin, I think doing anything in PostgreSQL around this beyond allowing DBAs to trust their server clocks is insane. The arguments for using and trusting ntpd is pretty much identical to the arguments for using and trusting the OS file systems. Oh, you don't want to implement our own NTP?

Re: [HACKERS] time-delayed standbys

2011-06-30 Thread Robert Haas
On Thu, Jun 30, 2011 at 1:51 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: I think doing anything in PostgreSQL around this beyond allowing DBAs to trust their server clocks is insane.  The arguments for using and trusting ntpd is pretty much identical to the arguments for using and

Re: [HACKERS] time-delayed standbys

2011-06-30 Thread Fujii Masao
On Fri, Jul 1, 2011 at 2:25 AM, Robert Haas robertmh...@gmail.com wrote: Some actions aren't even transactional, such as DROP DATABASE, amongst Good point.  We'd probably need to add a timestamp to the drop database record, as that's a case that people would likely want to defend against with

Re: [HACKERS] time-delayed standbys

2011-06-30 Thread Fujii Masao
On Fri, Jul 1, 2011 at 3:25 AM, Robert Haas robertmh...@gmail.com wrote: On Thu, Jun 30, 2011 at 1:51 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: I think doing anything in PostgreSQL around this beyond allowing DBAs to trust their server clocks is insane.  The arguments for using

Re: [HACKERS] time-delayed standbys

2011-06-30 Thread Jaime Casanova
Fujii Masao masao.fu...@gmail.com writes: On Fri, Jul 1, 2011 at 2:25 AM, Robert Haas robertmh...@gmail.com wrote: Some actions aren't even transactional, such as DROP DATABASE, amongst Good point.  We'd probably need to add a timestamp to the drop database record, as that's a case that

Re: [HACKERS] time-delayed standbys

2011-06-29 Thread Simon Riggs
On Wed, Jun 15, 2011 at 6:58 AM, Fujii Masao masao.fu...@gmail.com wrote: Or, we should implement new promote mode which finishes a recovery as soon as promote is requested (i.e., not replay all the available WAL records)? That's not a new feature. We had it in 8.4, but it was removed.

Re: [HACKERS] time-delayed standbys

2011-06-29 Thread Simon Riggs
On Thu, Jun 16, 2011 at 7:29 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, Jun 15, 2011 at 1:58 AM, Fujii Masao masao.fu...@gmail.com wrote: When the replication connection is terminated, the standby tries to read WAL files from the archive. In this case, there is no walreceiver process,

Re: [HACKERS] time-delayed standbys

2011-06-29 Thread Robert Haas
On Wed, Jun 29, 2011 at 4:00 AM, Simon Riggs si...@2ndquadrant.com wrote: On Thu, Jun 16, 2011 at 7:29 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, Jun 15, 2011 at 1:58 AM, Fujii Masao masao.fu...@gmail.com wrote: When the replication connection is terminated, the standby tries to read

Re: [HACKERS] time-delayed standbys

2011-06-29 Thread Simon Riggs
On Wed, Jun 29, 2011 at 1:24 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, Jun 29, 2011 at 4:00 AM, Simon Riggs si...@2ndquadrant.com wrote: On Thu, Jun 16, 2011 at 7:29 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, Jun 15, 2011 at 1:58 AM, Fujii Masao masao.fu...@gmail.com wrote:

Re: [HACKERS] time-delayed standbys

2011-06-29 Thread Robert Haas
On Wed, Jun 29, 2011 at 1:50 PM, Simon Riggs si...@2ndquadrant.com wrote: As implemented, the feature will work with either streaming replication or with file-based replication. That sounds like the exact opposite of yours and Fujii's comments above. Please explain. I think our comments

Re: [HACKERS] time-delayed standbys

2011-06-29 Thread Josh Berkus
Robert, I don't really see how that's any different from what happens now. If (for whatever reason) the master is generating WAL faster than a streaming standby can replay it, then the excess WAL is going to pile up someplace, and you might run out of disk space. Time-delaying the standby

Re: [HACKERS] time-delayed standbys

2011-06-29 Thread Josh Berkus
On 6/29/11 11:11 AM, Robert Haas wrote: If the standby gets far enough behind the master that the required files are no longer there, then it will switch to the archive, if available. One more thing: As I understand it (and my testing shows this), the standby *prefers* the archive logs, and

Re: [HACKERS] time-delayed standbys

2011-06-29 Thread Robert Haas
On Wed, Jun 29, 2011 at 9:54 PM, Josh Berkus j...@agliodbs.com wrote: I am not sure exactly how walreceiver handles it if the disk is full. I assume it craps out and eventually retries, so probably what will happen is that, after the standby's pg_xlog directory fills up, walreceiver will sit

Re: [HACKERS] time-delayed standbys

2011-06-29 Thread Fujii Masao
On Wed, Jun 29, 2011 at 11:14 AM, Robert Haas robertmh...@gmail.com wrote: On Wed, Jun 15, 2011 at 1:58 AM, Fujii Masao masao.fu...@gmail.com wrote: After we run pg_ctl promote, time-delayed replication should be disabled? Otherwise, failover might take very long time when we set

Re: [HACKERS] time-delayed standbys

2011-06-29 Thread Fujii Masao
On Thu, Jun 30, 2011 at 10:56 AM, Robert Haas robertmh...@gmail.com wrote: Nope, it gets stuck and stops there.  Replay doesn't advance unless you can somehow clear out some space manually; if the disk is full, the disk is full, and PostgreSQL doesn't remove WAL files without being able to

Re: [HACKERS] time-delayed standbys

2011-06-29 Thread Fujii Masao
On Thu, Jun 30, 2011 at 12:14 PM, Fujii Masao masao.fu...@gmail.com wrote: We should disable this feature also after recovery reaches the stop point (specified in recovery_target_xxx)? Another comment; it's very helpful to document the behavior of delayed standby when promoting or after

Re: [HACKERS] time-delayed standbys

2011-06-28 Thread Robert Haas
On Wed, Jun 15, 2011 at 1:58 AM, Fujii Masao masao.fu...@gmail.com wrote: After we run pg_ctl promote, time-delayed replication should be disabled? Otherwise, failover might take very long time when we set recovery_time_delay to high value. PFA a patch that I believe will disable

Re: [HACKERS] time-delayed standbys

2011-06-19 Thread Fujii Masao
On Fri, Jun 17, 2011 at 11:34 AM, Robert Haas robertmh...@gmail.com wrote: On Thu, Jun 16, 2011 at 10:10 PM, Fujii Masao masao.fu...@gmail.com wrote: According to the above page, one purpose of time-delayed replication is to protect against user mistakes on master. But, when an user notices his

Re: [HACKERS] time-delayed standbys

2011-06-16 Thread Robert Haas
On Wed, Jun 15, 2011 at 1:58 AM, Fujii Masao masao.fu...@gmail.com wrote: When the replication connection is terminated, the standby tries to read WAL files from the archive. In this case, there is no walreceiver process, so how does the standby calculate the clock difference? Good question.

Re: [HACKERS] time-delayed standbys

2011-06-16 Thread Fujii Masao
On Fri, Jun 17, 2011 at 3:29 AM, Robert Haas robertmh...@gmail.com wrote: Even if that were not an issue, I'm still more or less of the opinion that trying to solve the time synchronization problem is a rathole anyway.  To really solve this problem well, you're going to need the standby to

Re: [HACKERS] time-delayed standbys

2011-06-16 Thread Robert Haas
On Thu, Jun 16, 2011 at 10:10 PM, Fujii Masao masao.fu...@gmail.com wrote: According to the above page, one purpose of time-delayed replication is to protect against user mistakes on master. But, when an user notices his wrong operation on master, what should he do next? The WAL records of his

Re: [HACKERS] time-delayed standbys

2011-06-15 Thread Jaime Casanova
On Wed, Jun 15, 2011 at 12:58 AM, Fujii Masao masao.fu...@gmail.com wrote: http://forge.mysql.com/worklog/task.php?id=344 According to the above page, one purpose of time-delayed replication is to protect against user mistakes on master. But, when an user notices his wrong operation on

Re: [HACKERS] time-delayed standbys

2011-06-14 Thread Fujii Masao
On Thu, Apr 21, 2011 at 12:18 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, Apr 20, 2011 at 11:15 AM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: I am a bit concerned about the reliability of this approach.  If there is some network lag, or some lag in

Re: [HACKERS] time-delayed standbys

2011-05-11 Thread Heikki Linnakangas
On 07.05.2011 16:48, Robert Haas wrote: I was able to reproduce something very like this in unpatched master, just by letting recovery pause at a named restore point, and then resuming it. LOG: recovery stopping at restore point stop, time 2011-05-07 09:28:01.652958-04 LOG: recovery has

Re: [HACKERS] time-delayed standbys

2011-05-11 Thread Heikki Linnakangas
On 11.05.2011 08:29, Fujii Masao wrote: On Sat, May 7, 2011 at 10:48 PM, Robert Haasrobertmh...@gmail.com wrote: I was able to reproduce something very like this in unpatched master, just by letting recovery pause at a named restore point, and then resuming it. I was able to reproduce the

Re: [HACKERS] time-delayed standbys

2011-05-11 Thread Fujii Masao
On Wed, May 11, 2011 at 6:50 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I think we can just always call ShutdownWalRcv(). It should be gone if the server was promoted while streaming, but that's just an implementation detail of what the promotion code does. There's no

Re: [HACKERS] time-delayed standbys

2011-05-11 Thread Heikki Linnakangas
On 11.05.2011 14:16, Fujii Masao wrote: On Wed, May 11, 2011 at 6:50 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I think we can just always call ShutdownWalRcv(). It should be gone if the server was promoted while streaming, but that's just an implementation detail of

Re: [HACKERS] time-delayed standbys

2011-05-10 Thread Fujii Masao
On Sat, May 7, 2011 at 10:48 PM, Robert Haas robertmh...@gmail.com wrote: I was able to reproduce something very like this in unpatched master, just by letting recovery pause at a named restore point, and then resuming it. I was able to reproduce the same problem even in 9.0. When the standby

Re: [HACKERS] time-delayed standbys

2011-05-07 Thread Robert Haas
On Sat, Apr 23, 2011 at 9:46 PM, Jaime Casanova ja...@2ndquadrant.com wrote: On Tue, Apr 19, 2011 at 9:47 PM, Robert Haas robertmh...@gmail.com wrote: That is, a standby configured such that replay lags a prescribed amount of time behind the master. This seemed easy to implement, so I did.  

Re: [HACKERS] time-delayed standbys

2011-04-23 Thread Jaime Casanova
On Tue, Apr 19, 2011 at 9:47 PM, Robert Haas robertmh...@gmail.com wrote: That is, a standby configured such that replay lags a prescribed amount of time behind the master. This seemed easy to implement, so I did.  Patch (for 9.2, obviously) attached. This crashes when stoping recovery to a

  1   2   >