Re: [HACKERS] pg_receivexlog add synchronous mode

2014-08-12 Thread Michael Paquier
On Fri, Aug 8, 2014 at 5:10 PM, Fujii Masao masao.fu...@gmail.com wrote: Thanks for the review! Applied the patch. I noticed that the tab padding for the new option -F in the getops switch is incorrect. Attached patch corrects that. pgindent would have caught that anyway, but it doesn't hurt to

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-08-12 Thread Fujii Masao
On Tue, Aug 12, 2014 at 4:34 PM, Michael Paquier michael.paqu...@gmail.com wrote: On Fri, Aug 8, 2014 at 5:10 PM, Fujii Masao masao.fu...@gmail.com wrote: Thanks for the review! Applied the patch. I noticed that the tab padding for the new option -F in the getops switch is incorrect. Attached

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-08-08 Thread Fujii Masao
On Thu, Aug 7, 2014 at 5:24 PM, furu...@pm.nttdata.co.jp wrote: Okay, applied the patch. I heavily modified your patch based on the master that the refactoring patch has been applied. Attached is that modified version. Could you review that? Thank you for the patch. I did a review of the

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-08-07 Thread furuyao
Okay, applied the patch. I heavily modified your patch based on the master that the refactoring patch has been applied. Attached is that modified version. Could you review that? Thank you for the patch. I did a review of the patch. No problem in the patch. Behavior after the true return

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-08-06 Thread Fujii Masao
On Wed, Aug 6, 2014 at 2:34 PM, furu...@pm.nttdata.co.jp wrote: I have improved the patch by making following changes: 1. Since stream_stop() was redundant, stream_stop() at the time of WAL file closing was deleted. 2. Change the Flash judging timing for the readability of source

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-08-06 Thread furuyao
- break; /* ignore the rest of this XLogData packet */ + return true;/* ignore the rest of this XLogData packet */ For break statement at close of wal file, it is a return to true. It may be a behavior

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-08-06 Thread Fujii Masao
On Wed, Aug 6, 2014 at 5:10 PM, furu...@pm.nttdata.co.jp wrote: - break; /* ignore the rest of this XLogData packet */ + return true;/* ignore the rest of this XLogData packet */ For break statement at

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-08-05 Thread Fujii Masao
On Tue, Jul 29, 2014 at 7:07 PM, furu...@pm.nttdata.co.jp wrote: I have improved the patch by making following changes: 1. Since stream_stop() was redundant, stream_stop() at the time of WAL file closing was deleted. 2. Change the Flash judging timing for the readability of source code.

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-08-05 Thread Fujii Masao
On Tue, Aug 5, 2014 at 9:04 PM, Fujii Masao masao.fu...@gmail.com wrote: On Tue, Jul 29, 2014 at 7:07 PM, furu...@pm.nttdata.co.jp wrote: I have improved the patch by making following changes: 1. Since stream_stop() was redundant, stream_stop() at the time of WAL file closing was deleted.

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-08-05 Thread furuyao
I have improved the patch by making following changes: 1. Since stream_stop() was redundant, stream_stop() at the time of WAL file closing was deleted. 2. Change the Flash judging timing for the readability of source code. I have changed the Flash judging timing , from the

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-07-29 Thread furuyao
I have improved the patch by making following changes: 1. Since stream_stop() was redundant, stream_stop() at the time of WAL file closing was deleted. 2. Change the Flash judging timing for the readability of source code. I have changed the Flash judging timing , from the continuous

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-07-25 Thread furuyao
This patch was made by the following process. 1. post patch for additional pg_receivexlog synchronous mode. 2. In response to comment for the flush frequency, I revise the patch to do flush every consecutive message in reference to walreceiver. 3. The synchronization mode was necessary to

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-07-10 Thread furuyao
This patch is modified the comment. Each comment is coping as follows. Could you update the status of this patch from Waiting on Author to Needs Review when you post the revised version of the patch? Thank you for pointing out. From now on, I will update status when I post the patch. +

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-07-07 Thread Fujii Masao
On Fri, Jul 4, 2014 at 7:45 PM, furu...@pm.nttdata.co.jp wrote: Thanks for reviewing the patch! I think that this refactoring patch is useful for improving source code readability and making the future patches simpler, whether we adopt your patch or not. So, barring any objections, I'm

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-07-04 Thread furuyao
Thanks for reviewing the patch! I think that this refactoring patch is useful for improving source code readability and making the future patches simpler, whether we adopt your patch or not. So, barring any objections, I'm thinking to commit this refactoring patch. Committed! It is

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-07-03 Thread Fujii Masao
On Tue, Jul 1, 2014 at 10:11 PM, Fujii Masao masao.fu...@gmail.com wrote: On Mon, Jun 30, 2014 at 7:09 PM, furu...@pm.nttdata.co.jp wrote: Thanks for the review! +if (secs = 0) +secs = 1;/* Always sleep at least 1 sec */ + +sleeptime = secs *

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-07-01 Thread Fujii Masao
On Mon, Jun 30, 2014 at 7:09 PM, furu...@pm.nttdata.co.jp wrote: Thanks for the review! +if (secs = 0) +secs = 1;/* Always sleep at least 1 sec */ + +sleeptime = secs * 1000 + usecs / 1000; The above is the code which caused that problem.

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-06-30 Thread furuyao
Thanks for the review! +if (secs = 0) +secs = 1;/* Always sleep at least 1 sec */ + +sleeptime = secs * 1000 + usecs / 1000; The above is the code which caused that problem. 'usecs' should have been reset to zero when 'secs' are rounded up to

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-06-30 Thread furuyao
Synchronous(synchronous_commit = on) mode offers the ability to confirm WAL have been streamed in the same way as synchronous replication. If an output is used as a different disk from the directory where the transaction log should be stored. Prevent the loss of data due

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-06-27 Thread Fujii Masao
On Thu, Jun 26, 2014 at 7:01 PM, furu...@pm.nttdata.co.jp wrote: The patch looks somewhat complicated and bugs can be easily introduced because it tries to not only add new feature but also reorganize the main loop in HandleCopyStream at the same time. To keep the patch simple, I'm thinking

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-06-27 Thread Andres Freund
On 2014-06-05 19:13:34 +0900, furu...@pm.nttdata.co.jp wrote: -Original Message- Hi, On 2014-06-05 17:09:44 +0900, furu...@pm.nttdata.co.jp wrote: Synchronous(synchronous_commit = on) mode offers the ability to confirm WAL have been streamed in the same way as synchronous

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-06-26 Thread furuyao
The patch looks somewhat complicated and bugs can be easily introduced because it tries to not only add new feature but also reorganize the main loop in HandleCopyStream at the same time. To keep the patch simple, I'm thinking to firstly apply the attached patch which just refactors the main

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-06-25 Thread Fujii Masao
On Wed, Jun 25, 2014 at 3:50 AM, Fujii Masao masao.fu...@gmail.com wrote: On Tue, Jun 24, 2014 at 3:18 PM, furu...@pm.nttdata.co.jp wrote: I found that this patch breaks --status-interval option of pg_receivexlog when -m option which the patch introduced is supplied. When -m is set,

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-06-24 Thread furuyao
I found that this patch breaks --status-interval option of pg_receivexlog when -m option which the patch introduced is supplied. When -m is set, pg_receivexlog tries to send the feedback message as soon as it flushes WAL file even if status interval timeout has not been passed yet. If you

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-06-24 Thread Fujii Masao
On Tue, Jun 24, 2014 at 3:18 PM, furu...@pm.nttdata.co.jp wrote: I found that this patch breaks --status-interval option of pg_receivexlog when -m option which the patch introduced is supplied. When -m is set, pg_receivexlog tries to send the feedback message as soon as it flushes WAL file

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-06-19 Thread Fujii Masao
On Mon, Jun 16, 2014 at 7:03 PM, furu...@pm.nttdata.co.jp wrote: You introduced the state machine using the flag flush_flg into pg_receivexlog. That's complicated and would reduce the readability of the source code. I think that the logic should be simpler like walreceiver's one. Maybe I

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-06-16 Thread furuyao
You introduced the state machine using the flag flush_flg into pg_receivexlog. That's complicated and would reduce the readability of the source code. I think that the logic should be simpler like walreceiver's one. Maybe I found one problematic path as follows: 1. WAL is written and

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-06-11 Thread Fujii Masao
On Tue, Jun 10, 2014 at 5:01 PM, furu...@pm.nttdata.co.jp wrote: No. IIUC walreceiver does flush *less* frequently than what you implemented on pg_receivexlog. Your version of pg_receivexlog tries to do flush every time when it receives one WAL chunk. OTOH, walreceiver does flush only when

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-06-10 Thread furuyao
No. IIUC walreceiver does flush *less* frequently than what you implemented on pg_receivexlog. Your version of pg_receivexlog tries to do flush every time when it receives one WAL chunk. OTOH, walreceiver does flush only when there is no extra WAL chunk in receive buffer. IOW, after writing

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-06-06 Thread furuyao
-Original Message- Flush is not performed every time write, it is performed collectively like walrecever. I only glanced at this, but afaics you're only flushing at the end every WAL segment. That will result in absolutely horrible performance, right? Walreceiver does flush

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-06-06 Thread Fujii Masao
On Fri, Jun 6, 2014 at 8:05 PM, furu...@pm.nttdata.co.jp wrote: -Original Message- Flush is not performed every time write, it is performed collectively like walrecever. I only glanced at this, but afaics you're only flushing at the end every WAL segment. That will result in

[HACKERS] pg_receivexlog add synchronous mode

2014-06-05 Thread furuyao
Hi, This patch implements a pg_receivexlog add synchronous mode. Now, synchronous(synchronous_commit = remote_write) is supported. But synchronous(synchronous_commit = remote_write), if the server crashes then WAL file may not to be flushed to disk , causing data loss.

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-06-05 Thread Andres Freund
Hi, On 2014-06-05 17:09:44 +0900, furu...@pm.nttdata.co.jp wrote: Synchronous(synchronous_commit = on) mode offers the ability to confirm WAL have been streamed in the same way as synchronous replication. If an output is used as a different disk from the directory where the transaction log

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-06-05 Thread furuyao
-Original Message- Hi, On 2014-06-05 17:09:44 +0900, furu...@pm.nttdata.co.jp wrote: Synchronous(synchronous_commit = on) mode offers the ability to confirm WAL have been streamed in the same way as synchronous replication. If an output is used as a different disk from the