[HACKERS] pg_receivexlog add synchronous mode

2014-06-05 Thread furuyao
Hi, This patch implements a pg_receivexlog add synchronous mode. Now, synchronous(synchronous_commit = remote_write) is supported. But synchronous(synchronous_commit = remote_write), if the server crashes then WAL file may not to be flushed to disk , causing data loss.

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-06-05 Thread furuyao
-Original Message- Hi, On 2014-06-05 17:09:44 +0900, furu...@pm.nttdata.co.jp wrote: Synchronous(synchronous_commit = on) mode offers the ability to confirm WAL have been streamed in the same way as synchronous replication. If an output is used as a different disk from the

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-06-06 Thread furuyao
-Original Message- Flush is not performed every time write, it is performed collectively like walrecever. I only glanced at this, but afaics you're only flushing at the end every WAL segment. That will result in absolutely horrible performance, right? Walreceiver does flush

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-06-10 Thread furuyao
No. IIUC walreceiver does flush *less* frequently than what you implemented on pg_receivexlog. Your version of pg_receivexlog tries to do flush every time when it receives one WAL chunk. OTOH, walreceiver does flush only when there is no extra WAL chunk in receive buffer. IOW, after writing

Re: [HACKERS] pg_xlogdump --stats

2014-06-10 Thread furuyao
-Original Message- From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Abhijit Menon-Sen Sent: Wednesday, June 04, 2014 7:47 PM To: pgsql-hackers@postgresql.org Subject: [HACKERS] pg_xlogdump --stats Hi. Here's a patch to make

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-06-16 Thread furuyao
You introduced the state machine using the flag flush_flg into pg_receivexlog. That's complicated and would reduce the readability of the source code. I think that the logic should be simpler like walreceiver's one. Maybe I found one problematic path as follows: 1. WAL is written and

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-06-24 Thread furuyao
I found that this patch breaks --status-interval option of pg_receivexlog when -m option which the patch introduced is supplied. When -m is set, pg_receivexlog tries to send the feedback message as soon as it flushes WAL file even if status interval timeout has not been passed yet. If you

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-06-26 Thread furuyao
The patch looks somewhat complicated and bugs can be easily introduced because it tries to not only add new feature but also reorganize the main loop in HandleCopyStream at the same time. To keep the patch simple, I'm thinking to firstly apply the attached patch which just refactors the main

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-06-30 Thread furuyao
Thanks for the review! +if (secs = 0) +secs = 1;/* Always sleep at least 1 sec */ + +sleeptime = secs * 1000 + usecs / 1000; The above is the code which caused that problem. 'usecs' should have been reset to zero when 'secs' are rounded up to

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-06-30 Thread furuyao
Synchronous(synchronous_commit = on) mode offers the ability to confirm WAL have been streamed in the same way as synchronous replication. If an output is used as a different disk from the directory where the transaction log should be stored. Prevent the loss of data due

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-07-04 Thread furuyao
Thanks for reviewing the patch! I think that this refactoring patch is useful for improving source code readability and making the future patches simpler, whether we adopt your patch or not. So, barring any objections, I'm thinking to commit this refactoring patch. Committed! It is

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-07-10 Thread furuyao
This patch is modified the comment. Each comment is coping as follows. Could you update the status of this patch from Waiting on Author to Needs Review when you post the revised version of the patch? Thank you for pointing out. From now on, I will update status when I post the patch. +

Re: [HACKERS] timeout of pg_receivexlog --status-interval

2014-07-23 Thread furuyao
Hi, At 1047 line of receivelog.c:CopyStreamPoll(), we set NULL to timeoutptr variable. if the value of timeoutprt is set NULL then the process will wait until can read socket using by select() function as following. if (timeout_ms 0) timeoutptr = NULL; else

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-07-25 Thread furuyao
This patch was made by the following process. 1. post patch for additional pg_receivexlog synchronous mode. 2. In response to comment for the flush frequency, I revise the patch to do flush every consecutive message in reference to walreceiver. 3. The synchronization mode was necessary to

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-07-29 Thread furuyao
I have improved the patch by making following changes: 1. Since stream_stop() was redundant, stream_stop() at the time of WAL file closing was deleted. 2. Change the Flash judging timing for the readability of source code. I have changed the Flash judging timing , from the continuous

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-08-05 Thread furuyao
I have improved the patch by making following changes: 1. Since stream_stop() was redundant, stream_stop() at the time of WAL file closing was deleted. 2. Change the Flash judging timing for the readability of source code. I have changed the Flash judging timing , from the

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-08-06 Thread furuyao
- break; /* ignore the rest of this XLogData packet */ + return true;/* ignore the rest of this XLogData packet */ For break statement at close of wal file, it is a return to true. It may be a behavior

Re: [HACKERS] pg_receivexlog add synchronous mode

2014-08-07 Thread furuyao
Okay, applied the patch. I heavily modified your patch based on the master that the refactoring patch has been applied. Attached is that modified version. Could you review that? Thank you for the patch. I did a review of the patch. No problem in the patch. Behavior after the true return

[HACKERS] pg_receivexlog --status-interval add fsync feedback

2014-08-12 Thread furuyao
Hi all, This patch is to add setting to send status packets after fsync to --status-interval of pg_receivexlog. If -1 is specified to --status-interval, status packets is sent as soon as after fsync. Others are the same as when 0 is specified to --status-interval. When requested by the

Re: [HACKERS] pg_receivexlog --status-interval add fsync feedback

2014-08-13 Thread furuyao
I don't think that it's good idea to control that behavior by using --status-interval. I'm sure that there are some users who both want that behavior and want set the maximum interval between a feedback is sent back to the server because these settings are available in walreceiver. But your

Re: [HACKERS] pg_receivexlog and replication slots

2014-08-15 Thread furuyao
Actually I came up with the same need as Magnus, but a bit later, so... Attached is a patch to support physical slot creation and drop in pg_receivexlog with the addition of new options --create and --drop. It would be nice to have that in 9.4, but I will not the one deciding that at the end

Re: [HACKERS] pg_receivexlog --status-interval add fsync feedback

2014-08-18 Thread furuyao
Thanks for the review! One question is why reply_fsync is defined as volatile variable? Sorry I could not understand reason of that. It was affected to time_to_abort -- since it is unnecessary, it deletes. Currently patch modifies argument of some function (e.g., Handle CopyStream, Process

Re: [HACKERS] pg_receivexlog --status-interval add fsync feedback

2014-08-18 Thread furuyao
Thank you for updating the patch. I did not get error with applying, and compiling. It works fine. I think this function code has no problem. Could you please submit patch to commit fest app? Thanks for the review! As you pointed out, submitted patch to commit fest app. Regards, --

Re: [HACKERS] pg_receivexlog --status-interval add fsync feedback

2014-08-20 Thread furuyao
When replication slot is not specified in pg_receivexlog, the flush location in the feedback message always indicates invalid. So there seems to be no need to send the feedback as soon as fsync is issued, in that case. How should this option work when replication slot is not specified?

Re: [HACKERS] pg_receivexlog --status-interval add fsync feedback

2014-08-21 Thread furuyao
Thank you for updating the patch. I reviewed the patch. First of all, I think that we should not append the above message to section of '-r' option. (Or these message might not be needed at all) Whether flush location in feedback message is valid, is not depend on '-r' option. If we

Re: [HACKERS] pg_receivexlog --status-interval add fsync feedback

2014-09-04 Thread furuyao
Thanks for the review! I understand the attention message wasn't appropriate. To report the write location, even If you do not specify a replication slot. So the fix only appended messages. There was a description of the flush location section of '-S' option, but I intended to

Re: [HACKERS] pg_receivexlog --status-interval add fsync feedback

2014-09-29 Thread furuyao
On 09/05/2014 08:51 AM, furu...@pm.nttdata.co.jp wrote: Thanks for the review! I understand the attention message wasn't appropriate. To report the write location, even If you do not specify a replication slot. So the fix only appended messages. There was a description of the

Re: [HACKERS] pg_receivexlog --status-interval add fsync feedback

2014-10-07 Thread furuyao
On 09/29/2014 01:13 PM, furu...@pm.nttdata.co.jp wrote: I don't understand what this patch does. When would you want to use the new --reply-fsync option? Is there any reason *not* to use it? In other words, do we need an option for this, couldn't you just always send the feedback message

Re: [HACKERS] pg_receivexlog --status-interval add fsync feedback

2014-10-08 Thread furuyao
On 10/08/2014 07:23 AM, furu...@pm.nttdata.co.jp wrote: What set of options would you pass if you want to use it as a synchronous standby? And if you don't? Could we just have a single --synchronous flag, instead of -F and --reply-fsync? If you set synchronous_commit as remote_write,

Re: [HACKERS] pg_receivexlog --status-interval add fsync feedback

2014-10-08 Thread furuyao
What set of options would you pass if you want to use it as a synchronous standby? And if you don't? Could we just have a single --synchronous flag, instead of -F and --reply-fsync? If you set synchronous_commit as remote_write, the options would be different . The set of options

Re: [HACKERS] pg_receivexlog --status-interval add fsync feedback

2014-10-09 Thread furuyao
If we remove --fsync-interval, resoponse time to user will not be delay. Although, fsync will be executed multiple times in a short period. And there is no way to solve the problem without --fsync-interval, what should we do about it? I'm sorry, I didn't understand that. Here is

Re: [HACKERS] pg_receivexlog --status-interval add fsync feedback

2014-10-14 Thread furuyao
If we remove --fsync-interval, resoponse time to user will not be delay. Although, fsync will be executed multiple times in a short period. And there is no way to solve the problem without --fsync-interval, what should we do about it? I'm sorry, I didn't understand that.

Re: [HACKERS] pg_receivexlog --status-interval add fsync feedback

2014-10-17 Thread furuyao
In synchronous mode, pg_receivexlog should have similar logic as walreceiver does. OK. I understand that removing --fsync-interval has no problem. +1 for adding something like --synchronous option. To me, it sounds walreceiver-compatible mode rather than synchronous. Better to add

Re: [HACKERS] pg_receivexlog --status-interval add fsync feedback

2014-10-24 Thread furuyao
Sorry, I'm going around in the circle. But I'd like to say again, I don't think this is good idea. It prevents asynchronous pg_receivexlog from fsyncing WAL data and sending feedbacks more frequently at all. They are useful, for example, when we want to monitor the write location of

Re: [HACKERS] pg_receivexlog --status-interval add fsync feedback

2014-10-31 Thread furuyao
We seem to be going in circles. You suggested having two options, --feedback, and --fsync, which is almost exactly what Furuya posted originally. I objected to that, because I think that user interface is too complicated. Instead, I suggested having just a single option called

Re: [HACKERS] inherit support for foreign tables

2014-11-06 Thread furuyao
(2014/08/28 18:00), Etsuro Fujita wrote: (2014/08/22 11:51), Noah Misch wrote: Today's ANALYZE VERBOSE messaging for former inheritance parents (tables with relhassubclass = true but no pg_inherits.inhparent links) is deceptive, and I welcome a fix to omit the spurious message. As

Re: [HACKERS] pg_receivexlog --status-interval add fsync feedback

2014-11-10 Thread furuyao
On Fri, Oct 31, 2014 at 5:46 PM, furu...@pm.nttdata.co.jp wrote: We seem to be going in circles. You suggested having two options, --feedback, and --fsync, which is almost exactly what Furuya posted originally. I objected to that, because I think that user interface is too

[HACKERS] Customized Options Threshold Error

2014-11-14 Thread furuyao
Hi, When we specify a value which exceeds valid range in Customized Options , its behavior is different from Parameter Interaction via Configuration File behavior. In case of Parameter Interaction via Configuration File, it finish with FATAL error when it get threshold error. But in Customized

[HACKERS] PostgreSQL doesn't stop propley when --slot option is specified with pg_receivexlog.

2014-11-14 Thread furuyao
Hi, pg_ctl stop does't work propley, if --slot option is specified when WAL is flushed only it has switched. These processes still continue even after the posmaster failed:pg_receivexlog, walsender and logger. How to reproduce: 1.Start PostgreSQL 2.Create slot 3.Specify --slot option to

Re: [HACKERS] Customized Options Threshold Error

2014-11-19 Thread furuyao
When we specify a value which exceeds valid range in Customized Options , its behavior is different from Parameter Interaction via Configuration File behavior. In case of Parameter Interaction via Configuration File, it finish with FATAL error when it get threshold error. But in