RE: Allow logical replication to copy tables in binary format

2023-03-19 Thread Hayato Kuroda (Fujitsu)
options = lappend(options, makeDefElem("format", (Node *) makeString("binary"), -1)); ``` Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: [Proposal] Add foreign-server health checks infrastructure

2023-03-13 Thread Hayato Kuroda (Fujitsu)
Dear Katsuragi-san, > Hi, > > I updated the status of the patch to ready for committer. > > regards, Thank you so much for your reviewing! Now we can wait comments from senior members and committers. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-03-10 Thread Hayato Kuroda (Fujitsu)
/message-id/1339586927-13156-12-git-send-email-andres%402ndquadrant.com Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Rework LogicalOutputPluginWriterUpdateProgress

2023-03-08 Thread Hayato Kuroda (Fujitsu)
efs.list. --- Do we have to write a document for the breakage somewhere? I think we do not have to add appendix-obsolete-* file because we did not have any links for that, but we can add a warning in "Functions for Producing Output" subsection if needed. Best Regards, Hayato Kuroda FUJITSU LIMITED test.sh Description: test.sh

RE: Allow logical replication to copy tables in binary format

2023-03-07 Thread Hayato Kuroda (Fujitsu)
as changed and worker will restart..." or something, they can turn "binary" on again. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: [Proposal] Add foreign-server health checks infrastructure

2023-03-07 Thread Hayato Kuroda (Fujitsu)
f a > + * timeout occurred, -1 if an error or interrupt occurred. Note that if 0 is > + * returned and forConnCheck is requested, it means that the socket has not > + * matched POLLRDHUP event and the connection is not closed by the socket > peer. Fixed. [1]: https://www.postgresq

RE: Rework LogicalOutputPluginWriterUpdateProgress

2023-03-06 Thread Hayato Kuroda (Fujitsu)
u have any reasons why only they have? [1]: https://www.postgresql.org/message-id/CAHut%2BPsksiQHuv4A54R4w79TAvCu__PcuffKYY0V96e2z_sEvA%40mail.gmail.com Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: [Proposal] Add foreign-server health checks infrastructure

2023-03-06 Thread Hayato Kuroda (Fujitsu)
emits warnings if a disconnection is found. This > returns true > + * if disconnections cannot be found, otherwise returns false. > > I think false is returned only if disconnections are found and > true is returned in all other cases. So, modifying the description > like the

RE: [Proposal] Add foreign-server health checks infrastructure

2023-03-06 Thread Hayato Kuroda (Fujitsu)
er. Yes, but completely same statement cannot be used because these is not specified foreign server. How about: NULL is returned if there are no established connections or this function ... > 8. the document of postgres_fdw_can_verify_connection_states > + This function checks whether

RE: [Proposal] Add foreign-server health checks infrastructure

2023-03-03 Thread Hayato Kuroda (Fujitsu)
ected"? Changed. > 9. the document of postgres_fdw > The document of postgres_fdw_verify_connection_states_* is a little > bit old. Could you update it? Updated. IIUC postgres_fdw_verify_connection_states returns * true, if the connection is verified. * false, if th

RE: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2023-03-03 Thread Hayato Kuroda (Fujitsu)
pleByIndex() if idxIsRelationIdentityOrPK is false. 032_subscribe_use_index.pl ``` +# Testcase start: SUBSCRIPTION CREATE/DROP INDEX WORKS WITHOUT ISSUES ... +# Testcase end: SUBSCRIPTION RE-CALCULATES INDEX AFTER CREATE/DROP INDEX ``` There is still non-consistent case. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-03-02 Thread Hayato Kuroda (Fujitsu)
should be emphasized. Based on the understanding, I added them to the doc in new version patch. Please see [1]. [1]: https://www.postgresql.org/message-id/flat/TYAPR01MB586606CF3B585B6F8BE13A9CF5B29%40TYAPR01MB5866.jpnprd01.prod.outlook.com Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-03-02 Thread Hayato Kuroda (Fujitsu)
better, so changed. Please see new patch at [1] [1]: https://www.postgresql.org/message-id/flat/TYAPR01MB586606CF3B585B6F8BE13A9CF5B29%40TYAPR01MB5866.jpnprd01.prod.outlook.com Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-03-02 Thread Hayato Kuroda (Fujitsu)
he patch and I thought there were no similar one. > 7) In the expect we have specifically mention "for non-streaming > transaction", is the behavior different for streaming transaction, if > not we can change the message accordingly > +# The publisher waits for the replication to complete > +$node_publisher->wait_for_catchup('tap_sub_renamed'); > + > +# This test is successful only if at least the configured delay has elapsed. > +ok( time() - $publisher_insert_time >= $delay, > + "subscriber applies changes only after replication delay for > non-streaming transaction" > +); There is no difference, both of normal and streamed transaction could be delayed to apply. So removed. [1]: https://www.postgresql.org/message-id/flat/TYAPR01MB586606CF3B585B6F8BE13A9CF5B29%40TYAPR01MB5866.jpnprd01.prod.outlook.com Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-03-02 Thread Hayato Kuroda (Fujitsu)
32 min_send_delay; /* The minimum send delay */ > } logical; > } proto; > } WalRcvStreamOptions; > > ~ > > Should that comment mention the units are "(ms)" Added. Best Regards, Hayato Kuroda FUJITSU LIMITED v10-0001-Time-delayed-logical-replication-on-publisher-si.patch Description: v10-0001-Time-delayed-logical-replication-on-publisher-si.patch

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-03-01 Thread Hayato Kuroda (Fujitsu)
to no delay, it is > expected from users using this have an understanding of the trade-off. Yes, the trade-off should be emphasized. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Allow logical replication to copy tables in binary format

2023-03-01 Thread Hayato Kuroda (Fujitsu)
%40mail.gmail.com [3]: https://www.postgresql.org/message-id/TYAPR01MB5866968CF42FBAB73E8EEDF8F5AA9%40TYAPR01MB5866.jpnprd01.prod.outlook.com Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-03-01 Thread Hayato Kuroda (Fujitsu)
send the parameter at START_REPLICATION and the slot has been already created. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-28 Thread Hayato Kuroda (Fujitsu)
and I thought all of them should be included. Moreover, I used grammar checker and slightly reworded the commit message. [1]: https://man7.org/linux/man-pages/man3/send.3p.html Best Regards, Hayato Kuroda FUJITSU LIMITED v9-0001-Time-delayed-logical-replication-on-publisher-sid.patch Description: v9-0001-Time-delayed-logical-replication-on-publisher-sid.patch

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-27 Thread Hayato Kuroda (Fujitsu)
next version I will use grammar checker like Chat-GPT to modify commit messages... [1]: https://www.postgresql.org/message-id/CAA4eK1LjOm6-OHggYVH35dQ_v40jOXrJW0GFy3kuwTd2J48%3DUg%40mail.gmail.com [2]: https://www.postgresql.org/message-id/CAA4eK1K4uPbudrNdH%2B%3D_vN-Hpe9wYh%3D3vBS5Ww9dHn-LOWM

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-26 Thread Hayato Kuroda (Fujitsu)
e process spends time till a given period. Such a function is painful to read later. I think callbacks that have different purposes should not be mixed. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-23 Thread Hayato Kuroda (Fujitsu)
st way. TransactionId was added as an argument of functions. Best Regards, Hayato Kuroda FUJITSU LIMITED v7-0001-Time-delayed-logical-replication-on-publisher-sid.patch Description: v7-0001-Time-delayed-logical-replication-on-publisher-sid.patch

RE: Rework LogicalOutputPluginWriterUpdateProgress

2023-02-23 Thread Hayato Kuroda (Fujitsu)
needed. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Allow logical replication to copy tables in binary format

2023-02-22 Thread Hayato Kuroda (Fujitsu)
copy_data = false" should be rejected in parse_subscription_options() and AlterSubscription(). Is it right? I'm expecting that is done in next version. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-22 Thread Hayato Kuroda (Fujitsu)
table status updated in linkend="catalog-pg-subscription-rel">pg_subscription_rel ctname> > could be delayed in getting to the "ready" state, and also two-phase > (if specified) could be delayed in getting to "enabled"."? Oh, my Visual Studio Code did not dete

RE: [Proposal] Add foreign-server health checks infrastructure

2023-02-22 Thread Hayato Kuroda (Fujitsu)
(). It may be reasonable because the checking is not really done on this environment. How do you think? [1]: https://www.postgresql.org/message-id/TYAPR01MB586664F5ED2128E572EA95B0F5AA9%40TYAPR01MB5866.jpnprd01.prod.outlook.com Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: [Proposal] Add foreign-server health checks infrastructure

2023-02-22 Thread Hayato Kuroda (Fujitsu)
etPoll > > + * When neither forRead nor forWrite are set and timeout is disabled, > + * > + * - If the timeout is disabled, try to check the health of the socket > + * - Otherwise this immediately returns 0 > + * > + * Return >0 if condition is met, 0 if a timeout occurred

RE: Allow logical replication to copy tables in binary format

2023-02-21 Thread Hayato Kuroda (Fujitsu)
replication between different versions may be failed if binary format is specified. Therefore, I think we should accept copy_format = binary only when the major version of servers are the same. Note that this comments is not the request to the patch. Maybe the modification should be done not only for copy_format but also binary, and it may be out of scope the patch. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-21 Thread Hayato Kuroda (Fujitsu)
e. That's why I wrote "Sleep until we get reply from worker or we time out". > src/include/replication/logical.h > > 6. > + /* > + * The minimum delay, in milliseconds, by the publisher before sending > + * COMMIT/PREPARE record > + */ > + int32 min_send_delay; > > The comment is missing a period. Right, added. Best Regards, Hayato Kuroda FUJITSU LIMITED v5-0001-Time-delayed-logical-replication-on-publisher-sid.patch Description: v5-0001-Time-delayed-logical-replication-on-publisher-sid.patch

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-21 Thread Hayato Kuroda (Fujitsu)
> first place? > > > > I don't see the need to change the datatype of min_send_delay as > compared to what we have min_apply_delay. I think it is OK to change "long" to "unsinged long", because We use strtoul() for reading and should reject the minus value. Of course we can modify them, but I want to keep the consistency with proto_version part. [1]: https://www.postgresql.org/message-id/tyapr01mb5866c6bca4d9386d9c486033f5...@tyapr01mb5866.jpnprd01.prod.outlook.com Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-20 Thread Hayato Kuroda (Fujitsu)
saction" > +); > > It's not strictly an "apply delay". Maybe this comment only needs to > say like below: > > SUGGESTION > # This test is successful only if at least the configured delay has elapsed. Changed. New patch is available on [1]. [1]: https://www.postgresql.org/message-id/TYAPR01MB5866C6BCA4D9386D9C486033F5A59%40TYAPR01MB5866.jpnprd01.prod.outlook.com Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-20 Thread Hayato Kuroda (Fujitsu)
;... AFAIK > here we are just sleeping, not waiting for replies from anywhere (???) > > == > src/include/replication/logical.h > > 15. > @@ -64,6 +68,7 @@ typedef struct LogicalDecodingContext > LogicalOutputPluginWriterPrepareWrite prepare_write; > LogicalOutputPluginWriterWrite write; > LogicalOutputPluginWriterUpdateProgress update_progress; > + LogicalOutputPluginWriterDelay delay; > > ~ > > 15a. > Question: Is there some advantage to introducing another callback, > instead of just doing the delay inline? IIUC functions related with walsender should not be called directly, because there is a possibility that replication slots are manipulated from the backed. > 15b. > Should this be a more informative member name like 'delay_send'? Changed. > 16. > @@ -100,6 +105,8 @@ typedef struct LogicalDecodingContext > */ > bool twophase_opt_given; > > + int32 min_send_delay; > + > > Missing comment for this new member. Added. [1]: https://www.postgresql.org/message-id/caa4ek1+jwlavaophnz1ytiev_jomae6jgjmbe98oek2cg7x...@mail.gmail.com Best Regards, Hayato Kuroda FUJITSU LIMITED v4-0001-Time-delayed-logical-replication-on-publisher-sid.patch Description: v4-0001-Time-delayed-logical-replication-on-publisher-sid.patch

RE: Allow logical replication to copy tables in binary format

2023-02-19 Thread Hayato Kuroda (Fujitsu)
-cfbot/postgresql/commitfest/42/3840 Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: [Proposal] Add foreign-server health checks infrastructure

2023-02-19 Thread Hayato Kuroda (Fujitsu)
s 0 */ > Assert(PQcanConnCheck() && end_time == 0); > #if defined(POLLRDHUP) > input_fd.events = POLLRDHUP | POLLHUP | POLLNVAL; > #endif > } Fixed. > 8. PQconnCheck > +/* > + * Check whether PQconnCheck() can work well on this platform. > + * > + * Returns 1 if this can use PQconnCheck(), otherwise 0. > + */ > +int > +PQcanConnCheck(void) > +{ > +#if (defined(HAVE_POLL) && defined(POLLRDHUP)) > + return true; > +#else > + return false; > +#endif > +} > > ~ > > 8a. > "can work well" --> "works" > > ~ > > 8b. > Maybe better to say "true (1)" and "otherwise false (0)" Fixed. [1]: https://www.postgresql.org/message-id/TYAPR01MB58669C95604A0EB7BCC1B58CF5A49%40TYAPR01MB5866.jpnprd01.prod.outlook.com Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: [Proposal] Add foreign-server health checks infrastructure

2023-02-19 Thread Hayato Kuroda (Fujitsu)
t; I'm wondering if we should add kqueue support in this version, > because adding kqueue support introduces additional things to > be considered. What do you think about focusing on the main > functionality using poll in this patch and going for kqueue > support after this patch? I think it

RE: Is psSocketPoll doing the right thing?

2023-02-19 Thread Hayato Kuroda (Fujitsu)
o simply check the socket is sound without doing an > actual read/write, that behavior starts giving a harm by the false > answer. I checked man page of poll(3), and it said that POLLERR, POLLHUP, POLLNVAL is only valid in revents. Moreover, poll() has is clarified that it returns natural number if revent is larger than zero. So revent should be checked even if the returned value > 0. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-19 Thread Hayato Kuroda (Fujitsu)
mmit_prepared. Right, the transaction will be end when it receive PREPARE. Fixed. I've tested locally and the delay seemed to be occurred at PREPARE phase. > 8. > +# > +# If the subscription sets min_send_delay parameter, the logical replication > +# worker will delay the transaction a

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-16 Thread Hayato Kuroda (Fujitsu)
d directly from decode.c. So I implemented as callback in LogicalDecodingContext and it is called from decode.c if set. New patch was posted in [1]. [1]: https://www.postgresql.org/message-id/TYAPR01MB5866F00191375D0193320A4DF5A19%40TYAPR01MB5866.jpnprd01.prod.outlook.com Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-16 Thread Hayato Kuroda (Fujitsu)
ate_progress_txn() for this purpose but note > that we are discussing to change the same in thread [1]. > > [1] - > https://www.postgresql.org/message-id/20230210210423.r26ndnfmuifie4f6%40 > awork3.anarazel.de I did not reuse update_progress_txn() because we cannot use it straightforward, But I can change if we have better idea than present. New patch was posted in [1]. [1]: https://www.postgresql.org/message-id/TYAPR01MB5866F00191375D0193320A4DF5A19%40TYAPR01MB5866.jpnprd01.prod.outlook.com Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-16 Thread Hayato Kuroda (Fujitsu)
nt. When I read > through the last patch, I missed that WalSndDelay is actually a subset > of WalSndLoop. Although it can handle keep-alives correctly, I'm not > sure we can accept that structure.. No issues. I have kept the current implementation. Some bugs I found are also fixed. Best Regards, Hayato Kuroda FUJITSU LIMITED v2-0001-Time-delayed-logical-replication-on-publisher-sid.patch Description: v2-0001-Time-delayed-logical-replication-on-publisher-sid.patch

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-15 Thread Hayato Kuroda (Fujitsu)
I think in the latest patch, subscriber can respond to the keepalive packets from the peer. Also, publisher can respond to the peer. Could you please tell me if you know a case that publisher or subscriber cannot respond to the opposite side? Note that if we apply the publisher-side patch, we do

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-15 Thread Hayato Kuroda (Fujitsu)
in layer, the new LogicalOutputPlugin API was added. For now, I choose the output plugin layer but can consider to do it from the core if there is a better way. Could you please share your opinion? Note: thanks for Osumi-san to help implementing. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Exit walsender before confirming remote flush in logical replication

2023-02-10 Thread Hayato Kuroda (Fujitsu)
I want to ask other reviewers which one is better... [1]: https://www.postgresql.org/message-id/20230208.112717.1140830361804418505.horikyota.ntt%40gmail.com Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: [Proposal] Add foreign-server health checks infrastructure

2023-02-09 Thread Hayato Kuroda (Fujitsu)
ion_states' looks straightforward. > What do you think about this? Added. > 0004: > Sorry, I have not read 0004. I will read. No problem:-). [1]: https://www.postgresql.org/message-id/TYAPR01MB58664E039F45959AB321FA1FF5D99%40TYAPR01MB5866.jpnprd01.prod.outlook.com Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: [Proposal] Add foreign-server health checks infrastructure

2023-02-09 Thread Hayato Kuroda (Fujitsu)
Dear Horiguchi-san, Thank you for checking! The patch will be attached to another mail. > At Fri, 27 Jan 2023 06:57:01 +, "Hayato Kuroda (Fujitsu)" > wrote in > > I found cfbot failure, PSA fixed version. > > + Unlike , this function checks socket

RE: Exit walsender before confirming remote flush in logical replication

2023-02-09 Thread Hayato Kuroda (Fujitsu)
> parameters. How about the message like: > > ERROR: invalid value shutdown mode: "%s" Modified like enum parameters and hint message was also provided. New patch is attached in [1]. [1]: https://www.postgresql.org/message-id/TYAPR01MB586683FC450662990E356A0EF5D99%40TYAPR01MB5866.jpnprd01.prod.outlook.com Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Exit walsender before confirming remote flush in logical replication

2023-02-09 Thread Hayato Kuroda (Fujitsu)
obably, we can partly change this sentence like below, because we check > walsender's pid. > FROM: "... while waiting for apply to restart..." > TO: "... while waiting for the walsender to restart..." Right, fixed. Best Regards, Hayato Kuroda FUJITSU LIMITED v6-

RE: Exit walsender before confirming remote flush in logical replication

2023-02-08 Thread Hayato Kuroda (Fujitsu)
> > Dear Horiguchi-san, > > Thank you for checking the patch! PSA new version. PSA rebased patch that supports updated time-delayed patch[1]. [1]: https://www.postgresql.org/message-id/tyapr01mb5866c11daf8ab04f3cc181d3f5...@tyapr01mb5866.jpnprd01.prod.outlook.com Best Regards, H

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-08 Thread Hayato Kuroda (Fujitsu)
+# current timestamp that will be used as a comparison base. Even on slow > +# machines, this allows to have a predictable behavior when comparing the > +# delay between data insertion moment on publisher and replay time on > subscriber. > > Most of this comment is now redundant beca

RE: Exit walsender before confirming remote flush in logical replication

2023-02-08 Thread Hayato Kuroda (Fujitsu)
t; > /* minapplydelay affects START_REPLICATION option exit_before_confirming > */ Added just above the condition. > + * Reads all entrly of the list and consume if needed. > s/entrly/entries/ ? > ... This part is no longer needed. [1]: https://www.postgresql.org/message-id/TY

RE: Exit walsender before confirming remote flush in logical replication

2023-02-07 Thread Hayato Kuroda (Fujitsu)
ation once and then try to extend if we want. TBH I think adding physicalrep support seems not to be so hard, but I want to keep the patch smaller. The new patch will be attached soon in another mail. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Exit walsender before confirming remote flush in logical replication

2023-02-07 Thread Hayato Kuroda (Fujitsu)
se. Best Regards, Hayato Kuroda FUJITSU LIMITED v3-0001-Time-delayed-logical-replication-subscriber.patch Description: v3-0001-Time-delayed-logical-replication-subscriber.patch v3-0002-Extend-START_REPLICATION-command-to-accept-walsen.patch Description: v3-0002-Extend-START_REPLICATION-command-

RE: Exit walsender before confirming remote flush in logical replication

2023-02-07 Thread Hayato Kuroda (Fujitsu)
removed version number because the approach is completely changed. > > [1]: > https://www.postgresql.org/message-id/TYCPR01MB8373BA483A6D2C924C60 > 0968ed...@tycpr01mb8373.jpnprd01.prod.outlook.com I noticed that previous ones are rejected by cfbot, even if they passed on my envir

RE: Exit walsender before confirming remote flush in logical replication

2023-02-07 Thread Hayato Kuroda (Fujitsu)
ero to a non-zero value or non-zero to zero. Note that I removed version number because the approach is completely changed. [1]: https://www.postgresql.org/message-id/tycpr01mb8373ba483a6d2c924c600968ed...@tycpr01mb8373.jpnprd01.prod.outlook.com Best Regards, Hayato Kuroda FUJITSU LIMITED 0001-T

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-06 Thread Hayato Kuroda (Fujitsu)
gt; Since there are no translator considerations here why not write the > first error like: > > errmsg("invalid value for parameter \"min_apply_delay\": \"%s\"", > input_string) > > ~ > > 5b. > Since there are no translator considerat

RE: Perform streaming logical transactions by background workers and parallel apply

2023-02-06 Thread Hayato Kuroda (Fujitsu)
t adding new event. Thank you for your explanation. I think both of you said are reasonable. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Perform streaming logical transactions by background workers and parallel apply

2023-02-06 Thread Hayato Kuroda (Fujitsu)
y waits a shared message queue and it seems to be same as WAIT_EVENT_MQ_RECEIVE. Do you have a policy to reuse the event instead of adding a new event? Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Exit walsender before confirming remote flush in logical replication

2023-02-03 Thread Hayato Kuroda (Fujitsu)
if we are in smart shutdown mode * exit immediately if we are in fast shutdown mode Note that in both case, walsender does not wait the remote flush of WALs. For implementing that, I added new attribute to WalSndCtlData that indicates the shutdown status. Basically it is zero, but it will be changed by

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-30 Thread Hayato Kuroda (Fujitsu)
s://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=castoroides=2023-01-30%2012%3A00%3A07=configure#:~:text=checking%20alignment%20of%20int...%204 Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Perform streaming logical transactions by background workers and parallel apply

2023-01-30 Thread Hayato Kuroda (Fujitsu)
Dear Hou, Thank you for updating the patch! I checked your replies and new patch, and it seems good. Currently I have no comments Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Assertion failure in SnapBuildInitialSnapshot()

2023-01-28 Thread Hayato Kuroda (Fujitsu)
;effective_xmin is not updated. It is set as InvalidTransactionId at ReplicationSlotCreate() and no functions update that. Hence the slot acquired by the walsender may have Invalid effective_min. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Assertion failure in SnapBuildInitialSnapshot()

2023-01-27 Thread Hayato Kuroda (Fujitsu)
that uses ReplicationSlotsComputeRequiredXmin(true) and it acquires exclusive lock correctly, but it can avoid future bug. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: [Proposal] Add foreign-server health checks infrastructure

2023-01-26 Thread Hayato Kuroda (Fujitsu)
I found cfbot failure, PSA fixed version. Sorry for noise. Best Regards, Hayato Kuroda FUJITSU LIMITED v29-0003-add-test.patch Description: v29-0003-add-test.patch v29-0004-add-kqueue-support-for-PQconnCheck-and-PQcanConn.patch Description: v29-0004-add-kqueue-support-for-PQconnCheck

RE: Exit walsender before confirming remote flush in logical replication

2023-01-26 Thread Hayato Kuroda (Fujitsu)
stop at 4. In step4, got_SIGUSR2 is never set to on, so we must use got_STOPPING flag. [1]: https://www.postgresql.org/message-id/tycpr01mb58701a47f35fed0a2b399662f5...@tycpr01mb5870.jpnprd01.prod.outlook.com Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: [Proposal] Add foreign-server health checks infrastructure

2023-01-26 Thread Hayato Kuroda (Fujitsu)
Best Regards, Hayato Kuroda FUJITSU LIMITED v28-0001-Add-PQconnCheck-and-PQcanConnCheck-to-libpq.patch Description: v28-0001-Add-PQconnCheck-and-PQcanConnCheck-to-libpq.patch v28-0002-postgres_fdw-add-postgres_fdw_verify_connection_.patch Description: v28-0002-postgres_fdw-add

RE: Perform streaming logical transactions by background workers and parallel apply

2023-01-25 Thread Hayato Kuroda (Fujitsu)
el apply worker replayed all changes from file"? Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: [Proposal] Add foreign-server health checks infrastructure

2023-01-25 Thread Hayato Kuroda (Fujitsu)
n't these PQconnCheck and PQcanConnCheck? I think the first letter > following 'PQ' should be lower case. I cannot find any rules about it, but seems right. Fixed. Best Regards, Hayato Kuroda FUJITSU LIMITED v27-0001-Add-PQconnCheck-and-PQcanConnCheck-to-libpq.patch Description: v27-0001-Add-PQconn

RE: Perform streaming logical transactions by background workers and parallel apply

2023-01-24 Thread Hayato Kuroda (Fujitsu)
Dear Amit, > > I have updated the patch accordingly and it looks good to me. I'll > push this first patch early next week (Monday) unless there are more > comments. Thanks for updating. I checked v88-0001 and I have no objection. LGTM. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Perform streaming logical transactions by background workers and parallel apply

2023-01-24 Thread Hayato Kuroda (Fujitsu)
red seems not to be executed. This is because serializations are finished while handling PREPARE message and the final state of transaction does not affect that, right? I think it may be helpful to add a one line comment. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-24 Thread Hayato Kuroda (Fujitsu)
IALIZED VIEW is created on publisher and then the WAL is replicated to subscriber, but the transaction is not committed because logical replication does not support the statement. If we change the condition, we may the system may become inconsistent because the worker replies that all remote WALs are applied even if tablesync workers are synchronizing data. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Perform streaming logical transactions by background workers and parallel apply

2023-01-23 Thread Hayato Kuroda (Fujitsu)
o used to test the message passing between the leader and parallel apply workers." 4. 015_stream.pl ``` +# Ensure that the messages are serialized. ``` In other parts "changes" are used instead of "messages". Can you change the word? Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: [Proposal] Add foreign-server health checks infrastructure

2023-01-22 Thread Hayato Kuroda (Fujitsu)
e returned value like true/false. But IIUC libpq functions cannot define as "bool" datatype. E.g. PQconnectionNeedsPassword() returns true/false, but the function is defined as int. Best Regards, Hayato Kuroda FUJITSU LIMITED v26-0001-Add-PQConnCheck-and-PQCanConnCheck-to-libpq.p

RE: [Proposal] Add foreign-server health checks infrastructure

2023-01-22 Thread Hayato Kuroda (Fujitsu)
> Thank you for reviewing! PSA new patch set. Sorry, I missed the updated file in the patch. New version will be posted soon. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: [Proposal] Add foreign-server health checks infrastructure

2023-01-21 Thread Hayato Kuroda (Fujitsu)
servers. This check is performed by polling the socket and allows > > It seems better to add a description that states this function > checks all the connections. Like the description of > postgres_fdw_disconnect_all(). For example, "This function > checks the status of 'all

RE: Exit walsender before confirming remote flush in logical replication

2023-01-19 Thread Hayato Kuroda (Fujitsu)
/../PostgreSQL-Source-Dev/src/backend/replication/logical/decode.c:119 #12 0x0087f8c7 in XLogSendLogical () at ../../PostgreSQL-Source-Dev/src/backend/replication/walsender.c:3060 #13 0x0087ec5a in WalSndLoop (send_data=0x87f825 ) at ../../PostgreSQL-Source-Dev/src/backend/replication/wa

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-17 Thread Hayato Kuroda (Fujitsu)
ion as that is executed before the transaction. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Exit walsender before confirming remote flush in logical replication

2023-01-16 Thread Hayato Kuroda (Fujitsu)
there is no pending write. This is done in ProcessPendingWrites(). I added another termination path in the function. [1]: https://github.com/postgres/postgres/commit/985bd7d49726c9f178558491d31a570d47340459 Best Regards, Hayato Kuroda FUJITSU LIMITED v4-0001-Exit-walsender-before-confirmin

RE: Exit walsender before confirming remote flush in logical replication

2023-01-16 Thread Hayato Kuroda (Fujitsu)
ted abnormally before or while processing the request. connection to server was lost ``` [1]: https://github.com/postgres/postgres/blob/master/src/backend/replication/walsender.c#L3121 Best Regards, Hayato Kuroda FUJITSU LIMITED test_phy.sh Description: test_phy.sh

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-11 Thread Hayato Kuroda (Fujitsu)
tch v14 in [1]. [1]: https://www.postgresql.org/message-id/TYAPR01MB5866D0527B1B8D589F1C2551F5FC9%40TYAPR01MB5866.jpnprd01.prod.outlook.com Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-11 Thread Hayato Kuroda (Fujitsu)
site sides, so the WARNING cannot be raised. I considered that such a mechanism seemed to be complex. The discussion around [1] may be useful. [1]: https://www.postgresql.org/message-id/CAA4eK1Lq%2Bh8qo%2BrqGU-E%2BhwJKAHYocV54y4pvou4rLysCgYD-g%40mail.gmail.com Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-11 Thread Hayato Kuroda (Fujitsu)
be allowed to stream > parallel" (or something better) Your point that error messages are strange is right. And while checking other ones, I found they have very similar styles. Therefore I reworded ERROR messages in AlterSubscription() and parse_subscription_options() to follow them. W

RE: [Proposal] Add foreign-server health checks infrastructure

2023-01-11 Thread Hayato Kuroda (Fujitsu)
Dear hackers, I was not sure, but the cfbot could not be accepted the previous version. I made the patch again from HEAD(5f6401) without any changes, so I did not count up the version number. Best Regards, Hayato Kuroda FUJITSU LIMITED v24-0001-Add-PQConncheck-and-PQCanConncheck

RE: [Proposal] Add foreign-server health checks infrastructure

2023-01-10 Thread Hayato Kuroda (Fujitsu)
thout doing unique checks. IIUC a backend process cannot connect to the same foreign server by using different user mapping, so there is no possibility that the same name appears twice. If the user mapping is altered in the transaction, the cache entry is invalidated and will not be checke

RE: [Proposal] Add foreign-server health checks infrastructure

2023-01-10 Thread Hayato Kuroda (Fujitsu)
pproach was better. Best Regards, Hayato Kuroda FUJITSU LIMITED v23-0001-Add-PQConncheck-to-libpq.patch Description: v23-0001-Add-PQConncheck-to-libpq.patch v23-0002-postgres_fdw-add-postgres_fdw_verify_connection_.patch Description: v23-0002-postgres_fdw-add-postgres_fdw_verify_connection_.pa

RE: Exit walsender before confirming remote flush in logical replication

2022-12-28 Thread Hayato Kuroda (Fujitsu)
tps://wiki.postgresql.org/wiki/Streaming_Replication [2]: https://wiki.postgresql.org/wiki/Synchronous_Replication_9/2010_Proposal Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Exit walsender before confirming remote flush in logical replication

2022-12-27 Thread Hayato Kuroda (Fujitsu)
rver to shut down... failed pg_ctl: server does not shut down ``` Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Exit walsender before confirming remote flush in logical replication

2022-12-27 Thread Hayato Kuroda (Fujitsu)
and then remove them. It may have huge impact for other features and should not be accepted... Best Regards, Hayato Kuroda FUJITSU LIMITED v3-0001-Exit-walsender-before-confirming-remote-flush-in-.patch Description: v3-0001-Exit-walsender-before-confirming-remote-flush-in-.patch

RE: Time delayed LR (WAS Re: logical replication restrictions)

2022-12-27 Thread Hayato Kuroda (Fujitsu)
*/ > > This comment doesn't make much sense, I think this needs to be rephrased. Changed to simpler expression. We have also fixed wrong usage of wal_receiver_status_interval. We must convert the unit from [s] to [ms] when it is passed to WaitLatch(). Note that more than half of the mo

RE: Exit walsender before confirming remote flush in logical replication

2022-12-27 Thread Hayato Kuroda (Fujitsu)
ed 2, but I thought 1 seemed to be better. PSA the updated patch. Best Regards, Hayato Kuroda FUJITSU LIMITED v2-0001-Exit-walsender-before-confirming-remote-flush-in-.patch Description: v2-0001-Exit-walsender-before-confirming-remote-flush-in-.patch

RE: New strategies for freezing, advancing relfrozenxid early

2022-12-26 Thread Hayato Kuroda (Fujitsu)
Dear Peter, > Anyway, this assertion is wrong, and simply needs to be removed. > Thanks for the report Thanks for modifying for quickly! I found your commit in the remote repository. I will watch and report again if there are another issue. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Time delayed LR (WAS Re: logical replication restrictions)

2022-12-26 Thread Hayato Kuroda (Fujitsu)
s://cirrus-ci.com/task/5630405437554688 [2]: https://www.postgresql.org/message-id/TYAPR01MB5866B24104FD80B5D7E65C3EF5ED9%40TYAPR01MB5866.jpnprd01.prod.outlook.com Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: New strategies for freezing, advancing relfrozenxid early

2022-12-26 Thread Hayato Kuroda (Fujitsu)
res/commit/4ce3afb82ecfbf64d4f6247e725004e1da30f47c Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Exit walsender before confirming remote flush in logical replication

2022-12-26 Thread Hayato Kuroda (Fujitsu)
abase objects for a long time and it causes another issue. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Exit walsender before confirming remote flush in logical replication

2022-12-23 Thread Hayato Kuroda (Fujitsu)
335f6efa38ed...@tycpr01mb8373.jpnprd01.prod.outlook.com Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Force streaming every change in logical decoding

2022-12-23 Thread Hayato Kuroda (Fujitsu)
Dear Shi, > I also fixed Kuroda-san's comments[1]. Thanks for updating the patch! I confirmed that my comments were fixed and your patch could pass subscription and test_decoding tests. I think we can change the status to "Ready for Committer". Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Force streaming every change in logical decoding

2022-12-22 Thread Hayato Kuroda (Fujitsu)
can. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Force streaming every change in logical decoding

2022-12-21 Thread Hayato Kuroda (Fujitsu)
like (empty)? Or we can ignore now. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Time delayed LR (WAS Re: logical replication restrictions)

2022-12-21 Thread Hayato Kuroda (Fujitsu)
sage-id/TYAPR01MB586668E50FC2447AD7F92491F5E89%40TYAPR01MB5866.jpnprd01.prod.outlook.com Best Regards, Hayato Kuroda FUJITSU LIMITED

Exit walsender before confirming remote flush in logical replication

2022-12-21 Thread Hayato Kuroda (Fujitsu)
com [4]: https://github.com/postgres/postgres/commit/985bd7d49726c9f178558491d31a570d47340459 Best Regards, Hayato Kuroda FUJITSU LIMITED 0001-Exit-walsender-before-confirming-remote-flush-in-log.patch Description: 0001-Exit-walsender-before-confirming-remote-flush-in-log.patch

RE: Force streaming every change in logical decoding

2022-12-21 Thread Hayato Kuroda (Fujitsu)
thus hampering system's > performance. > Have seen many such cases in previous org. That never crossed my mind at all. Indeed, DBA may be confused, this proposal seems to be too optimized. I can withdraw this and we can go another way, maybe my previous approach. Best Regards, Hayato Kuroda FUJITSU LIMITED

<    1   2   3   4   5   6   >