RE: Random pg_upgrade test failure on drongo

2024-01-10 Thread Hayato Kuroda (Fujitsu)
e would be better as the problem is > > generic but it is currently shown for pg_largeobject. > > Yes, for sure. So let's keep it generic as you prefer. > > Thank you! Thanks for working the patch. I'm also OK to push the Amit's fix patch. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: speed up a logical replica setup

2024-01-11 Thread Hayato Kuroda (Fujitsu)
%40TY3PR01MB9889.jpnprd01.prod.outlook.com [3]: https://www.postgresql.org/message-id/CALDaNm098Jkbh%2Bye6zMj9Ro9j1bBe6FfPV80BFbs1%3DpUuTJ07g%40mail.gmail.com Best Regards, Hayato Kuroda FUJITSU LIMITED v4-0001-Creates-a-new-logical-replica-from-a-standby-serv.patch Description: v4-0001-Creates

RE: speed up a logical replica setup

2024-01-11 Thread Hayato Kuroda (Fujitsu)
may consider to enhance this tool later if we > have user demand for such options. OK, so let's drop it once and consider later as an enhancement. As Euler said, it leads additional ERRORs. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: speed up a logical replica setup

2024-01-11 Thread Hayato Kuroda (Fujitsu)
ber to a file as well? pg_upgrade did similar thing. Thought? > 5. Unnecessary Assert in drop_replication_slot() was removed. Instead, I fixed the code and keep the assert. > Cool. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: speed up a logical replica setup

2024-01-12 Thread Hayato Kuroda (Fujitsu)
orker" (PID 3669) exited with exit code 1 ``` But this is strange. I confirmed that the specified publication surely exists. Do you know the reason? ``` publisher=# SELECT pubname FROM pg_publication; pubname ----- pg_subscriber_5 (1 row) ``` Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Documentation to upgrade logical replication cluster

2024-01-15 Thread Hayato Kuroda (Fujitsu)
rchar(40)); +CREATE TABLE + + ``` Same tables were created, they must have another name. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: A failure in t/038_save_logical_slots_shutdown.pl

2024-01-15 Thread Hayato Kuroda (Fujitsu)
Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Random pg_upgrade test failure on drongo

2024-01-16 Thread Hayato Kuroda (Fujitsu)
Dear hackers, > > Thanks to both of you. I have pushed the patch. > I have been tracking the BF animals these days, and this failure has not seen anymore. I think we can close the topic. Again, thanks for all efforts. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: speed up a logical replica setup

2024-01-16 Thread Hayato Kuroda (Fujitsu)
.jpnprd01.prod.outlook.com Best Regards, Hayato Kuroda FUJITSU LIMITED v20240117-0001-Creates-a-new-logical-replica-from-a-stand.patch Description: v20240117-0001-Creates-a-new-logical-replica-from-a-stand.patch v20240117-0002-Address-some-comments-proposed-on-hackers.patch Description: v20240117

RE: speed up a logical replica setup

2024-01-21 Thread Hayato Kuroda (Fujitsu)
One solution is to create a publication before creating a consistent slot. Changes which came before creating the slot were surely replicated to the standby, so upcoming transactions can see the object. We are planning to patch set to fix the issue in this approach. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: speed up a logical replica setup

2024-01-22 Thread Hayato Kuroda (Fujitsu)
good verb "switch". So, how about using "pg_switchsubscriber"? Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Documentation to upgrade logical replication cluster

2024-01-24 Thread Hayato Kuroda (Fujitsu)
be allocated to step, not para. That's why the rendering is bit a strange. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: speed up a logical replica setup

2024-01-24 Thread Hayato Kuroda (Fujitsu)
cannot omit these, but setting smaller shared_buffers will reduce the start time. One approach is to overwrite the GUC to smaller value, but I think we cannot determine the appropriate value. Best Regards, Hayato Kuroda FUJITSU LIMITED run.sh Description: run.sh perf_result.xlsx Description

RE: speed up a logical replica setup

2024-01-25 Thread Hayato Kuroda (Fujitsu)
l variable. 19. C99-style has been allowed, so loop variables like "i" can be declared in the for-statement, like ``` for (int i = 0; i < MAX; i++) ``` 20. Some comments, docs, and outputs must be fixed when the name is changed. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: speed up a logical replica setup

2024-01-25 Thread Hayato Kuroda (Fujitsu)
> I think "convert" and "transform" fit for this case. However, "create", > > "convert" and "transform" have 6, 7 and 9 characters, respectively. I > > suggest > > that we avoid long names (subscriber already has 10 characters). My > > preference > > is pg_createsubscriber. > > That seems best to me. Just FYI - I'm ok to change the name to pg_createsubscriber. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: speed up a logical replica setup

2024-01-25 Thread Hayato Kuroda (Fujitsu)
ing is already specified as primary_conninfo so --publisher-conninfo may not be needed. The parameter does not contain database name, so --databases is still needed. I imagine like: 1. Parse options 2. Turn on standby 3. Verify the standby 4. Turn off standby 5. Get primary_conninfo from standby 6. Connect to primary (concat of primary_conninfo and an option is used) 7. Verify the primary ... How do you think? Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/global/

RE: speed up a logical replica setup

2024-01-25 Thread Hayato Kuroda (Fujitsu)
message-id/CAEG8a3%2BwL_2R8n12BmRz7yBP3EBNdHDhmdgxQFA9vS%2BzPR%2B3Kw%40mail.gmail.com [3]: https://www.postgresql.org/message-id/TY3PR01MB9889678E47B918F4D83A6FD8F57B2%40TY3PR01MB9889.jpnprd01.prod.outlook.com [4]: https://www.postgresql.org/message-id/TY3PR01MB9889C362FF76102C88FA1C29F56F2%40TY

RE: Documentation to upgrade logical replication cluster

2024-01-28 Thread Hayato Kuroda (Fujitsu)
Dear Vignesh, Thanks for updating the patch! For now, v4 patch LGTM. Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/

RE: speed up a logical replica setup

2024-01-30 Thread Hayato Kuroda (Fujitsu)
TY3PR01MB9889C362FF76102C88FA1C29F56F2%40TY3PR01MB9889.jpnprd01.prod.outlook.com [3]: https://www.postgresql.org/message-id/CAA4eK1JRgnhv_ySzuFjN7UaX9qxz5Hqcwew7Fk%3D%2BAbJbu0Kd9w%40mail.gmail.com Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/global/

RE: Improve eviction algorithm in ReorderBuffer

2024-01-30 Thread Hayato Kuroda (Fujitsu)
ch is O(n). This strategy minimizes the overheads of updating the transaction's memory counter. ``` s/pudate/update/ 08. IIUC, if more than 1024 transactions are running but they have small amount of changes, the performance may be degraded, right? Do you have a result in sucha a case? Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/

RE: Documentation to upgrade logical replication cluster

2024-01-30 Thread Hayato Kuroda (Fujitsu)
17.0 or later, then only logical slots on the primary are copied to the new standby. ``` Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/

RE: speed up a logical replica setup

2024-01-31 Thread Hayato Kuroda (Fujitsu)
Dear Fabrízio, Thanks for reporting. I understood that the issue occurred on v11 and v12. I will try to reproduce and check the reason. Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/global/

RE: speed up a logical replica setup

2024-01-31 Thread Hayato Kuroda (Fujitsu)
role and CREATE privilege on the specified database. > Not sure, but can we do the replication origin functions by these privilege? According to the doc[1], these ones seem not to be related. [1]: https://www.postgresql.org/docs/devel/functions-admin.html#FUNCTIONS-REPLICATION Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/global/

RE: speed up a logical replica setup

2024-01-31 Thread Hayato Kuroda (Fujitsu)
does not exist. Also, the physical slot is still remained on the primary. In short, "temp_replslot" should be "primary_slot_name". PSA a script file for reproducing. Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/ test_0201.sh Description: test_0201.sh

RE: Is this a problem in GenericXLogFinish()?

2024-02-04 Thread Hayato Kuroda (Fujitsu)
xlrec.is_prev_bucket_same_wrt is false) So, I think my patch (after removing elog(...) part) can fix the issue. Thought? [1]: ``` LOG: XXX: is_wbuf_registered: false CONTEXT: while vacuuming index "hash_cleanup_index" of relation "public.hash_cleanup_heap" STATEMENT: VA

RE: Is this a problem in GenericXLogFinish()?

2024-02-04 Thread Hayato Kuroda (Fujitsu)
buffers must not be PageSetLSN()'d. Other pages, e.g., metabuf, has already been followed the rule. I updated the patch based on the requirement. Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/ v2_avoid_registration.patch Description: v2_avoid_registration.patch

RE: Improve eviction algorithm in ReorderBuffer

2024-02-05 Thread Hayato Kuroda (Fujitsu)
1 > INSERT: > > HEAD: > 1810.192 ms > > HEAD w/ patch: > 2001.094 ms > > I set a large enough value to logical_decoding_work_mem not to evict > any transactions. I can see about about 10% performance regression in > this case. Thanks for running. I think this workload is the worst and an extreme case which would not be occurred on the real system (Such a system should be fixed), so we can say that the regression is up to -10%. I felt it could be negligible but how do other think? Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/

RE: speed up a logical replica setup

2024-02-06 Thread Hayato Kuroda (Fujitsu)
e server log during pg_createsubscriber, and we can see that walreceiver exits before receiving all the required changes. I cannot find a path yet, but the inconsistency lead the situation which walreceiver exists but startup remains. This also said that recovery status must be checked in

RE: speed up a logical replica setup

2024-02-06 Thread Hayato Kuroda (Fujitsu)
his works wrongly. Users still can set an arbitrary connection string as primary_conninfo. You just use longer string unintentionally. Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/ shorter_subconninfo.sh Description: shorter_subconninfo.sh

RE: speed up a logical replica setup

2024-02-06 Thread Hayato Kuroda (Fujitsu)
8902B992A4F2E99E1385EDF56F2%40TY3PR01MB9889.jpnprd01.prod.outlook.com Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/

RE: speed up a logical replica setup

2024-02-06 Thread Hayato Kuroda (Fujitsu)
al agents, and we can partially protect the database. But there is still a room. d) Overwrite both port and listen_addresses. This can protect databases perfectly but no one can monitor. Hmm, which one should be chosen? I prefer c) or d). Do you know how pglogical_create_subscriber does? Best R

RE: speed up a logical replica setup

2024-02-07 Thread Hayato Kuroda (Fujitsu)
d to change it to regular connections and use SQL functions (instead of replication commands). Is it a requirement for v16-0003? > No, it is not required by 0003. I forgot the decision that we force DBAs to open normal connection establishments for pg_createsubscriber. Please ignore... [1]: https

RE: speed up a logical replica setup

2024-02-07 Thread Hayato Kuroda (Fujitsu)
ic. > pglogical_create_subscriber does nothing [2][3]. > Oh, thanks. Just to confirm - pglogical set shared_preload_libraries to '', should we follow or not? Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/global/

RE: Improve eviction algorithm in ReorderBuffer

2024-02-08 Thread Hayato Kuroda (Fujitsu)
/CAD21AoB-7mPpKnLmBNfzfavG8AiTwEgAdVMuv%3DjzmAp9ex7eyQ%40mail.gmail.com Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/

RE: speed up a logical replica setup

2024-02-09 Thread Hayato Kuroda (Fujitsu)
n slot "pg_createsubscriber_16389_3884" on database "testdb" pg_createsubscriber: XXX: sleep 20s ``` Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/global/ N1.log Description: N1.log result.dat Description: result.dat server_start_20240209T115112.96

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-21 Thread Hayato Kuroda (Fujitsu)
dable when apply worker sends feedback message. That's why I wrote "Sleep until we get reply from worker or we time out". > src/include/replication/logical.h > > 6. > + /* > + * The minimum delay, in milliseconds, by the publisher before sending > + * COMMIT/PREPARE record > + */ > + int32 min_send_delay; > > The comment is missing a period. Right, added. Best Regards, Hayato Kuroda FUJITSU LIMITED v5-0001-Time-delayed-logical-replication-on-publisher-sid.patch Description: v5-0001-Time-delayed-logical-replication-on-publisher-sid.patch

RE: Allow logical replication to copy tables in binary format

2023-02-21 Thread Hayato Kuroda (Fujitsu)
a possibility that replication between different versions may be failed if binary format is specified. Therefore, I think we should accept copy_format = binary only when the major version of servers are the same. Note that this comments is not the request to the patch. Maybe the modification should be done not only for copy_format but also binary, and it may be out of scope the patch. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: [Proposal] Add foreign-server health checks infrastructure

2023-02-22 Thread Hayato Kuroda (Fujitsu)
> 7. pqSocketPoll > > + * When neither forRead nor forWrite are set and timeout is disabled, > + * > + * - If the timeout is disabled, try to check the health of the socket > + * - Otherwise this immediately returns 0 > + * > + * Return >0 if condition is met, 0 if a t

RE: [Proposal] Add foreign-server health checks infrastructure

2023-02-22 Thread Hayato Kuroda (Fujitsu)
may be reasonable because the checking is not really done on this environment. How do you think? [1]: https://www.postgresql.org/message-id/TYAPR01MB586664F5ED2128E572EA95B0F5AA9%40TYAPR01MB5866.jpnprd01.prod.outlook.com Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-22 Thread Hayato Kuroda (Fujitsu)
, there is a > possibility that the table status updated in linkend="catalog-pg-subscription-rel">pg_subscription_rel ctname> > could be delayed in getting to the "ready" state, and also two-phase > (if specified) could be delayed in getting to "enabled&qu

RE: Allow logical replication to copy tables in binary format

2023-02-22 Thread Hayato Kuroda (Fujitsu)
" and "copy_data = false" should be rejected in parse_subscription_options() and AlterSubscription(). Is it right? I'm expecting that is done in next version. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Rework LogicalOutputPluginWriterUpdateProgress

2023-02-23 Thread Hayato Kuroda (Fujitsu)
se? Or better name may be needed. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-23 Thread Hayato Kuroda (Fujitsu)
lest way. TransactionId was added as an argument of functions. Best Regards, Hayato Kuroda FUJITSU LIMITED v7-0001-Time-delayed-logical-replication-on-publisher-sid.patch Description: v7-0001-Time-delayed-logical-replication-on-publisher-sid.patch

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-26 Thread Hayato Kuroda (Fujitsu)
til the process spends time till a given period. Such a function is painful to read later. I think callbacks that have different purposes should not be mixed. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-27 Thread Hayato Kuroda (Fujitsu)
L". So fixed as you said. In next version I will use grammar checker like Chat-GPT to modify commit messages... [1]: https://www.postgresql.org/message-id/CAA4eK1LjOm6-OHggYVH35dQ_v40jOXrJW0GFy3kuwTd2J48%3DUg%40mail.gmail.com [2]: https://www.postgresql.org/message-id/CAA4eK1K4uPbudrNdH%

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-28 Thread Hayato Kuroda (Fujitsu)
the next version. Thanks! I checked and I thought all of them should be included. Moreover, I used grammar checker and slightly reworded the commit message. [1]: https://man7.org/linux/man-pages/man3/send.3p.html Best Regards, Hayato Kuroda FUJITSU LIMITED v9-0001-Time-delayed-logical-replication-on-publisher-sid.patch Description: v9-0001-Time-delayed-logical-replication-on-publisher-sid.patch

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-03-01 Thread Hayato Kuroda (Fujitsu)
because we send the parameter at START_REPLICATION and the slot has been already created. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Allow logical replication to copy tables in binary format

2023-03-01 Thread Hayato Kuroda (Fujitsu)
%40mail.gmail.com [3]: https://www.postgresql.org/message-id/TYAPR01MB5866968CF42FBAB73E8EEDF8F5AA9%40TYAPR01MB5866.jpnprd01.prod.outlook.com Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-03-01 Thread Hayato Kuroda (Fujitsu)
the default to no delay, it is > expected from users using this have an understanding of the trade-off. Yes, the trade-off should be emphasized. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-03-02 Thread Hayato Kuroda (Fujitsu)
+ int32 min_send_delay; /* The minimum send delay */ > } logical; > } proto; > } WalRcvStreamOptions; > > ~ > > Should that comment mention the units are "(ms)" Added. Best Regards, Hayato Kuroda FUJITSU LIMITED v10-0001-Time-delayed-logical-replication-on-publisher-si.patch Description: v10-0001-Time-delayed-logical-replication-on-publisher-si.patch

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-03-02 Thread Hayato Kuroda (Fujitsu)
ereport(ERROR, Changed. I grepped ereport() in the patch and I thought there were no similar one. > 7) In the expect we have specifically mention "for non-streaming > transaction", is the behavior different for streaming transaction, if > not we can change the message accordingly > +# The publisher waits for the replication to complete > +$node_publisher->wait_for_catchup('tap_sub_renamed'); > + > +# This test is successful only if at least the configured delay has elapsed. > +ok( time() - $publisher_insert_time >= $delay, > + "subscriber applies changes only after replication delay for > non-streaming transaction" > +); There is no difference, both of normal and streamed transaction could be delayed to apply. So removed. [1]: https://www.postgresql.org/message-id/flat/TYAPR01MB586606CF3B585B6F8BE13A9CF5B29%40TYAPR01MB5866.jpnprd01.prod.outlook.com Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-03-02 Thread Hayato Kuroda (Fujitsu)
ink it is better, so changed. Please see new patch at [1] [1]: https://www.postgresql.org/message-id/flat/TYAPR01MB586606CF3B585B6F8BE13A9CF5B29%40TYAPR01MB5866.jpnprd01.prod.outlook.com Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-03-02 Thread Hayato Kuroda (Fujitsu)
trade-off should be emphasized. Based on the understanding, I added them to the doc in new version patch. Please see [1]. [1]: https://www.postgresql.org/message-id/flat/TYAPR01MB586606CF3B585B6F8BE13A9CF5B29%40TYAPR01MB5866.jpnprd01.prod.outlook.com Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2023-03-03 Thread Hayato Kuroda (Fujitsu)
Index() if idxIsRelationIdentityOrPK is false. 032_subscribe_use_index.pl ``` +# Testcase start: SUBSCRIPTION CREATE/DROP INDEX WORKS WITHOUT ISSUES ... +# Testcase end: SUBSCRIPTION RE-CALCULATES INDEX AFTER CREATE/DROP INDEX ``` There is still non-consistent case. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: [Proposal] Add foreign-server health checks infrastructure

2023-03-03 Thread Hayato Kuroda (Fujitsu)
ot;Socket close is detected"? Changed. > 9. the document of postgres_fdw > The document of postgres_fdw_verify_connection_states_* is a little > bit old. Could you update it? Updated. IIUC postgres_fdw_verify_connection_states returns * true, if the connection is ver

RE: [Proposal] Add foreign-server health checks infrastructure

2023-03-06 Thread Hayato Kuroda (Fujitsu)
ment as the postgres_fdw_verify_connection_states > case (comment no.5) seems better. Yes, but completely same statement cannot be used because these is not specified foreign server. How about: NULL is returned if there are no established connections or this function ... > 8. the document o

RE: [Proposal] Add foreign-server health checks infrastructure

2023-03-06 Thread Hayato Kuroda (Fujitsu)
is function emits warnings if a disconnection is found. This > returns true > + * if disconnections cannot be found, otherwise returns false. > > I think false is returned only if disconnections are found and > true is returned in all other cases. So, modifying the description &g

RE: Rework LogicalOutputPluginWriterUpdateProgress

2023-03-06 Thread Hayato Kuroda (Fujitsu)
updates. Do you have any reasons why only they have? [1]: https://www.postgresql.org/message-id/CAHut%2BPsksiQHuv4A54R4w79TAvCu__PcuffKYY0V96e2z_sEvA%40mail.gmail.com Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: [Proposal] Add foreign-server health checks infrastructure

2023-03-07 Thread Hayato Kuroda (Fujitsu)
if a > + * timeout occurred, -1 if an error or interrupt occurred. Note that if 0 is > + * returned and forConnCheck is requested, it means that the socket has not > + * matched POLLRDHUP event and the connection is not closed by the socket > peer. Fixed. [1]: https://www.postgres

RE: Allow logical replication to copy tables in binary format

2023-03-07 Thread Hayato Kuroda (Fujitsu)
quot;binary option was changed and worker will restart..." or something, they can turn "binary" on again. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Rework LogicalOutputPluginWriterUpdateProgress

2023-03-08 Thread Hayato Kuroda (Fujitsu)
from typedefs.list. --- Do we have to write a document for the breakage somewhere? I think we do not have to add appendix-obsolete-* file because we did not have any links for that, but we can add a warning in "Functions for Producing Output" subsection if needed. Best Regards, Hayato Kuroda FUJITSU LIMITED test.sh Description: test.sh

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-03-10 Thread Hayato Kuroda (Fujitsu)
.postgresql.org/message-id/1339586927-13156-12-git-send-email-andres%402ndquadrant.com Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: [Proposal] Add foreign-server health checks infrastructure

2023-03-13 Thread Hayato Kuroda (Fujitsu)
Dear Katsuragi-san, > Hi, > > I updated the status of the patch to ready for committer. > > regards, Thank you so much for your reviewing! Now we can wait comments from senior members and committers. Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Allow logical replication to copy tables in binary format

2023-03-19 Thread Hayato Kuroda (Fujitsu)
options = lappend(options, makeDefElem("format", (Node *) makeString("binary"), -1)); ``` Best Regards, Hayato Kuroda FUJITSU LIMITED

Question: Do we have a rule to use "PostgreSQL" and "PostgreSQL" separately?

2023-03-20 Thread Hayato Kuroda (Fujitsu)
it? Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Data is copied twice when specifying both child and parent table in publication

2023-03-20 Thread Hayato Kuroda (Fujitsu)
_get_publication_tables ``` + pfree(elems); ``` Only elems is pfree()'d here, but how about other variable like pub_elem and pub_elem_tables? Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Question: Do we have a rule to use "PostgreSQL" and "PostgreSQL" separately?

2023-03-21 Thread Hayato Kuroda (Fujitsu)
Dear Daniel, Tom, > > On 20 Mar 2023, at 15:31, Tom Lane wrote: > > > > "Hayato Kuroda (Fujitsu)" writes: > >> While checking documentations, I found that one line notes our product as > >> "PostgreSQL", whereas another line notes >

RE: Data is copied twice when specifying both child and parent table in publication

2023-03-21 Thread Hayato Kuroda (Fujitsu)
about "use_extended_function" or something? Best Regards, Hayato Kuroda FUJITSU LIMITED

RE: Allow logical replication to copy tables in binary format

2023-03-22 Thread Hayato Kuroda (Fujitsu)
atop v19-0001. Please check 0002 patch. Best Regards, Hayato Kuroda FUJITSU LIMITED v19-0001-Allow-logical-replication-to-copy-table-in-binary.patch Description: v19-0001-Allow-logical-replication-to-copy-table-in-binary.patch v19-0002-Add-XML-ID-attributes-to-create_subscription.sgm.

PGdoc: add missing ID attribute to create_subscription.sgml

2023-03-22 Thread Hayato Kuroda (Fujitsu)
[email protected] [2]: https://www.postgresql.org/message-id/CAB8KJ=jpuQU9QJe4+RgWENrK5g9jhoysMw2nvTN_esoOU0=a...@mail.gmail.com [3]: https://www.postgresql.org/message-id/[email protected] Best Regards, Hayato Kuroda FUJITSU LIMITED 0001-Add-XML-ID-att

RE: Allow logical replication to copy tables in binary format

2023-03-22 Thread Hayato Kuroda (Fujitsu)
Dear Amit, > Pushed the 0001. It may be better to start a separate thread for 0002. Good job! I have started new thread [1] for 0002. [1]: https://www.postgresql.org/message-id/TYAPR01MB58667AE04D291924671E2051F5879%40TYAPR01MB5866.jpnprd01.prod.outlook.com Best Regards, Hayato Kuroda FUJI

RE: PGdoc: add missing ID attribute to create_subscription.sgml

2023-03-24 Thread Hayato Kuroda (Fujitsu)
> e.g. copy_data refers to -- origin > > e.g. origin refers to -- copy_data I have not added links because it was in the same page and I thought it was overkill. I checked a few reference pages, e.g. create_table.sgml and create_type.sgml, but I could not find any links that refer varliste

RE: PostgreSQL 17 Release Management Team & Feature Freeze

2024-03-17 Thread Hayato Kuroda (Fujitsu)
st script, not the feature. The issue has already been analyzed and the fix patch was pushed as f17529b710. Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/

RE: speed up a logical replica setup

2024-03-17 Thread Hayato Kuroda (Fujitsu)
;restart_standby_if_needed" or something. 40. ``` * XXX this code was extracted from BootStrapXLOG(). ``` So, can we extract the common part to somewhere? Since system identifier is related with the controldata file, I think it can be located in controldata_util.c. 41. You said like below in [1], but I could not find the related fix. Can you clarify? > That's a good point. We should state in the documentation that GUCs specified > in > the command-line options are ignored during the execution. [1]: https://www.postgresql.org/message-id/40595e73-c7e1-463a-b8be-49792e870007%40app.fastmail.com Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/global/

RE: PostgreSQL 17 Release Management Team & Feature Freeze

2024-03-18 Thread Hayato Kuroda (Fujitsu)
uld wait checking from you. Thanks. Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/

RE: speed up a logical replica setup

2024-03-18 Thread Hayato Kuroda (Fujitsu)
``` > > > > Sorry if we have already discussed. I think the registration can be moved > > just > > before the boot of the standby. Before that, the callback will be no-op. > > But it can also stay where it is. What is the advantage of moving it later? I thought we could reduce the risk of bugs. Previously some bugs were reported because the registration is too early. However, this is not a strong opinion. Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/

RE: Have pg_basebackup write "dbname" in "primary_conninfo"?

2024-03-19 Thread Hayato Kuroda (Fujitsu)
elow: PGDATABASE="user=kuroda dbname=postgres" pg_basebackup -D data_N2 -R -v The primary_conninfo written in the file will be: primary_conninfo = 'user=hayato ... dbname=''user=kuroda dbname=postgres''' [1]: https://www.postgresql.org/docs/devel/libpq-pgservice.html

RE: speed up a logical replica setup

2024-03-24 Thread Hayato Kuroda (Fujitsu)
ould be okay to extend it later as well. Both ways are OK, but I prefer to unify checks a bit. The number of working modes in the same executables should be reduced as much as possible. Also, I feel the current specification is acceptable. pg_upgrade checks one by one and exits soon in bad cases. If both old and new clusters have issues, the first run only reports the old one's issues. After DBA fixes and runs again, issues on the new cluster are output. [1]: https://www.postgresql.org/message-id/8d52c226-7e34-44f7-a919-759bf8d81541%40enterprisedb.com Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/

RE: speed up a logical replica setup

2024-03-25 Thread Hayato Kuroda (Fujitsu)
27;t think it is not completely needed. If we can agree a specification in sometime, it's OK for me to add them. If you ask me, I still prefer Tomas's approach. Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/

RE: speed up a logical replica setup

2024-03-25 Thread Hayato Kuroda (Fujitsu)
t until the replication starts. Alternative one is to ease the condition. How do you think? [1]: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=serinus&dt=2024-03-25%2013%3A03%3A07 [2]: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=culicidae&dt=2024-03-25%2013%3A53%3A58 Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/

RE: speed up a logical replica setup

2024-03-25 Thread Hayato Kuroda (Fujitsu)
onnect to another publisher as well - this may lead conflicts. This causes because both launcher/workers start after recovery finishes. So, based on the Ashutosh's point, should we remove such replication objects? Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/

RE: pg_upgrade and logical replication

2024-03-27 Thread Hayato Kuroda (Fujitsu)
A23%3A13 Yeah, I think it is an oversight in f17529. Previously subscriptions which receiving changes were confirmed to be caught up, I missed to add the line while restructuring the script. +1 for your fix. Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/

RE: Synchronizing slots from primary to standby

2024-03-28 Thread Hayato Kuroda (Fujitsu)
7.GA2361%40paquier.xyz Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/

RE: Improve eviction algorithm in ReorderBuffer

2024-03-29 Thread Hayato Kuroda (Fujitsu)
` Should be started with upper case? Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/

RE: Is this a problem in GenericXLogFinish()?

2024-04-04 Thread Hayato Kuroda (Fujitsu)
ge(XLogReaderState *record) writeopaque->hasho_nextblkno = xldata->nextblkno; } +if (xldata->ntups == 0 && +xldata->is_prim_bucket_same_wrt && +!xldata->is_prev_bucket_same_wrt) + elog(LOG, "XX

RE: Is this a problem in GenericXLogFinish()?

2024-04-09 Thread Hayato Kuroda (Fujitsu)
Dear Michael, > On Fri, Apr 05, 2024 at 06:22:58AM +0000, Hayato Kuroda (Fujitsu) wrote: > > Thanks for pointing out. Yes, we fixed a behavior by the commit aa5edbe37, > > but we missed the redo case. I made a fix patch based on the suggestion [1]. > > + bool

RE: Improve eviction algorithm in ReorderBuffer

2024-04-10 Thread Hayato Kuroda (Fujitsu)
>txn_node); ``` Since the number of stored transactions does not affect to the insert operation, we may able to add the node while creating ReorederBufferTXN and remove while cleaning up it. This can reduce branches in ReorderBufferChangeMemoryUpdate(). Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/ <>

RE: speed up a logical replica setup

2024-02-13 Thread Hayato Kuroda (Fujitsu)
pgrade's message like "check clusters only, don't > change any data": > + printf(_(" -d, --database=DBNAME database to > create a subscription\n")); > + printf(_(" -n, --dry-run stop before > modifying anything\n")); > + printf(_(" -t, --recovery-timeout=SECS seconds to wait > for recovery to end\n")); > + printf(_(" -r, --retainretain log file > after success\n")); > + printf(_(" -v, --verbose output verbose > messages\n")); > + printf(_(" -V, --version output version > information, then exit\n")); Changed. New patch is available in [2]. [1]: https://www.postgresql.org/message-id/TYCPR01MB1207713BEC5C379A05D65E342F54B2%40TYCPR01MB12077.jpnprd01.prod.outlook.com [2]: https://www.postgresql.org/message-id/TYCPR01MB12077A6BB424A025F04A8243DF54F2%40TYCPR01MB12077.jpnprd01.prod.outlook.com Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/

RE: speed up a logical replica setup

2024-02-13 Thread Hayato Kuroda (Fujitsu)
> the standby has been promoted once), we cannot resume the physical replication > as-is. IIUC the easiest method to retry is removing a cluster once and > restarting > from pg_basebackup. If so, no need to cleanup the standby because it is > corrupted. > We just say "Please

RE: speed up a logical replica setup

2024-02-13 Thread Hayato Kuroda (Fujitsu)
suggestions. I felt current code followed the style. Thought? New patch is available in [2]. [1]: https://www.postgresql.org/docs/devel/error-style-guide.html [2]: https://www.postgresql.org/message-id/TYCPR01MB12077A6BB424A025F04A8243DF54F2%40TYCPR01MB12077.jpnprd01.prod.outlook.com Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/

RE: pg_upgrade and logical replication

2024-02-13 Thread Hayato Kuroda (Fujitsu)
+++ t/004_subscription.pl .. ok All tests successful. Files=1, Tests=14, 9 wallclock secs ( 0.03 usr 0.00 sys + 0.55 cusr 1.08 csys = 1.66 CPU) Result: PASS ``` How do you think? [1]: https://www.postgresql.org/docs/devel/pgupgrade.html Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/ 0001-Fix-testcase.patch Description: 0001-Fix-testcase.patch

RE: pg_upgrade and logical replication

2024-02-13 Thread Hayato Kuroda (Fujitsu)
; @@ -181,139 +310,5 @@ is($result, qq(1), > "check the data is synced after enabling the subscription for > the table that was in init state" > ); > > -# cleanup > Removed. PSA a new version patch. Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.

RE: speed up a logical replica setup

2024-02-14 Thread Hayato Kuroda (Fujitsu)
ode2. * Can we really cleanup the standby in case of failure? Shouldn't we suggest to remove the target once? * Can we move outputs to stdout? [1]: https://www.postgresql.org/message-id/TYCPR01MB1207713BEC5C379A05D65E342F54B2%40TYCPR01MB12077.jpnprd01.prod.outlook.com Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/global/

RE: pg_upgrade and logical replication

2024-02-15 Thread Hayato Kuroda (Fujitsu)
reating objects as much as possible, but it may lead some confusion. I recreated a patch for creating pub/sub and dropping them at cleanup for every test cases. PSA a new version. Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/ v3-0001-Fix-testcase.patch Description: v3-0001-Fix-testcase.patch

RE: pg_upgrade and logical replication

2024-02-15 Thread Hayato Kuroda (Fujitsu)
ps://www.postgresql.org/message-id/TYCPR01MB12077B16EEDA360BA645B96F8F54C2%40TYCPR01MB12077.jpnprd01.prod.outlook.com Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/

RE: pg_upgrade and logical replication

2024-02-15 Thread Hayato Kuroda (Fujitsu)
egress_pub4" > +); Ditto. > With above fixes, the following can be removed: > # Initial setup > $publisher->safe_psql( > 'postgres', qq[ > CREATE TABLE tab_upgraded1(id int); > CREATE TABLE tab_upgraded2(id int); > ]); > $old_sub->safe_psql( > 'postgres', qq[ > CREATE TABLE tab_upgraded1(id int); > CREATE TABLE tab_upgraded2(id int); > ]); Yes, earlier definitions were removed instead. Also, some comments were adjusted based on these fixes. Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/ v4-0001-Fix-testcase.patch Description: v4-0001-Fix-testcase.patch

RE: speed up a logical replica setup

2024-02-15 Thread Hayato Kuroda (Fujitsu)
retry 2 or 3 times before give up using the pg_stat_wal_receiver query. Do you have a better plan? > It's good to determine the threshold. It can define the number of acceptable loss of walreceiver during the loop. I think we should retry at least the postmaster revives the walreceiver. The checking would work once per seconds, so more than 5 (or 10) may be better. Thought? Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/global/

RE: speed up a logical replica setup

2024-02-16 Thread Hayato Kuroda (Fujitsu)
gt; This is failing because slot name slot2 is used between node2->node3 > but pg_createsubscriber is checked for slot1, the slot which is used > for replication between node1->node2. > Thoughts? Right. The inconsistency is quite strange. Overall, I felt such a case must be rejected

RE: speed up a logical replica setup

2024-02-16 Thread Hayato Kuroda (Fujitsu)
25. The creation of subscriptions are not directory tested. @subnames contains the name of subscriptions, but it just assumes the number of them is more than two. Since it may be useful, I will post top-up patch on Monday, if there are no updating. [1]: https://www.postgresql.org/message-id/89ccf72b-59f9-4317-b9fd-1e6d20a0c3b1%40app.fastmail.com [2]: https://www.postgresql.org/message-id/TYCPR01MB1207713BEC5C379A05D65E342F54B2%40TYCPR01MB12077.jpnprd01.prod.outlook.com Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/global/

RE: pg_upgrade and logical replication

2024-02-18 Thread Hayato Kuroda (Fujitsu)
tables of subscription > # 'regress_sub5' are synchronized > $new_sub->append_conf('postgresql.conf', > "max_logical_replication_workers = 10"); > $new_sub->restart; > $new_sub->safe_psql('postgres', "ALTER SUBSCRIPTION regress_sub5 > ENABLE"); > $new_sub->wait_for_subscription_sync($publisher, 'regress_sub5'); > > Like: > # Enable the subscription > $new_sub->safe_psql('postgres', "ALTER SUBSCRIPTION regress_sub5 > ENABLE"); > > # Wait until all tables of subscription 'regress_sub5' are synchronized > $new_sub->wait_for_subscription_sync($publisher, 'regress_sub5'); Per comments from Amit [1], I did not change. [1]: https://www.postgresql.org/message-id/CAA4eK1Ls%2BRmJtTvOgaRXd%2BeHSY3x-KUE%3DsfEGQoU-JF_UzA62A%40mail.gmail.com Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/ v5-0001-Fix-testcase.patch Description: v5-0001-Fix-testcase.patch

  1   2   3   4   5   6   7   8   9   10   >