Re: Rethink the wait event names for postgres_fdw, dblink and etc

2023-10-05 Thread Masahiro Ikeda
were wrong for DblinkGetConnect: the wait event could be seen in other functions than dblink() and dblink_exec(). Thanks for modifying and committing. I agree your comments. Regards, -- Masahiro Ikeda NTT DATA CORPORATION

Re: Make --help output fit within 80 columns per line

2023-08-22 Thread Masahiro Ikeda
Hi, On 2023-08-22 22:57, torikoshia wrote: On 2023-08-21 13:08, Masahiro Ikeda wrote: (2) Is there any reason that only src/bin commands are targeted? I found that we also need to fix vacuumlo with the above test. I think it's better to fix it because it's a contrib module. $ vacuumlo

Re: Fix pg_stat_reset_single_table_counters function

2023-08-20 Thread Masahiro Ikeda
ot; in the collation tests so I have just used that, and applied the patch down to 15. Thanks! Regards, -- Masahiro Ikeda NTT DATA CORPORATION

Re: Fix pg_stat_reset_single_table_counters function

2023-08-20 Thread Masahiro Ikeda
On 2023-08-21 11:33, Kyotaro Horiguchi wrote: On 2023/08/15 14:13, Masahiro Ikeda wrote: On 2023-08-15 11:48, Masahiko Sawada wrote: +COMMENT ON DATABASE :current_database IS 'This is a test comment'; -- insert or update in 'pg_shdescription' I think the current_database should be quoted (see

Re: Make --help output fit within 80 columns per line

2023-08-20 Thread Masahiro Ikeda
ograms which is tested with program_help_ok $ find -name "*.pl" | xargs -I {} sh -c 'grep -e program_help_ok {}' Regards, -- Masahiro Ikeda NTT DATA CORPORATION

Re: Rethink the wait event names for postgres_fdw, dblink and etc

2023-08-20 Thread Masahiro Ikeda
shorter like pgfdw_we_receive. * Update documents. * Add some tests with pg_wait_events view. Regards, -- Masahiro Ikeda NTT DATA CORPORATIONFrom 025525eae164e33117d94f2a90a4219808072b3c Mon Sep 17 00:00:00 2001 From: Masahiro Ikeda Date: Mon, 21 Aug 2023 10:36:10 +0900 Subject: [PATCH] Make to use

Rethink the wait event names for postgres_fdw, dblink and etc

2023-08-17 Thread Masahiro Ikeda
ys welcome and appreciated. Regards, -- Masahiro Ikeda NTT DATA CORPORATIONFrom 73c4c6562509465bea75a9bbd273298bdf0ee85e Mon Sep 17 00:00:00 2001 From: Masahiro Ikeda Date: Fri, 18 Aug 2023 11:38:05 +0900 Subject: [PATCH] Make to use custom wait events for modules --- contrib/dblink/dblink.c

Re: WIP: new system catalog pg_wait_event

2023-08-17 Thread Masahiro Ikeda
Hi, On 2023-08-17 14:53, Drouvot, Bertrand wrote: On 8/17/23 3:53 AM, Masahiro Ikeda wrote: 1) The regular expression needs to be changed in generate-wait_event_types.pl. I have compared the documentation with the output of the pg_wait_event view and found the following differences

Re: Fix an entry in wait_event_names.txt

2023-08-17 Thread Masahiro Ikeda
patch to fix this entry in wait_event_names.txt (I did check that no other entries are in the same case). [1]: https://www.postgresql.org/message-id/735fbd560ae914c96faaa23cc8d9a118%40oss.nttdata.com Regards, +1. Thanks! Regards, -- Masahiro Ikeda NTT DATA CORPORATION

Re: WIP: new system catalog pg_wait_event

2023-08-16 Thread Masahiro Ikeda
On 2023-08-17 10:57, Michael Paquier wrote: On Thu, Aug 17, 2023 at 10:53:02AM +0900, Masahiro Ikeda wrote: BTW, although I think this is outside the scope of this patch, it might be a good idea to be able to add a description to the API for custom wait events. Somebody on twitter has raised

Re: WIP: new system catalog pg_wait_event

2023-08-16 Thread Masahiro Ikeda
tries */ 5) BTW, although I think this is outside the scope of this patch, it might be a good idea to be able to add a description to the API for custom wait events. Regards, -- Masahiro Ikeda NTT DATA CORPORATION

Re: Fix pg_stat_reset_single_table_counters function

2023-08-14 Thread Masahiro Ikeda
On 2023-08-15 11:48, Masahiko Sawada wrote: On Mon, Aug 14, 2023 at 5:12 PM Masahiro Ikeda wrote: I changed the table to check the stats from pg_database to pg_shdescription because the stats can update via the SQL interface COMMENT command. It seems to work well. +COMMENT ON DATABASE

Re: Support to define custom wait events for extensions

2023-08-14 Thread Masahiro Ikeda
On 2023-08-14 18:26, Michael Paquier wrote: On Mon, Aug 14, 2023 at 05:55:42PM +0900, Masahiro Ikeda wrote: I'm thinking a name like "contrib name + description summary" would be nice. The "contrib name" is namespace-like and the "description summary" is the s

Re: Support to define custom wait events for extensions

2023-08-14 Thread Masahiro Ikeda
On 2023-08-14 15:28, Michael Paquier wrote: On Mon, Aug 14, 2023 at 12:31:05PM +0900, Masahiro Ikeda wrote: Thanks! I confirmed the changes, and all tests passed. Okay, cool. I got some extra time today and applied that, with a few more tweaks. Thanks for applying master branch

Re: Fix pg_stat_reset_single_table_counters function

2023-08-14 Thread Masahiro Ikeda
, -- Masahiro Ikeda NTT DATA CORPORATIONFrom 215ef8ef68af30753cfcd4336b1f6bd9203ac014 Mon Sep 17 00:00:00 2001 From: Masahiro Ikeda Date: Mon, 14 Aug 2023 16:48:30 +0900 Subject: [PATCH] Fix pg_stat_reset_single_table_counters function. This commit revives the feature to reset statistics

Re: Support to define custom wait events for extensions

2023-08-13 Thread Masahiro Ikeda
in-core modules. Thanks! I confirmed the changes, and all tests passed. Regards, -- Masahiro Ikeda NTT DATA CORPORATION

Re: Fix pg_stat_reset_single_table_counters function

2023-08-09 Thread Masahiro Ikeda
On 2023-08-01 15:23, Masahiro Ikeda wrote: Hi, My colleague, Mitsuru Hinata (in CC), found the following issue. The documentation of pg_stat_reset_single_table_counters() says pg_stat_reset_single_table_counters ( oid ) → void Resets statistics for a single table or index in the current

Re: Support to define custom wait events for extensions

2023-08-09 Thread Masahiro Ikeda
losing the former entry. (https://commitfest.postgresql.org/43/4368/) Regards, -- Masahiro Ikeda NTT DATA CORPORATIONFrom 234cc7d852aebf78285ccde383f27ea35a27dad2 Mon Sep 17 00:00:00 2001 From: Masahiro Ikeda Date: Thu, 10 Aug 2023 10:44:41 +0900 Subject: [PATCH 2/2] poc: custom wait event for dbli

Re: Support to define custom wait events for extensions

2023-08-09 Thread Masahiro Ikeda
worrying about * Is 512(wee_hash_max_size) the maximum number of the custom wait events sufficient? * Is there any way to not force extensions that don't use shared memory for their use like dblink to acquire AddinShmemInitLock?; Regards, -- Masahiro Ikeda NTT DATA CORPORATIONFrom

Re: Support to define custom wait events for extensions

2023-08-08 Thread Masahiro Ikeda
I accidentally attached a patch in one previous email. But, you don't need to check it, sorry. (v1-0001-Change-to-manage-custom-wait-events-in-shared-hash.patch) Regards, -- Masahiro Ikeda NTT DATA CORPORATION

Re: Support to define custom wait events for extensions

2023-08-08 Thread Masahiro Ikeda
On 2023-08-08 10:05, Michael Paquier wrote: On Tue, Aug 08, 2023 at 09:39:02AM +0900, Masahiro Ikeda wrote: I am thinking a bit that we also need another hash where the key is a custom string. For extensions that have no dependencies with shared_preload_libraries, I think we need to avoid

Re: Support to define custom wait events for extensions

2023-08-07 Thread Masahiro Ikeda
On 2023-08-08 08:54, Michael Paquier wrote: On Wed, Aug 02, 2023 at 06:34:15PM +0900, Masahiro Ikeda wrote: On 2023-08-01 12:23, Andres Freund wrote: This is why the scheme as implemented doesn't really make sense to me. It'd be much easier to use if we had a shared hashtable

Re: Support to define custom wait events for extensions

2023-08-02 Thread Masahiro Ikeda
On 2023-08-01 12:23, Andres Freund wrote: Hi, On 2023-08-01 12:14:49 +0900, Michael Paquier wrote: On Tue, Aug 01, 2023 at 11:51:35AM +0900, Masahiro Ikeda wrote: > Thanks for committing the main patch. > > In my understanding, the rest works are > * to support WaitEventExten

Re: Support to define custom wait events for extensions

2023-08-01 Thread Masahiro Ikeda
On 2023-08-02 08:38, Ranier Vilela wrote: Hi, On Tue, Aug 01, 2023 at 11:51:35AM +0900, Masahiro Ikeda wrote: Thanks for committing the main patch. Latest head Ubuntu 64 bits gcc 13 64 bits ./configure --without-icu make clean make In file included from ../../src/include/pgstat.h:20

Fix pg_stat_reset_single_table_counters function

2023-08-01 Thread Masahiro Ikeda
, -- Masahiro Ikeda NTT DATA CORPORATIONFrom 561c4391c9dac30b5478637a6baf8c8689226da5 Mon Sep 17 00:00:00 2001 From: Masahiro Ikeda Date: Tue, 1 Aug 2023 13:46:00 +0900 Subject: [PATCH] Fix pg_stat_reset_single_table_counters function. This commit revives the feature to reset statistics

Re: Support to define custom wait events for extensions

2023-07-31 Thread Masahiro Ikeda
WAIT_EVENT_EXTENSION to custom wait events Do someone already works for them? If not, I'll consider how to realize them. Regards, -- Masahiro Ikeda NTT DATA CORPORATION

Re: Support to define custom wait events for extensions

2023-07-31 Thread Masahiro Ikeda
On 2023-07-31 16:28, Michael Paquier wrote: On Mon, Jul 31, 2023 at 03:53:27PM +0900, Masahiro Ikeda wrote: /* This should only be called for user-defined wait event. */ if (eventId < NUM_BUILTIN_WAIT_EVENT_EXTENSION) ereport(ER

Re: Support to define custom wait events for extensions

2023-07-31 Thread Masahiro Ikeda
ust wondering if it should also check the eventId that has been allocated though it needs to take the spinlock and GetWaitEventExtensionIdentifier() doesn't take it into account. Regards, -- Masahiro Ikeda NTT DATA CORPORATION

Re: Support to define custom wait events for extensions

2023-07-27 Thread Masahiro Ikeda
nextId; /* next ID to assign */ + slock_t mutex; /* protects the counter only */ +} WaitEventExtensionCounter; + +/* pointer to the shared memory */ +static WaitEventExtensionCounter * waitEventExtensionCounter; Regards, -- Masahiro

Re: Support worker_spi to execute the function dynamically.

2023-07-24 Thread Masahiro Ikeda
On 2023-07-24 12:01, Bharath Rupireddy wrote: I'm attaching the v3 patch. I verified it works and it looks good to me. Thanks to your work, I will be able to implement tests for custom wait events. Regards, -- Masahiro Ikeda NTT DATA CORPORATION

Re: Support worker_spi to execute the function dynamically.

2023-07-23 Thread Masahiro Ikeda
On 2023-07-22 01:05, Bharath Rupireddy wrote: On Fri, Jul 21, 2023 at 4:05 PM Masahiro Ikeda wrote: (2) Do we need "worker_spi.total_workers = 0" and "shared_preload_libraries = worker_spi" in dynamic.conf. Currently, the static bg workers will not

Re: Support worker_spi to execute the function dynamically.

2023-07-21 Thread Masahiro Ikeda
Hi, On 2023-07-20 18:39, Bharath Rupireddy wrote: On Thu, Jul 20, 2023 at 2:59 PM Michael Paquier wrote: On Thu, Jul 20, 2023 at 05:54:55PM +0900, Masahiro Ikeda wrote: > Yes, you're right. When I tried using worker_spi to test wait event, > I found the behavior. And thanks a lot fo

Re: Support worker_spi to execute the function dynamically.

2023-07-20 Thread Masahiro Ikeda
LOG: background worker "worker_spi" (PID 853621) exited with exit code 1 ``` It's better to remove "shared_preload_libraries = worker_spi" from the test configuration. I misunderstood that two background workers would be launched and waiting at the start of the test. Regar

Re: Support worker_spi to execute the function dynamically.

2023-07-20 Thread Masahiro Ikeda
On 2023-07-20 12:55, Michael Paquier wrote: On Thu, Jul 20, 2023 at 11:15:51AM +0900, Masahiro Ikeda wrote: While I'm working on the thread[1], I found that the function of worker_spi module fails if 'shared_preload_libraries' doesn't have worker_spi. I guess that you were patching worker_spi

Support worker_spi to execute the function dynamically.

2023-07-19 Thread Masahiro Ikeda
ge-id/flat/b9f5411acda0cf15c8fbb767702ff43e%40oss.nttdata.com Regards, -- Masahiro Ikeda NTT DATA CORPORATIONFrom c6e60c66c4066b4a01981ffae5a168901e7283eb Mon Sep 17 00:00:00 2001 From: Masahiro Ikeda Date: Thu, 20 Jul 2023 10:34:50 +0900 Subject: [PATCH] Support worker_spi to execute the function d

Re: Support to define custom wait events for extensions

2023-07-19 Thread Masahiro Ikeda
On 2023-07-19 12:52, Masahiro Ikeda wrote: Hi, I updated the patches. * v6-0001-Support-custom-wait-events-for-extensions.patch I updated the patch since the cfbot found a bug. * v7-0001-Support-custom-wait-events-for-extensions.patch Regards, -- Masahiro Ikeda NTT DATA CORPORATIONFrom

Re: Support to define custom wait events for extensions

2023-07-18 Thread Masahiro Ikeda
r the APIs are committed. The example mentioned in docs should be updated to the contrib modules codes, not the test module. Regards, -- Masahiro Ikeda NTT DATA CORPORATIONFrom 9c5c1f1f4f0be87cc06c7127396e7d228b665b8a Mon Sep 17 00:00:00 2001 From: Masahiro Ikeda Date: Wed, 19 Jul 2023 12:2

Re: Support to define custom wait events for extensions

2023-07-12 Thread Masahiro Ikeda
On 2023-07-13 09:12, Michael Paquier wrote: On Wed, Jul 12, 2023 at 05:46:31PM +0900, Michael Paquier wrote: On Wed, Jul 12, 2023 at 04:52:38PM +0900, Masahiro Ikeda wrote: If the behavior is unexpected, we need to change the current code. I have created a patch for the areas that I felt

Re: Support to define custom wait events for extensions

2023-07-12 Thread Masahiro Ikeda
it is not possible to connect to another backend and check the wait events on pg_stat_activity view. Please let me know if my understanding is wrong. Regards, -- Masahiro Ikeda NTT DATA CORPORATION

Re: Support to define custom wait events for extensions

2023-07-12 Thread Masahiro Ikeda
On 2023-07-12 02:39, Tristan Partin wrote: From bf06b8100cb747031959fe81a2d19baabc4838cf Mon Sep 17 00:00:00 2001 From: Masahiro Ikeda Date: Fri, 16 Jun 2023 11:53:29 +0900 Subject: [PATCH 1/2] Support custom wait events for extensions. + * This is indexed by event ID minus

Re: Support to define custom wait events for extensions

2023-07-12 Thread Masahiro Ikeda
On 2023-07-11 16:45, Michael Paquier wrote: On Fri, Jun 23, 2023 at 05:56:26PM +0900, Masahiro Ikeda wrote: I updated the patches to handle the warning mentioned by PostgreSQL Patch Tester, and removed unnecessary spaces. I have begun hacking on that, and the API layer inspired from

Re: PG 16 draft release notes ready

2023-06-30 Thread Masahiro Ikeda
, the feature is reverted with the commit. * https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=2535c74b1a6190cc42e13f6b6b55d94bff4b7dd6 Regards, -- Masahiro Ikeda NTT DATA CORPORATION

Re: Support to define custom wait events for extensions

2023-06-23 Thread Masahiro Ikeda
Hi, I updated the patches to handle the warning mentioned by PostgreSQL Patch Tester, and removed unnecessary spaces. Regards, -- Masahiro Ikeda NTT DATA CORPORATIONFrom 1bb78fa2cbe6b030cea7a570bec88bd4d68f314a Mon Sep 17 00:00:00 2001 From: Masahiro Ikeda Date: Fri, 23 Jun 2023 17:38:38 +0900

Re: Support to define custom wait events for extensions

2023-06-21 Thread Masahiro Ikeda
On 2023-06-20 18:26, Masahiro Ikeda wrote: The followings are TODO items. * to check that meson.build works since I tested with old command `make` now I test with meson and I updated the patches to work with it. My test procedure is the following. ``` export builddir=/mnt/tmp/build export

Re: Support to define custom wait events for extensions

2023-06-20 Thread Masahiro Ikeda
wait events for existing contrib modules (ex. postgres_fdw) Regards, -- Masahiro Ikeda NTT DATA CORPORATIONFrom 206ce9e1d74d1449b50cfc765936172de98687e8 Mon Sep 17 00:00:00 2001 From: Masahiro Ikeda Date: Tue, 20 Jun 2023 17:59:34 +0900 Subject: [PATCH 2/2] Add test codes for custom wait events

Re: Support to define custom wait events for extensions

2023-06-16 Thread Masahiro Ikeda
On 2023/06/17 1:16, Tristan Partin wrote: > I will take a look at your V2 when it is ready! I will also pass along > that this is wanted by Neon customers :). Thanks! Regards, -- Masahiro Ikeda NTT DATA CORPORATION

Re: Support to define custom wait events for extensions

2023-06-16 Thread Masahiro Ikeda
On 2023-06-16 16:46, Michael Paquier wrote: On Fri, Jun 16, 2023 at 11:14:05AM +0900, Masahiro Ikeda wrote: I tried to query on pg_stat_activity to check the background worker invoked by pg_prewarm. But, I found that pg_stat_activity doesn't show it although I may be missing something... So, I

Re: Support to define custom wait events for extensions

2023-06-15 Thread Masahiro Ikeda
. As a Neon user, I also feel the feature is important. Same as you. Thanks to Michael and Drouvot, I got to know the word tranche and the related existing code. From 59a118402e5e59685fb9e0fb086872e25a405736 Mon Sep 17 00:00:00 2001 From: Masahiro Ikeda Date: Thu, 15 Jun 2023 12:57:29 +0900

Re: Support to define custom wait events for extensions

2023-06-15 Thread Masahiro Ikeda
On 2023-06-15 22:21, Drouvot, Bertrand wrote: Hi, On 6/15/23 10:00 AM, Michael Paquier wrote: On Thu, Jun 15, 2023 at 03:06:01PM +0900, Masahiro Ikeda wrote: Currently, only one PG_WAIT_EXTENSION event can be used as a wait event for extensions. Therefore, in environments with multiple

Re: Support to define custom wait events for extensions

2023-06-15 Thread Masahiro Ikeda
Thanks for replying and your kind advice! On 2023-06-15 17:00, Michael Paquier wrote: On Thu, Jun 15, 2023 at 03:06:01PM +0900, Masahiro Ikeda wrote: In the core, the requested wait events are dynamically registered in shared memory. The extension then obtains the wait event information

Support to define custom wait events for extensions

2023-06-15 Thread Masahiro Ikeda
add regression code (but, it seems to be difficult) * others? (Please let me know) [1] https://www.postgresql.org/message-id/81290a48-b25c-22a5-31a6-3feff5864fe3%40gmail.com Regards, -- Masahiro Ikeda NTT DATA CORPORATIONFrom 59a118402e5e59685fb9e0fb086872e25a405736 Mon Sep 17 00:00:00 2001 From: Masah

Re: Introduce WAIT_EVENT_EXTENSION and WAIT_EVENT_BUFFER_PIN

2023-06-09 Thread Masahiro Ikeda
Hi, On 2023-06-09 13:26, Drouvot, Bertrand wrote: Hi, On 6/9/23 1:15 AM, Michael Paquier wrote: On Thu, Jun 08, 2023 at 10:57:55AM +0900, Masahiro Ikeda wrote: (Excuse me for cutting in, and this is not directly related to the thread.) +1. I'm interested in the feature. This is just

Re: Introduce WAIT_EVENT_EXTENSION and WAIT_EVENT_BUFFER_PIN

2023-06-07 Thread Masahiro Ikeda
able be useful for other users. IMO, at least, it's better to improve the specification that "Extension" wait event type has only the "Extension" wait event. [1] https://github.com/neondatabase/neon Regards, -- Masahiro Ikeda NTT DATA CORPORATION

Re: LogicalChanges* and LogicalSubxact* wait events are never reported

2021-10-21 Thread Masahiro Ikeda
On 2021/10/21 17:40, Amit Kapila wrote: > On Wed, Oct 20, 2021 at 3:46 PM Masahiro Ikeda > wrote: >> >> On 2021/10/20 18:17, Amit Kapila wrote: >>> On Wed, Oct 20, 2021 at 10:50 AM Michael Paquier >>> wrote: >>>> >>>> On

Re: LogicalChanges* and LogicalSubxact* wait events are never reported

2021-10-20 Thread Masahiro Ikeda
On 2021/10/20 18:17, Amit Kapila wrote: > On Wed, Oct 20, 2021 at 10:50 AM Michael Paquier wrote: >> >> On Wed, Oct 20, 2021 at 02:12:20PM +0900, Masahiro Ikeda wrote: >>> If my understanding is right, it's better to remove them since they make >>> users confus

LogicalChanges* and LogicalSubxact* wait events are never reported

2021-10-19 Thread Masahiro Ikeda
the attached patch. I confirmed that to make check-world passes all tests. [1] https://www.postgresql.org/message-id/CAA4eK1JV37jXUT5LeWzkBDNNnSntwQbLUZAj6m82QMiR1ZuuHQ%40mail.gmail.com Regards, -- Masahiro Ikeda NTT DATA CORPORATION diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml

Re: Allow escape in application_name

2021-09-06 Thread Masahiro Ikeda
On 2021/09/06 21:28, Fujii Masao wrote: > > > On 2021/09/06 16:48, Masahiro Ikeda wrote: >> On 2021-09-06 13:21, kuroda.hay...@fujitsu.com wrote: >>> Dear Ikeda-san, >>> >>>> I agree that this feature is useful. Thanks for working this. &g

Re: Allow escape in application_name

2021-09-06 Thread Masahiro Ikeda
1) is enough for the first step. If I have any time, I'll investigate it too. Regards, -- Masahiro Ikeda NTT DATA CORPORATION

Re: Allow escape in application_name

2021-09-05 Thread Masahiro Ikeda
cation_name and make it accept Non-ASCII. As a user, 3) is best for me. But it seems be out of scope this threads, so will you select 1)? Regards, -- Masahiro Ikeda NTT DATA CORPORATION

Re: RFC: Logging plan of the running query

2021-07-13 Thread Masahiro Ikeda
, the planning buffer usage is not so much useful. But, since the overhead to output buffer usages is not high and it's used for debugging use cases, I wonder it's not a bad idea to output buffer usages too. Thought? Regards, -- Masahiro Ikeda NTT DATA CORPORATION

Re: Fix comments of heap_prune_chain()

2021-07-13 Thread Masahiro Ikeda
could send an email with a new patch. So, please check it. https://www.postgresql.org/message-id/1aa07e2a-b715-5649-6c62-4fff96304d18%40oss.nttdata.com Regards, -- Masahiro Ikeda NTT DATA CORPORATION

Re: Fix comments of heap_prune_chain()

2021-07-13 Thread Masahiro Ikeda
essage instead. Thanks for your comments. I agree your suggestions. I also updated prstate->vistest to heap_prune_satisfies_vacuum of v1 patch because heap_prune_satisfies_vacuum() tests with not only prstate->vistest but also prstate->old_snap_xmin. I think it's more accurate repre

Re: Transactions involving multiple postgres foreign servers, take 2

2021-07-05 Thread Masahiro Ikeda
On 2021/06/30 10:05, Masahiko Sawada wrote: > On Fri, Jun 25, 2021 at 9:53 AM Masahiro Ikeda > wrote: >> >> Hi Jamison-san, sawada-san, >> >> Thanks for testing! >> >> FWIF, I tested using pgbench with "--rate=" option to know the serve

Re: Transactions involving multiple postgres foreign servers, take 2

2021-06-24 Thread Masahiro Ikeda
eSQL for Data Intensive Applications >From 12:27 says that how to solve unresolved prepared xacts. https://www.youtube.com/watch?v=AlF4C60FdlQ=PL3xUNnH4TdbsfndCMn02BqAAgGB0z7cwq Regards, -- Masahiro Ikeda NTT DATA CORPORATION

Re: Transactions involving multiple postgres foreign servers, take 2

2021-06-24 Thread Masahiro Ikeda
repared transactions cannot keep > up. Hi Jamison-san, sawada-san, Thanks for testing! FWIF, I tested using pgbench with "--rate=" option to know the server can execute transactions with stable throughput. As sawada-san said, the latest patch resolved second phase of 2PC asynchronously. So, it's difficult to control the stable throughput without "--rate=" option. I also worried what I should do when the error happened because to increase "max_prepared_foreign_transaction" doesn't work. Since too overloading may show the error, is it better to add the case to the HINT message? BTW, if sawada-san already develop to run the resolver processes in parallel, why don't you measure performance improvement? Although Robert-san, Tunakawa-san and so on are discussing what architecture is best, one discussion point is that there is a performance risk if adopting asynchronous approach. If we have promising solutions, I think we can make the discussion forward. In my understanding, there are three improvement idea. First is that to make the resolver processes run in parallel. Second is that to send "COMMIT/ABORT PREPARED" remote servers in bulk. Third is to stop syncing the WAL remove_fdwxact() after resolving is done, which I addressed in the mail sent at June 3rd, 13:56. Since third idea is not yet discussed, there may be my misunderstanding. -- Masahiro Ikeda NTT DATA CORPORATION

Re: Transactions involving multiple postgres foreign servers, take 2

2021-06-02 Thread Masahiro Ikeda
On 2021/05/25 21:59, Masahiko Sawada wrote: > On Fri, May 21, 2021 at 5:48 PM Masahiro Ikeda > wrote: >> >> On 2021/05/21 13:45, Masahiko Sawada wrote: >>> >>> Yes. We also might need to be careful about the order of foreign >>> transaction r

Re: Transactions involving multiple postgres foreign servers, take 2

2021-05-21 Thread Masahiro Ikeda
On 2021/05/21 13:45, Masahiko Sawada wrote: > On Fri, May 21, 2021 at 12:45 PM Masahiro Ikeda > wrote: >> >> >> >> On 2021/05/21 10:39, Masahiko Sawada wrote: >>> On Thu, May 20, 2021 at 1:26 PM Masahiro Ikeda >>> wrote: >>>> >

Re: Transactions involving multiple postgres foreign servers, take 2

2021-05-20 Thread Masahiro Ikeda
On 2021/05/21 10:39, Masahiko Sawada wrote: > On Thu, May 20, 2021 at 1:26 PM Masahiro Ikeda > wrote: >> >> >> On 2021/05/11 13:37, Masahiko Sawada wrote: >>> I've attached the updated patches that incorporated comments from >>> Zhihong and Ikeda-

Re: Transactions involving multiple postgres foreign servers, take 2

2021-05-19 Thread Masahiro Ikeda
pant/ - s/Foreign transactions involved in the current transaction/Foreign transaction participants involved in the current transaction/ Regards, -- Masahiro Ikeda NTT DATA CORPORATION

Re: wal stats questions

2021-05-19 Thread Masahiro Ikeda
On 2021/05/19 11:40, Fujii Masao wrote: > Thanks for updating the patch! I modified some comments slightly and > pushed that version of the patch. Thanks a lot! Regards, -- Masahiro Ikeda NTT DATA CORPORATION

Re: wal stats questions

2021-05-17 Thread Masahiro Ikeda
On 2021/05/17 22:34, Fujii Masao wrote: > > > On 2021/05/17 16:39, Masahiro Ikeda wrote: >> >> Thanks for your comments! >> >>>> + * is executed, wal records aren't nomally generated (although HOT >>>> makes >>> >>> n

Re: wal stats questions

2021-05-17 Thread Masahiro Ikeda
write/sync the WAL although it doesn't > > You use both 'wal' and 'WAL' in the comments, but are there any intension? No, I changed to use 'WAL' only. Although some other comments have 'wal' and 'WAL', it seems that 'WAL' is often used. Regards, -- Masahiro Ikeda NTT DATA CORPORATION diff --git a/

Re: wal stats questions

2021-05-12 Thread Masahiro Ikeda
On 2021/05/12 19:19, Fujii Masao wrote: > > > On 2021/05/11 18:46, Masahiro Ikeda wrote: >> >> >> On 2021/05/11 16:44, Fujii Masao wrote: >>> >>> >>> On 2021/04/28 9:10, Masahiro Ikeda wrote: >>>> >>>> >>>&g

Re: wal stats questions

2021-05-11 Thread Masahiro Ikeda
On 2021/05/11 16:44, Fujii Masao wrote: > > > On 2021/04/28 9:10, Masahiro Ikeda wrote: >> >> >> On 2021/04/27 21:56, Fujii Masao wrote: >>> >>> >>> On 2021/04/26 10:11, Masahiro Ikeda wrote: >>>> >>>> Fir

Re: wal stats questions

2021-04-27 Thread Masahiro Ikeda
On 2021/04/27 21:56, Fujii Masao wrote: > > > On 2021/04/26 10:11, Masahiro Ikeda wrote: >> >> First patch has only the changes for pg_stat_wal view. >> ("v6-0001-performance-improvements-of-reporting-wal-stats-without-introducing-a-new-variable.patch") &g

Re: Transactions involving multiple postgres foreign servers, take 2

2021-04-26 Thread Masahiro Ikeda
y. I think it's better since this is simpler than A. If I'm missing something, please let me know. Regards, -- Masahiro Ikeda NTT DATA CORPORATION

Re: wal stats questions

2021-04-25 Thread Masahiro Ikeda
On 2021/04/23 16:30, Fujii Masao wrote: > > > On 2021/04/23 10:25, Andres Freund wrote: >> Hi, >> >> On 2021-04-23 09:26:17 +0900, Masahiro Ikeda wrote: >>> On 2021/04/23 0:36, Andres Freund wrote: >>>> On Thu, Apr 22, 2021, at 06:42, Fujii

Re: wal stats questions

2021-04-22 Thread Masahiro Ikeda
 On 2021/04/23 0:36, Andres Freund wrote: > Hi > > On Thu, Apr 22, 2021, at 06:42, Fujii Masao wrote: >> On 2021/04/21 18:31, Masahiro Ikeda wrote: >>>> BTW, is it better to change PgStat_Counter from int64 to uint64 because> >>>> there aren't any c

Re: wal stats questions

2021-04-21 Thread Masahiro Ikeda
On 2021/04/21 15:08, torikoshia wrote: > On 2021-04-16 10:27, Masahiro Ikeda wrote: >> On 2021/04/13 9:33, Fujii Masao wrote: >>> >>> >>> On 2021/03/30 20:37, Masahiro Ikeda wrote: >>>> OK, I added the condition to the fast-return check. I not

Re: wal stats questions

2021-04-15 Thread Masahiro Ikeda
On 2021/04/13 9:33, Fujii Masao wrote: > > > On 2021/03/30 20:37, Masahiro Ikeda wrote: >> OK, I added the condition to the fast-return check. I noticed that I >> misunderstood that the purpose is to avoid expanding a clock check using WAL >> stats counters.

Re: Why reset pgstat during recovery

2021-04-05 Thread Masahiro Ikeda
erated as a one-time snapshot. This means that the stats counters saved at last may not be valid for the specific point in time. FWIW, there was a related discussion([1]) although the behavior is not changed yet. [1] https://www.postgresql.org/message-id/1416.1479760254%40sss.pgh.pa.us Regards, --

Re: wal stats questions

2021-03-30 Thread Masahiro Ikeda
On 2021/03/30 17:28, Kyotaro Horiguchi wrote: > At Tue, 30 Mar 2021 09:41:24 +0900, Masahiro Ikeda > wrote in >> I update the patch since there were my misunderstanding points. >> >> On 2021/03/26 16:20, Masahiro Ikeda wrote: >>> Thanks for many your suggestion

Re: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-29 Thread Masahiro Ikeda
On 2021/03/27 2:14, Andres Freund wrote: > Hi, > > On 2021-03-26 09:27:19 +0900, Masahiro Ikeda wrote: >> On 2021/03/25 19:48, Fujii Masao wrote: >>> Yes. So we should wait for the shared memory stats patch to be >>> committed before working on walreceiver stat

Re: wal stats questions

2021-03-29 Thread Masahiro Ikeda
I update the patch since there were my misunderstanding points. On 2021/03/26 16:20, Masahiro Ikeda wrote: > Thanks for many your suggestions! > I made the patch to handle the issues. > >> 1) What is the motivation to have both prevWalUsage and pgWalUsage, >>instead

Re: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-26 Thread Masahiro Ikeda
On 2021/03/26 9:54, Fujii Masao wrote: > On 2021/03/26 9:25, Masahiro Ikeda wrote: >> On 2021/03/25 21:23, Fujii Masao wrote: >>> On 2021/03/25 9:58, Andres Freund wrote: >>>> Also, won't this lead to postmaster now starting to complain about >>>> pgst

Re: wal stats questions

2021-03-26 Thread Masahiro Ikeda
e let me know. > Maybe there is the case where a backend generates no WAL records, > but just writes them because it needs to do write-ahead-logging > when flush the table data? > Ugh! I was missing a very large blob.. Ok, we need additional check > for non-pgWalUsage part. Sorry. R

Re: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-25 Thread Masahiro Ikeda
On 2021/03/25 19:48, Fujii Masao wrote: > > > On 2021/03/25 9:31, Masahiro Ikeda wrote: >> >> >> On 2021/03/24 18:36, Fujii Masao wrote: >>> >>> >>> On 2021/03/24 3:51, Andres Freund wrote: >>>> Hi, >>>> >>

Re: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-25 Thread Masahiro Ikeda
ildCrash() is never invoked due to the exit of pgstat. My understanding of Andres-san's comment is that is it ok to show like the following log message? ``` LOG: statistics collector process (PID 64991) exited with exit code 1 ``` Surely, other processes don't output the log like it. So, I agree to suppress the log message. FWIW, in immediate shutdown case, since pmdie() sets "pmState" variable to "PM_WAIT_BACKENDS", pgstat_start() won't be invoked. Regards, -- Masahiro Ikeda NTT DATA CORPORATION

Re: About to add WAL write/fsync statistics to pg_stat_wal view

2021-03-24 Thread Masahiro Ikeda
ause to collect stats in shared memory is very useful feature for users and it make a big change in design. So, I think it's beneficial to make an effort to move the shared memory stats thread forward (by reviewing or testing) instead of handling the issues in this thread. [1] https://www.postgresql.or

Re: wal stats questions

2021-03-24 Thread Masahiro Ikeda
_report_wal()? I think it's unnecessary because pgstat_report_stat() is already checked it. > Generally the various patches seems to to have a lot of the boilerplate > style comments (like "Prepare and send the message"), but very little in > the way of explaining the design. Sorry for that. I'll be careful. Regards, -- Masahiro Ikeda NTT DATA CORPORATION

Re: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-24 Thread Masahiro Ikeda
g the patch! I checked your patch and I don't have any comments. Regards, -- Masahiro Ikeda NTT DATA CORPORATION

Re: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-24 Thread Masahiro Ikeda
te it. [1] https://github.com/anarazel/postgres/compare/master...shmstat-before-split-2021-03-22 Regards, -- Masahiro Ikeda NTT DATA CORPORATION

Re: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-22 Thread Masahiro Ikeda
On 2021/03/23 11:40, Fujii Masao wrote: > > > On 2021/03/23 9:05, Masahiro Ikeda wrote: >> Yes. I attached the v5 patch based on v3 patch. >> I renamed SignalHandlerForUnsafeExit() and fixed the following comment. > > Thanks for updating the patch! > > When

Re: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-22 Thread Masahiro Ikeda
On 2021/03/22 23:59, Fujii Masao wrote: > > > On 2021/03/20 13:40, Masahiro Ikeda wrote: >> Sorry, there is no evidence we should call it. >> I thought that to call FreeWaitEventSet() is a manner after using >> CreateWaitEventSet() and the performance impact to call

Re: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-19 Thread Masahiro Ikeda
tchSupport(). Those wishing to disconnect from it should call ShutdownLatchSupport(). ``` Regards, -- Masahiro Ikeda NTT DATA CORPORATION

Re: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-18 Thread Masahiro Ikeda
EventSet() in the handler for the stats collector. Regards, -- Masahiro Ikeda NTT DATA CORPORATIONdiff --git a/src/backend/postmaster/interrupt.c b/src/backend/postmaster/interrupt.c index dd9136a942..2cc5a39ec5 100644 --- a/src/backend/postmaster/interrupt.c +++ b/src/backe

Re: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-16 Thread Masahiro Ikeda
quot;, statfile); unlink(statfile) } ``` I fixed it in the same way instead of checking the return value because IIUC, it's unimportant if an error has occurred. The log shows that just to try removing it. Though? Regards, -- Masahiro Ikeda NTT DATA CORPORATIONdiff --git a/src/back

RE: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-16 Thread Masahiro Ikeda
comments might be removed. How do you think? Hi, Kuroda-san. Thanks for your comments. I agreed that your idea. I combined them into one function and moved the comments to the calling function side. (v2-0001-pgstat_avoid_writing_on_sigquit.patch) Regards, -- Masahiro Ikeda NTT DATA CORPORATIONdiff

make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-15 Thread Masahiro Ikeda
://www.postgresql.org/message-id/flat/0A3221C70F24FB45833433255569204D1F5EF25A%40G01JPEXMBYT05 Regards, -- Masahiro Ikeda NTT DATA CORPORATIONdiff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c index b1e2d94951..03d191dfe6 100644 --- a/src/backend/postmaster/pgstat.c +++ b/src/backend

Re: About to add WAL write/fsync statistics to pg_stat_wal view

2021-03-14 Thread Masahiro Ikeda
On 2021/03/11 21:29, Masahiro Ikeda wrote: On 2021-03-11 11:52, Fujii Masao wrote: On 2021/03/11 9:38, Masahiro Ikeda wrote: On 2021-03-10 17:08, Fujii Masao wrote: IIUC the stats collector basically exits after checkpointer and walwriter exit. But there seems no guarantee that the stats

  1   2   >