Re: Refactor calculations to use instr_time

2023-02-19 Thread Kyotaro Horiguchi
omething more meaningful, like wal_usage_diff, WalUsageDiff or PendingWalUsage? Along those same lines, it occurs to me that the new struct should be named PgStat_PendingWalStats, instead of ..Usage. That change makes the name of the type and the variable consistent. regards. -- Kyotaro Horig

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-16 Thread Kyotaro Horiguchi
he keepalive packets from the peer. Also, publisher can respond to the > peer. > Could you please tell me if you know a case that publisher or subscriber > cannot > respond to the opposite side? Note that if we apply the publisher-side patch, > we > don't have to apply subscriber-side patch. Sorry about that again, I missed that part in the last patch as mentioned earlier.. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-15 Thread Kyotaro Horiguchi
that person should be aware of some additional subtle restrictions that are required compared to a non-delayed setups. > To delay the transaction in the output plugin layer, the new > LogicalOutputPlugin > API was added. For now, I choose the output plugin layer but can consider to > do > it from the core if there is a better way. > > Could you please share your opinion? > > Note: thanks for Osumi-san to help implementing. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2023-02-14 Thread Kyotaro Horiguchi
can happen. As for corss-views ambiguity, that is already present. Many columns in stats views share the same names with some other views. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Exit walsender before confirming remote flush in logical replication

2023-02-14 Thread Kyotaro Horiguchi
At Mon, 13 Feb 2023 17:13:43 -0800, Andres Freund wrote in > On 2023-02-14 10:05:40 +0900, Kyotaro Horiguchi wrote: > > What do you think about the need for explicitly specifying the > > default? I'm fine with specifying the default using a single word, > > such as

KeepLogSeg needs some fixes on behavior

2023-02-14 Thread Kyotaro Horiguchi
This is a derived thread form [1], that discusses some subtle behaviors of KeepLogSeg. 1: https://www.postgresql.org/message-id/20230213194131.hgzs6ropcvhda...@awork3.anarazel.de At Mon, 13 Feb 2023 11:41:31 -0800, Andres Freund wrote > Hi, > > On 2023-02-13 15:45:49 +0900, Kyotaro

Re: Use pg_pwritev_with_retry() instead of write() in dir_open_for_write() to avoid partial writes?

2023-02-14 Thread Kyotaro Horiguchi
rameters offset and size are in a different order from pwrite(fd, buf, count, offset). I perfer the arrangement suggested by Bharath. And isn't it better to use Min(remaining_size, BLCKSZ) instead of a bare if statement? regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Reconcile stats in find_tabstat_entry() and get rid of PgStat_BackendFunctionEntry

2023-02-14 Thread Kyotaro Horiguchi
ared tables. > * > * If no entry found, return NULL, don't create a new one The comment assumed that the function directly returns an entry from shared memory, but now it copies the entry's contents into a palloc'ed memory and stores the sums of some counters for the current transaction in it. Do you think we should update the comment to reflect this change? regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Reconcile stats in find_tabstat_entry() and get rid of PgStat_BackendFunctionEntry

2023-02-13 Thread Kyotaro Horiguchi
Thanks for the new version. At Mon, 13 Feb 2023 09:58:52 +0100, "Drouvot, Bertrand" wrote in > Hi, > > On 2/13/23 8:40 AM, Kyotaro Horiguchi wrote: > > At Mon, 13 Feb 2023 08:09:50 +0100, "Drouvot, Bertrand" > > wrote in > >>> I thi

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-13 Thread Kyotaro Horiguchi
ile delaying. It is not possible as far as we choke the stream at the subscriber end. It doesn't seem to be a practical problem, but IMHO I think he's right in terms of adherence to the wire protocol, which was also one of my own initial concern. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Fix GUC_NO_SHOW_ALL test scenario in 003_check_guc.pl

2023-02-13 Thread Kyotaro Horiguchi
At Mon, 13 Feb 2023 12:18:07 +0900, Michael Paquier wrote in > On Mon, Feb 13, 2023 at 11:27:58AM +0900, Kyotaro Horiguchi wrote: > > I think currently the output by --describe-config can be used only for > > consulting while editing a (possiblly broken) config file. Thus I

Re: Exit walsender before confirming remote flush in logical replication

2023-02-13 Thread Kyotaro Horiguchi
At Mon, 13 Feb 2023 08:27:01 +0530, Amit Kapila wrote in > On Mon, Feb 13, 2023 at 7:26 AM Kyotaro Horiguchi > wrote: > > > > IMHO I vaguely don't like that we lose a means to specify the default > > behavior here. And I'm not sure we definitely don't need other than

Re: Use pg_pwritev_with_retry() instead of write() in dir_open_for_write() to avoid partial writes?

2023-02-13 Thread Kyotaro Horiguchi
makes the iovec static variable > and points the zero buffer only once/for the first time to iovec. This > avoids for-loop on every call. As the patch itself, it seems forgetting to reset iov[0].iov_len after writing a partial block. retards. *1: https://gcc.gnu.org/onlinedocs/gcc-4.1.2/gcc/Designated-Inits.html -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Reconcile stats in find_tabstat_entry() and get rid of PgStat_BackendFunctionEntry

2023-02-12 Thread Kyotaro Horiguchi
't bother anyone, I don't mind to remove the flag. The patch will get far simpler by that. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Fix GUC_NO_SHOW_ALL test scenario in 003_check_guc.pl

2023-02-12 Thread Kyotaro Horiguchi
an on-session use. On the other hand, don't we need to remove the condition GUC_NOT_IN_SAMPLE from displayStruct? I think that help_config() should show a value if it is marked as !GUC_DISALLOW_IN_FILE even if it is GUC_NOT_IN_SAMPLE. I'm not sure whether there's any variable that are marked tha

Re: Exit walsender before confirming remote flush in logical replication

2023-02-12 Thread Kyotaro Horiguchi
ail.com IMHO I vaguely don't like that we lose a means to specify the default behavior here. And I'm not sure we definitely don't need other than flush and immedaite for both physical and logical replication. If it's not the case, I don't object to make it a Boolean. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-12 Thread Kyotaro Horiguchi
At Fri, 10 Feb 2023 10:34:49 +0530, Amit Kapila wrote in > On Fri, Feb 10, 2023 at 10:11 AM Amit Kapila wrote: > > > > On Fri, Feb 10, 2023 at 6:27 AM Kyotaro Horiguchi > > wrote: > > > We have suffered this kind of feedback silence many times. Thus I > >

Re: pg_usleep for multisecond delays

2023-02-09 Thread Kyotaro Horiguchi
e. And I think we are actully going to reduce or eliminate the use of pg_sleep(). regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Reconcile stats in find_tabstat_entry() and get rid of PgStat_BackendFunctionEntry

2023-02-09 Thread Kyotaro Horiguchi
callers but now that cost is imposed to the all callers along with additional palloc()/pfree() calls. That doesn't seem reasonable. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Is psSocketPoll doing the right thing?

2023-02-09 Thread Kyotaro Horiguchi
> Subject: is p*s*Socket.. Oops... At Thu, 9 Feb 2023 17:32:08 +0900, Fujii Masao wrote in > > > On 2023/02/09 11:50, Kyotaro Horiguchi wrote: > > Hello. > > While looking a patch, I found that pqSocketPoll passes through the > > result from poll(2) to the

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-09 Thread Kyotaro Horiguchi
Mmm. A part of the previous mail have gone anywhere for a uncertain reason and placed by a mysterious blank lines... At Fri, 10 Feb 2023 09:57:22 +0900 (JST), Kyotaro Horiguchi wrote in > At Thu, 9 Feb 2023 13:48:52 +0530, Amit Kapila > wrote in > > On Thu, Feb 9, 2023 at 10:4

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-09 Thread Kyotaro Horiguchi
At Thu, 9 Feb 2023 13:48:52 +0530, Amit Kapila wrote in > On Thu, Feb 9, 2023 at 10:45 AM Kyotaro Horiguchi > wrote: > > > > At Wed, 8 Feb 2023 09:03:03 +, "Hayato Kuroda (Fujitsu)" > > wrote in > > > Thank you for reviewing! PSA new version. &g

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-09 Thread Kyotaro Horiguchi
ation (which is doing WaitLatch anyway) is of insignificant > > concern, whereas having one assignment is simpler than having two IMO. > > > > Yeah, that sounds better to me as well. FWIW, I'm on board with this. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-08 Thread Kyotaro Horiguchi
us, it seems to me this may suppress feedbacks from being sent for an unexpectedly long time especially when min_apply_delay is shorter than wal_r_s_interval. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Is psSocketPoll doing the right thing?

2023-02-08 Thread Kyotaro Horiguchi
sockets, and returns 0 unless poll() itself failed to work. It doesn't seem to be the intended behavior since the function sets POLLERR to pollfd.events. (but the bit is ignored by poll(), though) Is the above diagnosis correct? regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: [Proposal] Add foreign-server health checks infrastructure

2023-02-08 Thread Kyotaro Horiguchi
&& !forRead && !forWrite, not causing side effects? regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2023-02-07 Thread Kyotaro Horiguchi
{}) > On 2023-01-25 16:56:17 +0900, Kyotaro Horiguchi wrote: > > At Tue, 24 Jan 2023 14:35:12 -0800, Andres Freund > > wrote in > > > > + write_chunk_s(fpout, ); > > > > + if (!read_chunk_s(fpin, >io.stats)) > > > > > &

Re: Exit walsender before confirming remote flush in logical replication

2023-02-07 Thread Kyotaro Horiguchi
prefer the following expression (Others may disagree:p): ((newsub->minapplydelay == 0) != (MySubscription->minapplydelay == 0)) And I think we need a comment for the term. For example, /* minapplydelay affects START_REPLICATION option exit_before_confirming */ + * Reads all entrly of the list and consume if needed. s/entrly/entries/ ? ... regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Make mesage at end-of-recovery less scary.

2023-02-06 Thread Kyotaro Horiguchi
following part of the + * record. */ static bool ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, I added a similar description to ValidXLogRecordHeader. regards. -- Kyotaro Horiguchi NTT Open Source Software Center >From c58ca4d5db52c75dec9882d158d5724e12617005 Mon Sep 17 00:00:00

Re: Where is the logig to create a table file?

2023-02-06 Thread Kyotaro Horiguchi
amed 1200 in the $PGDATA/base, So where > > is the logic code in the internal? > > > heapam_relation_set_new_filenode()->RelationCreateStorage() Or if you are searching for the logic to determin the file name, see GetNewRelFileNumber(). regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: In-placre persistance change of a relation

2023-02-06 Thread Kyotaro Horiguchi
thought on that line. Yes, indeed the marker files are a kind of undo log. Anyway, I'll split the current patch to two parts as suggested. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-06 Thread Kyotaro Horiguchi
k() unconditionally. -start_apply(XLogRecPtr origin_startpos) +start_apply(void) -LogicalRepApplyLoop(XLogRecPtr last_received) +LogicalRepApplyLoop(void) Does this patch requires this change? regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-06 Thread Kyotaro Horiguchi
nsidered this aspect. # "%d%s%s is outside the valid range for parameter \"%s\" (%d .. %d)" # also appears just once As for me, it seems to me a good practice to do that regadless of the number of duplicates to (semi)mechanically avoid duplicates. (But I believe I would do as Peter suggests by myself for the first cut, though:p) regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Exit walsender before confirming remote flush in logical replication

2023-02-01 Thread Kyotaro Horiguchi
uch troubles, as far as replication consistently is maintained. IMHO that is also true for the logical-delayed-replication case. > [1] - https://www.postgresql.org/docs/devel/app-pg-ctl.html regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-01 Thread Kyotaro Horiguchi
At Wed, 1 Feb 2023 08:38:11 +0530, Amit Kapila wrote in > On Wed, Feb 1, 2023 at 8:13 AM Kyotaro Horiguchi > wrote: > > > > At Tue, 31 Jan 2023 15:12:14 +0530, Amit Kapila > > wrote in > > > So, shall we check if the result of parse_int is in the range 0 and

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-31 Thread Kyotaro Horiguchi
At Tue, 31 Jan 2023 15:12:14 +0530, Amit Kapila wrote in > On Tue, Jan 31, 2023 at 1:40 PM Kyotaro Horiguchi > wrote: > > > > Hi, Kuroda-san, Thanks for the detailed study. > > > > At Tue, 31 Jan 2023 07:06:40 +, "Hayato Kuroda (Fujitsu)" > >

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-31 Thread Kyotaro Horiguchi
se, but the code will get uselessly complex without that assumption:p) regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-30 Thread Kyotaro Horiguchi
At Mon, 30 Jan 2023 14:24:31 +0530, Amit Kapila wrote in > On Mon, Jan 30, 2023 at 12:38 PM Kyotaro Horiguchi > wrote: > > > > At Mon, 30 Jan 2023 11:56:33 +0530, Amit Kapila > > wrote in > > > #define TimestampTzPlusMilliseconds(tz,ms) ((tz) + ((ms) * (int

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-29 Thread Kyotaro Horiguchi
At Mon, 30 Jan 2023 11:56:33 +0530, Amit Kapila wrote in > On Mon, Jan 30, 2023 at 9:43 AM Kyotaro Horiguchi > wrote: > > > > At Mon, 30 Jan 2023 08:51:05 +0530, Amit Kapila > > wrote in > > > On Mon, Jan 30, 2023 at 8:32 AM Kyotaro Horiguchi > > >

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-29 Thread Kyotaro Horiguchi
At Mon, 30 Jan 2023 08:51:05 +0530, Amit Kapila wrote in > On Mon, Jan 30, 2023 at 8:32 AM Kyotaro Horiguchi > wrote: > > > > At Sat, 28 Jan 2023 04:28:29 +, "Takamichi Osumi (Fujitsu)" > > wrote in > > > On Friday, January 27, 2023 8:00 PM Amit

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-29 Thread Kyotaro Horiguchi
and I agree that int32 is sufficient since I don't think more than 49 days delay is practical. On the other hand, maybe I wouldn't want to use int32 for intermediate calculations. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-25 Thread Kyotaro Horiguchi
At Wed, 25 Jan 2023 12:30:19 +0530, Amit Kapila wrote in > On Wed, Jan 25, 2023 at 11:57 AM Kyotaro Horiguchi > wrote: > > > > At Tue, 24 Jan 2023 12:19:04 +, "Takamichi Osumi (Fujitsu)" > > wrote in > > > Attached the patch v2

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2023-01-24 Thread Kyotaro Horiguchi
gt; be op_blocksize or just block_size? > > > > + b.io_object, > > + b.io_context, > > No, block wouldn't be helpful - we'd like to use this for something that isn't > uniform blocks. What does the field show in that case? The mean of operation size? Or one row per opration size? If the former, the name looks somewhat wrong. If the latter, block_size seems making sense. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: wake up logical workers after ALTER SUBSCRIPTION

2023-01-24 Thread Kyotaro Horiguchi
l the existing > uses of InvalidDsaPointer and DSM_HANDLE_INVALID and didn't notice anything > else that needed adjusting. There seems to be two cases for DSA_HANDLE_INVALID in dsa_get_handle and dsa_attach_in_place, one of which is Assert(), though. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-24 Thread Kyotaro Horiguchi
CONNECTION 'host=192.0.2.4 port=5432 user=foo dbname=foodb' +PUBLICATION mypub + WITH (copy_data = false, min_apply_delay = '4h'); + I'm not sure we need this additional example. We already have two exmaples one of which differs from the above only by actual values for PUBLICATION and WITH clauses. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-24 Thread Kyotaro Horiguchi
t_applied < end_lsn && > AllTablesyncsReady()) > + last_applied = > end_lsn; > ``` Maybe, the name "last_applied" made you confused. As I mentioned in another message, the variable points to the remote LSN of last "processed" 'w/k' messages. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-24 Thread Kyotaro Horiguchi
At Tue, 24 Jan 2023 14:22:19 +0530, Amit Kapila wrote in > On Tue, Jan 24, 2023 at 12:44 PM Peter Smith wrote: > > > > On Tue, Jan 24, 2023 at 5:58 PM Amit Kapila wrote: > > > > > > On Tue, Jan 24, 2023 at 8:15 AM Kyotaro Horiguchi > > > wrote: &g

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-24 Thread Kyotaro Horiguchi
y look at the both two varaibles instead. > > > > I think this is true without this patch, so why that has not been > followed in the first place? One comment, I see in this regard is as > below: > > /* It's legal to not pass a recvpos */ > if (recvpos < last_recvpos) > recvpos = last_recvpos; Sorry. I don't understand this. It is just a part of the ratchet mechanism for the last received lsn to report. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-24 Thread Kyotaro Horiguchi
place, I misunderstood maybe_start_skipping_changes(), which doesn't actually skip changes. So... sorry for the noise. For the record, I agree that the current order is right. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-24 Thread Kyotaro Horiguchi
At Tue, 24 Jan 2023 11:28:58 +0530, Amit Kapila wrote in > On Tue, Jan 24, 2023 at 6:17 AM Kyotaro Horiguchi > wrote: > > > > IMHO "foo > bar" is not an "option". I think we say "foo and bar are > > mutually exclusive options" but I th

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2023-01-24 Thread Kyotaro Horiguchi
At Tue, 24 Jan 2023 17:22:03 +0900 (JST), Kyotaro Horiguchi wrote in > +pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op) > +{ > + Assert(io_object < IOOBJECT_NUM_TYPES); > + Assert(io_context < IOCONTEXT_NUM_TYPES); > + Assert(i

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2023-01-24 Thread Kyotaro Horiguchi
I may be the only one who see the name as umbiguous between "total number of handled bytes" and "bytes hadled at an operation". Can't it be op_blocksize or just block_size? + b.io_object, + b.io_context, It's uncertain to me why only the two columns are prefixed b

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-23 Thread Kyotaro Horiguchi
Sorry, I forgot to write one comment. At Tue, 24 Jan 2023 11:45:35 +0900 (JST), Kyotaro Horiguchi wrote in + /* Should we delay the current transaction? */ + if (finish_ts) + maybe_delay_apply(xid, finish_ts); + if (!am_parallel_apply_worker

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-23 Thread Kyotaro Horiguchi
p; last_received == last_applied) The function is a static function and always called with a variable last_received that has the same scope with the function, as the first parameter. Thus we can remove the first parameter then let the function directly look at the both two varaibles instead. regards.

Re: Fix GetWALAvailability function code comments for WALAVAIL_REMOVED return value

2023-01-23 Thread Kyotaro Horiguchi
At Thu, 19 Jan 2023 18:43:52 -0500, Tom Lane wrote in > sirisha chamarthi writes: > > On Wed, Oct 19, 2022 at 7:59 PM Kyotaro Horiguchi > > wrote: > >> In short, the proposed fix alone seems fine to me. If we want to show > >> further deta

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-23 Thread Kyotaro Horiguchi
nd bar = y are.. options". I wrote a comment as "this should be more like human-speaking" and Euler seems having the same feeling for another error message. Concretely I would spell this as "min_apply_delay cannot be enabled when parallel streaming mode is enabled" or someth

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-15 Thread Kyotaro Horiguchi
At Tue, 10 Jan 2023 12:01:43 +0530, Amit Kapila wrote in > On Tue, Jan 10, 2023 at 11:16 AM Kyotaro Horiguchi > wrote: > > Although I don't see a technical difference between the two, all the > > other occurances including the just above (except test_shm_mq) use > > &qu

Re: Lazy allocation of pages required for verifying FPI consistency

2023-01-15 Thread Kyotaro Horiguchi
be said that that's not worth bothering. On the other hand I don't think it's great to waste 16kB * max_backends memory especially when it is clearly recognized and easily avoidable. I guess the reason for the code is more or less that. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Exit walsender before confirming remote flush in logical replication

2023-01-15 Thread Kyotaro Horiguchi
h.. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Exit walsender before confirming remote flush in logical replication

2023-01-15 Thread Kyotaro Horiguchi
r hand, logical recplication doesn't have a business with switchover so that assurarance is useless. Thus I think we can (practically) ignore apply_lsn at shutdown. It seems subtly irregular, though. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Lazy allocation of pages required for verifying FPI consistency

2023-01-12 Thread Kyotaro Horiguchi
> Thoughts? IMHO, it's a bit scaring to me to push down the execution stack by that large size. I tend to choose the (current) possible memory wasting only on startup process than digging stack deeply. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-11 Thread Kyotaro Horiguchi
too generic. Couldn't it be more specific? We need a explanation for the parameter in the function comment. + if (!am_parallel_apply_worker()) + { + Assert(ts > 0); + maybe_delay_apply(ts); It seems to me better that the if condition and assertion are checked inside maybe_delay_apply(). regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: mprove tab completion for ALTER EXTENSION ADD/DROP

2023-01-10 Thread Kyotaro Horiguchi
input. I didn't looked closer but maybe we need another way to do that. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: [PATCH] Add overlaps geometric operators that ignore point overlaps

2023-01-10 Thread Kyotaro Horiguchi
? > > If current behavior is correct, this todo item might need some update > (unless I missed anything) otherwise any suggestion is welcomed. I read the todo description as we may want *another set* of operators to do that, not to change the current behavior of the existing operators. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-09 Thread Kyotaro Horiguchi
E), -errmsg("unable to map dynamic shared memory segment"))); +errmsg("could not map dynamic shared memory segment"))); toc = shm_toc_attach(PG_TEST_SHM_MQ_MAGIC, dsm_segment_address(seg)); if (toc =

Re: ARRNELEMS Out-of-bounds possible errors

2022-12-23 Thread Kyotaro Horiguchi
At Fri, 23 Dec 2022 17:37:55 +0900 (JST), Kyotaro Horiguchi wrote in > At Thu, 22 Dec 2022 12:35:58 -0300, Ranier Vilela wrote > in > > Hi. > > > > Per Coverity. > > > > The commit ccff2d2 > > <https://github.com/postgres/postgres/co

Re: ARRNELEMS Out-of-bounds possible errors

2022-12-23 Thread Kyotaro Horiguchi
that function with a NULL escontext and the NULL turns ereturn() into ereport(). That doesn't seem to be changed by the commit. Of course teaching Coverity not to issue the false warnings would be another actual issue that we should do, maybe. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Force streaming every change in logical decoding

2022-12-22 Thread Kyotaro Horiguchi
parameters set by > CREATE SUBSCRIPTION) is enabled, otherwise, serialize each change. Looks fine. Thanks! regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Exit walsender before confirming remote flush in logical replication

2022-12-22 Thread Kyotaro Horiguchi
ng a kind of crafted feedback, which reports commit_data.end_lsn as flushpos? A little tweak is needed in send_feedback() but seems to work.. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Force streaming every change in logical decoding

2022-12-22 Thread Kyotaro Horiguchi
rmal' as Kuroda-san proposed, as the default value. Oh. I missed this, and agree to have the explicit default option. I slightly prefer "buffered" but "normal" also works fine for me. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Add LSN along with offset to error messages reported for WAL file read/write/validate header failures

2022-12-22 Thread Kyotaro Horiguchi
iated by appearance by readers, I think. (Yeah, we can identify that since the another parameter is identifiable alone.) Why don't we spell out the parameter simply as "segment number"? regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Force streaming every change in logical decoding

2022-12-21 Thread Kyotaro Horiguchi
y string, which is the default, +decoding will stream or serialize changes when +logical_decoding_work_mem is reached. With (really) fresh eyes, I took a bit long time to understand what the "streaming" option is. Couldn't we augment the description by a bit? regards. -- Kyotaro

Re: Inconsistency in reporting checkpointer stats

2022-12-21 Thread Kyotaro Horiguchi
gt; the main buffer pool. It sounds reasonable to exclude SRLU write from buffer writes. But I'm not sure its useful to count SLRU writes separately since it is under the noise level of buffer writes in reglular cases and the value doesn't lead to tuning. However I'm not strongly opposed to adding it either. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Inconsistency in reporting checkpointer stats

2022-12-21 Thread Kyotaro Horiguchi
At Mon, 19 Dec 2022 18:05:38 +0530, Bharath Rupireddy wrote in > On Fri, Dec 16, 2022 at 2:14 PM Kyotaro Horiguchi > wrote: > > > > In the first place I don't like that we count the same things twice. > > Couldn't we count the number only by any one of th

Re: (non) translatable string splicing

2022-12-20 Thread Kyotaro Horiguchi
> > clúster antiguo. > > I can't see how that could be mejor. :-) It was quite annoying but not untranslatable. But the "the" before "residen" looks like badly misplaced:p It should be a part of the inner text ("residen los.."). regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: (non) translatable string splicing

2022-12-20 Thread Kyotaro Horiguchi
blah" in that context. > Ugh. Perhaps we could just simplify the wordings as of "index on > blah", "index on OID %u", "TOAST table for blah" and "TOAST table for > OID %u" with newlines after each item? I'm fine with just removing " which ". but I don't understand about the extra newlines. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Inconsistency in reporting checkpointer stats

2022-12-16 Thread Kyotaro Horiguchi
ync will work fine. Assuming single process checkpointing there must be no problem to do that. (Anyway the current shared stats update for checkpointer is assuming single-process). Otherwise, in exchange with giving up the realtime nature, we can count the number only by CheckPointerStats.ckpt_

Re: plpgsq_plugin's stmt_end() is not called when an error is caught

2022-12-15 Thread Kyotaro Horiguchi
f, though. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: plpgsq_plugin's stmt_end() is not called when an error is caught

2022-12-14 Thread Kyotaro Horiguchi
than calling end callback at every nest level. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Time delayed LR (WAS Re: logical replication restrictions)

2022-12-14 Thread Kyotaro Horiguchi
At Thu, 15 Dec 2022 10:29:17 +0530, Amit Kapila wrote in > On Thu, Dec 15, 2022 at 10:11 AM Kyotaro Horiguchi > wrote: > > > > At Thu, 15 Dec 2022 09:18:55 +0530, Amit Kapila > > wrote in > > > On Thu, Dec 15, 2022 at 7:22 AM Kyotaro Horiguchi > > >

Re: Time delayed LR (WAS Re: logical replication restrictions)

2022-12-14 Thread Kyotaro Horiguchi
At Thu, 15 Dec 2022 09:18:55 +0530, Amit Kapila wrote in > On Thu, Dec 15, 2022 at 7:22 AM Kyotaro Horiguchi > wrote: > > > > At Wed, 14 Dec 2022 16:30:28 +0530, Amit Kapila > > wrote in > > > On Wed, Dec 14, 2022 at 4:16 PM Hayato Kuroda (Fujitsu) > &

Re: Time delayed LR (WAS Re: logical replication restrictions)

2022-12-14 Thread Kyotaro Horiguchi
At Thu, 15 Dec 2022 09:23:12 +0530, Amit Kapila wrote in > On Thu, Dec 15, 2022 at 7:16 AM Kyotaro Horiguchi > wrote: > > Allowing walsender to finish ignoring replication status > > wouldn't be great. > > > > Yes, that would be ideal. But do you know why that is

Re: Time delayed LR (WAS Re: logical replication restrictions)

2022-12-14 Thread Kyotaro Horiguchi
ill resolve many known issues including walsender's un-terminatability, possible buffer-full and status packet exchanging. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Time delayed LR (WAS Re: logical replication restrictions)

2022-12-14 Thread Kyotaro Horiguchi
> position > cannot be modified. I didn't look closer but the cause I guess is walsender doesn't die until all WAL has been sent, while logical delay chokes replication stream. Allowing walsender to finish ignoring replication status wouldn't be great. One idea is to let logical workers send delaying status. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: pg_upgrade: Make testing different transfer modes easier

2022-12-14 Thread Kyotaro Horiguchi
$oldnode->port, '-P', $newnode->port, > + $mode, > ], > > Minor nitpick, but while in there should we take the opportunity to add a > trailing comma on the other two array declarations which now ends with > --check? > It's good Perl practice and will make the code consistent. Otherwise look god to me. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Shortening the Scope of Critical Section in Creation of New MultiXacts

2022-12-13 Thread Kyotaro Horiguchi
on to the above, we cannot simply move the END_CRIT_SECTION() to there since RecordNewMultiXact() has another caller that doesn't start a critical section. By the way, I didn't tested this for myself but.. > This passes regression tests Maybe if you did the same with an assertion-build, you will get an

Re: Time delayed LR (WAS Re: logical replication restrictions)

2022-12-13 Thread Kyotaro Horiguchi
At Tue, 13 Dec 2022 17:05:35 +0530, Amit Kapila wrote in > On Tue, Dec 13, 2022 at 7:35 AM Kyotaro Horiguchi > wrote: > > > > At Mon, 12 Dec 2022 18:10:00 +0530, Amit Kapila > > wrote in > Yeah, I think ideally it will timeout but if we have a solution like >

Re: Time delayed LR (WAS Re: logical replication restrictions)

2022-12-12 Thread Kyotaro Horiguchi
At Tue, 13 Dec 2022 02:28:49 +, "Takamichi Osumi (Fujitsu)" wrote in > On Wednesday, December 7, 2022 12:00 PM Kyotaro Horiguchi > wrote: > > We couldn't reproduce this failure and > find the same type of failure on the cfbot from the past failures. > It s

Re: Time delayed LR (WAS Re: logical replication restrictions)

2022-12-12 Thread Kyotaro Horiguchi
ight need to adjust the wait_time in LogicalRepApplyLoop(). But, I'm not sure what will happen when walsender is blocked by buffer-full for a long time. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Time delayed LR (WAS Re: logical replication restrictions)

2022-12-12 Thread Kyotaro Horiguchi
Hello. At Mon, 12 Dec 2022 07:42:30 +, "Takamichi Osumi (Fujitsu)" wrote in > On Monday, December 12, 2022 2:54 PM Kyotaro Horiguchi > wrote: > > I asked about unexpected walsender termination caused by this feature but I > > think I didn't received an an

Re: Time delayed LR (WAS Re: logical replication restrictions)

2022-12-11 Thread Kyotaro Horiguchi
22-12-12 14:17:18.139 JST LOG: terminating walsender process due to > replication timeout > 2022-12-12 14:18:11.076 JST LOG: starting logical decoding for slot "s1" ... regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Improve WALRead() to suck data directly from WAL buffers when possible

2022-12-11 Thread Kyotaro Horiguchi
Sorry for the confusion. At Mon, 12 Dec 2022 12:06:36 +0900 (JST), Kyotaro Horiguchi wrote in > At Mon, 12 Dec 2022 11:57:17 +0900 (JST), Kyotaro Horiguchi > wrote in > > This patch copies the bleeding edge WAL page without recording the > > (next) insertion point nor ch

Re: Improve WALRead() to suck data directly from WAL buffers when possible

2022-12-11 Thread Kyotaro Horiguchi
At Mon, 12 Dec 2022 11:57:17 +0900 (JST), Kyotaro Horiguchi wrote in > This patch copies the bleeding edge WAL page without recording the > (next) insertion point nor checking whether all in-progress insertion > behind the target LSN have finished. Thus the copied page may hav

Re: Improve WALRead() to suck data directly from WAL buffers when possible

2022-12-11 Thread Kyotaro Horiguchi
perhaps we should manage how far the buffer is filled with valid content (or how far we can presend) in this feature. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Question regarding "Make archiver process an auxiliary process. commit"

2022-12-06 Thread Kyotaro Horiguchi
bout the first iteration of archive process, not periodical wakeups. So I think it is doable by considering how to handle incomplete archiving iterations. > The proposed idea is a behaviour change while this thread intends to clean > up some code that's > a result of the mentioned commit. So probably the proposed idea can be > discussed as a separate thread. Agreed. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Question regarding "Make archiver process an auxiliary process. commit"

2022-12-06 Thread Kyotaro Horiguchi
kes up occasionally to allow herself to be proactive. > */ So I think this is still valid. If we want to eliminate useless wakeups, archiver needs to remember whether the last iteration was fully done or not. But it seems to be a race condition is involved. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Time delayed LR (WAS Re: logical replication restrictions)

2022-12-06 Thread Kyotaro Horiguchi
. It fails for me on 64bit Linux.. (Rocky 8.7) > t/032_apply_delay.pl ... Dubious, test returned 29 (wstat 7424, > 0x1d00) > No subtests run .. > t/032_apply_delay.pl (Wstat: 7424 Tests: 0 Failed: 0) > Non-zero exit status: 29 > Parse errors: No plan f

Re: Transaction timeout

2022-12-06 Thread Kyotaro Horiguchi
ng the timeout enabled paths. And > you're right, it looks like there's a fair bit of optimization > potential. Thanks. I'll work on that. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Question regarding "Make archiver process an auxiliary process. commit"

2022-12-06 Thread Kyotaro Horiguchi
ehavior of the function. I think we should simplify the loop by removing last_copy_time and curtime in the "if (!time_to_stop)" block. Then we can remove the variable "timeout" and the "if (timeout > 0)" branch. Are you willing to work on this? regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Add LSN along with offset to error messages reported for WAL file read/write/validate header failures

2022-12-05 Thread Kyotaro Horiguchi
that the file names are typed in letter-by-letter, I rather prefer to allow lower-case letters:p) > > That said, let's also hear from others. > > Sure. Perhaps my set of suggestions will not get the majority, > though.. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

<    1   2   3   4   5   6   7   8   9   10   >