Re: Unsafe access BufferDescriptors array in BufferGetLSNAtomic()

Hi Tender, I’ve looked through the patch, and I believe there is a potential issue. The default size for BufferDescriptors appears to be 16,384. Passing and casting a negative buffer ID to a large unsigned integer in GetBufferDescriptor, and then using it as an array subscript, could potentiall

Re: Fwd: [BUG]: the walsender does not update its IO statistics until it exits

Hi, I performed some tests using elog(no so sure whether this is the proper way to do it) to monitor the new method. Here are the findings: • With PGSTAT_MIN_INTERVAL set to 1000ms, the number of flush reports was reduced to approximately 40–50 during the installcheck test suite. • With PGSTAT_I

Re: Fwd: [BUG]: the walsender does not update its IO statistics until it exits

Hi, I am OK with PGSTAT_MIN_INTERVAL. It has been used for flushing other statistics as well. And monitoring systems are generally configured to poll at one-second or longer intervals. > > I think that reporting at PGSTAT_MIN_INTERVAL is fine and more than > enough. I > mean, I 'm not sure that t

Re: Fwd: [BUG]: the walsender does not update its IO statistics until it exits

写道： > Hi, > > On Wed, Mar 19, 2025 at 02:26:47PM +0900, Michael Paquier wrote: > > On Wed, Mar 19, 2025 at 09:53:16AM +0800, Xuneng Zhou wrote: > > > I think these two conditions are good too. In a busy system, they are > met > > > frequently, so the flush ro

Re: Proposal: Limitations of palloc inside checkpointer

Hi, I’ve updated and rebased Maxim's patch version 4 with optimizations like ring buffer and capped max_requests proposed by Heikki. These are the summary of Changes in v5: • Replaced the linear array model in AbsorbSyncRequests() with a bounded ring buffer to avoid large memmove() operations wh

Re: Proposal: Limitations of palloc inside checkpointer

Hi, I've moved this entry to the July CommitFest. The CI reported an unused variable warning in patch v5, so I've addressed it by removing the unused one. Sorry for not catching the issue locally. Xuneng Zhou 于2025年4月10日周四 23:43写道： > Hi, > > I’ve updated and rebased Maxim

Re: BgBufferSync(): clarification about reusable_buffers variable

Hi, tks for working on this. I had a chance to look at this while googling BgBufferSync function. > > I also think tha reusable_buffers keep track of the number of reusable > buffers. BgBufferSync() calls SyncOneBuffer() with skip_recently_used > = true. In that case, if SyncOneBuffer() finds th

Re: BgBufferSync(): clarification about reusable_buffers variable

Here's a rebase. bg_buffer_sync_clarity.patch Description: Binary data

Re: Add an option to skip loading missing publication to avoid logical replication failure

Hi, A clear benefit of addressing this in code is to ensure that the user sees the log message, which can be valuable for trouble-shooting—even under race conditions. ereport(WARNING, errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),

Re: Add pg_buffercache_mark_dirty[_all] functions to the pg_buffercache

Hi, I’ve been trying to review this patch, and it struck me that we’re currently grabbing the content lock exclusively just to flip a header bit: if (!(buf_state & BM_DIRTY)) { LWLockAcquire(BufferDescriptorGetContentLock(desc), LW_EXCLUSIVE); MarkBufferDirty(buf); LWLockRelease(Buffer

Re: Add an option to skip loading missing publication to avoid logical replication failure

: > > > > On Wed, Apr 30, 2025 at 11:22 AM Tom Lane wrote: > > > > > > Xuneng Zhou pointed out on Discord that the test case added by > > > 7c99dc587 has caused repeated failures in CI --- though oddly, > > > it's not failed in the buildfar

Re: Add an option to skip loading missing publication to avoid logical replication failure

Yeh, tks for your clarification. I have a basic understanding of it now. I mean is this considered a bug or design defect in the codebase? If so, should we prevent it from occuring in general, not just for this specific test. vignesh C > > We have three processes involved in this scenario: > A

Re: Add an option to skip loading missing publication to avoid logical replication failure

+1, I was unable to reproduce this with lldb, not sure my way is appropriate or not. > > Can you be a little more specific about how you reproduced this? > I tried inserting sleep() calls in various likely-looking spots > and could not get a failure that way. > > regards,

Re: support virtual generated column not null constraint

Hi, jian he 于2025年3月6日周四 21:44写道： > On Wed, Feb 26, 2025 at 3:01 PM ego alter wrote: > > > > Hi, I’ve had a chance to review the patch. As I am still getting > familiar with executor part, questions and feedbacks could be relatively > trivial. > > > > There are two minor issues i want to discu

Re: support virtual generated column not null constraint

Hi, forget to add hackers to cc. Xuneng Zhou 于2025年3月8日周六 12:10写道： > > > Navneet Kumar 于2025年3月8日周六 02:09写道： > >> >> >>> This scenario fails >>> 1. CREATE TABLE person ( >>> id INT GENERATED BY DEFAULT AS IDENTITY, >>> first_

Re: per backend WAL statistics

Hi, Bertrand Drouvot 于2025年3月5日周三 21:03写道： > Hi, > > On Wed, Mar 05, 2025 at 05:45:57PM +0800, Xuneng Zhou wrote: > > Subject: Clarification Needed on WAL Pending Checks in Patchset > > > > Hi, > > > > Thank you for the patchset. I’ve spent some time lea

Re: per backend WAL statistics

Subject: Clarification Needed on WAL Pending Checks in Patchset Hi, Thank you for the patchset. I’ve spent some time learning and reviewing it and have 2 comments. I'm new to PostgreSQL hacking, so please bear with me if I make mistakes or say something that seems trivial. I noticed that in pat

Re: Proposal: Limitations of palloc inside checkpointer

Hi, The patch itself looks ok to me. I'm curious about the trade-offs between this incremental approach and the alternative of using palloc_extended() with the MCXT_ALLOC_HUGE flag. The approach of splitting the requests into fixed-size slices avoids OOM failures or process termination by the OOM

Fwd: [BUG]: the walsender does not update its IO statistics until it exits

Forgot to cc... -- Forwarded message - 发件人： Xuneng Zhou Date: 2025年3月13日周四 19:15 Subject: Re: [BUG]: the walsender does not update its IO statistics until it exits To: Bertrand Drouvot Hi, Thanks for working on this! I'm glad to see that the patch ( https://www.postgresq

Fwd: [BUG]: the walsender does not update its IO statistics until it exits

Sorry, forgot to cc this one too. -- Forwarded message - 发件人： Xuneng Zhou Date: 2025年3月13日周四 19:29 Subject: Re: [BUG]: the walsender does not update its IO statistics until it exits To: Bertrand Drouvot Hi, Given that PGSTAT_MIN_INTERVAL is the go-to setting for stats flushes

Re: support virtual generated column not null constraint

Hi, response from ChatGPT, seems correct: "Consing up" is an informal term derived from Lisp terminology. In this context, it means dynamically creating (allocating and constructing) a new NullTest node. Instead of reusing an existing node, the code allocates a fresh node—using PostgreSQL’s memor

Re: Add an option to skip loading missing publication to avoid logical replication failure

If the presumed theory regarding the cause of the issue is correct — as outlined in this email — and no data replication occurs in this scenario

Re: Add an option to skip loading missing publication to avoid logical replication failure

Hi, I was able to reproduce the failure by adding a 1-second sleep in the LogicalRepApplyLoop function However, I noticed that the tests under src/test/subscription run significantly slower— is this normal? Also, it looks like the patch mentioned in this thread addresses the issue: https://www.pos

Re: Add pg_buffercache_mark_dirty[_all] functions to the pg_buffercache

Hey, I noticed a couple of small clarity issues in the current version of patch for potential clean up: 1. Commit message wording Right now it says: “The pg_buffercache_mark_dirty_all() function provides an efficient way to dirty the entire buffer pool (e.g., ~550 ms vs. ~70 ms for 16 GB of sha

Re: Conflict detection for update_deleted in logical replication

Hi Zhijie, Thanks for the effort on the patches. I did a quick look on them before diving into the logic and discussion. Below are a few minor typos found in version 31. ⸻ 1. Spelling of “non-removable” [PATCH v31 1/7] In docs and code “removeable” vs. “removable” are used alternatively

Re: Add CHECK_FOR_INTERRUPTS in polling loop code path in XactLockTableWait

Hi all, Thank you for the earlier pushed patch—after testing it, things work well. Per Fujii’s and Kevin’s ideas and suggestions[1], [2], I’ve tried to extend it to add a new wait event and a progressive backoff mechanism for XactLockTableWait in logical replication scenarios. New wait event • X

Re: Proposal: Limitations of palloc inside checkpointer

Hi Alexander, Thanks a lot for reviewing! > I have few notes about that: > 1) Should we make CompactCheckpointerRequestQueue() process the queue > of checkpoint requests in smaller parts for the same reason we do this > in AbsorbSyncRequests()? That would require significant redesign of > the alg

Re: Add CHECK_FOR_INTERRUPTS in polling loop code path in XactLockTableWait

hidden coupling and is easier to maintain if additional contexts require different wait policies. If ConditionalXactLockTableWait ever needs backoff, it would benefit from the same explicit flag. On Tue, Jun 3, 2025 at 11:32 AM Xuneng Zhou wrote: > > Hi all, > > Thank you for the ea

Re: Add CHECK_FOR_INTERRUPTS in polling loop code path in XactLockTableWait

Hi Fujii, Thanks a lot for reviewing! > > This update amends the previous patch by replacing the global > > XactLockTableWait_us with a local wait_us variable inside > > XactLockTableWait. Now each call resets its backoff to 1 ms, doubling > > up to 1 s when oper == XLTW_None. This eliminates shar

Re: Add progressive backoff to XactLockTableWait functions

Hi Andres, Thanks for looking into this! On Tue, Jun 17, 2025 at 10:17 PM Andres Freund wrote: > Hi, > > On 2025-06-08 22:33:39 +0800, Xuneng Zhou wrote: > > This patch implements progressive backoff in XactLockTableWait() and > > ConditionalXactLockTableWait(). > &g

Re: Add progressive backoff to XactLockTableWait functions

Hi, On Sun, Jun 15, 2025 at 4:01 PM Xuneng Zhou wrote: > Hi, > > Attached is v2 of the patch to add threshold-based sleep to > XactLockTableWait functions. > > Changes from v1: > - Simplified approach based on Fujii's feedback [1]: instead of > exponential backoff,

Add new wait event to XactLockTableWait

Hi hackers, Currently, when XactLockTableWait() and ConditionalXactLockTableWait() sleep waiting for transactions to complete, they don't report any specific wait event to the statistics system. This means that backends stuck in these waits show up in pg_stat_activity with NULL wait_event_type and

Add progressive backoff to XactLockTableWait functions

Hi hackers, This patch implements progressive backoff in XactLockTableWait() and ConditionalXactLockTableWait(). As Kevin reported in this thread [1], XactLockTableWait() can enter a tight polling loop during logical replication slot creation on standby servers, sleeping for fixed 1ms intervals t

Re: Add CHECK_FOR_INTERRUPTS in polling loop code path in XactLockTableWait

Hi, > Thanks for the patch! I haven't reviewed it yet, but since this is > a v19 item, please add it to the next CommitFest so we don't lose > track of it. > > Also, I think it would be better to split the addition of the wait event > and the introduction of exponential backoff in XactLockTableWai

Re: Add new wait event to XactLockTableWait

Just CC. On Mon, Jun 9, 2025 at 10:57 AM Xuneng Zhou wrote: > > Hi Michael, > > Thanks for reviewing. > > On Mon, Jun 9, 2025 at 6:41 AM Michael Paquier wrote: > > > + pgstat_report_wait_start(WAIT_EVENT_XACT_DONE); > > [...] > > + pgstat

Re: Add new wait event to XactLockTableWait

Hi Fujii-san, Thanks for reviewing. > On 2025/06/09 7:41, Michael Paquier wrote: > > On Sun, Jun 08, 2025 at 10:30:45PM +0800, Xuneng Zhou wrote: > >> This is more problematic in logical replication scenarios where these > >> waits can be very long - for examp

Re: Add CHECK_FOR_INTERRUPTS in polling loop code path in XactLockTableWait

Hi, > Just idea, if XactLockTableWait() is expected to finish within a few seconds > after acquiring the lock, how about this approach: keep sleeping for 1ms > until the total sleep time reaches 10s (10s is just an example), > and after that, start doubling the sleep duration each cycle, up to > a

Re: Add progressive backoff to XactLockTableWait functions

Hi, Here's patch version 4. 1. The problem I conducted further investigation on this issue. The 1ms sleep in XactLockTableWait that falls back to polling was not problematic in certain scenarios prior to v16. It became an potential issue after the "Allow logical decoding on standbys" feature w

Re: Add progressive backoff to XactLockTableWait functions

Hi, Attached is v2 of the patch to add threshold-based sleep to XactLockTableWait functions. Changes from v1: - Simplified approach based on Fujii's feedback [1]: instead of exponential backoff, we now sleep 1ms for the first 5 seconds, then switch directly to 1s sleeps - Reduced the threshold

Re: Proposal: Limitations of palloc inside checkpointer

Hi all, Sorry—I forgot to Cc on my previous message. Resending here so they’re on the thread: On Wed, Jun 4, 2025 at 11:07 AM Xuneng Zhou wrote: > > Hi Alexander, > > Thanks again for the feedback! > > 1) Batch-processing CompactCheckpointerRequestQueue() and > AbsorbSync

Re: Proposal: Limitations of palloc inside checkpointer

Hi, Thanks for the feedback! > I think it would be good start point to use the same batch size of > CompactCheckpointerRequestQueue() and AbsorbSyncRequests() > So we’ll keep both batch processing and the request cap in place for now. > > > Right, but another point is to avoid lengthy holding o

Request to Expedite Account Activation Cool-Off Period

Hi, all I created a new PostgreSQL.org account today, as changing usernames is not possible. Could an administrator please assist in expediting my account activation. Thanks! Best regards, Xuneng new username: alterego

Re: Proposal: Limitations of palloc inside checkpointer

Hi, > > 3) Fill gaps by pulling from the tail instead of rewriting the whole > queue? > > > > I misunderstood at first—this is a generally helpful optimization. > > I'll integrate it into the current patch. > > Great, thank you. > I dug deeper into the “fill gaps from the tail” optimization and

Re: Add new wait event to XactLockTableWait

Hi, On Mon, Jun 9, 2025 at 12:52 PM Xuneng Zhou wrote: > > Hi, > > > Please find attached Version 2, incorporating the suggested changes. > > Apologies for the confusion — in the previous attempt, I mistakenly > named the patch file with a `0002-` prefix, thinking

Re: Add new wait event to XactLockTableWait

Hi, > Please find attached Version 2, incorporating the suggested changes. Apologies for the confusion — in the previous attempt, I mistakenly named the patch file with a `0002-` prefix, thinking it reflected the patch version rather than the patch series number. I've corrected the filename to f

Re: Add new wait event to XactLockTableWait

Hi Dilip, Thanks for looking into this! On Mon, Jun 9, 2025 at 6:56 PM Dilip Kumar wrote: > > Thanks for updating the patch! It looks good to me. > > > > I think we can mark it as "Ready for Committer" in the CommitFest. > > Unless there are any objections, I'll commit it once v19 development >

Re: Add new wait event to XactLockTableWait

Hi Fujii-san, On Mon, Jun 9, 2025 at 4:48 PM Fujii Masao wrote: > > Thanks for updating the patch! It looks good to me. > > I think we can mark it as "Ready for Committer" in the CommitFest. > Unless there are any objections, I'll commit it once v19 development opens. > I've marked it as "Ready

Re: Add progressive backoff to XactLockTableWait functions

Hi, Thanks for the feedback! On Thu, Jun 12, 2025 at 10:02 PM Fujii Masao wrote: > > > When I first suggested this idea, I used 10s as an example for > the maximum sleep time. But thinking more about it now, 10s might > be too long. Even if the target transaction has already finished, > XactLoc

Re: Add progressive backoff to XactLockTableWait functions

Hi all, I spent some extra time walking the code to see where XactLockTableWait() actually fires. A condensed recap: 1) Current call-paths A. Logical walsender (XLogSendLogical → … → SnapBuildWaitSnapshot) in cascading standby B. SQL slot functions (pg_logical_slot_get_changes[_peek]) create_lo

Re: Add progressive backoff to XactLockTableWait functions

Hi, After studying proarray and lmgr more closely, I have found several critical issues in the two patches and underestimated their complexity and subtlety. Sorry for posting immature patches that may have created noise. I now realize that making lock acquisition— (void) LockAcquire(&tag, ShareLo

Re: Add progressive backoff to XactLockTableWait functions

> > Thanks for the feedbacks! To be honest, I'm fine with v3, since it only increases the sleep time > after 5000 loop iterations, which has negligible performance impact. > But if these functions aren't intended to be used during recovery and > the loop shouldn't reach that many iterations, I'm a

Re: Add progressive backoff to XactLockTableWait functions

Hi, On Thu, Jul 3, 2025 at 9:30 AM Xuneng Zhou wrote: > Hi, > >> >> >>> On 2025-07-02 22:55:16 +0900, Fujii Masao wrote: >> >>>> On 2025/06/24 1:32, Xuneng Zhou wrote: >> >>>>> 3. The proposed solution >> >>

Re: Add progressive backoff to XactLockTableWait functions

Hi Andres, hi all, Thanks a lot for the advice. > My position is basically: > > 1) We should *never* add new long-duration polling loops to postgres. We've >regretted it every time. It just ends up masking bugs and biting us in >scenarios we didn't predict (increased wakeups increasing po

Re: Add progressive backoff to XactLockTableWait functions

02 23:04, Andres Freund wrote: > >>> Hi, > >>> > >>> On 2025-07-02 22:55:16 +0900, Fujii Masao wrote: > >>>> On 2025/06/24 1:32, Xuneng Zhou wrote: > >>>>> 3. The proposed solution > >>>>> > &

Re: Proposal: Limitations of palloc inside checkpointer

Hi, Patch v7 modifies CompactCheckpointerRequestQueue() to process requests incrementally in batches of CKPT_REQ_BATCH_SIZE (10,000), similar to the approach used in AbsorbSyncRequests(). This limits memory usage from O(num_requests) to O(batch_size) for both hash tables and skip arrays. - Hash

Re: Add progressive backoff to XactLockTableWait functions

Hi, I spent some time digging deeper into the KnownAssignedXids logic in procarray.c and made a few refinements. Below are some observations and questions I’d like to discuss. 1) Present usage (AFAICS) It appears that logical decoding is the only place that waits for an XID on a standby. The si

Re: Add progressive backoff to XactLockTableWait functions

Hi, On Thu, Jul 17, 2025 at 10:54 PM Xuneng Zhou wrote: > > Hi, > > After studying proarray and lmgr more closely, I have found several > critical issues in the two patches and underestimated their complexity > and subtlety. Sorry for posting immature patches that may have crea

Re: Proposal: Limitations of palloc inside checkpointer

Hi, Alexander Thanks for reviewing and feedback! > The v8 is attached. It's basically the same as v6, but has revised > commit message and comments. > > The patch for stable branches is also attached: it just limits the > maximum size of the checkpointer requests queue. LGTM. Best, Xuneng

Re: Bug in pg_dump --filter? - Invalid object types can be misinterpreted as valid

Hi Fujii-san, Thanks for working on this. On Sat, Aug 2, 2025 at 5:48 PM Fujii Masao wrote: > > Hi, > > It looks like pg_dump --filter can mistakenly treat invalid object types > in the filter file as valid ones. For example, the invalid type "table-data" > (probably a typo for "table_data") is

Re: Add progressive backoff to XactLockTableWait functions

Hi, On Wed, Jul 30, 2025 at 11:42 PM Xuneng Zhou wrote: > > Hi, > > On Mon, Jul 28, 2025 at 7:14 PM Xuneng Zhou wrote: > > > > Hi, > > > > On Thu, Jul 17, 2025 at 10:54 PM Xuneng Zhou wrote: > > > > > > Hi, > > > > >

Re: Implement waiting for wal lsn replay: reloaded

Hi, Thanks for working on this. I’ve just come across this thread and haven’t had a chance to dig into the patch yet, but I’m keen to review it soon. In the meantime, I have a quick question: is WAIT FOR REPLY intended mainly for user-defined functions, or can internal code invoke it as well? Du

Re: Bug in pg_dump --filter? - Invalid object types can be misinterpreted as valid

On Mon, Aug 4, 2025 at 11:18 PM Fujii Masao wrote: > > On Sun, Aug 3, 2025 at 3:03 PM Xuneng Zhou wrote: > > After testing, the patch LGTM. I noticed two very small possible nits: > > Thanks for the review! > > > > 1) Comment wording > > > > The loop no

Re: Proposal: Limitations of palloc inside checkpointer

Hi, On Thu, Aug 7, 2025 at 3:16 PM Xuneng Zhou wrote: > > Hi, Tom! > > Thanks for catching this. > > On Thu, Aug 7, 2025 at 2:58 AM Tom Lane wrote: > > > > Alexander Korotkov writes: > > > I'm going to push this if no objections. > > > >

Re: Proposal: Limitations of palloc inside checkpointer

Hi, Tom! Thanks for catching this. On Thu, Aug 7, 2025 at 2:58 AM Tom Lane wrote: > > Alexander Korotkov writes: > > I'm going to push this if no objections. > > I looked at these patches while preparing release notes, and > found an oversight. CheckpointerShmemInit does > > Checkpoint

Re: Implement waiting for wal lsn replay: reloaded

Hi Alexander! > > In the meantime, I have > > a quick question: is WAIT FOR REPLY intended mainly for user-defined > > functions, or can internal code invoke it as well? > > Currently, WaitForLSNReplay() is assumed to only be called from > backend, as corresponding shmem is allocated only per-back

Re: Possible inaccurate description of wal_compression in docs

Hi, On Sun, Aug 10, 2025 at 6:53 PM Jingtang Zhang wrote: > > Hi hackers~ > > I find the specification of wal_compression in our docs a little bit confusing > for it 'compress full page images when FPW is on or during a base backup', > which was true before v17. Since v17, we will also use full p

Re: Implement waiting for wal lsn replay: reloaded

Hi, > On Thu, Aug 7, 2025 at 6:01 PM Xuneng Zhou wrote: > > Thanks for working on this. > > > > I’ve just come across this thread and haven’t had a chance to dig into > > the patch yet, but I’m keen to review it soon. > > Great. Thank you for your attention

Re: Add progressive backoff to XactLockTableWait functions

Hi, Tom! Thanks for looking at this. On Fri, Aug 8, 2025 at 2:20 AM Tom Lane wrote: > > Xuneng Zhou writes: > > V9 replaces the original partitioned xid-wait htab with a single, > > unified one, reflecting the modest entry count and rare contention for > > waiting. T