Hello again. Well, I've been using the past few days to dig into the HOT/COOL+batching model and I think I've found the corners and have data to support it. Attached find v8 (on 9fa2c1e) and a tar.gz with my benchmark.
TL;DR:
Empirically-grounded data shows that HOT/COOL+batching:
- (a) provides scan resistance which is net new outside of the
ring buffer
- (b) is simpler (1-bit vs 0..5) <- that's a lot fewer bits, lol!
- (c) I've found no evidence of any signal provided by the 0..5
clock-sweep counter and direct evidence that it can, in
limited but not-unimaginable cases, become the root of
pathologically bad performance
- (d) batching de-contends nextVictimBuffer on NUMA +68-75% when
the sweep is hot, no regression otherwise. Not "always
faster," but "targeted de-contention that pays off exactly
when the sweep saturates, and never hurts when pooled.
I've tested with pgbench, HammerDB (TPROC-C/H), and custom code to
model and elicit the worst/best cases and the pay-off cases I could
imagine. I've not done this on local (non-cloud) hardware (I'm
building that out now!) but on EC2 metal instances which I'll argue
are close enough and all I had available (for now).
I may still be benchmarking in ways that you feel don't elicit the
best, most accurate results. Feel free to point that out and I'll
adapt and re-run the tests.
AI NOTICE:
Yes, this patch was co-developed/refined/tested/benchmarked using
AI/LLMs and even this email started off as LLM content. All the work
is steered by me and reviewed before submission by me, I stand behind
it warts and all. (gulp) ;-P
DETAILS:
I've done quite a bit of measurement. This reply is long because
it needs to be, legitimate requests for real data required thought
and time to execute, analyze, etc. Short version of what changed and
what I found:
- Two patches now, not three. I dropped the BufferAccessStrategy
removal; it removed real bulk-writer backpressure with no
replacement (Ants and Andres were both right), so it's out until
that's solved separately.
- I withdrew the v6/7 dirty-bulk "regression" and VACUUM "win" as
both were mismeasurements, detailed in my earlier replies.
- The batched sweep (0001) and the cooling evictor (0002) are
independent and I'm treating them as such; 0001 stands alone if 0002
is contentious, but used together is the real win.
- I built a sweep-bound microbenchmark that isolates the
nextVictimBuffer atomic, and it settles the batching question with
perf c2c evidence (below). It also shows honestly where batching
does *not* move the needle.
The series:
0001 Batch the clock sweep to reduce nextVictimBuffer atomic contention
(Jim Mlodgenski's idea; co-authored)
0002 Replace the 0..5 usage_count clock sweep with a 1-bit HOT/COOL
cooling-stage evictor (LeanStore/2Q-A1), rings kept
All numbers below are on AWS bare metal, working set sized to fit RAM so
misses are OS-page-cache-cheap (this is a CPU/TLB study, not a storage
one -- I'll say explicitly where that matters). Instrumentation (a
throwaway ticks-per-victim counter and a block-thrash function) is not
part of the patches; the repro kit is attached as repro.tgz_ (underscore
to avoid being picked up by CI I hope).
0001 -- batched sweep: does it help NUMA, and does it regress?
========================================================================
Andres, in the first reply on this thread you doubted that batching
independent of contention/usage-rate would help, and suspected it might
hurt. I set out to prove or disprove that, and the answer is: it depends
entirely on whether the clock sweep is the bottleneck, and I can now
show exactly when that is the case.
First, the fair-throughput question. Earlier I got confusing results
because I was running 2000-8000 backends against a 384-vCPU box -- 5-20x
oversubscribed -- which is a scheduler-storm test, not a batching test,
and there batching *did* look like it hurt (~-35%). With a connection
count at or below core count (the pooled configuration anyone runs in
production), on a 6-NUMA r8i.metal (384 vCPU), uniform pgbench-style
point lookups, stock vs batch{1,16,64} are all within ~1-2% at every
level -- batching neither helps nor hurts. So: no regression when
you're not oversubscribed; the -35% was an artifact of oversubscription,
which a connection pooler removes.
Why neutral there? Because in a normal point-lookup the executor +
protocol + tuple deform dwarf the one atomic; even stock only did ~2.9
clock ticks per victim, so nextVictimBuffer just wasn't hot enough to
matter.
To find out whether the *mechanism* is real at all, I built a
sweep-bound microbenchmark: a function that pins+releases random blocks
of a table 1.7x shared_buffers with the normal (non-ring) strategy, so
StrategyGetBuffer is essentially the only work. 360 concurrent loops (=
cores on the instance), 64GB shared_buffers, 6 NUMA nodes:
build block-evictions/sec vs stock
stock 2,096,314 --
bcs batch=1 2,138,485 +2% (single fetch-add: parity)
bcs batch=16 3,531,021 +68%
bcs batch=64 3,664,458 +75%
and the mechanism, straight from perf c2c (cross-node HITM on the
hottest line):
stock: StrategyGetBuffer's fetch-add on nextVictimBuffer is the #1
cross-node HITM line at 38.5%, spread across all 6 nodes.
batch=16/64: that same line drops to ~1-2% HITM.
So the batched claim does exactly what it claims: nextVictimBuffer is
touched ~1/N as often, its cross-socket bounce collapses, and when the
sweep is the bottleneck throughput rises 68-75%. batch=1 == stock
confirms the win is purely the batched atomic, not the evictor.
The honest framing, which I'd rather state than have pulled out of me:
this is a microbenchmark that isolates the atomic. It proves the
mechanism is real and bounds the ceiling; it does not claim a 68% win on
your OLTP box. On real mixed workloads at backends<=cores the sweep is
a small share of each query, so end-to-end it's neutral -- no
regression, no headline speedup -- until the sweep actually saturates
(large hot pool, heavy eviction, many NUMA nodes, which is the reported
field pathology). 0001 is a targeted de-contention that pays off
exactly when the hand is hot and costs nothing when it isn't; on a
single socket it compiles to the stock path (batch size 1).
I did also run HammerDB TPROC-C at backends<=cores; it's
write/lock/WAL-bound, the sweep is not the bottleneck, and batch=64 was
~5% *below* stock (the cooling bookkeeping isn't free when the sweep
isn't the cost). Reporting that too -- it's not a batching workload,
and I'm not going to pretend it is.
0002 -- cooling evictor: why the 0..5 counter, and the failure mode
========================================================================
Ants, you said the initial results were a promising indication that
G-CLOCK's 0..5 counter isn't carrying much value, and asked for a wider
access-pattern gamut. Here's the strongest thing I found.
The 0..5 counter has a pathological failure mode that the 1-bit HOT/COOL
clock does not. Stock must decrement a buffer from 5 to 0 before it can
evict it -- up to five visits per victim -- and it resets its
bounded-scan counter on every decrement, so under a hot,
continuously-evicting pool it never gives up early. With a large
shared_buffers and huge_pages off (a multi-GB BufferDescriptors array on
4KB pages), those repeated visits are dTLB misses. I measured
ticks-per-victim climbing to ~9 on stock under load where the 1-bit
clock stays flat (it demotes HOT->COOL in one pass; a victim is produced
in at most ~3 ticks, and typically 1-2). In the extreme this turns into
a throughput cliff / multi-second p99 for stock -- the field reports of
"raise shared_buffers and TPS falls off" on large no-huge-pages
machines. The 1-bit clock has no such cliff because it does bounded
work per victim by construction.
On scan resistance, which is the reason the 0..5 counter's replacement
has to be careful: a demand-loaded page is admitted COOL (probationary)
and only promoted to HOT on a genuine second touch, so a one-touch
sequential scan fills and drains the COOL stage without displacing the
HOT working set. I tested this adversarially -- an OLTP hot set that
fits in shared_buffers, hit by point lookups, while a concurrent seqscan
streams a table 4x shared_buffers, on local NVMe (no EBS/IOPS cap
masking the eviction rate):
OLTP hit ratio under the scan: stock 99.999% bcs 99.998%
OLTP resident set: flat in both
i.e. the cooling evictor's scan resistance is at parity with what the
BAS_BULKREAD ring gives today. (An earlier version of this patch also
removed that read ring on the theory that COOL admission made it
redundant; local-NVMe testing showed it wasn't quite -- the ring is
measurably tighter under concurrent scan -- so I kept the ring. That's
the dropped 0003.)
On the literature: you pointed at CLOCK-Pro, thanks -- the admission
model here (probationary COOL, promote on reuse) is the 2Q-A1 /
test-period idea and I'd rather adopt a better-justified promotion rule
from that line than defend "second touch" if the list prefers one. I
tried a stricter "promote only on a repeat reference" variant; it was
neutral on OLTP and didn't improve scan resistance, so I reverted it to
keep 0002 minimal, but it's an easy knob.
On backpressure: I'm not removing the rings, precisely because
BAS_BULKWRITE is backpressure, not just a pollution guard. If we ever
do want the rings gone, that backpressure needs a real replacement
first; I don't have one worth proposing, so it's out of scope here.
Relationship to the NUMA partitioned sweep
========================================================================
Tomas, this overlaps your NUMA series and I want to be clear it's not a
competitor. 0001 attacks the same nextVictimBuffer cross-node
contention your partitioned sweep does, but as a minimal,
placement-agnostic change: one fetch-add per batch instead of per tick,
no per-node partitioning, no buffer-placement changes, batch-size-1
fallback so it's a no-op off NUMA. The c2c data above is the same
contention your partitioning targets, measured in isolation.
I think these compose rather than conflict: a partitioned sweep still
advances a hand within each partition, and batching that per-partition
hand should give the same cross-node relief inside a partition that it
gives the global hand here. If your series lands, 0001 is a small
change on top (or moot, if the partition hands are already node-local
and uncontended). If it's useful I'm happy to test batching layered on
your patches -- I just didn't want to fold placement/partitioning into
this, since the evictor and the atomic de-contention stand on their own
and are far smaller.
What I'm claiming, and what I'm not
========================================================================
- 0002 is simpler (1 bit vs a 0..5 counter), has intrinsic scan
resistance at parity with the BAS_BULKREAD ring, and removes a real
large-pool/no-huge-pages cliff that the 0..5 counter can hit. It is
throughput-neutral on ordinary OLTP (within noise), which is the bar
for a replacement-policy change.
- 0001 de-contends nextVictimBuffer on NUMA: +68-75% when the sweep is
the bottleneck (c2c-proven mechanism), neutral otherwise, no-op off
NUMA, no regression when connections are kept at/under core count.
One idea would be to lower batch size to 1 when the number of
backends > cores available on the host to avoid the regression, but
that isn't in v8 nor have I tested it in practice.
- I am NOT claiming a general TPS win; on write/lock-bound or
non-sweep-bound workloads there isn't one, and I've shown that.
Repro kit attached (methodology, the microbenchmark, the ticks/victim
and block-thrash instruments, the residency-fraction model that predicts
when the 0..5 cliff appears, and the huge_pages on/off and
confound-discrimination notes). It's built to be re-run and attacked;
if any of this doesn't reproduce for you, that's the feedback I want.
Thanks for the reviews, they made the patch set smaller and the claims
honest.
best.
-gregFrom 40679bd604a4f55ebc460cdd4337fc3d370ec8ec Mon Sep 17 00:00:00 2001 From: Greg Burd <[email protected]> Date: Mon, 6 Jul 2026 17:25:00 -0400 Subject: [PATCH v8 1/2] Batch the clock sweep to reduce nextVictimBuffer atomic contention StrategyGetBuffer() advances the shared clock hand, nextVictimBuffer, with a pg_atomic_fetch_add_u32(..., 1) on every tick. On a multi-socket system the cache line holding that counter has to travel over the interconnect on each operation, pushing a sweep tick from ~20ns (same-socket, line warm in L1/L2) into the ~100-200ns range. Under eviction pressure with hundreds of backends in StrategyGetBuffer() concurrently, that single cache line becomes the dominant cost of the sweep, visible as elevated bus-cycles and cache-misses in a perf profile. Have each backend claim a run of consecutive buffer IDs from the shared hand with a single fetch-add and then iterate through them privately. The sweep still advances through the pool in order, each buffer is still visited exactly once per complete pass, and the meaning of the clock state is unchanged; only the temporal ordering of visits within a pass changes, which the algorithm does not depend on. The contended atomic now fires roughly once per batch rather than once per buffer. The batch is one cache line's worth of clock-hand values -- PG_CACHE_LINE_SIZE / sizeof(uint32) -- capped at NBuffers so a claim can never wrap the pool more than once. Batching only helps when the counter's cache line actually bounces between sockets, so it is enabled only on multi-node NUMA hardware (pg_numa_get_max_node() >= 1); on a single socket, or where libnuma is unavailable, the batch size stays 1 and the code path is byte-identical to the stock clock sweep. Wraparound handling is adjusted: with batching, several backends can each see a fetch-add return a value past NBuffers within the same pass. Any such backend takes buffer_strategy_lock, re-reads the counter, and if it is still out of range wraps it with a single CAS and increments completePasses. StrategySyncStart() continues to see a consistent (nextVictimBuffer, completePasses) pair. This is the batched-clock-sweep idea from Jim Mlodgenski's pgsql-hackers thread, adapted to derive the batch size from the platform cache-line size. Co-authored-by: Jim Mlodgenski <[email protected]> --- src/backend/storage/buffer/freelist.c | 152 ++++++++++++++++++-------- 1 file changed, 109 insertions(+), 43 deletions(-) diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c index fdb5bad..46104b3 100644 --- a/src/backend/storage/buffer/freelist.c +++ b/src/backend/storage/buffer/freelist.c @@ -22,6 +22,7 @@ #include "storage/proc.h" #include "storage/shmem.h" #include "storage/subsystems.h" +#include "port/pg_numa.h" #define INT_ACCESS_ONCE(var) ((int)(*((volatile int *)&(var)))) @@ -100,68 +101,110 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy, static void AddBufferToRing(BufferAccessStrategy strategy, BufferDesc *buf); +/* + * Per-backend state for the batched clock sweep. Each backend claims a run + * of consecutive clock-hand values with a single atomic fetch-add and then + * iterates through them privately, so the contended nextVictimBuffer cache + * line is touched roughly 1/batch as often. MyBatchPos is the next hand + * value to hand out; MyBatchEnd is one past the end of the claimed run. Both + * are absolute (monotonically increasing) hand values; the buffer id is the + * value modulo NBuffers. + */ +static uint32 MyBatchPos = 0; +static uint32 MyBatchEnd = 0; + +/* + * Number of clock-hand values a backend claims per atomic fetch-add, + * computed once at startup (see StrategyCtlShmemInit). When batching is + * enabled it is one cache line's worth of hand advance, so concurrent + * backends sweep non-overlapping, cache-line-sized runs of the pool; the + * global sweep order is preserved (each buffer is still visited exactly once + * per pass). Batching is enabled only on multi-node NUMA hardware; otherwise + * this stays 1 and the sweep is byte-identical to the stock clock. + */ +static uint32 ClockSweepBatchSize = 1; + /* * ClockSweepTick - Helper routine for StrategyGetBuffer() * - * Move the clock hand one buffer ahead of its current position and return the - * id of the buffer now under the hand. + * Return the next buffer to consider for eviction. Backends claim batches of + * consecutive buffer IDs from the shared clock hand, then iterate through + * them locally without further atomic operations. This preserves the global + * sweep order while reducing contention on the shared counter. */ static inline uint32 ClockSweepTick(void) { uint32 victim; - /* - * Atomically move hand ahead one buffer - if there's several processes - * doing this, this can lead to buffers being returned slightly out of - * apparent order. - */ - victim = - pg_atomic_fetch_add_u32(&StrategyControl->nextVictimBuffer, 1); - - if (victim >= NBuffers) + if (MyBatchPos >= MyBatchEnd) { - uint32 originalVictim = victim; - - /* always wrap what we look up in BufferDescriptors */ - victim = victim % NBuffers; - /* - * If we're the one that just caused a wraparound, force - * completePasses to be incremented while holding the spinlock. We - * need the spinlock so StrategySyncStart() can return a consistent - * value consisting of nextVictimBuffer and completePasses. + * Claim a fresh batch from the shared clock hand. This is the only + * atomic operation per batch, reducing contention by the batch size. */ - if (victim == 0) - { - uint32 expected; - uint32 wrapped; - bool success = false; + uint32 start; + uint32 batch_size = ClockSweepBatchSize; - expected = originalVictim + 1; + start = pg_atomic_fetch_add_u32(&StrategyControl->nextVictimBuffer, + batch_size); - while (!success) - { - /* - * Acquire the spinlock while increasing completePasses. That - * allows other readers to read nextVictimBuffer and - * completePasses in a consistent manner which is required for - * StrategySyncStart(). In theory delaying the increment - * could lead to an overflow of nextVictimBuffers, but that's - * highly unlikely and wouldn't be particularly harmful. - */ - SpinLockAcquire(&StrategyControl->buffer_strategy_lock); + if (start >= (uint32) NBuffers) + { + start = start % NBuffers; - wrapped = expected % NBuffers; + /* + * The counter has grown past NBuffers; try to wrap it back. We + * must hold the spinlock so StrategySyncStart() can read + * nextVictimBuffer and completePasses consistently. + * + * With batching, multiple backends may each land a fetch-add + * that returns a value past NBuffers in the same pass. After + * acquiring the spinlock we re-read the counter: if another + * backend already wrapped it below NBuffers we are done. + */ + SpinLockAcquire(&StrategyControl->buffer_strategy_lock); + { + uint32 current; + uint32 wrapped; - success = pg_atomic_compare_exchange_u32(&StrategyControl->nextVictimBuffer, - &expected, wrapped); - if (success) - StrategyControl->completePasses++; - SpinLockRelease(&StrategyControl->buffer_strategy_lock); + current = pg_atomic_read_u32(&StrategyControl->nextVictimBuffer); + if (current >= (uint32) NBuffers) + { + wrapped = current % NBuffers; + if (pg_atomic_compare_exchange_u32(&StrategyControl->nextVictimBuffer, + ¤t, wrapped)) + StrategyControl->completePasses++; + } } + SpinLockRelease(&StrategyControl->buffer_strategy_lock); + } + else if (start + batch_size > (uint32) NBuffers) + { + /* + * The fetch-add returned a value below NBuffers, but this batch + * spans the wrap point (start .. NBuffers-1 then 0 .. k are + * iterated locally via "% NBuffers"). That crossing is a + * complete pass too, but the wrap-and-increment path above only + * fires when the fetch-add return is itself >= NBuffers, so + * without this it would go uncounted and completePasses would + * drift low (it feeds bgwriter pacing / StrategySyncStart, not + * correctness). The batch is capped at NBuffers, so a batch + * spans the wrap at most once; count it under the spinlock for a + * consistent (nextVictimBuffer, completePasses) pair. + */ + SpinLockAcquire(&StrategyControl->buffer_strategy_lock); + StrategyControl->completePasses++; + SpinLockRelease(&StrategyControl->buffer_strategy_lock); } + + MyBatchPos = start; + MyBatchEnd = start + batch_size; } + + victim = MyBatchPos % NBuffers; + MyBatchPos++; + return victim; } @@ -408,6 +451,29 @@ StrategyCtlShmemInit(void *arg) /* No pending notification */ StrategyControl->bgwprocno = -1; + + /* + * Decide whether to batch the clock sweep. + * + * Batching claims a run of consecutive buffer IDs per atomic fetch-add so + * concurrent backends touch the shared nextVictimBuffer cache line ~1/batch + * as often -- a win only when that line actually bounces across sockets, + * i.e. on multi-node NUMA hardware. On a single socket the atomic is + * already node-local and batching would only make backends skip ahead for + * no benefit, so we fall back to batch size 1 there (byte-identical to the + * stock one-buffer-at-a-time clock sweep). + * + * Batch (> 1) only when libnuma reports more than one node + * (pg_numa_get_max_node() >= 1); pg_numa_init() returns -1 when NUMA is + * unavailable. (Benchmarking showed the win holds with or without huge + * pages, so huge-page availability is intentionally not part of the gate.) + */ + if (pg_numa_init() != -1 && + pg_numa_get_max_node() >= 1) + ClockSweepBatchSize = Min(PG_CACHE_LINE_SIZE / (uint32) sizeof(uint32), + (uint32) NBuffers); + else + ClockSweepBatchSize = 1; } -- 2.50.1
From 898e100f5e2c42d0804b0515310f51c357166fec Mon Sep 17 00:00:00 2001 From: Greg Burd <[email protected]> Date: Mon, 6 Jul 2026 19:22:45 -0400 Subject: [PATCH v8 2/2] Replace the usage_count clock sweep with a cooling-stage evictor Replace the 0..5 usage_count buffer-replacement policy with a cooling-stage clock (the LeanStore / 2Q-A1 model): a buffer is simply HOT (recently used) or COOL (an eviction candidate), with "pinned" being the existing refcount. There is no per-buffer access counter. - A demand-loaded page is admitted COOL (probationary), not HOT. A second access via PinBuffer promotes it COOL -> HOT (the rescue). So a page touched once -- a sequential scan -- fills and drains the COOL stage and is evicted from it without ever displacing the HOT working set. Scan resistance is intrinsic to the replacement algorithm, which is what lets a later commit remove the BufferAccessStrategy ring buffers entirely. - The foreground sweep in StrategyGetBuffer() reclaims an already-COOL, unpinned buffer, pinning it with a CAS so a racing PinBuffer always wins. Promotion (COOL -> HOT) and demotion (HOT -> COOL) are single-bit transitions; only the eviction claim is a CAS. - The background writer maintains the supply of COOL victims. As its LRU scan runs ahead of the clock hand it demotes HOT buffers to COOL, so the foreground finds a victim in a single pass rather than having to cool buffers itself. The demotion is demand-driven -- bounded by the predicted allocation for the next cycle -- so it stages just enough COOL buffers without cooling the whole pool, and it is done under the buffer header lock the scan already holds. A single second-chance reference bit (set by PinBuffer, cleared on the bgwriter's first pass over a HOT buffer) spares a recently-accessed buffer one cooling pass, keeping the genuinely hot set out of the COOL stage under scan pressure. BgBufferSync also tracks the reusable-buffer density it directly observes on a shorter smoothing window, so a burst of probationary/scan COOL pages is followed promptly rather than averaged away. Under a bulk-dirtying workload the per-cycle clean-write cap is raised from bgwriter_lru_maxpages to predicted demand so the bgwriter keeps supplying clean victims, rather than the foreground sweep having to flush dirty victims inline; normal workloads, where demand is below the cap, are unaffected. The 4-bit usage_count field is reinterpreted in place: bit 0 is the HOT/COOL state (BUF_COOLSTATE_ONE), bit 1 the reference bit (BUF_REFBIT). The 64-bit buffer-state layout -- refcount, flag and lock offsets and their StaticAsserts -- is unchanged; only the meaning of the field and the instructions that touch it change. A StaticAssert requires the field to be at least 2 bits wide so a future width change cannot push the reference bit into the flag bits. BM_MAX_USAGE_COUNT becomes BUF_COOLSTATE_HOT (1), so the pin fast path saturates at HOT. Local (temp-table) buffers get the same two-state treatment; being single-backend they need no background cooler. Because the reference bit shares the field, the full 4-bit value can be 0..3; readers that mean "the cooling state" must use BUF_STATE_GET_COOLSTATE(), which masks to bit 0 and returns only 0 (COOL) or 1 (HOT), never the raw field. contrib/pg_buffercache is updated accordingly: its usagecount column and the pg_buffercache_summary average report the cooling state (0 = COOL, 1 = HOT), and pg_buffercache_usage_counts() buckets on it. Using the raw 4-bit getter there would index its BM_MAX_USAGE_COUNT+1 = 2-element arrays with values up to 3 and overrun the stack; masking to the cooling bit keeps the index in range. The reference bit is deliberately not exposed as usagecount. Depends on the batched clock sweep from the previous commit. --- contrib/pg_buffercache/pg_buffercache_pages.c | 6 +- src/backend/storage/buffer/bufmgr.c | 130 ++++++++++++++---- src/backend/storage/buffer/freelist.c | 114 +++++++++------ src/backend/storage/buffer/localbuf.c | 13 +- src/include/storage/buf_internals.h | 63 +++++++-- 5 files changed, 239 insertions(+), 87 deletions(-) diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c index 5104559..b94e6f6 100644 --- a/contrib/pg_buffercache/pg_buffercache_pages.c +++ b/contrib/pg_buffercache/pg_buffercache_pages.c @@ -161,7 +161,7 @@ pg_buffercache_pages(PG_FUNCTION_ARGS) reldatabase = bufHdr->tag.dbOid; forknum = BufTagGetForkNum(&bufHdr->tag); blocknum = bufHdr->tag.blockNum; - usagecount = BUF_STATE_GET_USAGECOUNT(buf_state); + usagecount = BUF_STATE_GET_COOLSTATE(buf_state); pinning_backends = BUF_STATE_GET_REFCOUNT(buf_state); if (buf_state & BM_DIRTY) @@ -605,7 +605,7 @@ pg_buffercache_summary(PG_FUNCTION_ARGS) if (buf_state & BM_VALID) { buffers_used++; - usagecount_total += BUF_STATE_GET_USAGECOUNT(buf_state); + usagecount_total += BUF_STATE_GET_COOLSTATE(buf_state); if (buf_state & BM_DIRTY) buffers_dirty++; @@ -655,7 +655,7 @@ pg_buffercache_usage_counts(PG_FUNCTION_ARGS) CHECK_FOR_INTERRUPTS(); - usage_count = BUF_STATE_GET_USAGECOUNT(buf_state); + usage_count = BUF_STATE_GET_COOLSTATE(buf_state); usage_counts[usage_count]++; if (buf_state & BM_DIRTY) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 3908529..8689266 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -83,6 +83,7 @@ /* Bits in SyncOneBuffer's return value */ #define BUF_WRITTEN 0x01 #define BUF_REUSABLE 0x02 +#define BUF_COOLED 0x04 #define RELS_BSEARCH_THRESHOLD 20 @@ -634,7 +635,7 @@ static void PinBuffer_Locked(BufferDesc *buf); static void UnpinBuffer(BufferDesc *buf); static void UnpinBufferNoOwner(BufferDesc *buf); static void BufferSync(int flags); -static int SyncOneBuffer(int buf_id, bool skip_recently_used, +static int SyncOneBuffer(int buf_id, bool skip_recently_used, bool cool_if_hot, WritebackContext *wb_context); static void WaitIO(BufferDesc *buf); static void AbortBufferIO(Buffer buffer); @@ -2333,7 +2334,10 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, * checkpoints, except for their "init" forks, which need to be treated * just like permanent relations. */ - set_bits |= BM_TAG_VALID | BUF_USAGECOUNT_ONE; + set_bits |= BM_TAG_VALID; + /* Admit the newly loaded page COOL (probation); a second access via + * PinBuffer promotes it to HOT. This is what makes a one-touch scan + * self-evicting -- see the cooling-state notes in buf_internals.h. */ if (relpersistence == RELPERSISTENCE_PERMANENT || forkNum == INIT_FORKNUM) set_bits |= BM_PERMANENT; @@ -3002,7 +3006,9 @@ ExtendBufferedRelShared(BufferManagerRelation bmr, victim_buf_hdr->tag = tag; - set_bits |= BM_TAG_VALID | BUF_USAGECOUNT_ONE; + set_bits |= BM_TAG_VALID; + /* Admit COOL (probation); see the comment at the other admission + * site and the cooling-state notes in buf_internals.h. */ if (bmr.relpersistence == RELPERSISTENCE_PERMANENT || fork == INIT_FORKNUM) set_bits |= BM_PERMANENT; @@ -3332,21 +3338,17 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy, /* increase refcount */ buf_state += BUF_REFCOUNT_ONE; - if (strategy == NULL) - { - /* Default case: increase usagecount unless already max. */ - if (BUF_STATE_GET_USAGECOUNT(buf_state) < BM_MAX_USAGE_COUNT) - buf_state += BUF_USAGECOUNT_ONE; - } - else - { - /* - * Ring buffers shouldn't evict others from pool. Thus we - * don't make usagecount more than 1. - */ - if (BUF_STATE_GET_USAGECOUNT(buf_state) == 0) - buf_state += BUF_USAGECOUNT_ONE; - } + /* + * Accessing a resident buffer promotes it to HOT (the 2Q rescue): + * a page loaded COOL on probation becomes part of the hot working + * set on its second touch. BM_MAX_USAGE_COUNT is + * BUF_COOLSTATE_HOT (1), so this saturates at HOT and never + * overflows the field. We also set the second-chance ref bit so + * the bgwriter's next cooling pass spares this recently-used buffer. + */ + if (BUF_STATE_GET_COOLSTATE(buf_state) < BUF_COOLSTATE_HOT) + buf_state += BUF_COOLSTATE_ONE; + buf_state |= BUF_REFBIT; if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, buf_state)) @@ -3785,7 +3787,7 @@ BufferSync(int flags) */ if (pg_atomic_read_u64(&bufHdr->state) & BM_CHECKPOINT_NEEDED) { - if (SyncOneBuffer(buf_id, false, &wb_context) & BUF_WRITTEN) + if (SyncOneBuffer(buf_id, false, false, &wb_context) & BUF_WRITTEN) { TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id); PendingCheckpointerStats.buffers_written++; @@ -3876,6 +3878,19 @@ BgBufferSync(WritebackContext *wb_context) float smoothing_samples = 16; float scan_whole_pool_milliseconds = 120000.0; + /* + * The cleaner scan directly observes the reusable (COOL, unpinned) buffer + * density over the region it walks, which -- with the cooling-stage + * evictor -- is exactly the sweep's victim predicate. That observation is + * ground truth for the buffers about to be reused, whereas the strategy + * scan's positional proxy (strategy_delta/recent_alloc) blurs a pool whose + * COOL population is spatially clustered (a scan burst leaves whole regions + * COOL, hot OLTP regions not). So we let the cleaner's own sample adapt on + * a shorter window than the strategy proxy, tracking a burst of + * probationary/scan COOL pages within a cycle or two instead of lagging it. + */ + float cleaner_smoothing_samples = 4; + /* Used to compute how far we scan ahead */ long strategy_delta; int bufs_to_lap; @@ -3889,6 +3904,7 @@ BgBufferSync(WritebackContext *wb_context) int num_to_scan; int num_written; int reusable_buffers; + int write_limit; /* Variables for final smoothed_density update */ long new_strategy_delta; @@ -4063,8 +4079,22 @@ BgBufferSync(WritebackContext *wb_context) * Now write out dirty reusable buffers, working forward from the * next_to_clean point, until we have lapped the strategy scan, or cleaned * enough buffers to match our estimate of the next cycle's allocation - * requirements, or hit the bgwriter_lru_maxpages limit. - */ + * requirements, or hit the write limit. + * + * The per-cycle write cap is normally bgwriter_lru_maxpages. But under a + * bulk-dirtying workload (COPY, bulk UPDATE, VACUUM) the pool fills with + * dirty COOL buffers faster than that fixed cap can clean, so the + * foreground clock sweep is forced to flush dirty victims inline -- the + * very cost the cooling-stage evictor is meant to keep off the critical + * path. When predicted demand (upcoming_alloc_est) exceeds the fixed cap, + * raise the limit to meet demand so the bgwriter stays ahead and supplies + * clean victims. This stays bounded (by demand and by lapping the + * strategy point), so it cannot run away, and normal workloads are + * unaffected because there upcoming_alloc_est <= bgwriter_lru_maxpages. + */ + write_limit = bgwriter_lru_maxpages; + if (upcoming_alloc_est > write_limit) + write_limit = upcoming_alloc_est; num_to_scan = bufs_to_lap; num_written = 0; @@ -4073,7 +4103,13 @@ BgBufferSync(WritebackContext *wb_context) /* Execute the LRU scan */ while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est) { - int sync_state = SyncOneBuffer(next_to_clean, true, + /* + * cool_if_hot = false: with the cool-in-place foreground sweep, the + * clock hand demotes HOT buffers on every tick, so the bgwriter does + * writeback only and leaves all cooling to the sweep (no background + * pre-cooling, no double-cooling). + */ + int sync_state = SyncOneBuffer(next_to_clean, true, false, wb_context); if (++next_to_clean >= NBuffers) @@ -4086,7 +4122,7 @@ BgBufferSync(WritebackContext *wb_context) if (sync_state & BUF_WRITTEN) { reusable_buffers++; - if (++num_written >= bgwriter_lru_maxpages) + if (++num_written >= write_limit) { PendingBgWriterStats.maxwritten_clean++; break; @@ -4121,7 +4157,7 @@ BgBufferSync(WritebackContext *wb_context) { scans_per_alloc = (float) new_strategy_delta / (float) new_recent_alloc; smoothed_density += (scans_per_alloc - smoothed_density) / - smoothing_samples; + cleaner_smoothing_samples; #ifdef BGW_DEBUG elog(DEBUG2, "bgwriter: cleaner density alloc=%u scan=%ld density=%.2f new smoothed=%.2f", @@ -4140,16 +4176,26 @@ BgBufferSync(WritebackContext *wb_context) * If skip_recently_used is true, we don't write currently-pinned buffers, nor * buffers marked recently used, as these are not replacement candidates. * + * If cool_if_hot is true (the bgwriter's LRU scan), an unpinned HOT buffer is + * demoted HOT -> COOL as we pass it, pre-staging eviction candidates so the + * foreground clock sweep finds a COOL victim in a single pass instead of + * having to cool buffers itself (force_cool). The demotion is done under the + * buffer header lock we already hold, so it needs no CAS and cannot race a + * concurrent demotion. A concurrent PinBuffer promotes it back to HOT, which + * is the intended 2Q behavior (a re-accessed buffer is rescued). + * * Returns a bitmask containing the following flag bits: * BUF_WRITTEN: we wrote the buffer. * BUF_REUSABLE: buffer is available for replacement, ie, it has - * pin count 0 and usage count 0. + * pin count 0 and is COOL (an eviction candidate). + * BUF_COOLED: we demoted this buffer HOT -> COOL this call. * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) */ static int -SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) +SyncOneBuffer(int buf_id, bool skip_recently_used, bool cool_if_hot, + WritebackContext *wb_context) { BufferDesc *bufHdr = GetBufferDescriptor(buf_id); int result = 0; @@ -4171,8 +4217,38 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) */ buf_state = LockBufHdr(bufHdr); + /* + * Pre-cool with a second chance: if asked, act on an unpinned HOT buffer. + * If its ref bit is set (accessed since our last pass), clear the ref bit + * and leave it HOT -- a recently-used buffer earns one reprieve, keeping + * the hot working set out of the COOL stage under scan pressure. Only a + * HOT buffer whose ref bit is already clear is demoted HOT -> COOL, + * pre-staging it as an eviction candidate for the foreground sweep. We + * hold the header lock, so each transition is a plain masked store applied + * atomically by UnlockBufHdrExt; we re-lock to continue the dirty-write + * inspection below. + */ + if (cool_if_hot && + BUF_STATE_GET_REFCOUNT(buf_state) == 0 && + BUF_STATE_GET_COOLSTATE(buf_state) != BUF_COOLSTATE_COOL) + { + if (BUF_STATE_GET_REFBIT(buf_state)) + { + /* second chance: consume the ref bit, stay HOT */ + UnlockBufHdrExt(bufHdr, buf_state, 0, BUF_REFBIT, 0); + buf_state = LockBufHdr(bufHdr); + } + else + { + /* not re-accessed since last pass: demote to COOL */ + UnlockBufHdrExt(bufHdr, buf_state, 0, BUF_USAGECOUNT_MASK, 0); + buf_state = LockBufHdr(bufHdr); + result |= BUF_COOLED; + } + } + if (BUF_STATE_GET_REFCOUNT(buf_state) == 0 && - BUF_STATE_GET_USAGECOUNT(buf_state) == 0) + BUF_STATE_GET_COOLSTATE(buf_state) == BUF_COOLSTATE_COOL) { result |= BUF_REUSABLE; } diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c index 46104b3..808f133 100644 --- a/src/backend/storage/buffer/freelist.c +++ b/src/backend/storage/buffer/freelist.c @@ -49,6 +49,16 @@ typedef struct uint32 completePasses; /* Complete cycles of the clock-sweep */ pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */ + /* + * Number of clock-hand values a backend claims per atomic fetch-add. + * Computed once at startup (see StrategyCtlShmemInit). Kept in shared + * memory rather than a backend-local static so that EXEC_BACKEND children + * (which do not inherit the postmaster's statics) see the same value; a + * backend-local copy would silently reset to 1 there, disabling batching + * on Windows. + */ + uint32 batchSize; + /* * Bgworker process to be notified upon activity or -1 if none. See * StrategyNotifyBgWriter. @@ -113,17 +123,6 @@ static void AddBufferToRing(BufferAccessStrategy strategy, static uint32 MyBatchPos = 0; static uint32 MyBatchEnd = 0; -/* - * Number of clock-hand values a backend claims per atomic fetch-add, - * computed once at startup (see StrategyCtlShmemInit). When batching is - * enabled it is one cache line's worth of hand advance, so concurrent - * backends sweep non-overlapping, cache-line-sized runs of the pool; the - * global sweep order is preserved (each buffer is still visited exactly once - * per pass). Batching is enabled only on multi-node NUMA hardware; otherwise - * this stays 1 and the sweep is byte-identical to the stock clock. - */ -static uint32 ClockSweepBatchSize = 1; - /* * ClockSweepTick - Helper routine for StrategyGetBuffer() * @@ -144,7 +143,7 @@ ClockSweepTick(void) * atomic operation per batch, reducing contention by the batch size. */ uint32 start; - uint32 batch_size = ClockSweepBatchSize; + uint32 batch_size = StrategyControl->batchSize; start = pg_atomic_fetch_add_u32(&StrategyControl->nextVictimBuffer, batch_size); @@ -279,12 +278,30 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r */ pg_atomic_fetch_add_u32(&StrategyControl->numBufferAllocs, 1); - /* Use the "clock sweep" algorithm to find a free buffer */ + /* + * Use the cooling-stage clock sweep to find a victim. + * + * A buffer is HOT (recently used) or COOL (an eviction candidate). This + * is a COOL-IN-PLACE sweep: every tick past a HOT buffer demotes it one + * step toward COOL (respecting a single second-chance reference bit), and + * a COOL, unpinned buffer is claimed as the victim. Because every tick + * makes cooling progress, the sweep is self-staging: it never needs a + * background pre-cooler to keep COOL victims available, and it cannot get + * stuck in a "no COOL victim anywhere" state that forces a foreground + * escalation pass. Newly loaded pages are admitted COOL (see + * BufferAlloc), so scan resistance falls out of the algorithm. + * + * trycounter bounds the search: only a pinned buffer is "no progress". + * Demoting or reclaiming is progress and resets it. A full NBuffers pass + * that finds every buffer pinned fails, matching the stock "no unpinned + * buffers available" contract. + */ trycounter = NBuffers; for (;;) { uint64 old_buf_state; uint64 local_buf_state; + bool no_progress = false; buf = GetBufferDescriptor(ClockSweepTick()); @@ -297,25 +314,10 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r { local_buf_state = old_buf_state; - /* - * If the buffer is pinned or has a nonzero usage_count, we cannot - * use it; decrement the usage_count (unless pinned) and keep - * scanning. - */ - + /* If the buffer is pinned we cannot use it; keep scanning. */ if (BUF_STATE_GET_REFCOUNT(local_buf_state) != 0) { - if (--trycounter == 0) - { - /* - * We've scanned all the buffers without making any state - * changes, so all the buffers are pinned (or were when we - * looked at them). We could hope that someone will free - * one eventually, but it's probably better to fail than - * to risk getting stuck in an infinite loop. - */ - elog(ERROR, "no unpinned buffers available"); - } + no_progress = true; break; } @@ -326,9 +328,23 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r continue; } - if (BUF_STATE_GET_USAGECOUNT(local_buf_state) != 0) + if (BUF_STATE_GET_COOLSTATE(local_buf_state) != BUF_COOLSTATE_COOL) { - local_buf_state -= BUF_USAGECOUNT_ONE; + /* + * HOT buffer: cool it in place this tick. Apply a single + * second-chance reference bit: a HOT buffer whose ref bit is + * set (touched since it was last passed) has the ref bit + * cleared and stays HOT; only a HOT buffer whose ref bit is + * already clear is demoted HOT -> COOL. Either transition is + * progress toward a victim, so reset trycounter. We do NOT + * claim the buffer this tick -- a demoted buffer becomes a + * candidate for a later tick, giving it one more full sweep of + * grace before eviction. + */ + if (BUF_STATE_GET_REFBIT(local_buf_state)) + local_buf_state &= ~BUF_REFBIT; /* second chance: clear ref, stay HOT */ + else + local_buf_state &= ~BUF_USAGECOUNT_MASK; /* HOT -> COOL, clear ref bit */ if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, local_buf_state)) @@ -339,7 +355,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r } else { - /* pin the buffer if the CAS succeeds */ + /* COOL and unpinned: claim it. Pin if the CAS succeeds. */ local_buf_state += BUF_REFCOUNT_ONE; if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, @@ -356,6 +372,15 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r } } } + + /* + * Only a pinned buffer is no progress. A full NBuffers pass that + * makes no progress means every buffer is pinned, so fail rather than + * spin forever. (A failed CAS above is neither progress nor a full + * miss: we simply retry the same buffer.) + */ + if (no_progress && --trycounter == 0) + elog(ERROR, "no unpinned buffers available"); } } @@ -470,10 +495,10 @@ StrategyCtlShmemInit(void *arg) */ if (pg_numa_init() != -1 && pg_numa_get_max_node() >= 1) - ClockSweepBatchSize = Min(PG_CACHE_LINE_SIZE / (uint32) sizeof(uint32), - (uint32) NBuffers); + StrategyControl->batchSize = Min(PG_CACHE_LINE_SIZE / (uint32) sizeof(uint32), + (uint32) NBuffers); else - ClockSweepBatchSize = 1; + StrategyControl->batchSize = 1; } @@ -721,14 +746,17 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint64 *buf_state) /* * If the buffer is pinned we cannot use it under any circumstances. * - * If usage_count is 0 or 1 then the buffer is fair game (we expect 1, - * since our own previous usage of the ring element would have left it - * there, but it might've been decremented by clock-sweep since then). - * A higher usage_count indicates someone else has touched the buffer, - * so we shouldn't re-use it. + * If it is unpinned but has been promoted to HOT, another backend + * touched it since we last cycled past this ring slot, so it has + * joined the working set and we must not recycle it -- tell the caller + * to get a fresh victim from the sweep instead. This is the 1-bit + * cooling-state analog of the stock ring's "usage_count > 1 means + * someone else touched it" test: a slot the ring keeps reusing and + * nobody else pins stays COOL, and an out-of-ring PinBuffer() is + * exactly what promotes it to HOT. */ - if (BUF_STATE_GET_REFCOUNT(local_buf_state) != 0 - || BUF_STATE_GET_USAGECOUNT(local_buf_state) > 1) + if (BUF_STATE_GET_REFCOUNT(local_buf_state) != 0 || + BUF_STATE_GET_COOLSTATE(local_buf_state) != BUF_COOLSTATE_COOL) break; /* See equivalent code in PinBuffer() */ diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 4870c8e..21c0809 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -167,7 +167,7 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, buf_state = pg_atomic_read_u64(&bufHdr->state); buf_state &= ~(BUF_FLAG_MASK | BUF_USAGECOUNT_MASK); - buf_state |= BM_TAG_VALID | BUF_USAGECOUNT_ONE; + buf_state |= BM_TAG_VALID; /* admit COOL (probation) */ pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); *foundPtr = false; @@ -248,9 +248,10 @@ GetLocalVictimBuffer(void) { uint64 buf_state = pg_atomic_read_u64(&bufHdr->state); - if (BUF_STATE_GET_USAGECOUNT(buf_state) > 0) + if (BUF_STATE_GET_COOLSTATE(buf_state) != BUF_COOLSTATE_COOL) { - buf_state -= BUF_USAGECOUNT_ONE; + /* HOT: give it a second chance, cool it and keep scanning. */ + buf_state -= BUF_COOLSTATE_ONE; pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); trycounter = NLocBuffer; } @@ -454,7 +455,7 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr, victim_buf_hdr->tag = tag; - buf_state |= BM_TAG_VALID | BUF_USAGECOUNT_ONE; + buf_state |= BM_TAG_VALID; /* admit COOL (probation) */ pg_atomic_unlocked_write_u64(&victim_buf_hdr->state, buf_state); @@ -839,9 +840,9 @@ PinLocalBuffer(BufferDesc *buf_hdr, bool adjust_usagecount) NLocalPinnedBuffers++; buf_state += BUF_REFCOUNT_ONE; if (adjust_usagecount && - BUF_STATE_GET_USAGECOUNT(buf_state) < BM_MAX_USAGE_COUNT) + BUF_STATE_GET_COOLSTATE(buf_state) < BUF_COOLSTATE_HOT) { - buf_state += BUF_USAGECOUNT_ONE; + buf_state += BUF_COOLSTATE_ONE; } pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state); diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h index e4ff561..131df12 100644 --- a/src/include/storage/buf_internals.h +++ b/src/include/storage/buf_internals.h @@ -67,6 +67,50 @@ StaticAssertDecl(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + BUF_FLAG_BITS + BUF_L #define BUF_USAGECOUNT_ONE \ (UINT64CONST(1) << BUF_REFCOUNT_BITS) +/* + * Cooling state (LeanStore / 2Q-A1 cooling-stage clock sweep). + * + * The field historically used for the 0..5 usage_count now holds a single + * cooling-state bit: HOT (recently accessed, not an eviction candidate) or + * COOL (an eviction candidate). We reuse BUF_USAGECOUNT_ONE as the unit so + * the buffer-state bit geography -- refcount, flag, and lock offsets, and the + * 64-bit StaticAsserts -- is unchanged; only the meaning of the field and the + * instructions that touch it change. + * + * A demand-loaded page is admitted COOL (probation); a second access promotes + * it to HOT (the rescue). The sweep prefers COOL victims and demotes HOT to + * COOL only when a full pass finds no COOL victim. So a one-touch scan fills + * and drains the COOL stage without displacing the HOT working set -- scan + * resistance intrinsic to the replacement algorithm. + */ +#define BUF_COOLSTATE_COOL 0 +#define BUF_COOLSTATE_HOT 1 +#define BUF_COOLSTATE_ONE BUF_USAGECOUNT_ONE + +/* + * Second-chance reference bit, bit 1 of the (former usagecount) field, one + * position above the cooling-state bit. PinBuffer sets it on every access. + * The bgwriter's pre-cooling gives a HOT buffer a second chance: the first + * time it passes a HOT buffer whose ref bit is set, it clears the ref bit and + * leaves the buffer HOT; only a HOT buffer whose ref bit is already clear (not + * re-accessed since the previous bgwriter pass) is demoted to COOL. This + * keeps genuinely-hot pages out of the COOL stage (protecting the working set + * from being cooled under scan pressure) while leaving the foreground sweep a + * single-pass search over the pre-staged COOL buffers. A separate bit (not a + * count) so it stays a plain masked store under the header lock. + */ +#define BUF_REFBIT (UINT64CONST(2) << BUF_REFCOUNT_BITS) +#define BUF_STATE_GET_REFBIT(state) (((state) & BUF_REFBIT) != 0) + +/* + * The cooling state occupies bit 0 and the reference bit occupies bit 1 of + * the (former usagecount) field, so the field must be at least 2 bits wide. + * Assert it here so a future change to BUF_USAGECOUNT_BITS cannot silently + * push BUF_REFBIT up into the flag bits. + */ +StaticAssertDecl(BUF_USAGECOUNT_BITS >= 2, + "cooling state + reference bit need at least 2 bits in the usagecount field"); + /* flags related definitions */ #define BUF_FLAG_SHIFT \ (BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS) @@ -92,6 +136,11 @@ StaticAssertDecl(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + BUF_FLAG_BITS + BUF_L #define BUF_STATE_GET_USAGECOUNT(state) \ ((uint32)(((state) & BUF_USAGECOUNT_MASK) >> BUF_USAGECOUNT_SHIFT)) +/* Cooling state (HOT/COOL) from buffer state -- bit 0 of the field only, so + * the second-chance ref bit (bit 1) does not perturb the HOT/COOL test. */ +#define BUF_STATE_GET_COOLSTATE(state) \ + ((uint32) (((state) >> BUF_USAGECOUNT_SHIFT) & 1)) + /* * Flags for buffer descriptors * @@ -134,17 +183,15 @@ StaticAssertDecl(MAX_BACKENDS_BITS <= (BUF_LOCK_BITS - 2), /* - * The maximum allowed value of usage_count represents a tradeoff between - * accuracy and speed of the clock-sweep buffer management algorithm. A - * large value (comparable to NBuffers) would approximate LRU semantics. - * But it can take as many as BM_MAX_USAGE_COUNT+1 complete cycles of the - * clock-sweep hand to find a free buffer, so in practice we don't want the - * value to be very large. + * The cooling state is a single bit (HOT/COOL); the maximum value stored in + * the field is therefore BUF_COOLSTATE_HOT. Retained under the historical + * name BM_MAX_USAGE_COUNT so the pin fast path ("promote unless already at + * max") reads naturally. */ -#define BM_MAX_USAGE_COUNT 5 +#define BM_MAX_USAGE_COUNT BUF_COOLSTATE_HOT StaticAssertDecl(BM_MAX_USAGE_COUNT < (UINT64CONST(1) << BUF_USAGECOUNT_BITS), - "BM_MAX_USAGE_COUNT doesn't fit in BUF_USAGECOUNT_BITS bits"); + "cooling state doesn't fit in BUF_USAGECOUNT_BITS bits"); /* * Buffer tag identifies which disk block the buffer contains. -- 2.50.1
repro.tgz_
Description: Binary data
