While working on recent round of s_lock.h cleanup, I started wondering whether we really need architecture-specific implementations of anything in that file besides SPIN_DELAY(). We've been slowly moving away from spinlocks, and the compiler builtins have been available for ~20 years, so I doubt all of this hand-rolled code is still buying us much. I'd like to see how much of it we can get rid of, and the attached patches are a first step in that direction. This batch just removes some redundancy between platforms.
0001 removes the parts of the ARM section that are redundant with the generic __sync code added by commit 1c72d82c25. The TAS_SPIN() and SPIN_DELAY() definitions for AArch64 stay, and the generated code shouldn't change. 0002 merges the i386 section into the x86-64 one. The only real difference is that i386 does an unlocked test before every xchg, whereas x86-64 only does so in TAS_SPIN(). The comment in the i386 section suggests that i386 should probably follow x86-64's lead, but nobody has tested it. Since we've disclaimed interest in micro-optimizing for 32-bit builds (see commit cb7b2e5e8e), IMHO we should just do it. 0003 makes the unlocked test the default for TAS_SPIN() and removes the platform-specific versions for x86, PPC, and AArch64 (see commits b03d196be0, bc2a050d40, and 3d0b4b1068). Nothing changes for those three, but everything else (e.g., s390, SPARC, MIPS, RISC-V, 32-bit ARM, and all MSVC builds) will now do the unlocked test while waiting for a contended spinlock. We've thus far been hesitant to add the unlocked test to platforms without evidence of improvements, but that means a platform goes without it until someone happens to notice. AArch64 is the best example. It didn't get the unlocked test until v18, more than a decade after we added support for it, despite it being a clear win there. That's one of the main reasons I'd like to make it the default. AFAICT that reasoning should basically apply everywhere, and a platform can still override TAS_SPIN() if it turns out to hurt. Thoughts? -- nathan
>From dacba8aa9cc7404449aaacc830969c57be7d1ceb Mon Sep 17 00:00:00 2001 From: Nathan Bossart <[email protected]> Date: Fri, 20 Nov 2026 12:48:02 -0600 Subject: [PATCH v1 1/3] Remove redundant spinlock code for ARM/ARM64. The ARM/ARM64 section of s_lock.h defines slock_t, tas(), and S_UNLOCK() the same way as the "no platform-specific knowledge" section added by commit 1c72d82c25. That commit left the ARM/ARM64 section alone because it has some special knowledge, but that only amounts to the TAS_SPIN() and SPIN_DELAY() definitions for ARM64. This commit removes everything else so that these platforms use the generic code instead. The generated code should be the same as before. --- src/include/storage/s_lock.h | 25 +++---------------------- 1 file changed, 3 insertions(+), 22 deletions(-) diff --git a/src/include/storage/s_lock.h b/src/include/storage/s_lock.h index 7e049dfac18..02a3feb0d65 100644 --- a/src/include/storage/s_lock.h +++ b/src/include/storage/s_lock.h @@ -231,26 +231,10 @@ spin_delay(void) /* - * On ARM and ARM64, we use __sync_lock_test_and_set(int *, int) if available. - * - * We use the int-width variant of the builtin because it works on more chips - * than other widths. + * For ARM/ARM64, we use the "no platform-specific knowledge" flavors below, + * which use __sync_lock_test_and_set() and friends. However, we do use a + * custom TAS_SPIN() and SPIN_DELAY() for ARM64. */ -#if defined(__arm__) || defined(__aarch64__) -#ifdef HAVE_GCC__SYNC_INT32_TAS - -#define TAS(lock) tas(lock) - -typedef int slock_t; - -static inline int -tas(volatile slock_t *lock) -{ - return __sync_lock_test_and_set(lock, 1); -} - -#define S_UNLOCK(lock) __sync_lock_release(lock) - #if defined(__aarch64__) /* @@ -274,8 +258,6 @@ spin_delay(void) } #endif /* __aarch64__ */ -#endif /* HAVE_GCC__SYNC_INT32_TAS */ -#endif /* __arm__ || __aarch64__ */ /* S/390 and S/390x Linux (32- and 64-bit zSeries) */ @@ -513,7 +495,6 @@ do \ * provides __sync_lock_test_and_set(), use that. Prefer the int-width * version over the char-width version if we have both, on the rather dubious * grounds that that's known to be more likely to work in the ARM ecosystem. - * (But we dealt with ARM above.) */ #if !defined(TAS) -- 2.55.0
>From eb5f3fd323f90174fc15d72f2b93d69e8d3da151 Mon Sep 17 00:00:00 2001 From: Nathan Bossart <[email protected]> Date: Fri, 20 Nov 2026 12:48:14 -0600 Subject: [PATCH v1 2/3] Merge i386 and x86-64 code in s_lock.h. The i386 and x86-64 sections of s_lock.h are nearly identical. The only real difference is that i386 does an unlocked test of the lock before every xchg, whereas x86-64 only does so while spinning (i.e., in TAS_SPIN()). Both commit b03d196be0 and the comment in the i386 section suggest that i386 should probably follow x86-64's lead, but nobody has tested it. Since we've disclaimed interest in micro-optimizing for 32-bit builds (see commit cb7b2e5e8e), let's just merge the i386 section into the x86-64 one. While at it, condense the comments in spin_delay(). --- src/include/storage/s_lock.h | 83 +++++------------------------------- 1 file changed, 10 insertions(+), 73 deletions(-) diff --git a/src/include/storage/s_lock.h b/src/include/storage/s_lock.h index 02a3feb0d65..0cab0410b5b 100644 --- a/src/include/storage/s_lock.h +++ b/src/include/storage/s_lock.h @@ -114,76 +114,11 @@ */ -#ifdef __i386__ /* 32-bit i386 */ - -typedef unsigned char slock_t; - -#define TAS(lock) tas(lock) - -static inline int -tas(volatile slock_t *lock) -{ - slock_t _res = 1; - - /* - * Use a non-locking test before asserting the bus lock. Note that the - * extra test appears to be a small loss on some x86 platforms and a small - * win on others; it's by no means clear that we should keep it. - * - * When this was last tested, we didn't have separate TAS() and TAS_SPIN() - * macros. Nowadays it probably would be better to do a non-locking test - * in TAS_SPIN() but not in TAS(), like on x86_64, but no-one's done the - * testing to verify that. Without some empirical evidence, better to - * leave it alone. - */ - __asm__ __volatile__( - " cmpb $0,%1 \n" - " jne TAS%=_out \n" - " lock \n" - " xchgb %0,%1 \n" - "TAS%=_out: \n" -: "+q"(_res), "+m"(*lock) -: /* no inputs */ -: "memory", "cc"); - return (int) _res; -} - -#define SPIN_DELAY() spin_delay() - -static inline void -spin_delay(void) -{ - /* - * This sequence is equivalent to the PAUSE instruction ("rep" is - * ignored by old IA32 processors if the following instruction is - * not a string operation); the IA-32 Architecture Software - * Developer's Manual, Vol. 3, Section 7.7.2 describes why using - * PAUSE in the inner loop of a spin lock is necessary for good - * performance: - * - * The PAUSE instruction improves the performance of IA-32 - * processors supporting Hyper-Threading Technology when - * executing spin-wait loops and other routines where one - * thread is accessing a shared lock or semaphore in a tight - * polling loop. When executing a spin-wait loop, the - * processor can suffer a severe performance penalty when - * exiting the loop because it detects a possible memory order - * violation and flushes the core processor's pipeline. The - * PAUSE instruction provides a hint to the processor that the - * code sequence is a spin-wait loop. The processor uses this - * hint to avoid the memory order violation and prevent the - * pipeline flush. In addition, the PAUSE instruction - * de-pipelines the spin-wait loop to prevent it from - * consuming execution resources excessively. - */ - __asm__ __volatile__( - " rep; nop \n"); -} - -#endif /* __i386__ */ - - -#ifdef __x86_64__ /* AMD Opteron, Intel EM64T */ +/* + * For x86 (both 32-bit and 64-bit), we have custom asm blocks for both TAS() + * and SPIN_DELAY(). + */ +#if defined(__i386__) || defined(__x86_64__) typedef unsigned char slock_t; @@ -220,14 +155,16 @@ static inline void spin_delay(void) { /* - * Adding a PAUSE in the spin delay loop is demonstrably a no-op on - * Opteron, but it may be of some use on EM64T, so we keep it. + * "rep; nop" is the PAUSE instruction. Intel recommends it in spin-wait + * loops: it hints that the loop is a spin-wait, which avoids a pipeline + * flush when the loop exits, and it keeps the loop from hogging execution + * resources. Processors that predate PAUSE treat it as a plain NOP. */ __asm__ __volatile__( " rep; nop \n"); } -#endif /* __x86_64__ */ +#endif /* __i386__ || __x86_64__ */ /* -- 2.55.0
>From fe0febb064cfcdee2bd96fd5d3bf5954fc3cdf1d Mon Sep 17 00:00:00 2001 From: Nathan Bossart <[email protected]> Date: Fri, 20 Nov 2026 14:05:02 -0600 Subject: [PATCH v1 3/3] Use unlocked test before test-and-set in TAS_SPIN() by default. TAS_SPIN() is used when waiting for a contended spinlock. On x86, PPC, and ARM64, it does an unlocked test of the lock and only retries the atomic test-and-set once the lock appears free (see commits b03d196be0, bc2a050d40, and 3d0b4b1068). Everywhere else, it is the same as TAS(). This commit makes the unlocked test the default, which allows us to remove the three platform-specific definitions. The link to Intel's paper in the x86 comment is dead, so I didn't bother preserving it. We've thus far been hesitant to add the unlocked test to platforms without evidence of improvements, but the reasoning should apply everywhere. Note that this only changes how other platforms (e.g., s390, SPARC, MIPS, RISC-V, 32-bit ARM, and all MSVC builds) wait for a contended spinlock. The first attempt to acquire a spinlock still uses TAS(). --- src/include/storage/s_lock.h | 32 ++++++-------------------------- 1 file changed, 6 insertions(+), 26 deletions(-) diff --git a/src/include/storage/s_lock.h b/src/include/storage/s_lock.h index 0cab0410b5b..a61e6d6eb95 100644 --- a/src/include/storage/s_lock.h +++ b/src/include/storage/s_lock.h @@ -38,10 +38,10 @@ * * int TAS_SPIN(slock_t *lock) * Like TAS(), but this version is used when waiting for a lock - * previously found to be contended. By default, this is the - * same as TAS(), but on some architectures it's better to poll a - * contended lock using an unlocked instruction and retry the - * atomic test-and-set only when it appears free. + * previously found to be contended. By default, this polls a + * contended lock using an unlocked instruction and retries the atomic + * test-and-set only when it appears free. On some architectures, it + * may be better to override this to be the same as TAS(). * * TAS() and TAS_SPIN() are NOT part of the API, and should never be called * directly. A platform must provide either S_LOCK() or a TAS() for the @@ -124,17 +124,6 @@ typedef unsigned char slock_t; #define TAS(lock) tas(lock) -/* - * On Intel EM64T, it's a win to use a non-locking test before the xchg proper, - * but only when spinning. - * - * See also Implementing Scalable Atomic Locks for Multi-Core Intel(tm) EM64T - * and IA32, by Michael Chynoweth and Mary R. Lee. As of this writing, it is - * available at: - * http://software.intel.com/en-us/articles/implementing-scalable-atomic-locks-for-multi-core-intel-em64t-and-ia32-architectures - */ -#define TAS_SPIN(lock) (*(lock) ? 1 : TAS(lock)) - static inline int tas(volatile slock_t *lock) { @@ -170,16 +159,10 @@ spin_delay(void) /* * For ARM/ARM64, we use the "no platform-specific knowledge" flavors below, * which use __sync_lock_test_and_set() and friends. However, we do use a - * custom TAS_SPIN() and SPIN_DELAY() for ARM64. + * custom SPIN_DELAY() for ARM64. */ #if defined(__aarch64__) -/* - * On ARM64, it's a win to use a non-locking test before the TAS proper. It - * may be a win on 32-bit ARM, too, but nobody's tested it yet. - */ -#define TAS_SPIN(lock) (*(lock) ? 1 : TAS(lock)) - #define SPIN_DELAY() spin_delay() static inline void @@ -302,9 +285,6 @@ typedef unsigned int slock_t; #define TAS(lock) tas(lock) -/* On PPC, it's a win to use a non-locking test before the lwarx */ -#define TAS_SPIN(lock) (*(lock) ? 1 : TAS(lock)) - /* * The second operand of addi can hold a constant zero or a register number, * hence constraint "=&b" to avoid allocating r0. "b" stands for "address @@ -605,7 +585,7 @@ extern void s_unlock(volatile slock_t *lock); * so we only provide a default when there is a TAS to base it on. */ #if !defined(TAS_SPIN) && defined(TAS) -#define TAS_SPIN(lock) TAS(lock) +#define TAS_SPIN(lock) (*(lock) ? 1 : TAS(lock)) #endif /* ! TAS_SPIN && TAS */ -- 2.55.0
