On Wed, Aug 12, 2026 at 1:15 PM Hongtao Liu <[email protected]> wrote:
>
> On Wed, Aug 12, 2026 at 7:59 AM H.J. Lu <[email protected]> wrote:
> >
> > On Tue, Aug 11, 2026 at 7:15 PM H.J. Lu <[email protected]> wrote:
> > >
> > > On Tue, Aug 11, 2026 at 11:32 AM Hongtao Liu <[email protected]> wrote:
> > > >
> > > > On Wed, Jul 22, 2026 at 2:24 PM H.J. Lu <[email protected]> wrote:
> > > > >
> > > > > Changes in v2:
> > > > >
> > > > > 1. Add TARGET_128BIT_ATOMIC_ENABLED.
> > > > > 2. Remove __atomic Builtins reference from -mcx16.
> > > > > 3. Update -m128bit-atomic documentation.
> > > > > 4. Issue an error for -m128bit-atomic with -m32.
> > > > >
> > > > > --
> > > > > H.J.
> > > > > --
> > > > > With the silicon vendor guarantees from Intel, AMD, Hygon and Zhaoxin
> > > > > in:
> > > > >
> > > > > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104688
> > > > >
> > > > > many software developers would happily use inline 128-bit atomic loads
> > > > > and stores in their programs because they only target compatible CPUs.
> > > > > Add -m128bit-atomic to generate 128-bit atomic loads and stores to
> > > > > avoid
> > > > > the overhead of calling into libatomic. Enable -m128bit-atomic in
> > > > > 64-bit
> > > > > mode by default if supported by the targeting processor, which is one
> > > > > of
> > > > > x86-64-v3 capable processors as well as AVX capable processors from
> > > > > Intel,
> > > > > AMD, Hygon and Zhaoxin.
> > > > >
> > > > > gcc/
> > > > >
> > > > > PR target/94649
> > > > > PR target/126293
> > > > > * common/config/i386/i386-cpuinfo.h (ix86_decode_cpu_info): New
> > > > > function.
> > > > > * config/i386/i386-options.cc
> > > > > (ix86_option_override_internal): Issue an error for -m128bit-atomic
> > > > > in 32-bit mode. Turn on -mcx16 if -m128bit-atomic is enabled.
> > > > > Enable -m128bit-atomic in 64-bit mode by default if supported by
> > > > > the targeting processor.
> > > > > * config/i386/i386.h (TARGET_128BIT_ATOMIC_ENABLED): New.
> > > > > * config/i386/i386.opt (ix86_flags): New Variable.
> > > > > (m128bit-atomic): New option.
> > > > > * config/i386/i386.opt.urls: Regenerated.
> > > > > * config/i386/sync.md (atomic_loadti): New pattern.
> > > > > (atomic_loadti_sse): Likewise.
> > > > > (atomic_storeti): Likewise.
> > > > > (atomic_storeti_sse): Likewise.
> > > > > * doc/invoke.texi: Remove __atomic Builtins reference from -mcx16.
> > > > > Document -m128bit-atomic.
> > > > >
> > > >
> > > > + || (subtype >= INTEL_COREI7_GRANITERAPIDS
> > > > + && subtype <= INTEL_COREI7_PANTHERLAKE)
> > > > + || subtype == INTEL_COREI7_DIAMONDRAPIDS
> > > > + || subtype <= INTEL_COREI7_NOVALAKE)
> > > > + {
> > > > + vendor = VENDOR_INTEL;
> > > > + type = INTEL_COREI7;
> > > > + }
> > > > || substype <= INTEL_COREI7_NOVALAKE classifies AMD/Zhaoxin CPUs as
> > > > Intel, is it a typo of *subtype == NTEL_COREI7_NOVALAKE*
> > >
> > > Fixed.
> > >
> > > >
> > > > + if (TARGET_128BIT_ATOMIC_P (opts->x_ix86_flags))
> > > > + {
> > > > + if (!TARGET_64BIT_P (opts->x_ix86_isa_flags))
> > > > + error ("%<-m128bit-atomic%> not supported for 32-bit code");
> > > > + else if (!TARGET_CX16_P (opts_set->x_ix86_isa_flags2))
> > > > + {
> > > > + /* Enable CMPXCHG16B when -m128bit-atomic is enabled. */
> > > > + opts->x_ix86_isa_flags2 |= OPTION_MASK_ISA2_CX16;
> > > > + }
> > > > + }
> > > >
> > > > With global -march=skylake, a function marked target("arch=x86-64")
> > > > still emits movdqa atomic loads. The non-explicit bit needs clearing
> > > > before deriving the default for each target option set.
> > >
> > > Fixed.
> > >
> > > > +Variable
> > > > +int ix86_flags = 0
> > > >
> > > > Can we reuse ix86_target_flags, m128bit-atomic only need one bit.
> > >
> > > Fixed.
> > >
> > > > +(define_expand "atomic_loadti"
> > > > + [(set (match_operand:TI 0 "nonimmediate_operand")
> > > > + (unspec:TI [(match_operand:TI 1 "memory_operand")
> > > > + (match_operand:SI 2 "const_int_operand")]
> > > > + UNSPEC_LDA))]
> > > > + "TARGET_128BIT_ATOMIC_ENABLED"
> > > > +{
> > > > + emit_insn (gen_atomic_loadti_sse (operands[0], operands[1]));
> > > > + DONE;
> > > > +})
> > > >
> > > > change operands[0] from nonimmediate_operand to register_operand?
> > >
> > > It is done on purpose the same way as
> > >
> > > (define_expand "atomic_load<mode>"
> > > [(set (match_operand:ATOMIC 0 "nonimmediate_operand")
> > > (unspec:ATOMIC [(match_operand:ATOMIC 1 "memory_operand")
> > > (match_operand:SI 2 "const_int_operand")]
> > > UNSPEC_LDA))]
> > > ""
> This one either generates atomic_loaddi_fpu which supports
> memory_operand for operands[0], or manually handles memory operand for
> operands[0].
> But atomic_loadti expander directly generates atomic_loadti_sse which
> will hit unrecognized insn when operands[0] is memory_operand. For
> simplicity, maybe change nonimmediate_operand to register_operand.
Changed to register_operand in v4.
> 169 /* For DImode on 32-bit, we can use the FPU to perform the load. */
> 170 if (<MODE>mode == DImode && !TARGET_64BIT)
> 171 emit_insn (gen_atomic_loaddi_fpu
> 172 (operands[0], operands[1],
> 173 assign_stack_temp (DImode, GET_MODE_SIZE (DImode))));
> 174 else
> 175 {
> 176 rtx dst = operands[0];
> 177
> 178 if (MEM_P (dst))
> 179 dst = gen_reg_rtx (<MODE>mode);
> 180
> 181 emit_move_insn (dst, operands[1]);
> 182
> 183 /* Fix up the destination if needed. */
> 184 if (dst != operands[0])
> 185 emit_move_insn (operands[0], dst);
>
>
>
> >153;; Importantly, *no* processor makes atomicity guarantees for larger
> >154;; accesses. In particular, there's no way to perform an atomic TImode
> >155;; move, despite the apparent applicability of MOVDQA et al.
>
> We may need to change the comments in sync.md:153 after 128-bit atomic
> load is supported.
I replaced them with
;; AVX capable processors from Intel, AMD, Hygon and Zhaoxin guarantee
;; that 128-bit aligned vector loads and stores are atomic.
Here is the v4 patch. OK for master if there are no regressions?
Thanks.
--
H.J.
From 6ad207c9040687cf843e8766c43dd666d4c67c67 Mon Sep 17 00:00:00 2001
From: "H.J. Lu" <[email protected]>
Date: Fri, 17 Jul 2026 08:35:34 +0800
Subject: [PATCH v4] x86-64: Add -m128bit-atomic
With the silicon vendor guarantees from Intel, AMD, Hygon and Zhaoxin in:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104688
many software developers would happily use inline 128-bit atomic loads
and stores in their programs because they only target compatible CPUs.
Add -m128bit-atomic to generate 128-bit atomic loads and stores to avoid
the overhead of calling into libatomic. Enable -m128bit-atomic in 64-bit
mode by default if supported by the targeting processor, which is one of
x86-64-v3 capable processors as well as AVX capable processors from Intel,
AMD, Hygon and Zhaoxin.
gcc/
PR target/94649
PR target/126293
* common/config/i386/i386-cpuinfo.h (ix86_decode_cpu_info): New
function.
* config/i386/i386-options.cc
(ix86_option_override_internal): Issue an error for -m128bit-atomic
in 32-bit mode. Turn on -mcx16 if -m128bit-atomic is enabled.
Enable -m128bit-atomic in 64-bit mode by default if supported by
the targeting processor.
* config/i386/i386.h (TARGET_128BIT_ATOMIC_ENABLED): New.
* config/i386/i386.opt (m128bit-atomic): New option.
* config/i386/i386.opt.urls: Regenerated.
* config/i386/sync.md: Update comments for 128-bit aligned vector
loads and stores.
(atomic_loadti): New pattern.
(atomic_loadti_sse): Likewise.
(atomic_storeti): Likewise.
(atomic_storeti_sse): Likewise.
* doc/invoke.texi: Remove __atomic Builtins reference from -mcx16.
Document -m128bit-atomic.
gcc/testsuite/
PR target/94649
PR target/126293
* g++.target/i386/pr94649-1.C: New test.
* gcc.target/i386/pr126293-1a.c: Likewise.
* gcc.target/i386/pr126293-1b.c: Likewise.
* gcc.target/i386/pr126293-1c.c: Likewise.
* gcc.target/i386/pr126293-1d.c: Likewise.
* gcc.target/i386/pr126293-2a.c: Likewise.
* gcc.target/i386/pr126293-2b.c: Likewise.
* gcc.target/i386/pr126293-3a.c: Likewise.
* gcc.target/i386/pr126293-3b.c: Likewise.
* gcc.target/i386/pr126293-4a.c: Likewise.
* gcc.target/i386/pr126293-4b.c: Likewise.
* gcc.target/i386/pr126293-4c.c: Likewise.
* gcc.target/i386/pr126293-4d.c: Likewise.
* gcc.target/i386/pr126293-4e.c: Likewise.
* gcc.target/i386/pr126293-4f.c: Likewise.
* gcc.target/i386/pr126293-4g.c: Likewise.
* gcc.target/i386/pr126293-4h.c: Likewise.
* gcc.target/i386/pr126293-4i.c: Likewise.
* gcc.target/i386/pr126293-4j.c: Likewise.
* gcc.target/i386/pr126293-4k.c: Likewise.
* gcc.target/i386/pr126293-4l.c: Likewise.
* gcc.target/i386/pr126293-4m.c: Likewise.
* gcc.target/i386/pr126293-4n.c: Likewise.
* gcc.target/i386/pr126293-4o.c: Likewise.
* gcc.target/i386/pr126293-4p.c: Likewise.
* gcc.target/i386/pr126293-4q.c: Likewise.
* gcc.target/i386/pr126293-4r.c: Likewise.
* gcc.target/i386/pr126293-4s.c: Likewise.
* gcc.target/i386/pr126293-5.c: Likewise.
* gcc.target/i386/pr126293-6.c: Likewise.
* gcc.target/i386/pr126293-7.c: Likewise.
* gcc.target/i386/pr126293-8.c: Likewise.
* gcc.target/i386/pr126293-9.c: Likewise.
* gcc.target/i386/pr126293-10.c: Likewise.
* gcc.target/i386/pr126293-11.c: Likewise.
* gcc.target/i386/pr126293-12.c: Likewise.
* gcc.target/i386/pr126293-13.c: Likewise.
* gcc.target/i386/pr126293-14.c: Likewise.
* gcc.target/i386/pr126293-15.c: Likewise.
* gcc.target/i386/pr126293-16.c: Likewise.
* gcc.target/i386/pr126293-17.c: Likewise.
Signed-off-by: H.J. Lu <[email protected]>
---
gcc/common/config/i386/i386-cpuinfo.h | 126 ++++++++++++++++++++
gcc/config/i386/i386-options.cc | 50 ++++++++
gcc/config/i386/i386.h | 9 ++
gcc/config/i386/i386.opt | 4 +
gcc/config/i386/i386.opt.urls | 3 +
gcc/config/i386/sync.md | 60 +++++++++-
gcc/doc/invoke.texi | 24 +++-
gcc/testsuite/g++.target/i386/pr94649-1.C | 17 +++
gcc/testsuite/gcc.target/i386/pr126293-10.c | 26 ++++
gcc/testsuite/gcc.target/i386/pr126293-11.c | 24 ++++
gcc/testsuite/gcc.target/i386/pr126293-12.c | 29 +++++
gcc/testsuite/gcc.target/i386/pr126293-13.c | 28 +++++
gcc/testsuite/gcc.target/i386/pr126293-14.c | 31 +++++
gcc/testsuite/gcc.target/i386/pr126293-15.c | 30 +++++
gcc/testsuite/gcc.target/i386/pr126293-16.c | 3 +
gcc/testsuite/gcc.target/i386/pr126293-17.c | 4 +
gcc/testsuite/gcc.target/i386/pr126293-1a.c | 22 ++++
gcc/testsuite/gcc.target/i386/pr126293-1b.c | 6 +
gcc/testsuite/gcc.target/i386/pr126293-1c.c | 6 +
gcc/testsuite/gcc.target/i386/pr126293-1d.c | 6 +
gcc/testsuite/gcc.target/i386/pr126293-2a.c | 23 ++++
gcc/testsuite/gcc.target/i386/pr126293-2b.c | 55 +++++++++
gcc/testsuite/gcc.target/i386/pr126293-3a.c | 21 ++++
gcc/testsuite/gcc.target/i386/pr126293-3b.c | 45 +++++++
gcc/testsuite/gcc.target/i386/pr126293-4a.c | 31 +++++
gcc/testsuite/gcc.target/i386/pr126293-4b.c | 19 +++
gcc/testsuite/gcc.target/i386/pr126293-4c.c | 19 +++
gcc/testsuite/gcc.target/i386/pr126293-4d.c | 19 +++
gcc/testsuite/gcc.target/i386/pr126293-4e.c | 19 +++
gcc/testsuite/gcc.target/i386/pr126293-4f.c | 19 +++
gcc/testsuite/gcc.target/i386/pr126293-4g.c | 19 +++
gcc/testsuite/gcc.target/i386/pr126293-4h.c | 19 +++
gcc/testsuite/gcc.target/i386/pr126293-4i.c | 19 +++
gcc/testsuite/gcc.target/i386/pr126293-4j.c | 19 +++
gcc/testsuite/gcc.target/i386/pr126293-4k.c | 19 +++
gcc/testsuite/gcc.target/i386/pr126293-4l.c | 19 +++
gcc/testsuite/gcc.target/i386/pr126293-4m.c | 19 +++
gcc/testsuite/gcc.target/i386/pr126293-4n.c | 19 +++
gcc/testsuite/gcc.target/i386/pr126293-4o.c | 19 +++
gcc/testsuite/gcc.target/i386/pr126293-4p.c | 19 +++
gcc/testsuite/gcc.target/i386/pr126293-4q.c | 19 +++
gcc/testsuite/gcc.target/i386/pr126293-4r.c | 20 ++++
gcc/testsuite/gcc.target/i386/pr126293-4s.c | 20 ++++
gcc/testsuite/gcc.target/i386/pr126293-5.c | 25 ++++
gcc/testsuite/gcc.target/i386/pr126293-6.c | 24 ++++
gcc/testsuite/gcc.target/i386/pr126293-7.c | 23 ++++
gcc/testsuite/gcc.target/i386/pr126293-8.c | 24 ++++
gcc/testsuite/gcc.target/i386/pr126293-9.c | 23 ++++
48 files changed, 1140 insertions(+), 6 deletions(-)
create mode 100644 gcc/testsuite/g++.target/i386/pr94649-1.C
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-10.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-11.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-12.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-13.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-14.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-15.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-16.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-17.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-1a.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-1b.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-1c.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-1d.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-2a.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-2b.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-3a.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-3b.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-4a.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-4b.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-4c.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-4d.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-4e.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-4f.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-4g.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-4h.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-4i.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-4j.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-4k.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-4l.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-4m.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-4n.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-4o.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-4p.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-4q.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-4r.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-4s.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-5.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-6.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-7.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-8.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr126293-9.c
diff --git a/gcc/common/config/i386/i386-cpuinfo.h b/gcc/common/config/i386/i386-cpuinfo.h
index 5b9bf5db68b..dd3b4e46d8b 100644
--- a/gcc/common/config/i386/i386-cpuinfo.h
+++ b/gcc/common/config/i386/i386-cpuinfo.h
@@ -300,3 +300,129 @@ enum processor_features
#define M_VENDOR(a) (a)
#define M_CPU_TYPE(a) (M_CPU_TYPE_START + int (a))
#define M_CPU_SUBTYPE(a) (M_CPU_SUBTYPE_START + a)
+
+#ifdef __cplusplus
+/* Extract processor VENDOR, TYPE and SUBTYPE from MODEL. */
+
+static inline void
+ix86_decode_cpu_info (int model, processor_vendor &vendor,
+ processor_types &type,
+ processor_subtypes &subtype)
+{
+ if (model > M_CPU_SUBTYPE_START)
+ {
+ subtype = (processor_subtypes) (model - M_CPU_SUBTYPE_START);
+ if (subtype <= INTEL_COREI7_SANDYBRIDGE
+ || (subtype >= INTEL_COREI7_IVYBRIDGE
+ && subtype <= INTEL_COREI7_ICELAKE_SERVER)
+ || (subtype >= INTEL_COREI7_CASCADELAKE
+ && subtype <= INTEL_COREI7_ALDERLAKE)
+ || subtype == INTEL_COREI7_ROCKETLAKE
+ || (subtype >= INTEL_COREI7_GRANITERAPIDS
+ && subtype <= INTEL_COREI7_PANTHERLAKE)
+ || subtype == INTEL_COREI7_DIAMONDRAPIDS
+ || subtype == INTEL_COREI7_NOVALAKE)
+ {
+ vendor = VENDOR_INTEL;
+ type = INTEL_COREI7;
+ }
+ else if (subtype >= AMDFAM10H_BARCELONA
+ && subtype <= AMDFAM10H_ISTANBUL)
+ {
+ vendor = VENDOR_AMD;
+ type = AMDFAM10H;
+ }
+ else if (subtype >= AMDFAM15H_BDVER1
+ && subtype <= AMDFAM15H_BDVER4)
+ {
+ vendor = VENDOR_AMD;
+ type = AMDFAM15H;
+ }
+ else if (subtype == AMDFAM17H_ZNVER1
+ || subtype == AMDFAM17H_ZNVER2)
+ {
+ vendor = VENDOR_AMD;
+ type = AMDFAM17H;
+ }
+ else if (subtype == AMDFAM19H_ZNVER3
+ || subtype == AMDFAM19H_ZNVER4)
+ {
+ vendor = VENDOR_AMD;
+ type = AMDFAM19H;
+ }
+ else if (subtype == AMDFAM1AH_ZNVER5
+ || subtype == AMDFAM1AH_ZNVER6)
+ {
+ vendor = VENDOR_AMD;
+ type = AMDFAM1AH;
+ }
+ else if (subtype == ZHAOXIN_FAM7H_LUJIAZUI
+ || subtype == ZHAOXIN_FAM7H_YONGFENG
+ || subtype == ZHAOXIN_FAM7H_SHIJIDADAO)
+ {
+ vendor = VENDOR_ZHAOXIN;
+ type = ZHAOXIN_FAM7H;
+ }
+ else if (subtype >= HYGONFAM18H_C86_4G_M4
+ && subtype <= HYGONFAM18H_C86_4G_M8)
+ {
+ vendor = VENDOR_HYGON;
+ type = HYGONFAM18H;
+ }
+ else
+ {
+ /* Set VENDOR to VENDOR_OTHER and TYPE to CPU_TYPE_MAX for
+ unsupported SUBTYPE. */
+ vendor = VENDOR_OTHER;
+ type = CPU_TYPE_MAX;
+ }
+ }
+ else if (model > M_CPU_TYPE_START)
+ {
+ type = (processor_types) (model - M_CPU_TYPE_START);
+ if ((type >= INTEL_BONNELL && type <= INTEL_COREI7)
+ || type == INTEL_SILVERMONT
+ || (type >= INTEL_GOLDMONT && type <= INTEL_TREMONT)
+ || (type >= INTEL_SIERRAFOREST
+ && type <= INTEL_CLEARWATERFOREST))
+ {
+ vendor = VENDOR_INTEL;
+ subtype = CPU_SUBTYPE_MAX;
+ }
+ else if (type == AMDFAM10H
+ || type == AMDFAM15H
+ || (type >= AMD_BTVER1 && type <= AMDFAM17H)
+ || type == AMDFAM19H
+ || type == AMDFAM1AH)
+ {
+ vendor = VENDOR_AMD;
+ subtype = CPU_SUBTYPE_MAX;
+ }
+ else if (type == ZHAOXIN_FAM7H)
+ {
+ vendor = VENDOR_ZHAOXIN;
+ subtype = CPU_SUBTYPE_MAX;
+ }
+ else if (type == HYGONFAM18H)
+ {
+ vendor = VENDOR_HYGON;
+ subtype = CPU_SUBTYPE_MAX;
+ }
+ else
+ {
+ /* Set VENDOR to VENDOR_OTHER and SUBTYPE to CPU_SUBTYPE_MAX
+ for unsupported TYPE. */
+ vendor = VENDOR_OTHER;
+ subtype = CPU_SUBTYPE_MAX;
+ }
+ }
+ else
+ {
+ vendor = (processor_vendor) model;
+ /* Set TYPE to CPU_TYPE_MAX and SUBTYPE to CPU_SUBTYPE_MAX for
+ unknown TYPE and SUBTYPE. */
+ type = CPU_TYPE_MAX;
+ subtype = CPU_SUBTYPE_MAX;
+ }
+}
+#endif
diff --git a/gcc/config/i386/i386-options.cc b/gcc/config/i386/i386-options.cc
index afbd96b79c3..6bb5e1a08d9 100644
--- a/gcc/config/i386/i386-options.cc
+++ b/gcc/config/i386/i386-options.cc
@@ -2305,6 +2305,11 @@ ix86_option_override_internal (bool main_args_p,
sorry ("%i-bit mode not compiled in",
(opts->x_ix86_isa_flags & OPTION_MASK_ISA_64BIT) ? 64 : 32);
+ processor_vendor ix86_vendor = VENDOR_OTHER;
+ processor_types ix86_type = CPU_TYPE_MAX;
+ processor_subtypes ix86_subtype = CPU_SUBTYPE_MAX;
+ wide_int_bitmask ix86_pta = 0;
+
/* Last processor_alias_table must point to "generic" entry. */
gcc_checking_assert (strcmp (processor_alias_table[pta_size - 1].name,
"generic") == 0);
@@ -2340,6 +2345,10 @@ ix86_option_override_internal (bool main_args_p,
ix86_schedule = processor_alias_table[i].schedule;
ix86_arch = processor_alias_table[i].processor;
+ ix86_pta = processor_alias_table[i].flags;
+
+ ix86_decode_cpu_info (processor_alias_table[i].model,
+ ix86_vendor, ix86_type, ix86_subtype);
/* Default cpu tuning to the architecture, unless the table
entry requests not to do this. Used by the x86-64 psABI
@@ -2528,6 +2537,47 @@ ix86_option_override_internal (bool main_args_p,
XDELETEVEC (s);
}
+ /* Disable -m128bit-atomic if not set on command-line. */
+ if (!TARGET_128BIT_ATOMIC_P (opts_set->x_ix86_target_flags))
+ opts->x_ix86_target_flags &= ~OPTION_MASK_128BIT_ATOMIC;
+
+ if (TARGET_128BIT_ATOMIC_P (opts->x_ix86_target_flags))
+ {
+ if (!TARGET_64BIT_P (opts->x_ix86_isa_flags))
+ error ("%<-m128bit-atomic%> not supported for 32-bit code");
+ else if (!TARGET_CX16_P (opts_set->x_ix86_isa_flags2))
+ {
+ /* Enable CMPXCHG16B when -m128bit-atomic is enabled. */
+ opts->x_ix86_isa_flags2 |= OPTION_MASK_ISA2_CX16;
+ }
+ }
+ else if (!TARGET_128BIT_ATOMIC_P (opts_set->x_ix86_target_flags)
+ && TARGET_64BIT_P (opts->x_ix86_isa_flags)
+ && ((TARGET_AVX_P (opts->x_ix86_isa_flags)
+ && (ix86_vendor == VENDOR_INTEL
+ || ix86_vendor == VENDOR_AMD
+ || ix86_vendor == VENDOR_HYGON
+ || ix86_vendor == VENDOR_ZHAOXIN))
+ || (ix86_pta & PTA_X86_64_V3) == PTA_X86_64_V3
+ || ix86_type == AMDFAM17H
+ || ix86_type == AMDFAM19H
+ || ix86_type == AMDFAM1AH
+ || ix86_type == HYGONFAM18H
+ || (ix86_type >= INTEL_SIERRAFOREST
+ && ix86_type <= INTEL_CLEARWATERFOREST)
+ || ix86_subtype == INTEL_COREI7_SANDYBRIDGE
+ || (ix86_subtype >= AMDFAM17H_ZNVER1
+ && ix86_subtype <= INTEL_COREI7_ROCKETLAKE)
+ || (ix86_subtype >= AMDFAM19H_ZNVER4
+ && ix86_subtype <= HYGONFAM18H_C86_4G_M8)))
+ {
+ /* Turn on -m128bit-atomic in 64-bit mode by default if supported
+ by the targeting processor, which is one of x86-64-v3 capable
+ processors as well as AVX capable processors from Intel, AMD,
+ Hygon and Zhaoxin. */
+ opts->x_ix86_target_flags |= OPTION_MASK_128BIT_ATOMIC;
+ }
+
set_ix86_tune_features (opts, ix86_tune, opts->x_ix86_dump_tunes);
ix86_recompute_optlev_based_flags (opts, opts_set);
diff --git a/gcc/config/i386/i386.h b/gcc/config/i386/i386.h
index d4082a84059..26c3b73a50f 100644
--- a/gcc/config/i386/i386.h
+++ b/gcc/config/i386/i386.h
@@ -3085,6 +3085,15 @@ extern void debug_dispatch_window (int);
#define TARGET_RECIP_VEC_DIV ((recip_mask & RECIP_MASK_VEC_DIV) != 0)
#define TARGET_RECIP_VEC_SQRT ((recip_mask & RECIP_MASK_VEC_SQRT) != 0)
+/* -m128bit-atomic requires CMPXCHG16B and SSE2.
+
+ Note: Pre-Tiger Lake (Desktop/Mobile): Generations including Kaby
+ Lake, Coffee Lake, and Comet Lake (e.g., Pentium Gold G5400, Celeron
+ G5900, N4020) do not support AVX. However, 128-bit aligned SSE loads
+ and stores are atomic on these processors. Should AVX be required? */
+#define TARGET_128BIT_ATOMIC_ENABLED \
+ (TARGET_CX16 && TARGET_SSE2 && TARGET_128BIT_ATOMIC)
+
/* Use 128-bit AVX instructions in the auto-vectorizer. */
#define TARGET_PREFER_AVX128 (prefer_vector_width_type == PVW_AVX128)
/* Use 256-bit AVX instructions in the auto-vectorizer. */
diff --git a/gcc/config/i386/i386.opt b/gcc/config/i386/i386.opt
index 2cdc71a2324..57ba3fbfa8f 100644
--- a/gcc/config/i386/i386.opt
+++ b/gcc/config/i386/i386.opt
@@ -1043,6 +1043,10 @@ mgeneral-regs-only
Target RejectNegative Mask(GENERAL_REGS_ONLY) Var(ix86_target_flags) Save
Generate code which uses only the general registers.
+m128bit-atomic
+Target Mask(128BIT_ATOMIC) Var(ix86_target_flags) Save
+Generate cmpxchg16b, 128-bit atomic vector loads and stores for the x86-64 architecture.
+
mshstk
Target Mask(ISA_SHSTK) Var(ix86_isa_flags) Save
Enable shadow stack built-in functions from Control-flow Enforcement
diff --git a/gcc/config/i386/i386.opt.urls b/gcc/config/i386/i386.opt.urls
index d0c58ca3526..d52e6de8014 100644
--- a/gcc/config/i386/i386.opt.urls
+++ b/gcc/config/i386/i386.opt.urls
@@ -464,6 +464,9 @@ UrlSuffix(gcc/x86-Options.html#index-mstack-protector-guard-symbol)
mgeneral-regs-only
UrlSuffix(gcc/x86-Options.html#index-mgeneral-regs-only-2)
+m128bit-atomic
+UrlSuffix(gcc/x86-Options.html#index-m128bit-atomic)
+
mshstk
UrlSuffix(gcc/x86-Options.html#index-mshstk)
diff --git a/gcc/config/i386/sync.md b/gcc/config/i386/sync.md
index d9f822f4821..19bf1757aee 100644
--- a/gcc/config/i386/sync.md
+++ b/gcc/config/i386/sync.md
@@ -150,9 +150,8 @@ (define_expand "mem_thread_fence"
;;
;; Note that the TARGET_CMPXCHG8B test below is a stand-in for "Pentium".
;;
-;; Importantly, *no* processor makes atomicity guarantees for larger
-;; accesses. In particular, there's no way to perform an atomic TImode
-;; move, despite the apparent applicability of MOVDQA et al.
+;; AVX capable processors from Intel, AMD, Hygon and Zhaoxin guarantee
+;; that 128-bit aligned vector loads and stores are atomic.
(define_mode_iterator ATOMIC
[QI HI SI
@@ -388,6 +387,61 @@ (define_insn "storedi_via_sse"
[(set_attr "type" "ssemov")
(set_attr "mode" "DI")])
+(define_expand "atomic_loadti"
+ [(set (match_operand:TI 0 "register_operand")
+ (unspec:TI [(match_operand:TI 1 "memory_operand")
+ (match_operand:SI 2 "const_int_operand")]
+ UNSPEC_LDA))]
+ "TARGET_128BIT_ATOMIC_ENABLED"
+{
+ emit_insn (gen_atomic_loadti_sse (operands[0], operands[1]));
+ DONE;
+})
+
+(define_insn_and_split "atomic_loadti_sse"
+ [(set (match_operand:TI 0 "register_operand" "=v")
+ (unspec:TI [(match_operand:TI 1 "memory_operand" "m")]
+ UNSPEC_LDX_ATOMIC))]
+ "TARGET_128BIT_ATOMIC_ENABLED"
+ "#"
+ "&& reload_completed"
+ [(set (match_dup 0) (match_dup 1))])
+
+(define_expand "atomic_storeti"
+ [(set (match_operand:TI 0 "memory_operand")
+ (unspec:TI [(match_operand:TI 1 "nonimmediate_operand")
+ (match_operand:SI 2 "const_int_operand")]
+ UNSPEC_STA))]
+ "TARGET_128BIT_ATOMIC_ENABLED"
+{
+ enum memmodel model = memmodel_from_int (INTVAL (operands[2]));
+
+ /* Use V1TImode to force vector register for atomic store. */
+ rtx src = gen_reg_rtx (V1TImode);
+ rtx op1 = gen_lowpart (V1TImode, operands[1]);
+ emit_move_insn (src, op1);
+ emit_insn (gen_atomic_storeti_sse (operands[0], src));
+
+ /* ... followed by an MFENCE, if required. */
+ if (is_mm_seq_cst (model))
+ emit_insn (gen_mem_thread_fence (operands[2]));
+ DONE;
+})
+
+(define_insn_and_split "atomic_storeti_sse"
+ [(set (match_operand:TI 0 "memory_operand" "=m")
+ (unspec:TI [(match_operand:V1TI 1 "nonimmediate_operand" "v")]
+ UNSPEC_STX_ATOMIC))]
+ "TARGET_128BIT_ATOMIC_ENABLED"
+ "#"
+ "&& reload_completed"
+ [(const_int 0)]
+{
+ rtx src = gen_lowpart (TImode, operands[1]);
+ emit_move_insn (operands[0], src);
+ DONE;
+})
+
(define_expand "atomic_compare_and_swap<mode>"
[(match_operand:QI 0 "register_operand") ;; bool success output
(match_operand:SWI124 1 "register_operand") ;; oldval output
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index 7c65c924271..2e53a8c8a7e 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -1576,7 +1576,7 @@ See RS/6000 and PowerPC Options.
-mstack-protector-guard-offset=@var{offset}
-mstack-protector-guard-symbol=@var{symbol}
-mgeneral-regs-only -mcall-ms2sysv-xlogues -mtls-dialect=@var{type}
--mrelax-cmpxchg-loop
+-m128bit-atomic -mrelax-cmpxchg-loop
-mindirect-branch=@var{choice} -mfunction-return=@var{choice}
-mindirect-branch-register -mharden-sls=@var{choice}
-mindirect-branch-cs-prefix -mapx-inline-asm-use-gpr32
@@ -36726,8 +36726,7 @@ This option enables GCC to generate @code{CMPXCHG16B} instructions in 64-bit
code to implement compare-and-exchange operations on 16-byte aligned 128-bit
objects. This is useful for atomic updates of data structures exceeding one
machine word in size. The compiler uses this instruction to implement
-@ref{__sync Builtins}. However, for @ref{__atomic Builtins} operating on
-128-bit integers, a library call is always used.
+@ref{__sync Builtins}.
@opindex msahf
@opindex mno-sahf
@@ -37292,6 +37291,25 @@ Generate code that uses only the general-purpose registers. This
prevents the compiler from using floating-point, vector, mask and bound
registers.
+@opindex m128bit-atomic
+@opindex mno-128bit-atomic
+@item -m128bit-atomic
+Generate cmpxchg16b, 128-bit atomic vector load and store instructions
+for the x86-64 architecture. This is safe to use only on x86-64-v3
+capable processors as well as AVX capable processors from Intel, AMD,
+Hygon and Zhaoxin, which guarantee that 128-bit aligned vector loads
+and stores are atomic. This option requires SSE2 and CMPXCHG16B. It
+implies @option{-mcx16} which enables CMPXCHG16B. This option is
+enabled in 64-bit mode by default if supported by the targeting
+processor. For @ref{__atomic Builtins} operating on 128-bit integers,
+this option generates atomic instructions directly, instead of calling
+a library function in the @file{libatomic} runtime library.
+
+Note that when SSE2 is disabled, i.e., by @option{-mgeneral-regs-only},
+128-bit atomic operations are unsupported and @file{libatomic} shouldn't
+be used since it may use SSE2 or AVX instructions to implement 128-bit
+atomic operations.
+
@opindex mrelax-cmpxchg-loop
@opindex mno-relax-cmpxchg-loop
@item -mrelax-cmpxchg-loop
diff --git a/gcc/testsuite/g++.target/i386/pr94649-1.C b/gcc/testsuite/g++.target/i386/pr94649-1.C
new file mode 100644
index 00000000000..7871e319134
--- /dev/null
+++ b/gcc/testsuite/g++.target/i386/pr94649-1.C
@@ -0,0 +1,17 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -std=c++11 -march=x86-64 -m128bit-atomic" } */
+/* { dg-final { scan-assembler-times "lock;?\[ \\t\]+cmpxchg16b" 1 } } */
+
+#include <atomic>
+
+struct alignas(16) a
+{
+ long x;
+ long y;
+};
+
+bool
+cmpxchg(std::atomic<a>& data, a expected, a newval)
+{
+ return std::atomic_compare_exchange_weak(&data, &expected, newval);
+}
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-10.c b/gcc/testsuite/gcc.target/i386/pr126293-10.c
new file mode 100644
index 00000000000..462042c29bb
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-10.c
@@ -0,0 +1,26 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -fno-pic -std=c23 -march=x86-64 -m128bit-atomic" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc'). */
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-linux* } {^\t?\.} } } */
+
+/*
+**func:
+**.LFB0:
+** .cfi_startproc
+** fldt 8\(%rsp\)
+** fstpt -24\(%rsp\)
+** movdqa -24\(%rsp\), %xmm0
+** movaps %xmm0, store\(%rip\)
+** lock orq \$0, \(%rsp\)
+** ret
+** .cfi_endproc
+**...
+*/
+
+extern _Atomic long double store;
+
+void
+func (long double i)
+{
+ store = i;
+}
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-11.c b/gcc/testsuite/gcc.target/i386/pr126293-11.c
new file mode 100644
index 00000000000..aa65acda009
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-11.c
@@ -0,0 +1,24 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -fno-pic -std=c23 -march=x86-64 -m128bit-atomic" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc'). */
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-linux* } {^\t?\.} } } */
+
+/*
+**func:
+**.LFB0:
+** .cfi_startproc
+** movdqa load\(%rip\), %xmm0
+** movaps %xmm0, -24\(%rsp\)
+** fldt -24\(%rsp\)
+** ret
+** .cfi_endproc
+**...
+*/
+
+extern _Atomic long double load;
+
+long double
+func (void)
+{
+ return load;
+}
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-12.c b/gcc/testsuite/gcc.target/i386/pr126293-12.c
new file mode 100644
index 00000000000..065b779c409
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-12.c
@@ -0,0 +1,29 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -fno-pic -std=c23 -march=x86-64 -m128bit-atomic" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc'). */
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-linux* } {^\t?\.} } } */
+
+/*
+**func:
+**.LFB0:
+** .cfi_startproc
+** movaps %xmm0, store\(%rip\)
+** lock orq \$0, \(%rsp\)
+** ret
+** .cfi_endproc
+**...
+*/
+
+typedef float vector __attribute__((vector_size (16)));
+typedef struct
+{
+ vector v;
+} v;
+
+extern _Atomic v store;
+
+void
+func (v i)
+{
+ store = i;
+}
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-13.c b/gcc/testsuite/gcc.target/i386/pr126293-13.c
new file mode 100644
index 00000000000..a46eb736347
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-13.c
@@ -0,0 +1,28 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -fno-pic -std=c23 -march=x86-64 -m128bit-atomic" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc'). */
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-linux* } {^\t?\.} } } */
+
+/*
+**func:
+**.LFB0:
+** .cfi_startproc
+** movdqa load\(%rip\), %xmm0
+** ret
+** .cfi_endproc
+**...
+*/
+
+typedef float vector __attribute__((vector_size (16)));
+typedef struct
+{
+ vector v;
+} v;
+
+extern _Atomic v load;
+
+v
+func (void)
+{
+ return load;
+}
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-14.c b/gcc/testsuite/gcc.target/i386/pr126293-14.c
new file mode 100644
index 00000000000..14d9822b476
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-14.c
@@ -0,0 +1,31 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -fno-pic -std=c23 -march=x86-64 -m128bit-atomic" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc'). */
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-linux* } {^\t?\.} } } */
+
+/*
+**func:
+**.LFB0:
+** .cfi_startproc
+** movq %rdi, %xmm0
+** movq %rsi, %xmm1
+** punpcklqdq %xmm1, %xmm0
+** movaps %xmm0, store\(%rip\)
+** lock orq \$0, \(%rsp\)
+** ret
+** .cfi_endproc
+**...
+*/
+
+typedef struct
+{
+ int a[4];
+} s;
+
+extern _Atomic s store;
+
+void
+func (s i)
+{
+ store = i;
+}
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-15.c b/gcc/testsuite/gcc.target/i386/pr126293-15.c
new file mode 100644
index 00000000000..058e432058d
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-15.c
@@ -0,0 +1,30 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -fno-pic -std=c23 -march=x86-64 -m128bit-atomic" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc'). */
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-linux* } {^\t?\.} } } */
+
+/*
+**func:
+**.LFB0:
+** .cfi_startproc
+** movdqa load\(%rip\), %xmm0
+** movaps %xmm0, -24\(%rsp\)
+** movq -24\(%rsp\), %rax
+** movq -16\(%rsp\), %rdx
+** ret
+** .cfi_endproc
+**...
+*/
+
+typedef struct
+{
+ int a[4];
+} s;
+
+_Atomic s load;
+
+s
+func (void)
+{
+ return load;
+}
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-16.c b/gcc/testsuite/gcc.target/i386/pr126293-16.c
new file mode 100644
index 00000000000..4c4d486bc30
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-16.c
@@ -0,0 +1,3 @@
+/* { dg-do compile { target ia32 } } */
+/* { dg-additional-options "-O2 -march=x86-64 -m128bit-atomic" } */
+/* { dg-error "'-m128bit-atomic' not supported for 32-bit code" "" { target *-*-* } 0 } */
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-17.c b/gcc/testsuite/gcc.target/i386/pr126293-17.c
new file mode 100644
index 00000000000..aafee4797fb
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-17.c
@@ -0,0 +1,4 @@
+/* { dg-do compile { target ia32 } } */
+/* { dg-additional-options "-O2 -march=x86-64-v3" } */
+
+#include <x86intrin.h>
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-1a.c b/gcc/testsuite/gcc.target/i386/pr126293-1a.c
new file mode 100644
index 00000000000..cd34926a377
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-1a.c
@@ -0,0 +1,22 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -march=x86-64 -m128bit-atomic" } */
+/* { dg-final { scan-assembler-times "lock;?\[ \\t\]+cmpxchg16b" 8 } } */
+
+#include <stdint.h>
+
+#define FUNC_ATOMIC(TYPE, OP) \
+__attribute__ ((noinline, noclone)) \
+TYPE f_##TYPE##_##OP##_fetch (TYPE *a, TYPE b) \
+{ \
+ return __atomic_##OP##_fetch (a, b, __ATOMIC_RELAXED); \
+} \
+__attribute__ ((noinline, noclone)) \
+TYPE f_##TYPE##_fetch_##OP (TYPE *a, TYPE b) \
+{ \
+ return __atomic_fetch_##OP (a, b, __ATOMIC_RELAXED); \
+}
+
+FUNC_ATOMIC (__int128_t, and)
+FUNC_ATOMIC (__int128_t, nand)
+FUNC_ATOMIC (__int128_t, or)
+FUNC_ATOMIC (__int128_t, xor)
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-1b.c b/gcc/testsuite/gcc.target/i386/pr126293-1b.c
new file mode 100644
index 00000000000..bbd74e4c73a
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-1b.c
@@ -0,0 +1,6 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -march=x86-64 -mno-cx16 -m128bit-atomic" } */
+/* { dg-final { scan-assembler-times "call\[ \\t\]+__atomic_fetch_" 4 } } */
+/* { dg-final { scan-assembler-times "jmp\[ \\t\]+__atomic_fetch_" 4 } } */
+
+#include "pr126293-1a.c"
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-1c.c b/gcc/testsuite/gcc.target/i386/pr126293-1c.c
new file mode 100644
index 00000000000..85fe25cae8b
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-1c.c
@@ -0,0 +1,6 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -march=x86-64 -m128bit-atomic -mno-cx16" } */
+/* { dg-final { scan-assembler-times "call\[ \\t\]+__atomic_fetch_" 4 } } */
+/* { dg-final { scan-assembler-times "jmp\[ \\t\]+__atomic_fetch_" 4 } } */
+
+#include "pr126293-1a.c"
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-1d.c b/gcc/testsuite/gcc.target/i386/pr126293-1d.c
new file mode 100644
index 00000000000..6ca19cb1060
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-1d.c
@@ -0,0 +1,6 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -march=x86-64 -m128bit-atomic" } */
+/* { dg-final { scan-assembler-times "lock;?\[ \\t\]+cmpxchg16b" 8 } } */
+
+#include <x86intrin.h>
+#include "pr126293-1a.c"
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-2a.c b/gcc/testsuite/gcc.target/i386/pr126293-2a.c
new file mode 100644
index 00000000000..d2a11579d2f
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-2a.c
@@ -0,0 +1,23 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -march=x86-64 -m128bit-atomic -mrelax-cmpxchg-loop" } */
+/* { dg-final { scan-assembler-times "rep;?\[ \\t\]+nop" 8 } } */
+/* { dg-final { scan-assembler-times "lock;?\[ \\t\]+cmpxchg16b" 8 } } */
+
+#include <stdint.h>
+
+#define FUNC_ATOMIC(TYPE, OP) \
+__attribute__ ((noinline, noclone)) \
+TYPE f_##TYPE##_##OP##_fetch (TYPE *a, TYPE b) \
+{ \
+ return __atomic_##OP##_fetch (a, b, __ATOMIC_RELAXED); \
+} \
+__attribute__ ((noinline, noclone)) \
+TYPE f_##TYPE##_fetch_##OP (TYPE *a, TYPE b) \
+{ \
+ return __atomic_fetch_##OP (a, b, __ATOMIC_RELAXED); \
+}
+
+FUNC_ATOMIC (__int128_t, and)
+FUNC_ATOMIC (__int128_t, nand)
+FUNC_ATOMIC (__int128_t, or)
+FUNC_ATOMIC (__int128_t, xor)
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-2b.c b/gcc/testsuite/gcc.target/i386/pr126293-2b.c
new file mode 100644
index 00000000000..b74579c7cb0
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-2b.c
@@ -0,0 +1,55 @@
+/* { dg-do run { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -march=x86-64 -m128bit-atomic" } */
+
+#include <stdlib.h>
+#include "pr126293-2a.c"
+
+#define FUNC_ATOMIC_RELAX(TYPE, OP) \
+__attribute__ ((noinline, noclone, target ("relax-cmpxchg-loop"))) \
+TYPE relax_##TYPE##_##OP##_fetch (TYPE *a, TYPE b) \
+{ \
+ return __atomic_##OP##_fetch (a, b, __ATOMIC_RELAXED); \
+} \
+__attribute__ ((noinline, noclone, target ("relax-cmpxchg-loop"))) \
+TYPE relax_##TYPE##_fetch_##OP (TYPE *a, TYPE b) \
+{ \
+ return __atomic_fetch_##OP (a, b, __ATOMIC_RELAXED); \
+}
+
+FUNC_ATOMIC_RELAX (__int128_t, and)
+FUNC_ATOMIC_RELAX (__int128_t, nand)
+FUNC_ATOMIC_RELAX (__int128_t, or)
+FUNC_ATOMIC_RELAX (__int128_t, xor)
+
+#define TEST_ATOMIC_FETCH_LOGIC(TYPE, OP) \
+{ \
+ TYPE a = 11, b = 101, res, exp; \
+ TYPE c = 11, d = 101; \
+ res = relax_##TYPE##_##OP##_fetch (&a, b); \
+ exp = f_##TYPE##_##OP##_fetch (&c, d); \
+ if (res != exp || a != c) \
+ abort (); \
+ a = c = 21, b = d = 92; \
+ res = relax_##TYPE##_fetch_##OP (&a, b); \
+ exp = f_##TYPE##_fetch_##OP (&c, d); \
+ if (res != exp || a != c) \
+ abort (); \
+}
+
+__attribute__((noinline))
+static void
+do_test (void)
+{
+ TEST_ATOMIC_FETCH_LOGIC (__int128_t, and)
+ TEST_ATOMIC_FETCH_LOGIC (__int128_t, nand)
+ TEST_ATOMIC_FETCH_LOGIC (__int128_t, or)
+ TEST_ATOMIC_FETCH_LOGIC (__int128_t, xor)
+}
+
+int
+main (void)
+{
+ if (__builtin_cpu_supports ("cmpxchg16b"))
+ do_test ();
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-3a.c b/gcc/testsuite/gcc.target/i386/pr126293-3a.c
new file mode 100644
index 00000000000..48630f5f6b6
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-3a.c
@@ -0,0 +1,21 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -march=x86-64 -m128bit-atomic -mrelax-cmpxchg-loop" } */
+/* { dg-final { scan-assembler-times "lock;?\[ \\t\]+cmpxchg16b" 1 } } */
+
+#include <stdint.h>
+
+#define FUNC_CMPXCHG(TYPE) \
+__attribute__ ((noinline, noclone)) \
+TYPE f_##TYPE##_cmpxchg (TYPE *lock, TYPE newval, TYPE oldval) \
+{ \
+ do \
+ { \
+ newval = oldval | 1; \
+ } while (! __atomic_compare_exchange_n (lock, &oldval, newval, \
+ 0, __ATOMIC_RELEASE, \
+ __ATOMIC_RELAXED)); \
+ return *lock; \
+}
+
+
+FUNC_CMPXCHG (__int128_t)
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-3b.c b/gcc/testsuite/gcc.target/i386/pr126293-3b.c
new file mode 100644
index 00000000000..1e9d1872f8e
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-3b.c
@@ -0,0 +1,45 @@
+/* { dg-do run { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -march=x86-64 -m128bit-atomic" } */
+
+#include <stdlib.h>
+#include "pr126293-3a.c"
+
+#define FUNC_CMPXCHG_RELAX(TYPE) \
+__attribute__ ((noinline, noclone, target ("relax-cmpxchg-loop"))) \
+TYPE relax_##TYPE##_cmpxchg (TYPE *lock, TYPE newval, TYPE oldval) \
+{ \
+ do \
+ { \
+ newval = oldval | 1; \
+ } while (! __atomic_compare_exchange_n (lock, &oldval, newval, \
+ 0, __ATOMIC_RELEASE, \
+ __ATOMIC_RELAXED)); \
+ return *lock; \
+}
+
+FUNC_CMPXCHG_RELAX (__int128_t)
+
+#define TEST_CMPXCHG_LOOP(TYPE) \
+{ \
+ TYPE a = 11, b = 20, c = 11, res, exp; \
+ TYPE d = 11, e = 20, f = 11; \
+ res = relax_##TYPE##_cmpxchg (&a, b, c); \
+ exp = f_##TYPE##_cmpxchg (&d, e, f); \
+ if (res != exp || a != d) \
+ abort (); \
+}
+
+__attribute__((noinline))
+static void
+do_test (void)
+{
+ TEST_CMPXCHG_LOOP (__int128_t)
+}
+
+int
+main (void)
+{
+ if (__builtin_cpu_supports ("cmpxchg16b"))
+ do_test ();
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-4a.c b/gcc/testsuite/gcc.target/i386/pr126293-4a.c
new file mode 100644
index 00000000000..547f5fc9469
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-4a.c
@@ -0,0 +1,31 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -fno-pic -std=c23 -march=x86-64 -m128bit-atomic" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc'). */
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-linux* } {^\t?\.} } } */
+
+/*
+**func:
+**.LFB0:
+** .cfi_startproc
+** movq %rdi, %xmm0
+** movq %rsi, %xmm1
+** punpcklqdq %xmm1, %xmm0
+** movaps %xmm0, store\(%rip\)
+** lock orq \$0, \(%rsp\)
+** ret
+** .cfi_endproc
+**...
+*/
+
+#ifndef ATTRIBUTE
+#define ATTRIBUTE
+#endif
+
+extern _Atomic __uint128_t store;
+
+ATTRIBUTE
+void
+func (__uint128_t i)
+{
+ store = i;
+}
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-4b.c b/gcc/testsuite/gcc.target/i386/pr126293-4b.c
new file mode 100644
index 00000000000..8a3ec685db8
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-4b.c
@@ -0,0 +1,19 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -fno-pic -std=c23 -march=diamondrapids" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc'). */
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-linux* } {^\t?\.} } } */
+
+/*
+**func:
+**.LFB0:
+** .cfi_startproc
+** vmovq %rdi, %xmm1
+** vpinsrq \$1, %rsi, %xmm1, %xmm0
+** vmovdqa %xmm0, store\(%rip\)
+** lock orq \$0, \(%rsp\)
+** ret
+** .cfi_endproc
+**...
+*/
+
+#include "pr126293-4a.c"
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-4c.c b/gcc/testsuite/gcc.target/i386/pr126293-4c.c
new file mode 100644
index 00000000000..f1848cdb543
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-4c.c
@@ -0,0 +1,19 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -fno-pic -std=c23 -march=nehalem -mavx" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc'). */
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-linux* } {^\t?\.} } } */
+
+/*
+**func:
+**.LFB0:
+** .cfi_startproc
+** vmovq %rdi, %xmm1
+** vpinsrq \$1, %rsi, %xmm1, %xmm0
+** vmovdqa %xmm0, store\(%rip\)
+** lock orq \$0, \(%rsp\)
+** ret
+** .cfi_endproc
+**...
+*/
+
+#include "pr126293-4a.c"
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-4d.c b/gcc/testsuite/gcc.target/i386/pr126293-4d.c
new file mode 100644
index 00000000000..39caeee001b
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-4d.c
@@ -0,0 +1,19 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -fno-pic -std=c23 -march=znver1" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc'). */
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-linux* } {^\t?\.} } } */
+
+/*
+**func:
+**.LFB0:
+** .cfi_startproc
+** vmovq %rdi, %xmm1
+** vpinsrq \$1, %rsi, %xmm1, %xmm0
+** vmovdqa %xmm0, store\(%rip\)
+** lock orq \$0, \(%rsp\)
+** ret
+** .cfi_endproc
+**...
+*/
+
+#include "pr126293-4a.c"
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-4e.c b/gcc/testsuite/gcc.target/i386/pr126293-4e.c
new file mode 100644
index 00000000000..9981ae138b5
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-4e.c
@@ -0,0 +1,19 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -fno-pic -std=c23 -march=yongfeng" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc'). */
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-linux* } {^\t?\.} } } */
+
+/*
+**func:
+**.LFB0:
+** .cfi_startproc
+** vmovq %rdi, %xmm1
+** vpinsrq \$1, %rsi, %xmm1, %xmm0
+** vmovdqa %xmm0, store\(%rip\)
+** lock orq \$0, \(%rsp\)
+** ret
+** .cfi_endproc
+**...
+*/
+
+#include "pr126293-4a.c"
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-4f.c b/gcc/testsuite/gcc.target/i386/pr126293-4f.c
new file mode 100644
index 00000000000..3931a3e8bcc
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-4f.c
@@ -0,0 +1,19 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -fno-pic -std=c23 -march=diamondrapids -mno-avx" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc'). */
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-linux* } {^\t?\.} } } */
+
+/*
+**func:
+**.LFB0:
+** .cfi_startproc
+** movq %rdi, %xmm0
+** pinsrq \$1, %rsi, %xmm0
+** movaps %xmm0, store\(%rip\)
+** lock orq \$0, \(%rsp\)
+** ret
+** .cfi_endproc
+**...
+*/
+
+#include "pr126293-4a.c"
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-4g.c b/gcc/testsuite/gcc.target/i386/pr126293-4g.c
new file mode 100644
index 00000000000..77ff16507ef
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-4g.c
@@ -0,0 +1,19 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -fno-pic -std=c23 -march=sandybridge -mno-avx" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc'). */
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-linux* } {^\t?\.} } } */
+
+/*
+**func:
+**.LFB0:
+** .cfi_startproc
+** movq %rdi, %xmm0
+** pinsrq \$1, %rsi, %xmm0
+** movaps %xmm0, store\(%rip\)
+** lock orq \$0, \(%rsp\)
+** ret
+** .cfi_endproc
+**...
+*/
+
+#include "pr126293-4a.c"
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-4h.c b/gcc/testsuite/gcc.target/i386/pr126293-4h.c
new file mode 100644
index 00000000000..6327e4185fe
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-4h.c
@@ -0,0 +1,19 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -fno-pic -std=c23 -march=ivybridge -mno-avx" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc'). */
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-linux* } {^\t?\.} } } */
+
+/*
+**func:
+**.LFB0:
+** .cfi_startproc
+** movq %rdi, %xmm0
+** pinsrq \$1, %rsi, %xmm0
+** movaps %xmm0, store\(%rip\)
+** lock orq \$0, \(%rsp\)
+** ret
+** .cfi_endproc
+**...
+*/
+
+#include "pr126293-4a.c"
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-4i.c b/gcc/testsuite/gcc.target/i386/pr126293-4i.c
new file mode 100644
index 00000000000..db9f8b49ec3
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-4i.c
@@ -0,0 +1,19 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -fno-pic -std=c23 -march=znver1 -mno-avx" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc'). */
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-linux* } {^\t?\.} } } */
+
+/*
+**func:
+**.LFB0:
+** .cfi_startproc
+** movq %rdi, %xmm0
+** pinsrq \$1, %rsi, %xmm0
+** movaps %xmm0, store\(%rip\)
+** lock orq \$0, \(%rsp\)
+** ret
+** .cfi_endproc
+**...
+*/
+
+#include "pr126293-4a.c"
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-4j.c b/gcc/testsuite/gcc.target/i386/pr126293-4j.c
new file mode 100644
index 00000000000..1628319b5b5
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-4j.c
@@ -0,0 +1,19 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -fno-pic -std=c23 -march=lujiazui" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc'). */
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-linux* } {^\t?\.} } } */
+
+/*
+**func:
+**.LFB0:
+** .cfi_startproc
+** movq %rsi, %rdx
+** movl \$5, %ecx
+** movq %rdi, %rsi
+** movl \$store, %edi
+** jmp __atomic_store_16
+** .cfi_endproc
+**...
+*/
+
+#include "pr126293-4a.c"
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-4k.c b/gcc/testsuite/gcc.target/i386/pr126293-4k.c
new file mode 100644
index 00000000000..76a37927575
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-4k.c
@@ -0,0 +1,19 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -fno-pic -std=c23 -march=skylake -mno-cx16" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc'). */
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-linux* } {^\t?\.} } } */
+
+/*
+**func:
+**.LFB0:
+** .cfi_startproc
+** movq %rsi, %rdx
+** movl \$5, %ecx
+** movq %rdi, %rsi
+** movl \$store, %edi
+** jmp __atomic_store_16
+** .cfi_endproc
+**...
+*/
+
+#include "pr126293-4a.c"
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-4l.c b/gcc/testsuite/gcc.target/i386/pr126293-4l.c
new file mode 100644
index 00000000000..4a80ed40efa
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-4l.c
@@ -0,0 +1,19 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -fno-pic -std=c23 -march=skylake -mno-sse" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc'). */
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-linux* } {^\t?\.} } } */
+
+/*
+**func:
+**.LFB0:
+** .cfi_startproc
+** movq %rsi, %rdx
+** movl \$5, %ecx
+** movq %rdi, %rsi
+** movl \$store, %edi
+** jmp __atomic_store_16
+** .cfi_endproc
+**...
+*/
+
+#include "pr126293-4a.c"
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-4m.c b/gcc/testsuite/gcc.target/i386/pr126293-4m.c
new file mode 100644
index 00000000000..4ea11e00ff3
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-4m.c
@@ -0,0 +1,19 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -fno-pic -std=c23 -march=grandridge" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc'). */
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-linux* } {^\t?\.} } } */
+
+/*
+**func:
+**.LFB0:
+** .cfi_startproc
+** vmovq %rdi, %xmm1
+** vpinsrq \$1, %rsi, %xmm1, %xmm0
+** vmovdqa %xmm0, store\(%rip\)
+** lock orq \$0, \(%rsp\)
+** ret
+** .cfi_endproc
+**...
+*/
+
+#include "pr126293-4a.c"
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-4n.c b/gcc/testsuite/gcc.target/i386/pr126293-4n.c
new file mode 100644
index 00000000000..f0f92b429ce
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-4n.c
@@ -0,0 +1,19 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -fno-pic -std=c23 -march=skylake -mno-128bit-atomic" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc'). */
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-linux* } {^\t?\.} } } */
+
+/*
+**func:
+**.LFB0:
+** .cfi_startproc
+** movq %rsi, %rdx
+** movl \$5, %ecx
+** movq %rdi, %rsi
+** movl \$store, %edi
+** jmp __atomic_store_16
+** .cfi_endproc
+**...
+*/
+
+#include "pr126293-4a.c"
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-4o.c b/gcc/testsuite/gcc.target/i386/pr126293-4o.c
new file mode 100644
index 00000000000..6b94a7d59a4
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-4o.c
@@ -0,0 +1,19 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -fno-pic -std=c23 -march=c86-4g-m8" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc'). */
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-linux* } {^\t?\.} } } */
+
+/*
+**func:
+**.LFB0:
+** .cfi_startproc
+** vmovq %rdi, %xmm1
+** vpinsrq \$1, %rsi, %xmm1, %xmm0
+** vmovdqa %xmm0, store\(%rip\)
+** lock orq \$0, \(%rsp\)
+** ret
+** .cfi_endproc
+**...
+*/
+
+#include "pr126293-4a.c"
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-4p.c b/gcc/testsuite/gcc.target/i386/pr126293-4p.c
new file mode 100644
index 00000000000..9b2de450c5d
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-4p.c
@@ -0,0 +1,19 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -fno-pic -std=c23 -march=x86-64-v3" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc'). */
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-linux* } {^\t?\.} } } */
+
+/*
+**func:
+**.LFB0:
+** .cfi_startproc
+** vmovq %rdi, %xmm1
+** vpinsrq \$1, %rsi, %xmm1, %xmm0
+** vmovdqa %xmm0, store\(%rip\)
+** lock orq \$0, \(%rsp\)
+** ret
+** .cfi_endproc
+**...
+*/
+
+#include "pr126293-4a.c"
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-4q.c b/gcc/testsuite/gcc.target/i386/pr126293-4q.c
new file mode 100644
index 00000000000..4268a297c2c
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-4q.c
@@ -0,0 +1,19 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -fno-pic -std=c23 -march=x86-64-v4" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc'). */
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-linux* } {^\t?\.} } } */
+
+/*
+**func:
+**.LFB0:
+** .cfi_startproc
+** vmovq %rdi, %xmm1
+** vpinsrq \$1, %rsi, %xmm1, %xmm0
+** vmovdqa %xmm0, store\(%rip\)
+** lock orq \$0, \(%rsp\)
+** ret
+** .cfi_endproc
+**...
+*/
+
+#include "pr126293-4a.c"
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-4r.c b/gcc/testsuite/gcc.target/i386/pr126293-4r.c
new file mode 100644
index 00000000000..45e8bb7e55c
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-4r.c
@@ -0,0 +1,20 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -fno-pic -std=c23 -march=x86-64" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc'). */
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-linux* } {^\t?\.} } } */
+
+/*
+**func:
+**.LFB0:
+** .cfi_startproc
+** vmovq %rdi, %xmm1
+** vpinsrq \$1, %rsi, %xmm1, %xmm0
+** vmovdqa %xmm0, store\(%rip\)
+** lock orq \$0, \(%rsp\)
+** ret
+** .cfi_endproc
+**...
+*/
+
+#define ATTRIBUTE __attribute__ ((target("arch=skylake")))
+#include "pr126293-4a.c"
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-4s.c b/gcc/testsuite/gcc.target/i386/pr126293-4s.c
new file mode 100644
index 00000000000..38eb8bdac28
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-4s.c
@@ -0,0 +1,20 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -fno-pic -std=c23 -march=skylake" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc'). */
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-linux* } {^\t?\.} } } */
+
+/*
+**func:
+**.LFB0:
+** .cfi_startproc
+** movq %rsi, %rdx
+** movl \$5, %ecx
+** movq %rdi, %rsi
+** movl \$store, %edi
+** jmp __atomic_store_16
+** .cfi_endproc
+**...
+*/
+
+#define ATTRIBUTE __attribute__ ((target("arch=x86-64")))
+#include "pr126293-4a.c"
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-5.c b/gcc/testsuite/gcc.target/i386/pr126293-5.c
new file mode 100644
index 00000000000..2f0794132c9
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-5.c
@@ -0,0 +1,25 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -fno-pic -std=c23 -march=x86-64 -m128bit-atomic" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc'). */
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-linux* } {^\t?\.} } } */
+
+/*
+**func:
+**.LFB0:
+** .cfi_startproc
+** movdqa load\(%rip\), %xmm0
+** movaps %xmm0, -24\(%rsp\)
+** movq -24\(%rsp\), %rax
+** movq -16\(%rsp\), %rdx
+** ret
+** .cfi_endproc
+**...
+*/
+
+_Atomic __uint128_t load;
+
+__uint128_t
+func (void)
+{
+ return load;
+}
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-6.c b/gcc/testsuite/gcc.target/i386/pr126293-6.c
new file mode 100644
index 00000000000..5ad0b4b6a7e
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-6.c
@@ -0,0 +1,24 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -fno-pic -std=c23 -march=x86-64 -m128bit-atomic" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc'). */
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-linux* } {^\t?\.} } } */
+
+/*
+**func:
+**.LFB0:
+** .cfi_startproc
+** movaps %xmm0, store\(%rip\)
+** lock orq \$0, \(%rsp\)
+** ret
+** .cfi_endproc
+**...
+*/
+
+typedef int vector __attribute__((vector_size (16)));
+extern _Atomic vector store;
+
+void
+func (vector i)
+{
+ store = i;
+}
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-7.c b/gcc/testsuite/gcc.target/i386/pr126293-7.c
new file mode 100644
index 00000000000..0f5073ea985
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-7.c
@@ -0,0 +1,23 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -fno-pic -std=c23 -march=x86-64 -m128bit-atomic" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc'). */
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-linux* } {^\t?\.} } } */
+
+/*
+**func:
+**.LFB0:
+** .cfi_startproc
+** movdqa load\(%rip\), %xmm0
+** ret
+** .cfi_endproc
+**...
+*/
+
+typedef int vector __attribute__((vector_size (16)));
+extern _Atomic vector load;
+
+vector
+func (void)
+{
+ return load;
+}
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-8.c b/gcc/testsuite/gcc.target/i386/pr126293-8.c
new file mode 100644
index 00000000000..5c9abe02dd3
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-8.c
@@ -0,0 +1,24 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -fno-pic -std=c23 -march=x86-64 -m128bit-atomic" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc'). */
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-linux* } {^\t?\.} } } */
+
+/*
+**func:
+**.LFB0:
+** .cfi_startproc
+** movaps %xmm0, store\(%rip\)
+** lock orq \$0, \(%rsp\)
+** ret
+** .cfi_endproc
+**...
+*/
+
+typedef float vector __attribute__((vector_size (16)));
+extern _Atomic vector store;
+
+void
+func (vector i)
+{
+ store = i;
+}
diff --git a/gcc/testsuite/gcc.target/i386/pr126293-9.c b/gcc/testsuite/gcc.target/i386/pr126293-9.c
new file mode 100644
index 00000000000..623498711f7
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr126293-9.c
@@ -0,0 +1,23 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-additional-options "-O2 -fno-pic -std=c23 -march=x86-64 -m128bit-atomic" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc'). */
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-linux* } {^\t?\.} } } */
+
+/*
+**func:
+**.LFB0:
+** .cfi_startproc
+** movdqa load\(%rip\), %xmm0
+** ret
+** .cfi_endproc
+**...
+*/
+
+typedef float vector __attribute__((vector_size (16)));
+extern _Atomic vector load;
+
+vector
+func (void)
+{
+ return load;
+}
--
2.55.0
