From: Reshma Roy <[email protected]>
Transform uniform gather operations into optimized scalar load + broadcast
instead of emulated gather with N individual scalar loads.
BEFORE (Emulated Gather - 8 loads for VF=8):
for each lane i in 0..7:
offset[i] = BIT_FIELD_REF<offset_vector, i>
addr[i] = base + offset[i] * scale
result[i] = MEM[addr[i]]
result_vector = {result[0], ..., result[7]}
AFTER (Broadcast - 1 load):
offset = BIT_FIELD_REF<offset_vector, 0>
addr = base + offset * scale
scalar_val = MEM[addr]
result_vector = {scalar_val, scalar_val,...}
v3 changes:
- Dumping: remove the out-of-place transform-time dump and report the
uniform-offset broadcast decision during analysis in get_load_store_type;
- update tests to scan for the new message.
- Formatting : re-indent the emulated-gather costing block under the new
else, fix "k > 0" spacing, and drop the spurious trailing-whitespace change.
- Update test cases, included more test cases for bot_and_expr as well.
gcc/ChangeLog:
* tree-vect-stmts.cc (vectorizable_load): Generate broadcast for
uniform gather offsets.
gcc/testsuite/ChangeLog:
* g++.dg/vect/vect-uniform-broadcast-and.c: New test.
* g++.dg/vect/vect-uniform-broadcast-neg-and.c: New test.
* g++.dg/vect/vect-uniform-broadcast-neg.c: New test.
* g++.dg/vect/vect-uniform-broadcast-op2-var.c: New test.
* g++.dg/vect/vect-uniform-broadcast.c: New test.
---
Hi Richard,
The comments were addressed and the updated patch is attached.
Bootstrapped and tested on x86_64
Thanks,
Reshma Roy
Original Message-----
> From: Richard Biener <[email protected]>
> Sent: Tuesday, June 30, 2026 6:01 PM
> To: Roy, Reshma <[email protected]>
> Cc: [email protected]; Kumar, Venkataramanan
> <[email protected]>
> Subject: Re: [PATCH v2 2/2] Loop Vectorizer: Generate broadcast for uniform
> gather
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On Wed, 24 Jun 2026, [email protected] wrote:
>
> > From: Reshma Roy <[email protected]>
> >
> > Transform uniform gather operations into optimized scalar load +
> > broadcast instead of emulated gather with N individual scalar loads.
> >
> > BEFORE (Emulated Gather - 8 loads for VF=8):
> > for each lane i in 0..7:
> > offset[i] = BIT_FIELD_REF<offset_vector, i>
> > addr[i] = base + offset[i] * scale
> > result[i] = MEM[addr[i]]
> > result_vector = {result[0], ..., result[7]}
> >
> > AFTER (Broadcast - 1 load):
> > offset = BIT_FIELD_REF<offset_vector, 0>
> > addr = base + offset * scale
> > scalar_val = MEM[addr]
> > result_vector = VEC_DUPLICATE_EXPR(scalar_val)
> >
> > v2 changes:
> > - Keep the full loop (k = 0 ??? const_nunits-1) but only perform the
> > scalar load once and reuse elt for all lanes
> > - Since VEC_DUPLICATE_EXPR is for VLA/scalable vector types only
> > removed that and added constructor
> > - Removed the SSA copy by refactoring the code
> >
> > gcc/ChangeLog:
> >
> > * tree-vect-stmts.cc (vectorizable_load): Generate broadcast for
> > uniform gather offsets.
> >
> > gcc/testsuite/ChangeLog:
> >
> > * g++.dg/vect/vect-uniform-broadcast-neg.c: New test.
> > * g++.dg/vect/vect-uniform-broadcast-op2-var.c: New test.
> > * g++.dg/vect/vect-uniform-broadcast.c: New test.
> >
> > ---
> >
> > Hi Richard,
> >
> > The comments were addressed and the updated patch is attached.
> >
> > Thanks,
> > Reshma Roy
> >
> > Original Message-----
> > > From: Richard Biener <[email protected]>
> > > Sent: Tuesday, June 9, 2026 7:31 PM
> > > To: Roy, Reshma <[email protected]>
> > > Cc: [email protected]; Kumar, Venkataramanan
> > > <[email protected]>
> > > Subject: Re: [PATCH 2/2] Loop Vectorizer: Generate broadcast for
> > > uniform gather offsets
> > >
> > > Caution: This message originated from an External Source. Use proper
> > > caution when opening attachments, clicking links, or responding.
> > >
> > >
> > > On Mon, 25 May 2026, [email protected] wrote:
> > >
> > > > From: Reshma Roy <[email protected]>
> > > >
> > > > Transform uniform gather operations into optimized scalar load +
> > > > broadcast instead of emulated gather with N individual scalar loads.
> > > >
> > > > BEFORE (Emulated Gather - 8 loads for VF=8):
> > > > for each lane i in 0..7:
> > > > offset[i] = BIT_FIELD_REF<offset_vector, i>
> > > > addr[i] = base + offset[i] * scale
> > > > result[i] = MEM[addr[i]]
> > > > result_vector = {result[0], ..., result[7]}
> > > >
> > > > AFTER (Broadcast - 1 load):
> > > > offset = BIT_FIELD_REF<offset_vector, 0>
> > > > addr = base + offset * scale
> > > > scalar_val = MEM[addr]
> > > > result_vector = VEC_DUPLICATE_EXPR(scalar_val)
> > > >
> > > > gcc/ChangeLog:
> > > >
> > > > * tree-vect-stmts.cc (vectorizable_load): Generate broadcast for
> > > > uniform gather offsets.
> > > >
> > > > gcc/testsuite/ChangeLog:
> > > > * g++.dg/vect/vect-uniform-broadcast-neg.c: New test.
> > > > * g++.dg/vect/vect-uniform-broadcast-op2-var.c: New test.
> > > > * g++.dg/vect/vect-uniform-broadcast.c: New test.
> > > >
> > > > ---
> > > > .../g++.dg/vect/vect-uniform-broadcast-neg.c | 29 +++++++++
> > > > .../vect/vect-uniform-broadcast-op2-var.c | 38 ++++++++++++
> > > > .../g++.dg/vect/vect-uniform-broadcast.c | 29 +++++++++
> > > > gcc/tree-vect-stmts.cc | 59 ++++++++++++++++---
> > > > 4 files changed, 148 insertions(+), 7 deletions(-) create mode
> > > > 100644 gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg.c
> > > > create mode 100644
> > > > gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-op2-var.c
> > > > create mode 100644
> > > > gcc/testsuite/g++.dg/vect/vect-uniform-broadcast.c
> > > >
> > > > diff --git
> > > > a/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg.c
> > > > b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg.c
> > > > new file mode 100644
> > > > index 00000000000..590706d3b51
> > > > --- /dev/null
> > > > +++ b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg.c
> > > > @@ -0,0 +1,29 @@
> > > > +/* { dg-do compile } */
> > > > +/* { dg-additional-options "-O3 -std=c++11 -march=znver5
> > > > +-fdump-tree-vect-details" } */
> > > > +/* Negative test: loop i starts at 1 -> SCEV {1, +, 1}. POC only
> > > > handles
> > > > + {0, +, 1}, so offset is not treated as uniform -> no broadcast
> > > > + (vec_duplicate_expr must not appear for this gather). */
> > > > +#include <bitset> #include <vector> class Foo {
> > > > + public:
> > > > + void fun (unsigned boardsize, bool check);
> > > > + std::vector<int> m_mcowner;
> > > > +};
> > > > +void Foo::fun (unsigned boardsize, bool check) {
> > > > + std::bitset<21*21> blacksq;
> > > > + __asm__ ("" : "+g" (blacksq));
> > > > + for (int i = 0; i < boardsize; i++) {
> > > > + if (i % 2 == 0)
> > > > + blacksq[i] = true;
> > > > + }
> > > > + unsigned int i = 1;
> > > > + do {
> > > > + if (blacksq[i])
> > > > + m_mcowner[i]++;
> > > > + } while (++i < blacksq.size ()); }
> > > > +/* { dg-final { scan-tree-dump-not "\.vec_duplicate_expr" "vect"
> > > > +} } */
> > > > +
> > > > diff --git
> > > > a/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-op2-var.c
> > > > b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-op2-var.c
> > > > new file mode 100644
> > > > index 00000000000..b212b3d56be
> > > > --- /dev/null
> > > > +++ b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-op2-var.c
> > > > @@ -0,0 +1,38 @@
> > > > +/* { dg-do compile } */
> > > > +/* { dg-additional-options "-O3 -std=c++11 -march=znver5
> > > > +-fdump-tree-vect-details" } */
> > > > +/* Gather load with PLUS (hi, lo). The second operand of the offset
> > > > PLUS is a
> > > > + variable, hence does not emit vec_duplicate_expr. The inner loop
> > > > still
> > > > + vectorizes as a normal gather */
> > > > +
> > > > +#include <vector>
> > > > +
> > > > +class Foo
> > > > +{
> > > > + public:
> > > > + void fun (unsigned boardsize);
> > > > + std::vector<int> m_mcowner;
> > > > +};
> > > > +
> > > > +void
> > > > +Foo::fun (unsigned boardsize)
> > > > +{
> > > > + unsigned nwords = (21 * 21 + 63) / 64 + 2;
> > > > + std::vector<unsigned long> words (nwords, 0);
> > > > + __asm__ ("" : "+g" (words));
> > > > + for (unsigned i = 0; i < boardsize; i++)
> > > > + if (i % 2 == 0)
> > > > + words[i / 64] |= 1UL << (i % 64);
> > > > + unsigned int i = 0;
> > > > + do
> > > > + {
> > > > + unsigned hi = i / 64;
> > > > + unsigned lo = i & 1u;
> > > > + /* the second operand is ensured to be loop-varying by the SCEV
> > > > path */
> > > > + if (words[hi + lo])
> > > > + m_mcowner[i]++;
> > > > + }
> > > > + while (++i < 21 * 21);
> > > > +}
> > > > +
> > > > +/* { dg-final { scan-tree-dump-not "\.vec_duplicate_expr" "vect"
> > > > +} } */
> > > > +/* { dg-final { scan-tree-dump "gs_offset_uniform_p is set to: 0"
> > > > +"vect" } } */
> > > > diff --git a/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast.c
> > > > b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast.c
> > > > new file mode 100644
> > > > index 00000000000..818b7a63c00
> > > > --- /dev/null
> > > > +++ b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast.c
> > > > @@ -0,0 +1,29 @@
> > > > +/* { dg-do compile } */
> > > > +/* { dg-additional-options "-O3 -std=c++11 -march=znver5
> > > > +-fdump-tree-vect-details" } */
> > > > +/* Test case for generating uniform broadcast instead of gather
> > > > + when there is uniform value accross the vector lane */
> > > > +#include <bitset> #include <vector> class Foo {
> > > > + public:
> > > > + void fun (unsigned boardsize, bool check);
> > > > + std::vector<int> m_mcowner;
> > > > +};
> > > > +void Foo::fun( unsigned boardsize, bool check) {
> > > > + std::bitset<21*21> blacksq;
> > > > + __asm__ ("" : "+g" (blacksq));
> > > > + for (int i = 0; i < boardsize; i++) {
> > > > + if(i%2 == 0)
> > > > + blacksq[i] = true;
> > > > + }
> > > > + unsigned int i = 0;
> > > > + do{
> > > > + if (blacksq[i])
> > > > + m_mcowner[i]++;
> > > > + }while(++i < blacksq.size());
> > > > +
> > > > +}
> > > > +/* { dg-final { scan-tree-dump "\.vec_duplicate_expr" "vect" } }
> > > > +*/
> > > > +
> > > > diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc index
> > > > b68fc5072af..71ff138c069 100644
> > > > --- a/gcc/tree-vect-stmts.cc
> > > > +++ b/gcc/tree-vect-stmts.cc
> > > > @@ -11198,6 +11198,25 @@ vectorizable_load (vec_info *vinfo,
> > > > unsigned HOST_WIDE_INT const_nunits = nunits.to_constant ();
> > > > if (costing_p)
> > > > {
> > > > + if (slp_node->gs_offset_uniform_p)
> > > > + {
> > > > + if (dump_enabled_p ())
> > > > + dump_printf_loc (MSG_NOTE, vect_location,
> > > > + "computing cost for broadcast
> > > > \n");
> > > > + /* Broadcast optimization: 1 extract + 1 load
> > > > + + 1 broadcast. */
> > > > + inside_cost = record_stmt_cost (cost_vec, 1,
> > > > + vec_to_scalar,
> > > > slp_node,
> > > > + 0, vect_body);
> > > > + inside_cost += record_stmt_cost (cost_vec, 1,
> > > > scalar_load,
> > > > + slp_node, 0,
> > > > vect_body);
> > > > + inside_cost += record_stmt_cost (cost_vec, 1,
> > > > + scalar_to_vec,
> > > > slp_node,
> > > > + 0, vect_body);
> > > > + }
> > > > + else
> > > > + {
> > > > +
> > > > /* For emulated gathers N offset vector element
> > > > offset add is consumed by the load). */
> > > > inside_cost = record_stmt_cost (cost_vec,
> > > > const_nunits, @@ -11211,6 +11230,7 @@ vectorizable_load (vec_info
> > > > *vinfo,
> > > > inside_cost
> > > > = record_stmt_cost (cost_vec, 1, vec_construct,
> > > > slp_node, 0, vect_body);
> > > > + }
> > > > continue;
> > > > }
> > > > tree offset_vectype = TREE_TYPE (vec_offsets[0]); @@
> > > > -11228,7 +11248,12 @@ vectorizable_load (vec_info *vinfo,
> > > > tree idx_type = TREE_TYPE (TREE_TYPE (vec_offset));
> > > > tree scale = size_int (SLP_TREE_GS_SCALE (slp_node));
> > > > tree ltype = build_aligned_type (TREE_TYPE (vectype),
> > > > align);
> > > > - for (unsigned k = 0; k < const_nunits; ++k)
> > > > + tree broadcast_vec = NULL_TREE;
> > > > + /* If all lanes are uniform then generate broadcast
> > > > otherwise
> > > > + emulated gathers. */
> > > > + unsigned HOST_WIDE_INT vec_lane_loads
> > > > + = slp_node->gs_offset_uniform_p ? 1 : const_nunits;
> > >
> > > instead of this
> > >
> > > > + for (unsigned k = 0; k < vec_lane_loads; ++k)
> > > > {
> > > > tree boff = size_binop (MULT_EXPR, TYPE_SIZE (idx_type),
> > > > bitsize_int (k +
> > > > elt_offset)); @@ -11247,13 +11272,33 @@ vectorizable_load (vec_info
> > > > *vinfo,
> > > > new_stmt = gimple_build_assign (elt, ref);
> > > > gimple_set_vuse (new_stmt, gimple_vuse (gsi_stmt
> > > > (*gsi)));
> > > > gimple_seq_add_stmt (&stmts, new_stmt);
> > > > - CONSTRUCTOR_APPEND_ELT (ctor_elts, NULL_TREE, elt);
> > > > + if (!slp_node->gs_offset_uniform_p)
> > > > + CONSTRUCTOR_APPEND_ELT (ctor_elts, NULL_TREE,
> > > > elt);
> > > > + else
> > > > + /* Broadcast to all lanes. */
> > > > + broadcast_vec = gimple_build (&stmts,
> > > > + VEC_DUPLICATE_EXPR,
> > > > + vectype, elt);
> > >
> > > break from the loop here. Also use gimple_build_vector_from_val, a
> > > VEC_DUPLICATE_EXPR is only valid for VLA vectors, a broadcast for
> > > non-VLA vectors should use a uniform CONSTRUCTOR.
> > Done.
> > >
> > > > + }
> > > > + if (!slp_node->gs_offset_uniform_p)
> > > > + {
> > > > + gsi_insert_seq_before (gsi, stmts, GSI_SAME_STMT);
> > > > + new_stmt = gimple_build_assign (NULL_TREE,
> > > > + build_constructor
> > > > (vectype,
> > > > +
> > > > ctor_elts)
> > > > + );
> > > > + data_ref = NULL_TREE;
> > > > + }
> > > > + else
> > > > + {
> > > > + gsi_insert_seq_before (gsi, stmts, GSI_SAME_STMT);
> > > > + new_stmt = gimple_build_assign (NULL_TREE,
> > > > + broadcast_vec);
> > >
> > > you get a SSA copy from this, some refactoring should avoid that.
> > > Or simply build that uniform vector above by keeping the loop but
> > > only load 'elt' once (but add it nunits time).
> > Refactored the code such that SSA copy is removed.
> > >
> > > > + data_ref = NULL_TREE;
> > > > + if (dump_enabled_p ())
> > > > + dump_printf_loc (MSG_OPTIMIZED_LOCATIONS,
> > > > vect_location,
> > > > + "generating broadcast instead of
> > > > gather "
> > > > + "loads (uniform offset
> > > > + lanes)\n");
> > > > }
> > > > - gsi_insert_seq_before (gsi, stmts, GSI_SAME_STMT);
> > > > - new_stmt = gimple_build_assign (NULL_TREE,
> > > > - build_constructor (vectype,
> > > > -
> > > > ctor_elts));
> > > > - data_ref = NULL_TREE;
> > > > }
> > > >
> > > > vec_dest = vect_create_destination_var (scalar_dest,
> > > > vectype);
> > > >
> > >
> > > --
> > > Richard Biener <[email protected]>
> > > SUSE Software Solutions Germany GmbH, Frankenstrasse 146, 90461
> > > Nuernberg, Germany;
> > > GF: Jochen Jaser, Andrew McDonald, Werner Knoblich; (HRB 36809, AG
> > > Nuernberg)
> >
> >
> > .../g++.dg/vect/vect-uniform-broadcast-neg.c | 29 ++++++++
> > .../vect/vect-uniform-broadcast-op2-var.c | 38 ++++++++++
> > .../g++.dg/vect/vect-uniform-broadcast.c | 29 ++++++++
> > gcc/tree-vect-stmts.cc | 71 ++++++++++++++-----
> > 4 files changed, 149 insertions(+), 18 deletions(-) create mode
> > 100644 gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg.c
> > create mode 100644
> > gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-op2-var.c
> > create mode 100644 gcc/testsuite/g++.dg/vect/vect-uniform-broadcast.c
> >
> > diff --git a/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg.c
> > b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg.c
> > new file mode 100644
> > index 00000000000..590706d3b51
> > --- /dev/null
> > +++ b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg.c
> > @@ -0,0 +1,29 @@
> > +/* { dg-do compile } */
> > +/* { dg-additional-options "-O3 -std=c++11 -march=znver5
> > +-fdump-tree-vect-details" } */
> > +/* Negative test: loop i starts at 1 -> SCEV {1, +, 1}. POC only handles
> > + {0, +, 1}, so offset is not treated as uniform -> no broadcast
> > + (vec_duplicate_expr must not appear for this gather). */ #include
> > +<bitset> #include <vector> class Foo {
> > + public:
> > + void fun (unsigned boardsize, bool check);
> > + std::vector<int> m_mcowner;
> > +};
> > +void Foo::fun (unsigned boardsize, bool check) {
> > + std::bitset<21*21> blacksq;
> > + __asm__ ("" : "+g" (blacksq));
> > + for (int i = 0; i < boardsize; i++) {
> > + if (i % 2 == 0)
> > + blacksq[i] = true;
> > + }
> > + unsigned int i = 1;
> > + do {
> > + if (blacksq[i])
> > + m_mcowner[i]++;
> > + } while (++i < blacksq.size ());
> > +}
> > +/* { dg-final { scan-tree-dump-not "\.vec_duplicate_expr" "vect" } }
> > +*/
> > +
> > diff --git
> > a/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-op2-var.c
> > b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-op2-var.c
> > new file mode 100644
> > index 00000000000..b212b3d56be
> > --- /dev/null
> > +++ b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-op2-var.c
> > @@ -0,0 +1,38 @@
> > +/* { dg-do compile } */
> > +/* { dg-additional-options "-O3 -std=c++11 -march=znver5
> > +-fdump-tree-vect-details" } */
> > +/* Gather load with PLUS (hi, lo). The second operand of the offset PLUS
> > is a
> > + variable, hence does not emit vec_duplicate_expr. The inner loop still
> > + vectorizes as a normal gather */
> > +
> > +#include <vector>
> > +
> > +class Foo
> > +{
> > + public:
> > + void fun (unsigned boardsize);
> > + std::vector<int> m_mcowner;
> > +};
> > +
> > +void
> > +Foo::fun (unsigned boardsize)
> > +{
> > + unsigned nwords = (21 * 21 + 63) / 64 + 2;
> > + std::vector<unsigned long> words (nwords, 0);
> > + __asm__ ("" : "+g" (words));
> > + for (unsigned i = 0; i < boardsize; i++)
> > + if (i % 2 == 0)
> > + words[i / 64] |= 1UL << (i % 64);
> > + unsigned int i = 0;
> > + do
> > + {
> > + unsigned hi = i / 64;
> > + unsigned lo = i & 1u;
> > + /* the second operand is ensured to be loop-varying by the SCEV path
> > */
> > + if (words[hi + lo])
> > + m_mcowner[i]++;
> > + }
> > + while (++i < 21 * 21);
> > +}
> > +
> > +/* { dg-final { scan-tree-dump-not "\.vec_duplicate_expr" "vect" } }
> > +*/
> > +/* { dg-final { scan-tree-dump "gs_offset_uniform_p is set to: 0"
> > +"vect" } } */
> > diff --git a/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast.c
> > b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast.c
> > new file mode 100644
> > index 00000000000..818b7a63c00
> > --- /dev/null
> > +++ b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast.c
> > @@ -0,0 +1,29 @@
> > +/* { dg-do compile } */
> > +/* { dg-additional-options "-O3 -std=c++11 -march=znver5
> > +-fdump-tree-vect-details" } */
> > +/* Test case for generating uniform broadcast instead of gather
> > + when there is uniform value accross the vector lane */ #include
> > +<bitset> #include <vector> class Foo {
> > + public:
> > + void fun (unsigned boardsize, bool check);
> > + std::vector<int> m_mcowner;
> > +};
> > +void Foo::fun( unsigned boardsize, bool check) {
> > + std::bitset<21*21> blacksq;
> > + __asm__ ("" : "+g" (blacksq));
> > + for (int i = 0; i < boardsize; i++) {
> > + if(i%2 == 0)
> > + blacksq[i] = true;
> > + }
> > + unsigned int i = 0;
> > + do{
> > + if (blacksq[i])
> > + m_mcowner[i]++;
> > + }while(++i < blacksq.size());
> > +
> > +}
> > +/* { dg-final { scan-tree-dump "\.vec_duplicate_expr" "vect" } } */
> > +
> > diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc index
> > afd5109c9f7..44282f20873 100644
> > --- a/gcc/tree-vect-stmts.cc
> > +++ b/gcc/tree-vect-stmts.cc
> > @@ -11207,6 +11207,25 @@ vectorizable_load (vec_info *vinfo,
> > unsigned HOST_WIDE_INT const_nunits = nunits.to_constant ();
> > if (costing_p)
> > {
> > + if (slp_node->gs_offset_uniform_p)
> > + {
> > + if (dump_enabled_p ())
> > + dump_printf_loc (MSG_NOTE, vect_location,
> > + "computing cost for broadcast \n");
> > + /* Broadcast optimization: 1 extract + 1 load
> > + + 1 broadcast. */
> > + inside_cost = record_stmt_cost (cost_vec, 1,
> > + vec_to_scalar, slp_node,
> > + 0, vect_body);
> > + inside_cost += record_stmt_cost (cost_vec, 1,
> > scalar_load,
> > + slp_node, 0,
> > vect_body);
> > + inside_cost += record_stmt_cost (cost_vec, 1,
> > + scalar_to_vec,
> > slp_node,
> > + 0, vect_body);
> > + }
> > + else
> > + {
> > +
> > /* For emulated gathers N offset vector element
> > offset add is consumed by the load). */
>
> this block now needs re-indenting.
Done.
>
> > inside_cost = record_stmt_cost (cost_vec, 1,
> > vec_deconstruct, @@ -11219,6 +11238,7 @@ vectorizable_load (vec_info *vinfo,
> > inside_cost
> > = record_stmt_cost (cost_vec, 1, vec_construct,
> > slp_node, 0, vect_body);
> > + }
> > continue;
> > }
> > tree offset_vectype = TREE_TYPE (vec_offsets[0]); @@
> > -11236,25 +11256,37 @@ vectorizable_load (vec_info *vinfo,
> > tree idx_type = TREE_TYPE (TREE_TYPE (vec_offset));
> > tree scale = size_int (SLP_TREE_GS_SCALE (slp_node));
> > tree ltype = build_aligned_type (TREE_TYPE (vectype),
> > align);
> > + tree first_elt = NULL_TREE;
> > for (unsigned k = 0; k < const_nunits; ++k)
> > {
> > - tree boff = size_binop (MULT_EXPR, TYPE_SIZE (idx_type),
> > - bitsize_int (k + elt_offset));
> > - tree idx = gimple_build (&stmts, BIT_FIELD_REF, idx_type,
> > - vec_offset, TYPE_SIZE (idx_type),
> > - boff);
> > - idx = gimple_convert (&stmts, sizetype, idx);
> > - idx = gimple_build (&stmts, MULT_EXPR, sizetype, idx,
> > scale);
> > - tree ptr = gimple_build (&stmts, PLUS_EXPR,
> > - TREE_TYPE (dataref_ptr),
> > - dataref_ptr, idx);
> > - ptr = gimple_convert (&stmts, ptr_type_node, ptr);
> > - tree elt = make_ssa_name (TREE_TYPE (vectype));
> > - tree ref = build2 (MEM_REF, ltype, ptr,
> > - build_int_cst (ref_type, 0));
> > - new_stmt = gimple_build_assign (elt, ref);
> > - gimple_set_vuse (new_stmt, gimple_vuse (gsi_stmt (*gsi)));
> > - gimple_seq_add_stmt (&stmts, new_stmt);
> > + tree elt;
> > + /* If all lanes are uniorm then generate broadcast otherwise
> > + emulated gathers. */
> > + if (slp_node->gs_offset_uniform_p && k>0)
>
> k > 0
Done.
>
> > + elt = first_elt;
> > + else
> > + {
> > +
> > + tree boff = size_binop (MULT_EXPR, TYPE_SIZE (idx_type),
> > + bitsize_int (k + elt_offset));
> > + tree idx = gimple_build (&stmts, BIT_FIELD_REF,
> > idx_type,
> > + vec_offset, TYPE_SIZE
> > (idx_type),
> > + boff);
> > + idx = gimple_convert (&stmts, sizetype, idx);
> > + idx = gimple_build (&stmts, MULT_EXPR, sizetype, idx,
> > + scale);
> > + tree ptr = gimple_build (&stmts, PLUS_EXPR,
> > + TREE_TYPE (dataref_ptr),
> > + dataref_ptr, idx);
> > + ptr = gimple_convert (&stmts, ptr_type_node, ptr);
> > + elt = make_ssa_name (TREE_TYPE (vectype));
> > + tree ref = build2 (MEM_REF, ltype, ptr,
> > + build_int_cst (ref_type, 0));
> > + new_stmt = gimple_build_assign (elt, ref);
> > + gimple_set_vuse (new_stmt, gimple_vuse (gsi_stmt
> > (*gsi)));
> > + gimple_seq_add_stmt (&stmts, new_stmt);
> > + first_elt = elt;
> > + }
> > CONSTRUCTOR_APPEND_ELT (ctor_elts, NULL_TREE, elt);
> > }
> > gsi_insert_seq_before (gsi, stmts, GSI_SAME_STMT); @@
> > -11262,8 +11294,11 @@ vectorizable_load (vec_info *vinfo,
> > build_constructor (vectype,
> > ctor_elts));
> > data_ref = NULL_TREE;
> > + if (slp_node->gs_offset_uniform_p && dump_enabled_p ())
> > + dump_printf_loc (MSG_OPTIMIZED_LOCATIONS, vect_location,
> > + "generating broadcast instead of gather "
> > + "loads (uniform offset lanes)\n");
>
> The extra dump prints are a bit out-of-place, we probably want to dump (in
> general)
> the gather/scatter kind that's being used, but during analysis and for all
> cases. We
> might already do, if we do you want to ament that.
I could not find a place during the analysis where the dumps are already
done, hence i have moved this dump to a location in the analysis which i
think is sufficient.
>
> > }
> > -
>
> please avoid spurious whitespace changes
Done.
>
> > vec_dest = vect_create_destination_var (scalar_dest, vectype);
> > /* DATA_REF is null if we've already built the statement. */
> > if (data_ref)
> >
>
> --
> Richard Biener <[email protected]>
> SUSE Software Solutions Germany GmbH,
> Frankenstrasse 146, 90461 Nuernberg, Germany;
> GF: Jochen Jaser, Andrew McDonald, Werner Knoblich; (HRB 36809, AG
> Nuernberg)
.../g++.dg/vect/vect-uniform-broadcast-and.c | 33 ++++++
.../vect/vect-uniform-broadcast-neg-and.c | 40 +++++++
.../g++.dg/vect/vect-uniform-broadcast-neg.c | 29 +++++
.../vect/vect-uniform-broadcast-op2-var.c | 37 +++++++
.../g++.dg/vect/vect-uniform-broadcast.c | 29 +++++
gcc/tree-vect-stmts.cc | 100 ++++++++++++------
6 files changed, 235 insertions(+), 33 deletions(-)
create mode 100644 gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-and.c
create mode 100644 gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg-and.c
create mode 100644 gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg.c
create mode 100644 gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-op2-var.c
create mode 100644 gcc/testsuite/g++.dg/vect/vect-uniform-broadcast.c
diff --git a/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-and.c
b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-and.c
new file mode 100644
index 00000000000..2736aa73131
--- /dev/null
+++ b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-and.c
@@ -0,0 +1,33 @@
+/* { dg-do compile { target { x86_64-*-* i?86-*-* } } } */
+/* { dg-additional-options "-O3 -march=znver5 -fdump-tree-vect-details" } */
+/* { dg-additional-options "-mprefer-vector-width=128" } */
+/* Gather load table[i & 4]: mask 4 affects only bit 2, so (4 & (N - 1)) == 0
+ when N = 4. -mprefer-vector-width=128 encourages 4-wide int vectors on
+ x86 so i & 4 is uniform per vector chunk and broadcast is possible. */
+
+#include <stdio.h>
+
+#define N 32
+
+int block_values[8] = {10, 20, 30, 40, 50, 60, 70, 80};
+int result[N];
+
+void
+tile_fill (int *__restrict out, int *__restrict table, int n,
+ int *__restrict in)
+{
+ for (int i = 0; i < n; i++)
+ {
+ out[i] = table[i & 4];
+ in[i] = out[i] + 10;
+ }
+}
+
+void
+use_tile_fill (void)
+{
+ tile_fill (result, block_values, N, result);
+ __asm__ volatile ("" : : "r" (result) : "memory");
+}
+
+/* { dg-final { scan-tree-dump "gather offset is uniform across vector lanes"
"vect" } } */
diff --git a/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg-and.c
b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg-and.c
new file mode 100644
index 00000000000..82a85834661
--- /dev/null
+++ b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg-and.c
@@ -0,0 +1,40 @@
+/* { dg-do compile } */
+/* { dg-additional-options "-O3 -march=znver5 -fdump-tree-vect-details" } */
+/* Negative test: the offset must NOT be reported as uniform for ANY candidate
+ vector size the vectorizer tries.
+
+ NOTE: a right-shift offset (i >> k) cannot be used for a negative test here.
+ The vectorizer analyzes every candidate vector size (32, 16, 8, 4, 2), and
+ for a right shift the offset is uniform per chunk iff N - 1 < 2^k. At the
+ smallest candidate N = 2 this is 1 < 2^k, which holds for every k >= 1, so
+ *some* candidate always reports "uniform" and scan-tree-dump-not would fail.
+
+ For BIT_AND the offset is uniform iff (k & (N - 1)) == 0. With an odd mask
+ (k = 1) bit 0 is set, and N - 1 has bit 0 set for every N >= 2, so the
+ offset is non-uniform for ALL candidate vector sizes. The broadcast
+ optimization must never trigger. */
+
+#include <stdio.h>
+
+#define N 512
+
+short block_values[64];
+short result[N];
+
+void
+tile_fill (short *__restrict out, short *__restrict table, int n)
+{
+ for (int i = 0; i < n; i++)
+ out[i] = table[i & 1];
+}
+
+void
+use_tile_fill (void)
+{
+ tile_fill (result, block_values, N);
+ __asm__ volatile ("" : : "r" (result) : "memory");
+}
+
+/* The offset is not uniform across the vector lanes, so no broadcast should be
+ reported. */
+/* { dg-final { scan-tree-dump-not "gather offset is uniform across vector
lanes" "vect" } } */
diff --git a/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg.c
b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg.c
new file mode 100644
index 00000000000..b366753ec62
--- /dev/null
+++ b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg.c
@@ -0,0 +1,29 @@
+/* { dg-do compile { target { x86_64-*-* i?86-*-* } } } */
+/* { dg-additional-options "-O3 -std=c++11 -march=znver5
-fdump-tree-vect-details" } */
+/* Negative test: loop i starts at 1 -> SCEV {1, +, 1}. POC only handles
+ {0, +, 1}, so offset is not treated as uniform -> no broadcast
+ (offset must not be reported as uniform for this gather). */
+#include <bitset>
+#include <vector>
+class Foo
+{
+ public:
+ void fun (unsigned boardsize, bool check);
+ std::vector<int> m_mcowner;
+};
+void Foo::fun (unsigned boardsize, bool check)
+{
+ std::bitset<21*21> blacksq;
+ __asm__ ("" : "+g" (blacksq));
+ for (int i = 0; i < boardsize; i++) {
+ if (i % 2 == 0)
+ blacksq[i] = true;
+ }
+ unsigned int i = 1;
+ do {
+ if (blacksq[i])
+ m_mcowner[i]++;
+ } while (++i < blacksq.size ());
+}
+/* { dg-final { scan-tree-dump-not "gather offset is uniform across each
individual" "vect" } } */
+
diff --git a/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-op2-var.c
b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-op2-var.c
new file mode 100644
index 00000000000..97b18fabcc9
--- /dev/null
+++ b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-op2-var.c
@@ -0,0 +1,37 @@
+/* { dg-do compile { target { x86_64-*-* i?86-*-* } } } */
+/* { dg-additional-options "-O3 -std=c++11 -march=znver5
-fdump-tree-vect-details" } */
+/* Gather load with PLUS (hi, lo). The second operand of the offset PLUS is a
+ variable, hence the offset is not uniform and no broadcast is generated.
+ The inner loop still vectorizes as a normal gather. */
+
+#include <vector>
+
+class Foo
+{
+ public:
+ void fun (unsigned boardsize);
+ std::vector<int> m_mcowner;
+};
+
+void
+Foo::fun (unsigned boardsize)
+{
+ unsigned nwords = (21 * 21 + 63) / 64 + 2;
+ std::vector<unsigned long> words (nwords, 0);
+ __asm__ ("" : "+g" (words));
+ for (unsigned i = 0; i < boardsize; i++)
+ if (i % 2 == 0)
+ words[i / 64] |= 1UL << (i % 64);
+ unsigned int i = 0;
+ do
+ {
+ unsigned hi = i / 64;
+ unsigned lo = i & 1u;
+ /* the second operand is ensured to be loop-varying by the SCEV path */
+ if (words[hi + lo])
+ m_mcowner[i]++;
+ }
+ while (++i < 21 * 21);
+}
+
+/* { dg-final { scan-tree-dump-not "gather offset is uniform across each
individual" "vect" } } */
diff --git a/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast.c
b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast.c
new file mode 100644
index 00000000000..507bc6e4e3e
--- /dev/null
+++ b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast.c
@@ -0,0 +1,29 @@
+/* { dg-do compile { target { x86_64-*-* i?86-*-* } } } */
+/* { dg-additional-options "-O3 -std=c++11 -march=znver5
-fdump-tree-vect-details" } */
+/* Test case for generating uniform broadcast instead of gather
+ when there is uniform value accross the vector lane */
+#include <bitset>
+#include <vector>
+class Foo
+{
+ public:
+ void fun (unsigned boardsize, bool check);
+ std::vector<int> m_mcowner;
+};
+void Foo::fun( unsigned boardsize, bool check)
+{
+ std::bitset<21*21> blacksq;
+ __asm__ ("" : "+g" (blacksq));
+ for (int i = 0; i < boardsize; i++) {
+ if(i%2 == 0)
+ blacksq[i] = true;
+ }
+ unsigned int i = 0;
+ do{
+ if (blacksq[i])
+ m_mcowner[i]++;
+ }while(++i < blacksq.size());
+
+}
+/* { dg-final { scan-tree-dump "gather offset is uniform across each
individual" "vect" } } */
+
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index c1776f78f53..a79db05756e 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -2168,10 +2168,11 @@ get_load_store_type (vec_info *vinfo, stmt_vec_info
stmt_info,
}
else if (STMT_VINFO_GATHER_SCATTER_P (stmt_info))
{
- if (dump_enabled_p ())
- dump_printf_loc (MSG_NOTE, vect_location,
- "gs_offset_uniform_p is set to: %d\n",
- SLP_TREE_GS_OFFSET_UNIFORM_P (slp_node));
+ if (SLP_TREE_GS_OFFSET_UNIFORM_P (slp_node) && dump_enabled_p ())
+ dump_printf_loc (MSG_OPTIMIZED_LOCATIONS, vect_location,
+ "gather offset is uniform across each individual "
+ "vector's lanes; using scalar-load + broadcast "
+ "instead of emulated gather\n");
slp_tree offset_node = SLP_TREE_CHILDREN (slp_node)[0];
tree offset_vectype = SLP_TREE_VECTYPE (offset_node);
int scale = SLP_TREE_GS_SCALE (slp_node);
@@ -11203,18 +11204,39 @@ vectorizable_load (vec_info *vinfo,
unsigned HOST_WIDE_INT const_nunits = nunits.to_constant ();
if (costing_p)
{
- /* For emulated gathers N offset vector element
- offset add is consumed by the load). */
- inside_cost = record_stmt_cost (cost_vec, 1, vec_deconstruct,
- slp_node, 0, vect_body);
- /* N scalar loads plus gathering them into a
- vector. */
- inside_cost
- = record_stmt_cost (cost_vec, const_nunits, scalar_load,
- slp_node, 0, vect_body);
- inside_cost
- = record_stmt_cost (cost_vec, 1, vec_construct,
- slp_node, 0, vect_body);
+ if (slp_node->gs_offset_uniform_p)
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, vect_location,
+ "computing cost for broadcast \n");
+ /* Broadcast optimization: 1 extract + 1 load
+ + 1 broadcast. */
+ inside_cost = record_stmt_cost (cost_vec, 1,
+ vec_to_scalar, slp_node,
+ 0, vect_body);
+ inside_cost += record_stmt_cost (cost_vec, 1, scalar_load,
+ slp_node, 0, vect_body);
+ inside_cost += record_stmt_cost (cost_vec, 1,
+ scalar_to_vec, slp_node,
+ 0, vect_body);
+ }
+ else
+ {
+
+ /* For emulated gathers N offset vector element
+ offset add is consumed by the load). */
+ inside_cost = record_stmt_cost (cost_vec, 1,
+ vec_deconstruct, slp_node,
+ 0, vect_body);
+ /* N scalar loads plus gathering them into a
+ vector. */
+ inside_cost
+ = record_stmt_cost (cost_vec, const_nunits, scalar_load,
+ slp_node, 0, vect_body);
+ inside_cost
+ = record_stmt_cost (cost_vec, 1, vec_construct,
+ slp_node, 0, vect_body);
+ }
continue;
}
tree offset_vectype = TREE_TYPE (vec_offsets[0]);
@@ -11232,25 +11254,37 @@ vectorizable_load (vec_info *vinfo,
tree idx_type = TREE_TYPE (TREE_TYPE (vec_offset));
tree scale = size_int (SLP_TREE_GS_SCALE (slp_node));
tree ltype = build_aligned_type (TREE_TYPE (vectype), align);
+ tree first_elt = NULL_TREE;
for (unsigned k = 0; k < const_nunits; ++k)
{
- tree boff = size_binop (MULT_EXPR, TYPE_SIZE (idx_type),
- bitsize_int (k + elt_offset));
- tree idx = gimple_build (&stmts, BIT_FIELD_REF, idx_type,
- vec_offset, TYPE_SIZE (idx_type),
- boff);
- idx = gimple_convert (&stmts, sizetype, idx);
- idx = gimple_build (&stmts, MULT_EXPR, sizetype, idx, scale);
- tree ptr = gimple_build (&stmts, PLUS_EXPR,
- TREE_TYPE (dataref_ptr),
- dataref_ptr, idx);
- ptr = gimple_convert (&stmts, ptr_type_node, ptr);
- tree elt = make_ssa_name (TREE_TYPE (vectype));
- tree ref = build2 (MEM_REF, ltype, ptr,
- build_int_cst (ref_type, 0));
- new_stmt = gimple_build_assign (elt, ref);
- gimple_set_vuse (new_stmt, gimple_vuse (gsi_stmt (*gsi)));
- gimple_seq_add_stmt (&stmts, new_stmt);
+ tree elt;
+ /* If all lanes are uniform then generate broadcast otherwise
+ emulated gathers. */
+ if (slp_node->gs_offset_uniform_p && k > 0)
+ elt = first_elt;
+ else
+ {
+
+ tree boff = size_binop (MULT_EXPR, TYPE_SIZE (idx_type),
+ bitsize_int (k + elt_offset));
+ tree idx = gimple_build (&stmts, BIT_FIELD_REF, idx_type,
+ vec_offset, TYPE_SIZE (idx_type),
+ boff);
+ idx = gimple_convert (&stmts, sizetype, idx);
+ idx = gimple_build (&stmts, MULT_EXPR, sizetype, idx,
+ scale);
+ tree ptr = gimple_build (&stmts, PLUS_EXPR,
+ TREE_TYPE (dataref_ptr),
+ dataref_ptr, idx);
+ ptr = gimple_convert (&stmts, ptr_type_node, ptr);
+ elt = make_ssa_name (TREE_TYPE (vectype));
+ tree ref = build2 (MEM_REF, ltype, ptr,
+ build_int_cst (ref_type, 0));
+ new_stmt = gimple_build_assign (elt, ref);
+ gimple_set_vuse (new_stmt, gimple_vuse (gsi_stmt (*gsi)));
+ gimple_seq_add_stmt (&stmts, new_stmt);
+ first_elt = elt;
+ }
CONSTRUCTOR_APPEND_ELT (ctor_elts, NULL_TREE, elt);
}
gsi_insert_seq_before (gsi, stmts, GSI_SAME_STMT);
--
2.34.1
