On Tue, 14 Jul 2026, [email protected] wrote: > From: Reshma Roy <[email protected]> > > Transform uniform gather operations into optimized scalar load + broadcast > instead of emulated gather with N individual scalar loads. > > BEFORE (Emulated Gather - 8 loads for VF=8): > for each lane i in 0..7: > offset[i] = BIT_FIELD_REF<offset_vector, i> > addr[i] = base + offset[i] * scale > result[i] = MEM[addr[i]] > result_vector = {result[0], ..., result[7]} > > AFTER (Broadcast - 1 load): > offset = BIT_FIELD_REF<offset_vector, 0> > addr = base + offset * scale > scalar_val = MEM[addr] > result_vector = {scalar_val, scalar_val,...} > > v3 changes: > - Dumping: remove the out-of-place transform-time dump and report the > uniform-offset broadcast decision during analysis in get_load_store_type; > - update tests to scan for the new message. > - Formatting : re-indent the emulated-gather costing block under the new > else, fix "k > 0" spacing, and drop the spurious trailing-whitespace > change. > - Update test cases, included more test cases for bot_and_expr as well. > > gcc/ChangeLog: > > * tree-vect-stmts.cc (vectorizable_load): Generate broadcast for > uniform gather offsets. > > gcc/testsuite/ChangeLog: > > * g++.dg/vect/vect-uniform-broadcast-and.c: New test. > * g++.dg/vect/vect-uniform-broadcast-neg-and.c: New test. > * g++.dg/vect/vect-uniform-broadcast-neg.c: New test. > * g++.dg/vect/vect-uniform-broadcast-op2-var.c: New test. > * g++.dg/vect/vect-uniform-broadcast.c: New test.
Please make sure to add a testcase with an SLP opportunity, you can use gcc.dg/vect/vect-gather-1.c as example. I think accessing both table[i & 4] and table[i & 4 + 1] might work (uniformity of table & 4 + 1 would not work, but the thing would be grouped I guess?), another case would be table[i & 4] and table[(i+1) & 4], the latter with a step of two for i of course. Just experiment a bit, see the review on 1/2 - I think we want some explicit handling of more than one SLP lane (even if just rejecting the uniformity there). For the next re-spin please squash the series. > --- > > Hi Richard, > > The comments were addressed and the updated patch is attached. > > Bootstrapped and tested on x86_64 > > Thanks, > Reshma Roy > > Original Message----- > > From: Richard Biener <[email protected]> > > Sent: Tuesday, June 30, 2026 6:01 PM > > To: Roy, Reshma <[email protected]> > > Cc: [email protected]; Kumar, Venkataramanan > > <[email protected]> > > Subject: Re: [PATCH v2 2/2] Loop Vectorizer: Generate broadcast for uniform > > gather > > > > Caution: This message originated from an External Source. Use proper caution > > when opening attachments, clicking links, or responding. > > > > > > On Wed, 24 Jun 2026, [email protected] wrote: > > > > > From: Reshma Roy <[email protected]> > > > > > > Transform uniform gather operations into optimized scalar load + > > > broadcast instead of emulated gather with N individual scalar loads. > > > > > > BEFORE (Emulated Gather - 8 loads for VF=8): > > > for each lane i in 0..7: > > > offset[i] = BIT_FIELD_REF<offset_vector, i> > > > addr[i] = base + offset[i] * scale > > > result[i] = MEM[addr[i]] > > > result_vector = {result[0], ..., result[7]} > > > > > > AFTER (Broadcast - 1 load): > > > offset = BIT_FIELD_REF<offset_vector, 0> > > > addr = base + offset * scale > > > scalar_val = MEM[addr] > > > result_vector = VEC_DUPLICATE_EXPR(scalar_val) > > > > > > v2 changes: > > > - Keep the full loop (k = 0 ??? const_nunits-1) but only perform the > > > scalar load once and reuse elt for all lanes > > > - Since VEC_DUPLICATE_EXPR is for VLA/scalable vector types only > > > removed that and added constructor > > > - Removed the SSA copy by refactoring the code > > > > > > gcc/ChangeLog: > > > > > > * tree-vect-stmts.cc (vectorizable_load): Generate broadcast for > > > uniform gather offsets. > > > > > > gcc/testsuite/ChangeLog: > > > > > > * g++.dg/vect/vect-uniform-broadcast-neg.c: New test. > > > * g++.dg/vect/vect-uniform-broadcast-op2-var.c: New test. > > > * g++.dg/vect/vect-uniform-broadcast.c: New test. > > > > > > --- > > > > > > Hi Richard, > > > > > > The comments were addressed and the updated patch is attached. > > > > > > Thanks, > > > Reshma Roy > > > > > > Original Message----- > > > > From: Richard Biener <[email protected]> > > > > Sent: Tuesday, June 9, 2026 7:31 PM > > > > To: Roy, Reshma <[email protected]> > > > > Cc: [email protected]; Kumar, Venkataramanan > > > > <[email protected]> > > > > Subject: Re: [PATCH 2/2] Loop Vectorizer: Generate broadcast for > > > > uniform gather offsets > > > > > > > > Caution: This message originated from an External Source. Use proper > > > > caution when opening attachments, clicking links, or responding. > > > > > > > > > > > > On Mon, 25 May 2026, [email protected] wrote: > > > > > > > > > From: Reshma Roy <[email protected]> > > > > > > > > > > Transform uniform gather operations into optimized scalar load + > > > > > broadcast instead of emulated gather with N individual scalar loads. > > > > > > > > > > BEFORE (Emulated Gather - 8 loads for VF=8): > > > > > for each lane i in 0..7: > > > > > offset[i] = BIT_FIELD_REF<offset_vector, i> > > > > > addr[i] = base + offset[i] * scale > > > > > result[i] = MEM[addr[i]] > > > > > result_vector = {result[0], ..., result[7]} > > > > > > > > > > AFTER (Broadcast - 1 load): > > > > > offset = BIT_FIELD_REF<offset_vector, 0> > > > > > addr = base + offset * scale > > > > > scalar_val = MEM[addr] > > > > > result_vector = VEC_DUPLICATE_EXPR(scalar_val) > > > > > > > > > > gcc/ChangeLog: > > > > > > > > > > * tree-vect-stmts.cc (vectorizable_load): Generate broadcast for > > > > > uniform gather offsets. > > > > > > > > > > gcc/testsuite/ChangeLog: > > > > > * g++.dg/vect/vect-uniform-broadcast-neg.c: New test. > > > > > * g++.dg/vect/vect-uniform-broadcast-op2-var.c: New test. > > > > > * g++.dg/vect/vect-uniform-broadcast.c: New test. > > > > > > > > > > --- > > > > > .../g++.dg/vect/vect-uniform-broadcast-neg.c | 29 +++++++++ > > > > > .../vect/vect-uniform-broadcast-op2-var.c | 38 ++++++++++++ > > > > > .../g++.dg/vect/vect-uniform-broadcast.c | 29 +++++++++ > > > > > gcc/tree-vect-stmts.cc | 59 > > > > > ++++++++++++++++--- > > > > > 4 files changed, 148 insertions(+), 7 deletions(-) create mode > > > > > 100644 gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg.c > > > > > create mode 100644 > > > > > gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-op2-var.c > > > > > create mode 100644 > > > > > gcc/testsuite/g++.dg/vect/vect-uniform-broadcast.c > > > > > > > > > > diff --git > > > > > a/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg.c > > > > > b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg.c > > > > > new file mode 100644 > > > > > index 00000000000..590706d3b51 > > > > > --- /dev/null > > > > > +++ b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg.c > > > > > @@ -0,0 +1,29 @@ > > > > > +/* { dg-do compile } */ > > > > > +/* { dg-additional-options "-O3 -std=c++11 -march=znver5 > > > > > +-fdump-tree-vect-details" } */ > > > > > +/* Negative test: loop i starts at 1 -> SCEV {1, +, 1}. POC only > > > > > handles > > > > > + {0, +, 1}, so offset is not treated as uniform -> no broadcast > > > > > + (vec_duplicate_expr must not appear for this gather). */ > > > > > +#include <bitset> #include <vector> class Foo { > > > > > + public: > > > > > + void fun (unsigned boardsize, bool check); > > > > > + std::vector<int> m_mcowner; > > > > > +}; > > > > > +void Foo::fun (unsigned boardsize, bool check) { > > > > > + std::bitset<21*21> blacksq; > > > > > + __asm__ ("" : "+g" (blacksq)); > > > > > + for (int i = 0; i < boardsize; i++) { > > > > > + if (i % 2 == 0) > > > > > + blacksq[i] = true; > > > > > + } > > > > > + unsigned int i = 1; > > > > > + do { > > > > > + if (blacksq[i]) > > > > > + m_mcowner[i]++; > > > > > + } while (++i < blacksq.size ()); } > > > > > +/* { dg-final { scan-tree-dump-not "\.vec_duplicate_expr" "vect" > > > > > +} } */ > > > > > + > > > > > diff --git > > > > > a/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-op2-var.c > > > > > b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-op2-var.c > > > > > new file mode 100644 > > > > > index 00000000000..b212b3d56be > > > > > --- /dev/null > > > > > +++ b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-op2-var.c > > > > > @@ -0,0 +1,38 @@ > > > > > +/* { dg-do compile } */ > > > > > +/* { dg-additional-options "-O3 -std=c++11 -march=znver5 > > > > > +-fdump-tree-vect-details" } */ > > > > > +/* Gather load with PLUS (hi, lo). The second operand of the offset > > > > > PLUS is a > > > > > + variable, hence does not emit vec_duplicate_expr. The inner loop > > > > > still > > > > > + vectorizes as a normal gather */ > > > > > + > > > > > +#include <vector> > > > > > + > > > > > +class Foo > > > > > +{ > > > > > + public: > > > > > + void fun (unsigned boardsize); > > > > > + std::vector<int> m_mcowner; > > > > > +}; > > > > > + > > > > > +void > > > > > +Foo::fun (unsigned boardsize) > > > > > +{ > > > > > + unsigned nwords = (21 * 21 + 63) / 64 + 2; > > > > > + std::vector<unsigned long> words (nwords, 0); > > > > > + __asm__ ("" : "+g" (words)); > > > > > + for (unsigned i = 0; i < boardsize; i++) > > > > > + if (i % 2 == 0) > > > > > + words[i / 64] |= 1UL << (i % 64); > > > > > + unsigned int i = 0; > > > > > + do > > > > > + { > > > > > + unsigned hi = i / 64; > > > > > + unsigned lo = i & 1u; > > > > > + /* the second operand is ensured to be loop-varying by the > > > > > SCEV path */ > > > > > + if (words[hi + lo]) > > > > > + m_mcowner[i]++; > > > > > + } > > > > > + while (++i < 21 * 21); > > > > > +} > > > > > + > > > > > +/* { dg-final { scan-tree-dump-not "\.vec_duplicate_expr" "vect" > > > > > +} } */ > > > > > +/* { dg-final { scan-tree-dump "gs_offset_uniform_p is set to: 0" > > > > > +"vect" } } */ > > > > > diff --git a/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast.c > > > > > b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast.c > > > > > new file mode 100644 > > > > > index 00000000000..818b7a63c00 > > > > > --- /dev/null > > > > > +++ b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast.c > > > > > @@ -0,0 +1,29 @@ > > > > > +/* { dg-do compile } */ > > > > > +/* { dg-additional-options "-O3 -std=c++11 -march=znver5 > > > > > +-fdump-tree-vect-details" } */ > > > > > +/* Test case for generating uniform broadcast instead of gather > > > > > + when there is uniform value accross the vector lane */ > > > > > +#include <bitset> #include <vector> class Foo { > > > > > + public: > > > > > + void fun (unsigned boardsize, bool check); > > > > > + std::vector<int> m_mcowner; > > > > > +}; > > > > > +void Foo::fun( unsigned boardsize, bool check) { > > > > > + std::bitset<21*21> blacksq; > > > > > + __asm__ ("" : "+g" (blacksq)); > > > > > + for (int i = 0; i < boardsize; i++) { > > > > > + if(i%2 == 0) > > > > > + blacksq[i] = true; > > > > > + } > > > > > + unsigned int i = 0; > > > > > + do{ > > > > > + if (blacksq[i]) > > > > > + m_mcowner[i]++; > > > > > + }while(++i < blacksq.size()); > > > > > + > > > > > +} > > > > > +/* { dg-final { scan-tree-dump "\.vec_duplicate_expr" "vect" } } > > > > > +*/ > > > > > + > > > > > diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc index > > > > > b68fc5072af..71ff138c069 100644 > > > > > --- a/gcc/tree-vect-stmts.cc > > > > > +++ b/gcc/tree-vect-stmts.cc > > > > > @@ -11198,6 +11198,25 @@ vectorizable_load (vec_info *vinfo, > > > > > unsigned HOST_WIDE_INT const_nunits = nunits.to_constant > > > > > (); > > > > > if (costing_p) > > > > > { > > > > > + if (slp_node->gs_offset_uniform_p) > > > > > + { > > > > > + if (dump_enabled_p ()) > > > > > + dump_printf_loc (MSG_NOTE, vect_location, > > > > > + "computing cost for broadcast > > > > > \n"); > > > > > + /* Broadcast optimization: 1 extract + 1 load > > > > > + + 1 broadcast. */ > > > > > + inside_cost = record_stmt_cost (cost_vec, 1, > > > > > + vec_to_scalar, > > > > > slp_node, > > > > > + 0, vect_body); > > > > > + inside_cost += record_stmt_cost (cost_vec, 1, > > > > > scalar_load, > > > > > + slp_node, 0, > > > > > vect_body); > > > > > + inside_cost += record_stmt_cost (cost_vec, 1, > > > > > + scalar_to_vec, > > > > > slp_node, > > > > > + 0, vect_body); > > > > > + } > > > > > + else > > > > > + { > > > > > + > > > > > /* For emulated gathers N offset vector element > > > > > offset add is consumed by the load). */ > > > > > inside_cost = record_stmt_cost (cost_vec, > > > > > const_nunits, @@ -11211,6 +11230,7 @@ vectorizable_load (vec_info > > > > > *vinfo, > > > > > inside_cost > > > > > = record_stmt_cost (cost_vec, 1, vec_construct, > > > > > slp_node, 0, vect_body); > > > > > + } > > > > > continue; > > > > > } > > > > > tree offset_vectype = TREE_TYPE (vec_offsets[0]); @@ > > > > > -11228,7 +11248,12 @@ vectorizable_load (vec_info *vinfo, > > > > > tree idx_type = TREE_TYPE (TREE_TYPE (vec_offset)); > > > > > tree scale = size_int (SLP_TREE_GS_SCALE (slp_node)); > > > > > tree ltype = build_aligned_type (TREE_TYPE (vectype), > > > > > align); > > > > > - for (unsigned k = 0; k < const_nunits; ++k) > > > > > + tree broadcast_vec = NULL_TREE; > > > > > + /* If all lanes are uniform then generate broadcast > > > > > otherwise > > > > > + emulated gathers. */ > > > > > + unsigned HOST_WIDE_INT vec_lane_loads > > > > > + = slp_node->gs_offset_uniform_p ? 1 : const_nunits; > > > > > > > > instead of this > > > > > > > > > + for (unsigned k = 0; k < vec_lane_loads; ++k) > > > > > { > > > > > tree boff = size_binop (MULT_EXPR, TYPE_SIZE > > > > > (idx_type), > > > > > bitsize_int (k + > > > > > elt_offset)); @@ -11247,13 +11272,33 @@ vectorizable_load (vec_info > > > > > *vinfo, > > > > > new_stmt = gimple_build_assign (elt, ref); > > > > > gimple_set_vuse (new_stmt, gimple_vuse (gsi_stmt > > > > > (*gsi))); > > > > > gimple_seq_add_stmt (&stmts, new_stmt); > > > > > - CONSTRUCTOR_APPEND_ELT (ctor_elts, NULL_TREE, elt); > > > > > + if (!slp_node->gs_offset_uniform_p) > > > > > + CONSTRUCTOR_APPEND_ELT (ctor_elts, NULL_TREE, > > > > > elt); > > > > > + else > > > > > + /* Broadcast to all lanes. */ > > > > > + broadcast_vec = gimple_build (&stmts, > > > > > + VEC_DUPLICATE_EXPR, > > > > > + vectype, elt); > > > > > > > > break from the loop here. Also use gimple_build_vector_from_val, a > > > > VEC_DUPLICATE_EXPR is only valid for VLA vectors, a broadcast for > > > > non-VLA vectors should use a uniform CONSTRUCTOR. > > > Done. > > > > > > > > > + } > > > > > + if (!slp_node->gs_offset_uniform_p) > > > > > + { > > > > > + gsi_insert_seq_before (gsi, stmts, GSI_SAME_STMT); > > > > > + new_stmt = gimple_build_assign (NULL_TREE, > > > > > + build_constructor > > > > > (vectype, > > > > > + > > > > > ctor_elts) > > > > > + ); > > > > > + data_ref = NULL_TREE; > > > > > + } > > > > > + else > > > > > + { > > > > > + gsi_insert_seq_before (gsi, stmts, GSI_SAME_STMT); > > > > > + new_stmt = gimple_build_assign (NULL_TREE, > > > > > + broadcast_vec); > > > > > > > > you get a SSA copy from this, some refactoring should avoid that. > > > > Or simply build that uniform vector above by keeping the loop but > > > > only load 'elt' once (but add it nunits time). > > > Refactored the code such that SSA copy is removed. > > > > > > > > > + data_ref = NULL_TREE; > > > > > + if (dump_enabled_p ()) > > > > > + dump_printf_loc (MSG_OPTIMIZED_LOCATIONS, > > > > > vect_location, > > > > > + "generating broadcast instead of > > > > > gather " > > > > > + "loads (uniform offset > > > > > + lanes)\n"); > > > > > } > > > > > - gsi_insert_seq_before (gsi, stmts, GSI_SAME_STMT); > > > > > - new_stmt = gimple_build_assign (NULL_TREE, > > > > > - build_constructor > > > > > (vectype, > > > > > - > > > > > ctor_elts)); > > > > > - data_ref = NULL_TREE; > > > > > } > > > > > > > > > > vec_dest = vect_create_destination_var (scalar_dest, > > > > > vectype); > > > > > > > > > > > > > -- > > > > Richard Biener <[email protected]> > > > > SUSE Software Solutions Germany GmbH, Frankenstrasse 146, 90461 > > > > Nuernberg, Germany; > > > > GF: Jochen Jaser, Andrew McDonald, Werner Knoblich; (HRB 36809, AG > > > > Nuernberg) > > > > > > > > > .../g++.dg/vect/vect-uniform-broadcast-neg.c | 29 ++++++++ > > > .../vect/vect-uniform-broadcast-op2-var.c | 38 ++++++++++ > > > .../g++.dg/vect/vect-uniform-broadcast.c | 29 ++++++++ > > > gcc/tree-vect-stmts.cc | 71 ++++++++++++++----- > > > 4 files changed, 149 insertions(+), 18 deletions(-) create mode > > > 100644 gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg.c > > > create mode 100644 > > > gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-op2-var.c > > > create mode 100644 gcc/testsuite/g++.dg/vect/vect-uniform-broadcast.c > > > > > > diff --git a/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg.c > > > b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg.c > > > new file mode 100644 > > > index 00000000000..590706d3b51 > > > --- /dev/null > > > +++ b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg.c > > > @@ -0,0 +1,29 @@ > > > +/* { dg-do compile } */ > > > +/* { dg-additional-options "-O3 -std=c++11 -march=znver5 > > > +-fdump-tree-vect-details" } */ > > > +/* Negative test: loop i starts at 1 -> SCEV {1, +, 1}. POC only handles > > > + {0, +, 1}, so offset is not treated as uniform -> no broadcast > > > + (vec_duplicate_expr must not appear for this gather). */ #include > > > +<bitset> #include <vector> class Foo { > > > + public: > > > + void fun (unsigned boardsize, bool check); > > > + std::vector<int> m_mcowner; > > > +}; > > > +void Foo::fun (unsigned boardsize, bool check) { > > > + std::bitset<21*21> blacksq; > > > + __asm__ ("" : "+g" (blacksq)); > > > + for (int i = 0; i < boardsize; i++) { > > > + if (i % 2 == 0) > > > + blacksq[i] = true; > > > + } > > > + unsigned int i = 1; > > > + do { > > > + if (blacksq[i]) > > > + m_mcowner[i]++; > > > + } while (++i < blacksq.size ()); > > > +} > > > +/* { dg-final { scan-tree-dump-not "\.vec_duplicate_expr" "vect" } } > > > +*/ > > > + > > > diff --git > > > a/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-op2-var.c > > > b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-op2-var.c > > > new file mode 100644 > > > index 00000000000..b212b3d56be > > > --- /dev/null > > > +++ b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-op2-var.c > > > @@ -0,0 +1,38 @@ > > > +/* { dg-do compile } */ > > > +/* { dg-additional-options "-O3 -std=c++11 -march=znver5 > > > +-fdump-tree-vect-details" } */ > > > +/* Gather load with PLUS (hi, lo). The second operand of the offset PLUS > > > is a > > > + variable, hence does not emit vec_duplicate_expr. The inner loop still > > > + vectorizes as a normal gather */ > > > + > > > +#include <vector> > > > + > > > +class Foo > > > +{ > > > + public: > > > + void fun (unsigned boardsize); > > > + std::vector<int> m_mcowner; > > > +}; > > > + > > > +void > > > +Foo::fun (unsigned boardsize) > > > +{ > > > + unsigned nwords = (21 * 21 + 63) / 64 + 2; > > > + std::vector<unsigned long> words (nwords, 0); > > > + __asm__ ("" : "+g" (words)); > > > + for (unsigned i = 0; i < boardsize; i++) > > > + if (i % 2 == 0) > > > + words[i / 64] |= 1UL << (i % 64); > > > + unsigned int i = 0; > > > + do > > > + { > > > + unsigned hi = i / 64; > > > + unsigned lo = i & 1u; > > > + /* the second operand is ensured to be loop-varying by the SCEV > > > path */ > > > + if (words[hi + lo]) > > > + m_mcowner[i]++; > > > + } > > > + while (++i < 21 * 21); > > > +} > > > + > > > +/* { dg-final { scan-tree-dump-not "\.vec_duplicate_expr" "vect" } } > > > +*/ > > > +/* { dg-final { scan-tree-dump "gs_offset_uniform_p is set to: 0" > > > +"vect" } } */ > > > diff --git a/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast.c > > > b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast.c > > > new file mode 100644 > > > index 00000000000..818b7a63c00 > > > --- /dev/null > > > +++ b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast.c > > > @@ -0,0 +1,29 @@ > > > +/* { dg-do compile } */ > > > +/* { dg-additional-options "-O3 -std=c++11 -march=znver5 > > > +-fdump-tree-vect-details" } */ > > > +/* Test case for generating uniform broadcast instead of gather > > > + when there is uniform value accross the vector lane */ #include > > > +<bitset> #include <vector> class Foo { > > > + public: > > > + void fun (unsigned boardsize, bool check); > > > + std::vector<int> m_mcowner; > > > +}; > > > +void Foo::fun( unsigned boardsize, bool check) { > > > + std::bitset<21*21> blacksq; > > > + __asm__ ("" : "+g" (blacksq)); > > > + for (int i = 0; i < boardsize; i++) { > > > + if(i%2 == 0) > > > + blacksq[i] = true; > > > + } > > > + unsigned int i = 0; > > > + do{ > > > + if (blacksq[i]) > > > + m_mcowner[i]++; > > > + }while(++i < blacksq.size()); > > > + > > > +} > > > +/* { dg-final { scan-tree-dump "\.vec_duplicate_expr" "vect" } } */ > > > + > > > diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc index > > > afd5109c9f7..44282f20873 100644 > > > --- a/gcc/tree-vect-stmts.cc > > > +++ b/gcc/tree-vect-stmts.cc > > > @@ -11207,6 +11207,25 @@ vectorizable_load (vec_info *vinfo, > > > unsigned HOST_WIDE_INT const_nunits = nunits.to_constant (); > > > if (costing_p) > > > { > > > + if (slp_node->gs_offset_uniform_p) > > > + { > > > + if (dump_enabled_p ()) > > > + dump_printf_loc (MSG_NOTE, vect_location, > > > + "computing cost for broadcast \n"); > > > + /* Broadcast optimization: 1 extract + 1 load > > > + + 1 broadcast. */ > > > + inside_cost = record_stmt_cost (cost_vec, 1, > > > + vec_to_scalar, > > > slp_node, > > > + 0, vect_body); > > > + inside_cost += record_stmt_cost (cost_vec, 1, > > > scalar_load, > > > + slp_node, 0, > > > vect_body); > > > + inside_cost += record_stmt_cost (cost_vec, 1, > > > + scalar_to_vec, > > > slp_node, > > > + 0, vect_body); > > > + } > > > + else > > > + { > > > + > > > /* For emulated gathers N offset vector element > > > offset add is consumed by the load). */ > > > > this block now needs re-indenting. > Done. > > > > > inside_cost = record_stmt_cost (cost_vec, 1, > > > vec_deconstruct, @@ -11219,6 +11238,7 @@ vectorizable_load (vec_info > > > *vinfo, > > > inside_cost > > > = record_stmt_cost (cost_vec, 1, vec_construct, > > > slp_node, 0, vect_body); > > > + } > > > continue; > > > } > > > tree offset_vectype = TREE_TYPE (vec_offsets[0]); @@ > > > -11236,25 +11256,37 @@ vectorizable_load (vec_info *vinfo, > > > tree idx_type = TREE_TYPE (TREE_TYPE (vec_offset)); > > > tree scale = size_int (SLP_TREE_GS_SCALE (slp_node)); > > > tree ltype = build_aligned_type (TREE_TYPE (vectype), > > > align); > > > + tree first_elt = NULL_TREE; > > > for (unsigned k = 0; k < const_nunits; ++k) > > > { > > > - tree boff = size_binop (MULT_EXPR, TYPE_SIZE (idx_type), > > > - bitsize_int (k + elt_offset)); > > > - tree idx = gimple_build (&stmts, BIT_FIELD_REF, idx_type, > > > - vec_offset, TYPE_SIZE (idx_type), > > > - boff); > > > - idx = gimple_convert (&stmts, sizetype, idx); > > > - idx = gimple_build (&stmts, MULT_EXPR, sizetype, idx, > > > scale); > > > - tree ptr = gimple_build (&stmts, PLUS_EXPR, > > > - TREE_TYPE (dataref_ptr), > > > - dataref_ptr, idx); > > > - ptr = gimple_convert (&stmts, ptr_type_node, ptr); > > > - tree elt = make_ssa_name (TREE_TYPE (vectype)); > > > - tree ref = build2 (MEM_REF, ltype, ptr, > > > - build_int_cst (ref_type, 0)); > > > - new_stmt = gimple_build_assign (elt, ref); > > > - gimple_set_vuse (new_stmt, gimple_vuse (gsi_stmt (*gsi))); > > > - gimple_seq_add_stmt (&stmts, new_stmt); > > > + tree elt; > > > + /* If all lanes are uniorm then generate broadcast > > > otherwise > > > + emulated gathers. */ > > > + if (slp_node->gs_offset_uniform_p && k>0) > > > > k > 0 > Done. > > > > > + elt = first_elt; > > > + else > > > + { > > > + > > > + tree boff = size_binop (MULT_EXPR, TYPE_SIZE > > > (idx_type), > > > + bitsize_int (k + elt_offset)); > > > + tree idx = gimple_build (&stmts, BIT_FIELD_REF, > > > idx_type, > > > + vec_offset, TYPE_SIZE > > > (idx_type), > > > + boff); > > > + idx = gimple_convert (&stmts, sizetype, idx); > > > + idx = gimple_build (&stmts, MULT_EXPR, sizetype, idx, > > > + scale); > > > + tree ptr = gimple_build (&stmts, PLUS_EXPR, > > > + TREE_TYPE (dataref_ptr), > > > + dataref_ptr, idx); > > > + ptr = gimple_convert (&stmts, ptr_type_node, ptr); > > > + elt = make_ssa_name (TREE_TYPE (vectype)); > > > + tree ref = build2 (MEM_REF, ltype, ptr, > > > + build_int_cst (ref_type, 0)); > > > + new_stmt = gimple_build_assign (elt, ref); > > > + gimple_set_vuse (new_stmt, gimple_vuse (gsi_stmt > > > (*gsi))); > > > + gimple_seq_add_stmt (&stmts, new_stmt); > > > + first_elt = elt; > > > + } > > > CONSTRUCTOR_APPEND_ELT (ctor_elts, NULL_TREE, elt); > > > } > > > gsi_insert_seq_before (gsi, stmts, GSI_SAME_STMT); @@ > > > -11262,8 +11294,11 @@ vectorizable_load (vec_info *vinfo, > > > build_constructor (vectype, > > > > > > ctor_elts)); > > > data_ref = NULL_TREE; > > > + if (slp_node->gs_offset_uniform_p && dump_enabled_p ()) > > > + dump_printf_loc (MSG_OPTIMIZED_LOCATIONS, vect_location, > > > + "generating broadcast instead of gather " > > > + "loads (uniform offset lanes)\n"); > > > > The extra dump prints are a bit out-of-place, we probably want to dump (in > > general) > > the gather/scatter kind that's being used, but during analysis and for all > > cases. We > > might already do, if we do you want to ament that. > I could not find a place during the analysis where the dumps are already > done, hence i have moved this dump to a location in the analysis which i > think is sufficient. > > > > > } > > > - > > > > please avoid spurious whitespace changes > Done. > > > > > vec_dest = vect_create_destination_var (scalar_dest, vectype); > > > /* DATA_REF is null if we've already built the statement. */ > > > if (data_ref) > > > > > > > -- > > Richard Biener <[email protected]> > > SUSE Software Solutions Germany GmbH, > > Frankenstrasse 146, 90461 Nuernberg, Germany; > > GF: Jochen Jaser, Andrew McDonald, Werner Knoblich; (HRB 36809, AG > > Nuernberg) > > > > .../g++.dg/vect/vect-uniform-broadcast-and.c | 33 ++++++ > .../vect/vect-uniform-broadcast-neg-and.c | 40 +++++++ > .../g++.dg/vect/vect-uniform-broadcast-neg.c | 29 +++++ > .../vect/vect-uniform-broadcast-op2-var.c | 37 +++++++ > .../g++.dg/vect/vect-uniform-broadcast.c | 29 +++++ > gcc/tree-vect-stmts.cc | 100 ++++++++++++------ > 6 files changed, 235 insertions(+), 33 deletions(-) > create mode 100644 gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-and.c > create mode 100644 gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg-and.c > create mode 100644 gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg.c > create mode 100644 gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-op2-var.c > create mode 100644 gcc/testsuite/g++.dg/vect/vect-uniform-broadcast.c > > diff --git a/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-and.c > b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-and.c > new file mode 100644 > index 00000000000..2736aa73131 > --- /dev/null > +++ b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-and.c > @@ -0,0 +1,33 @@ > +/* { dg-do compile { target { x86_64-*-* i?86-*-* } } } */ put the target restriction > +/* { dg-additional-options "-O3 -march=znver5 -fdump-tree-vect-details" } */ > +/* { dg-additional-options "-mprefer-vector-width=128" } */ On these. Please avoid -march=znver5 - tuning should be not relevant and we shouldn't need gather either? So you only want the -mprefer-vector-width=128 conditional on x86. -fdump-tree-vect-details is always added by the harness. This might be also more suitable for gcc.dg/vect/costmodel/x86_64/ where I'd expect that costing would favor SSE vector width and there -march=znver5 would be appropriate. > +/* Gather load table[i & 4]: mask 4 affects only bit 2, so (4 & (N - 1)) == 0 > + when N = 4. -mprefer-vector-width=128 encourages 4-wide int vectors on > + x86 so i & 4 is uniform per vector chunk and broadcast is possible. */ > + > +#include <stdio.h> > + > +#define N 32 > + > +int block_values[8] = {10, 20, 30, 40, 50, 60, 70, 80}; > +int result[N]; > + > +void > +tile_fill (int *__restrict out, int *__restrict table, int n, > + int *__restrict in) > +{ > + for (int i = 0; i < n; i++) > + { > + out[i] = table[i & 4]; > + in[i] = out[i] + 10; > + } > +} > + > +void > +use_tile_fill (void) > +{ > + tile_fill (result, block_values, N, result); > + __asm__ volatile ("" : : "r" (result) : "memory"); > +} > + > +/* { dg-final { scan-tree-dump "gather offset is uniform across vector > lanes" "vect" } } */ > diff --git a/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg-and.c > b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg-and.c > new file mode 100644 > index 00000000000..82a85834661 > --- /dev/null > +++ b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg-and.c > @@ -0,0 +1,40 @@ > +/* { dg-do compile } */ > +/* { dg-additional-options "-O3 -march=znver5 -fdump-tree-vect-details" } */ needs to be restricted to x86. Why do you need -O3? The analysis should run even with plain SSE? So why additonal options at all? > +/* Negative test: the offset must NOT be reported as uniform for ANY > candidate > + vector size the vectorizer tries. > + > + NOTE: a right-shift offset (i >> k) cannot be used for a negative test > here. > + The vectorizer analyzes every candidate vector size (32, 16, 8, 4, 2), and > + for a right shift the offset is uniform per chunk iff N - 1 < 2^k. At the > + smallest candidate N = 2 this is 1 < 2^k, which holds for every k >= 1, so > + *some* candidate always reports "uniform" and scan-tree-dump-not would > fail. > + > + For BIT_AND the offset is uniform iff (k & (N - 1)) == 0. With an odd > mask > + (k = 1) bit 0 is set, and N - 1 has bit 0 set for every N >= 2, so the > + offset is non-uniform for ALL candidate vector sizes. The broadcast > + optimization must never trigger. */ > + > +#include <stdio.h> > + > +#define N 512 > + > +short block_values[64]; > +short result[N]; > + > +void > +tile_fill (short *__restrict out, short *__restrict table, int n) > +{ > + for (int i = 0; i < n; i++) > + out[i] = table[i & 1]; > +} > + > +void > +use_tile_fill (void) > +{ > + tile_fill (result, block_values, N); > + __asm__ volatile ("" : : "r" (result) : "memory"); > +} > + > +/* The offset is not uniform across the vector lanes, so no broadcast should > be > + reported. */ > +/* { dg-final { scan-tree-dump-not "gather offset is uniform across vector > lanes" "vect" } } */ > diff --git a/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg.c > b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg.c > new file mode 100644 > index 00000000000..b366753ec62 > --- /dev/null > +++ b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg.c > @@ -0,0 +1,29 @@ > +/* { dg-do compile { target { x86_64-*-* i?86-*-* } } } */ > +/* { dg-additional-options "-O3 -std=c++11 -march=znver5 > -fdump-tree-vect-details" } */ > +/* Negative test: loop i starts at 1 -> SCEV {1, +, 1}. POC only handles > + {0, +, 1}, so offset is not treated as uniform -> no broadcast > + (offset must not be reported as uniform for this gather). */ Same. > +#include <bitset> > +#include <vector> > +class Foo > +{ > + public: > + void fun (unsigned boardsize, bool check); > + std::vector<int> m_mcowner; > +}; > +void Foo::fun (unsigned boardsize, bool check) > +{ > + std::bitset<21*21> blacksq; > + __asm__ ("" : "+g" (blacksq)); > + for (int i = 0; i < boardsize; i++) { > + if (i % 2 == 0) > + blacksq[i] = true; > + } > + unsigned int i = 1; > + do { > + if (blacksq[i]) > + m_mcowner[i]++; > + } while (++i < blacksq.size ()); > +} > +/* { dg-final { scan-tree-dump-not "gather offset is uniform across each > individual" "vect" } } */ > + > diff --git a/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-op2-var.c > b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-op2-var.c > new file mode 100644 > index 00000000000..97b18fabcc9 > --- /dev/null > +++ b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-op2-var.c > @@ -0,0 +1,37 @@ > +/* { dg-do compile { target { x86_64-*-* i?86-*-* } } } */ > +/* { dg-additional-options "-O3 -std=c++11 -march=znver5 > -fdump-tree-vect-details" } */ > +/* Gather load with PLUS (hi, lo). The second operand of the offset PLUS is a > + variable, hence the offset is not uniform and no broadcast is generated. > + The inner loop still vectorizes as a normal gather. */ > + same (you don't verify it gets vectorized as gather) > +#include <vector> > + > +class Foo > +{ > + public: > + void fun (unsigned boardsize); > + std::vector<int> m_mcowner; > +}; > + > +void > +Foo::fun (unsigned boardsize) > +{ > + unsigned nwords = (21 * 21 + 63) / 64 + 2; > + std::vector<unsigned long> words (nwords, 0); > + __asm__ ("" : "+g" (words)); > + for (unsigned i = 0; i < boardsize; i++) > + if (i % 2 == 0) > + words[i / 64] |= 1UL << (i % 64); > + unsigned int i = 0; > + do > + { > + unsigned hi = i / 64; > + unsigned lo = i & 1u; > + /* the second operand is ensured to be loop-varying by the SCEV path */ > + if (words[hi + lo]) > + m_mcowner[i]++; > + } > + while (++i < 21 * 21); > +} > + > +/* { dg-final { scan-tree-dump-not "gather offset is uniform across each > individual" "vect" } } */ > diff --git a/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast.c > b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast.c > new file mode 100644 > index 00000000000..507bc6e4e3e > --- /dev/null > +++ b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast.c > @@ -0,0 +1,29 @@ > +/* { dg-do compile { target { x86_64-*-* i?86-*-* } } } */ > +/* { dg-additional-options "-O3 -std=c++11 -march=znver5 > -fdump-tree-vect-details" } */ no need to add -std=c++11, the default is c++20 already. Similar comments as to the first testcase. > +/* Test case for generating uniform broadcast instead of gather > + when there is uniform value accross the vector lane */ > +#include <bitset> > +#include <vector> > +class Foo > +{ > + public: > + void fun (unsigned boardsize, bool check); > + std::vector<int> m_mcowner; > +}; > +void Foo::fun( unsigned boardsize, bool check) > +{ > + std::bitset<21*21> blacksq; > + __asm__ ("" : "+g" (blacksq)); > + for (int i = 0; i < boardsize; i++) { > + if(i%2 == 0) > + blacksq[i] = true; > + } > + unsigned int i = 0; > + do{ > + if (blacksq[i]) > + m_mcowner[i]++; > + }while(++i < blacksq.size()); > + > +} > +/* { dg-final { scan-tree-dump "gather offset is uniform across each > individual" "vect" } } */ > + > diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc > index c1776f78f53..a79db05756e 100644 > --- a/gcc/tree-vect-stmts.cc > +++ b/gcc/tree-vect-stmts.cc > @@ -2168,10 +2168,11 @@ get_load_store_type (vec_info *vinfo, stmt_vec_info > stmt_info, > } > else if (STMT_VINFO_GATHER_SCATTER_P (stmt_info)) > { > - if (dump_enabled_p ()) > - dump_printf_loc (MSG_NOTE, vect_location, > - "gs_offset_uniform_p is set to: %d\n", > - SLP_TREE_GS_OFFSET_UNIFORM_P (slp_node)); > + if (SLP_TREE_GS_OFFSET_UNIFORM_P (slp_node) && dump_enabled_p ()) > + dump_printf_loc (MSG_OPTIMIZED_LOCATIONS, vect_location, > + "gather offset is uniform across each individual " > + "vector's lanes; using scalar-load + broadcast " > + "instead of emulated gather\n"); > slp_tree offset_node = SLP_TREE_CHILDREN (slp_node)[0]; > tree offset_vectype = SLP_TREE_VECTYPE (offset_node); > int scale = SLP_TREE_GS_SCALE (slp_node); > @@ -11203,18 +11204,39 @@ vectorizable_load (vec_info *vinfo, > unsigned HOST_WIDE_INT const_nunits = nunits.to_constant (); > if (costing_p) > { > - /* For emulated gathers N offset vector element > - offset add is consumed by the load). */ > - inside_cost = record_stmt_cost (cost_vec, 1, vec_deconstruct, > - slp_node, 0, vect_body); > - /* N scalar loads plus gathering them into a > - vector. */ > - inside_cost > - = record_stmt_cost (cost_vec, const_nunits, scalar_load, > - slp_node, 0, vect_body); > - inside_cost > - = record_stmt_cost (cost_vec, 1, vec_construct, > - slp_node, 0, vect_body); > + if (slp_node->gs_offset_uniform_p) SLP_TREE_GS_OFFSET_UNIFORM_P > + { > + if (dump_enabled_p ()) > + dump_printf_loc (MSG_NOTE, vect_location, > + "computing cost for broadcast \n"); > + /* Broadcast optimization: 1 extract + 1 load > + + 1 broadcast. */ > + inside_cost = record_stmt_cost (cost_vec, 1, > + vec_to_scalar, slp_node, > + 0, vect_body); > + inside_cost += record_stmt_cost (cost_vec, 1, scalar_load, > + slp_node, 0, vect_body); > + inside_cost += record_stmt_cost (cost_vec, 1, > + scalar_to_vec, slp_node, > + 0, vect_body); > + } > + else > + { > + > + /* For emulated gathers N offset vector element > + offset add is consumed by the load). */ > + inside_cost = record_stmt_cost (cost_vec, 1, > + vec_deconstruct, slp_node, > + 0, vect_body); > + /* N scalar loads plus gathering them into a > + vector. */ > + inside_cost > + = record_stmt_cost (cost_vec, const_nunits, scalar_load, > + slp_node, 0, vect_body); > + inside_cost > + = record_stmt_cost (cost_vec, 1, vec_construct, > + slp_node, 0, vect_body); > + } > continue; > } > tree offset_vectype = TREE_TYPE (vec_offsets[0]); > @@ -11232,25 +11254,37 @@ vectorizable_load (vec_info *vinfo, > tree idx_type = TREE_TYPE (TREE_TYPE (vec_offset)); > tree scale = size_int (SLP_TREE_GS_SCALE (slp_node)); > tree ltype = build_aligned_type (TREE_TYPE (vectype), align); > + tree first_elt = NULL_TREE; > for (unsigned k = 0; k < const_nunits; ++k) > { > - tree boff = size_binop (MULT_EXPR, TYPE_SIZE (idx_type), > - bitsize_int (k + elt_offset)); > - tree idx = gimple_build (&stmts, BIT_FIELD_REF, idx_type, > - vec_offset, TYPE_SIZE (idx_type), > - boff); > - idx = gimple_convert (&stmts, sizetype, idx); > - idx = gimple_build (&stmts, MULT_EXPR, sizetype, idx, scale); > - tree ptr = gimple_build (&stmts, PLUS_EXPR, > - TREE_TYPE (dataref_ptr), > - dataref_ptr, idx); > - ptr = gimple_convert (&stmts, ptr_type_node, ptr); > - tree elt = make_ssa_name (TREE_TYPE (vectype)); > - tree ref = build2 (MEM_REF, ltype, ptr, > - build_int_cst (ref_type, 0)); > - new_stmt = gimple_build_assign (elt, ref); > - gimple_set_vuse (new_stmt, gimple_vuse (gsi_stmt (*gsi))); > - gimple_seq_add_stmt (&stmts, new_stmt); > + tree elt; > + /* If all lanes are uniform then generate broadcast otherwise > + emulated gathers. */ > + if (slp_node->gs_offset_uniform_p && k > 0) SLP_TREE_GS_OFFSET_UNIFORM_P > + elt = first_elt; use (*ctor_elts)[0].value instead of the new first_elt variable > + else > + { > + > + tree boff = size_binop (MULT_EXPR, TYPE_SIZE (idx_type), > + bitsize_int (k + elt_offset)); > + tree idx = gimple_build (&stmts, BIT_FIELD_REF, idx_type, > + vec_offset, TYPE_SIZE (idx_type), > + boff); > + idx = gimple_convert (&stmts, sizetype, idx); > + idx = gimple_build (&stmts, MULT_EXPR, sizetype, idx, > + scale); > + tree ptr = gimple_build (&stmts, PLUS_EXPR, > + TREE_TYPE (dataref_ptr), > + dataref_ptr, idx); > + ptr = gimple_convert (&stmts, ptr_type_node, ptr); > + elt = make_ssa_name (TREE_TYPE (vectype)); > + tree ref = build2 (MEM_REF, ltype, ptr, > + build_int_cst (ref_type, 0)); > + new_stmt = gimple_build_assign (elt, ref); > + gimple_set_vuse (new_stmt, gimple_vuse (gsi_stmt (*gsi))); > + gimple_seq_add_stmt (&stmts, new_stmt); > + first_elt = elt; > + } > CONSTRUCTOR_APPEND_ELT (ctor_elts, NULL_TREE, elt); > } > gsi_insert_seq_before (gsi, stmts, GSI_SAME_STMT); > -- Richard Biener <[email protected]> SUSE Software Solutions Germany GmbH, Frankenstrasse 146, 90461 Nuernberg, Germany; GF: Stefan Gaiser, Jochen Jaser, Abhinav Puri; (HRB 36809, AG Nuernberg)
