From: Reshma Roy <[email protected]>
Analyze and generate broadcast for uniform gather offsets
Implement analysis using SCEV information and formal proof to detect when
gather/scatter offset expressions evaluate to the same value for all lanes of
each vector of N lanes. For example, if N=8, iterations
0-7 of "i >> 6" all evaluate to 0, enabling optimization to broadcast.
Example: For "M[i >> 6]" with N=8:
- Iterations 0-7: i >> 6 = 0 (uniform)
- Iterations 64-71: i >> 6 = 1 (uniform)
Here we can optimize each vector iteration to load once + broadcast
Then transform uniform gather operations into optimized scalar load + broadcast
instead of emulated gather with N individual scalar loads.
BEFORE (Emulated Gather - 8 loads for N=8):
for each lane i in 0..7:
offset[i] = BIT_FIELD_REF<offset_vector, i>
addr[i] = base + offset[i] * scale
result[i] = MEM[addr[i]]
result_vector = {result[0], ..., result[7]}
AFTER (Broadcast - 1 load):
offset = BIT_FIELD_REF<offset_vector, 0>
addr = base + offset * scale
scalar_val = MEM[addr]
result_vector = {scalar_val, scalar_val,...}
gcc/ChangeLog:
* tree-vect-data-refs.cc (vect_check_gather_load_offset_uniform): New.
Return whether the gather offset is the same for all lanes of
each vector of N = CONST_NUNITS lanes in LOOP.
* tree-vect-stmts.cc (get_load_store_type): Initialize
gs_offset_uniform_p. Call vect_check_gather_load_offset_uniform
and set gs_offset_uniform_p.
(vectorizable_load): Generate broadcast for uniform gather offsets.
* tree-vectorizer.h (struct vect_load_store_data): New field gs_offset_
uniform_p for flagging uniformity.
gcc/testsuite/ChangeLog:
* g++.dg/vect/vect-uniform-broadcast-neg.cc: New test.
* g++.dg/vect/vect-uniform-broadcast-op2-var.cc: New test.
* g++.dg/vect/vect-uniform-broadcast.cc: New test.
* gcc.dg/vect/vect-gather-uniform-and.c: New test.
* gcc.dg/vect/vect-gather-uniform-epil-1.c: New test.
* gcc.dg/vect/vect-gather-uniform-mpeel-1.c: New test.
* gcc.dg/vect/vect-gather-uniform-neg-and.c: New test.
* gcc.dg/vect/vect-gather-uniform-peel-1.c: New test.
* gcc.dg/vect/vect-gather-uniform-peel-2.c: New test.
* gcc.dg/vect/vect-gather-uniform-peel-3.c: New test.
* gcc.dg/vect/vect-gather-uniform-run-1.c: New test.
* gcc.dg/vect/vect-gather-uniform-run-2.c: New test.
* gcc.dg/vect/vect-gather-uniform-run-3.c: New test.
* gcc.dg/vect/vect-gather-uniform-slp-1.c: New test.
* gcc.dg/vect/vect-gather-uniform-slp-2.c: New test.
---
Hi Richard,
This v6 patch updates most of the comments you have mentioned in the
last review except the following:
- Broadcast for IFN and builtin gathers (your comment about
-mtune-ctrl=use_gather_*). v6 still only changes the emulated gather path.
I have a version that handles all gather kinds, but it needs extra
restrictions for masked loops (a conditional load, or a vector copy that is
fully inactive in the last iteration of a masked loop, must not do the
unconditional scalar load). I would like to send it as a separate
follow-up patch.
- Computing the scalar offset from the scalar IV instead of extracting
lane 0 of the offset vector. As you suggested, this is left as a
follow-up optimization.
- Handle arbitrary step beyond {B,+,1} where B is the arbitrary base.
v6 changes:
- Fix wrong code with peeling for alignment and epilogue loops. The
uniformity check assumes each vector starts at an iteration that is a
multiple of the lane count, which peeling and epilogues break.
For example, with -mtune=atom the loop
for (int i = 0; i < n; i++)
out[i + 1] = table[i >> 2];
is peeled by 3, so the first vector covers i = 3..6 and i >> 2 is
{0,1,1,1}, but v5 broadcast lane 0.
- Make the decision in get_load_store_type, where the peeling and
epilogue state is known. Do not use the broadcast when
LOOP_VINFO_PEELING_FOR_ALIGNMENT is nonzero or LOOP_VINFO_EPILOGUE_P
is set. Masked peeling also implies LOOP_VINFO_PEELING_FOR_ALIGNMENT
is nonzero, so it is rejected too. It could not cause wrong code in
v5: an emulated gather cannot be masked, so a masked loop uses a real
gather, which the broadcast does not handle. The masked-peeling test
guards the planned follow-up for real gathers.
- Call vect_check_gather_load_offset_uniform only from there, for
single-lane gather loads (not scatters), using the SLP node's offset
child.
- Store the result in vect_load_store_data so costing and code
generation agree. Remove gather_scatter_info::offset_uniform (not
set on all paths) and the SLP node field.
- In the helper, move the dump below the trivial exits, use %wd, and
rewrite the comment in terms of the lane count N.
- Add runtime tests for all targets, the reviewer's testcase, and x86
tests for peeling, epilogue and masked peeling.
- Rename the C++ tests to .cc, move the C ones to gcc.dg/vect, remove
stdio.h, and fix typos and formatting.
- As you suggested, ran the patch through an LLM-assisted review and no issues
on x86_64, but on LoongArch (native vector division, no gathers) a
signed i / k could wrap to negative values under -fwrapv and was wrongly
treated as uniform, so TRUNC_DIV_EXPR now bails out when the dividend is signed
and its overflow wraps.
Bootstrapped and regression tested on x86_64-pc-linux-gnu
Thanks
Reshma
.../g++.dg/vect/vect-uniform-broadcast-neg.cc | 38 +++++
.../vect/vect-uniform-broadcast-op2-var.cc | 42 ++++++
.../g++.dg/vect/vect-uniform-broadcast.cc | 35 +++++
.../gcc.dg/vect/vect-gather-uniform-and.c | 31 +++++
.../gcc.dg/vect/vect-gather-uniform-epil-1.c | 46 ++++++
.../gcc.dg/vect/vect-gather-uniform-mpeel-1.c | 47 +++++++
.../gcc.dg/vect/vect-gather-uniform-neg-and.c | 36 +++++
.../gcc.dg/vect/vect-gather-uniform-peel-1.c | 35 +++++
.../gcc.dg/vect/vect-gather-uniform-peel-2.c | 42 ++++++
.../gcc.dg/vect/vect-gather-uniform-peel-3.c | 46 ++++++
.../gcc.dg/vect/vect-gather-uniform-run-1.c | 66 +++++++++
.../gcc.dg/vect/vect-gather-uniform-run-2.c | 37 +++++
.../gcc.dg/vect/vect-gather-uniform-run-3.c | 63 +++++++++
.../gcc.dg/vect/vect-gather-uniform-slp-1.c | 41 ++++++
.../gcc.dg/vect/vect-gather-uniform-slp-2.c | 39 ++++++
gcc/tree-vect-data-refs.cc | 131 ++++++++++++++++++
gcc/tree-vect-stmts.cc | 114 +++++++++++----
gcc/tree-vectorizer.h | 5 +
18 files changed, 865 insertions(+), 29 deletions(-)
create mode 100644 gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg.cc
create mode 100644 gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-op2-var.cc
create mode 100644 gcc/testsuite/g++.dg/vect/vect-uniform-broadcast.cc
create mode 100644 gcc/testsuite/gcc.dg/vect/vect-gather-uniform-and.c
create mode 100644 gcc/testsuite/gcc.dg/vect/vect-gather-uniform-epil-1.c
create mode 100644 gcc/testsuite/gcc.dg/vect/vect-gather-uniform-mpeel-1.c
create mode 100644 gcc/testsuite/gcc.dg/vect/vect-gather-uniform-neg-and.c
create mode 100644 gcc/testsuite/gcc.dg/vect/vect-gather-uniform-peel-1.c
create mode 100644 gcc/testsuite/gcc.dg/vect/vect-gather-uniform-peel-2.c
create mode 100644 gcc/testsuite/gcc.dg/vect/vect-gather-uniform-peel-3.c
create mode 100644 gcc/testsuite/gcc.dg/vect/vect-gather-uniform-run-1.c
create mode 100644 gcc/testsuite/gcc.dg/vect/vect-gather-uniform-run-2.c
create mode 100644 gcc/testsuite/gcc.dg/vect/vect-gather-uniform-run-3.c
create mode 100644 gcc/testsuite/gcc.dg/vect/vect-gather-uniform-slp-1.c
create mode 100644 gcc/testsuite/gcc.dg/vect/vect-gather-uniform-slp-2.c
diff --git a/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg.cc
b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg.cc
new file mode 100644
index 00000000000..81db1e2a2dd
--- /dev/null
+++ b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-neg.cc
@@ -0,0 +1,38 @@
+/* Negative test: the loop starts at i = 1, so the SCEV of i is {1, +, 1}.
+ A non-zero base B is only accepted when B is a multiple of the number of
+ vector lanes, and 1 is not. A vector chunk can then span two bitset
+ words (e.g. i = 61..64), so the word offset i / 64 is not uniform across
+ the lanes and no broadcast must be generated. The conditional store
+ needs a masked store, so enable AVX2. */
+/* { dg-do compile { target { x86_64-*-* i?86-*-* } } } */
+/* { dg-additional-options "-mavx2 -mprefer-vector-width=128" } */
+
+#include <bitset>
+#include <vector>
+
+class Foo
+{
+public:
+ void fun (unsigned boardsize, bool check);
+ std::vector<int> m_mcowner;
+};
+
+void
+Foo::fun (unsigned boardsize, bool check)
+{
+ std::bitset<21*21> blacksq;
+ __asm__ ("" : "+g" (blacksq));
+ for (int i = 0; i < boardsize; i++)
+ if (i % 2 == 0)
+ blacksq[i] = true;
+ unsigned int i = 1;
+ do
+ {
+ if (blacksq[i])
+ m_mcowner[i]++;
+ }
+ while (++i < blacksq.size ());
+}
+
+/* { dg-final { scan-tree-dump "loop vectorized" "vect" } } */
+/* { dg-final { scan-tree-dump-not "computing cost for broadcast" "vect" } } */
diff --git a/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-op2-var.cc
b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-op2-var.cc
new file mode 100644
index 00000000000..474b8fff217
--- /dev/null
+++ b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast-op2-var.cc
@@ -0,0 +1,42 @@
+/* Gather load with PLUS (hi, lo). The second operand of the offset PLUS is
+ a variable, hence the offset is not uniform and no broadcast is generated.
+ The inner loop still vectorizes as a normal gather. The conditional store
+ needs a masked store, so enable AVX2. */
+/* { dg-do compile { target { x86_64-*-* i?86-*-* } } } */
+/* { dg-additional-options "-mavx2 -mprefer-vector-width=128" } */
+
+#include <vector>
+
+class Foo
+{
+public:
+ void fun (unsigned boardsize);
+ std::vector<int> m_mcowner;
+};
+
+void
+Foo::fun (unsigned boardsize)
+{
+ unsigned nwords = (21 * 21 + 63) / 64 + 2;
+ std::vector<unsigned long> words (nwords, 0);
+ __asm__ ("" : "+g" (words));
+ for (unsigned i = 0; i < boardsize; i++)
+ if (i % 2 == 0)
+ words[i / 64] |= 1UL << (i % 64);
+ unsigned int i = 0;
+ do
+ {
+ unsigned hi = i / 64;
+ unsigned lo = i & 1u;
+ /* The second operand is loop-varying, so the offset is not a
+ shift, division or AND of the IV. */
+ if (words[hi + lo])
+ m_mcowner[i]++;
+ }
+ while (++i < 21 * 21);
+}
+
+/* { dg-final { scan-tree-dump "loop vectorized" "vect" } } */
+/* The emulated gather extracts each lane of the offset vector. */
+/* { dg-final { scan-tree-dump "BIT_FIELD_REF <vect_" "vect" } } */
+/* { dg-final { scan-tree-dump-not "computing cost for broadcast" "vect" } } */
diff --git a/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast.cc
b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast.cc
new file mode 100644
index 00000000000..a47001e05ec
--- /dev/null
+++ b/gcc/testsuite/g++.dg/vect/vect-uniform-broadcast.cc
@@ -0,0 +1,35 @@
+/* Test case for generating a scalar load and broadcast instead of a gather
+ when the gather offset is uniform across the vector lanes. The
+ conditional store needs a masked store, so enable AVX2. */
+/* { dg-do compile { target { x86_64-*-* i?86-*-* } } } */
+/* { dg-additional-options "-mavx2 -mprefer-vector-width=128" } */
+
+#include <bitset>
+#include <vector>
+
+class Foo
+{
+public:
+ void fun (unsigned boardsize, bool check);
+ std::vector<int> m_mcowner;
+};
+
+void
+Foo::fun (unsigned boardsize, bool check)
+{
+ std::bitset<21*21> blacksq;
+ __asm__ ("" : "+g" (blacksq));
+ for (int i = 0; i < boardsize; i++)
+ if (i % 2 == 0)
+ blacksq[i] = true;
+ unsigned int i = 0;
+ do
+ {
+ if (blacksq[i])
+ m_mcowner[i]++;
+ }
+ while (++i < blacksq.size ());
+}
+
+/* { dg-final { scan-tree-dump "loop vectorized" "vect" } } */
+/* { dg-final { scan-tree-dump "computing cost for broadcast" "vect" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-and.c
b/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-and.c
new file mode 100644
index 00000000000..bcd54373ed6
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-and.c
@@ -0,0 +1,31 @@
+/* Gather load table[i & 4]: mask 4 affects only bit 2, so (4 & (N - 1)) == 0
+ when N = 4. -mprefer-vector-width=128 encourages 4-wide int vectors on
+ x86 so i & 4 is uniform per vector chunk and broadcast is possible. */
+/* { dg-do compile { target { x86_64-*-* i?86-*-* } } } */
+/* { dg-additional-options "-mprefer-vector-width=128" } */
+
+#define N 32
+
+int block_values[8] = {10, 20, 30, 40, 50, 60, 70, 80};
+int result[N];
+
+void
+tile_fill (int *__restrict out, int *__restrict table, int n,
+ int *__restrict in)
+{
+ for (int i = 0; i < n; i++)
+ {
+ out[i] = table[i & 4];
+ in[i] = out[i] + 10;
+ }
+}
+
+void
+use_tile_fill (void)
+{
+ tile_fill (result, block_values, N, result);
+ __asm__ volatile ("" : : "r" (result) : "memory");
+}
+
+/* { dg-final { scan-tree-dump "loop vectorized" "vect" } } */
+/* { dg-final { scan-tree-dump "computing cost for broadcast" "vect" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-epil-1.c
b/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-epil-1.c
new file mode 100644
index 00000000000..42b92f4890e
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-epil-1.c
@@ -0,0 +1,46 @@
+/* Epilogue vectorization. The main loop uses 8 lanes and peels 7
+ iterations for alignment; i >> 2 is not uniform over 8 lanes, so the main
+ loop uses the full emulated gather. The 4-lane epilogue starts at
+ i = 7 + 8 * k, which is not a multiple of 4, so the broadcast must not be
+ used there either. The tune-ctrl options keep the gather emulated and
+ -fno-vect-cost-model keeps the main loop at 8 lanes. */
+/* { dg-do run { target { x86_64-*-* i?86-*-* } } } */
+/* { dg-require-effective-target avx2_runtime } */
+/* { dg-additional-options "-mavx2 -mtune=atom
-mtune-ctrl=^use_gather_2parts,^use_gather_4parts,^use_gather_8parts" } */
+
+#include "tree-vect.h"
+
+#define N 2048
+
+int table[N];
+int out[N + 64] __attribute__((aligned (64)));
+
+void __attribute__((noipa))
+f (int n)
+{
+ for (int i = 0; i < n; i++)
+ out[i + 1] = table[i >> 2];
+}
+
+int
+main (void)
+{
+ check_vect ();
+
+ volatile int n = 1013;
+ for (int i = 0; i < N; i++)
+ table[i] = i * 7 + 3;
+
+ f (n);
+
+#pragma GCC novector
+ for (int i = 0; i < n; i++)
+ if (out[i + 1] != table[i >> 2])
+ __builtin_abort ();
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump "epilogue loop vectorized using 16 byte
vectors" "vect" } } */
+/* { dg-final { scan-tree-dump "not using broadcast for uniform gather offset
because of peeling or epilogue" "vect" } } */
+/* { dg-final { scan-tree-dump-not "computing cost for broadcast" "vect" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-mpeel-1.c
b/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-mpeel-1.c
new file mode 100644
index 00000000000..88453e1d2fa
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-mpeel-1.c
@@ -0,0 +1,47 @@
+/* Peeling with a mask. With AVX-512 and partial vectors the vectorizer
+ aligns the store by starting the first vector iteration before i = 0 and
+ masking off the leading lanes, so lane 0 is inactive and every chunk is
+ shifted. Masked peeling implies LOOP_VINFO_PEELING_FOR_ALIGNMENT != 0,
+ so the broadcast is rejected. An emulated gather cannot be masked, so
+ this loop uses a real gather, which the broadcast does not handle yet.
+ The test guards against extending the broadcast to real gathers without
+ rejecting masked peeling. */
+/* { dg-do run { target { x86_64-*-* i?86-*-* } } } */
+/* { dg-require-effective-target avx512f_runtime } */
+/* { dg-additional-options "-mavx512f -mprefer-vector-width=512 -mtune=atom
--param vect-partial-vector-usage=2" } */
+
+#include "tree-vect.h"
+
+#define N 2048
+
+int table[N];
+int out[N + 64] __attribute__((aligned (64)));
+
+void __attribute__((noipa))
+f (int n)
+{
+ for (int i = 0; i < n; i++)
+ out[i + 1] = table[i >> 4];
+}
+
+int
+main (void)
+{
+ check_vect ();
+
+ volatile int n = 1013;
+ for (int i = 0; i < N; i++)
+ table[i] = i * 7 + 3;
+
+ f (n);
+
+#pragma GCC novector
+ for (int i = 0; i < n; i++)
+ if (out[i + 1] != table[i >> 4])
+ __builtin_abort ();
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump "misalignment for fully-masked loop" "vect" } }
*/
+/* { dg-final { scan-tree-dump-not "computing cost for broadcast" "vect" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-neg-and.c
b/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-neg-and.c
new file mode 100644
index 00000000000..bf5f330a2f9
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-neg-and.c
@@ -0,0 +1,36 @@
+/* Negative test: the offset must NOT be reported as uniform for ANY candidate
+ vector size the vectorizer tries.
+
+ NOTE: a right-shift offset (i >> k) cannot be used for a negative test here.
+ The vectorizer analyzes every candidate vector size (32, 16, 8, 4, 2), and
+ for a right shift the offset is uniform per chunk iff N - 1 < 2^k. At the
+ smallest candidate N = 2 this is 1 < 2^k, which holds for every k >= 1, so
+ *some* candidate always reports "uniform" and scan-tree-dump-not would fail.
+
+ For BIT_AND the offset is uniform iff (k & (N - 1)) == 0. With an odd mask
+ (k = 1) bit 0 is set, and N - 1 has bit 0 set for every N >= 2, so the
+ offset is non-uniform for ALL candidate vector sizes. The broadcast
+ optimization must never trigger. */
+/* { dg-do compile { target { x86_64-*-* i?86-*-* } } } */
+
+#define N 512
+
+int block_values[64];
+int result[N];
+
+void
+tile_fill (int *__restrict out, int *__restrict table, int n)
+{
+ for (int i = 0; i < n; i++)
+ out[i] = table[i & 1];
+}
+
+void
+use_tile_fill (void)
+{
+ tile_fill (result, block_values, N);
+ __asm__ volatile ("" : : "r" (result) : "memory");
+}
+
+/* { dg-final { scan-tree-dump "loop vectorized" "vect" } } */
+/* { dg-final { scan-tree-dump-not "computing cost for broadcast" "vect" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-peel-1.c
b/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-peel-1.c
new file mode 100644
index 00000000000..24a2da8f5fe
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-peel-1.c
@@ -0,0 +1,35 @@
+/* The store to out[i + 1] is misaligned, so the vectorizer peels for alignment
+ by a known amount (3 for 4 x int). The vector loop then starts at i = 3 and
+ the first chunk {3,4,5,6} gives i >> 2 = {0,1,1,1}, which is not uniform.
*/
+
+#include "tree-vect.h"
+
+#define N 1024
+
+int table[N];
+int out[N + 16];
+
+void __attribute__((noipa))
+f (int n)
+{
+ for (int i = 0; i < n; i++)
+ out[i + 1] = table[i >> 2];
+}
+
+int
+main (void)
+{
+ check_vect ();
+
+ for (int i = 0; i < N; i++)
+ table[i] = i;
+
+ f (N - 8);
+
+#pragma GCC novector
+ for (int i = 0; i < N - 8; i++)
+ if (out[i + 1] != table[i >> 2])
+ __builtin_abort ();
+
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-peel-2.c
b/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-peel-2.c
new file mode 100644
index 00000000000..7f777b40c41
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-peel-2.c
@@ -0,0 +1,42 @@
+/* Peeling for alignment by a known amount, forced with -mtune=atom. The
+ vector loop does not start at a multiple of the vector length, so the
+ broadcast must not be used. */
+/* { dg-do run { target { x86_64-*-* i?86-*-* } } } */
+/* { dg-additional-options "-mtune=atom -mprefer-vector-width=128" } */
+
+#include "tree-vect.h"
+
+#define N 2048
+
+int table[N];
+int out[N + 64] __attribute__((aligned (64)));
+
+void __attribute__((noipa))
+f (int n)
+{
+ for (int i = 0; i < n; i++)
+ out[i + 1] = table[i >> 2];
+}
+
+int
+main (void)
+{
+ check_vect ();
+
+ volatile int n = 1013;
+ for (int i = 0; i < N; i++)
+ table[i] = i * 7 + 3;
+
+ f (n);
+
+#pragma GCC novector
+ for (int i = 0; i < n; i++)
+ if (out[i + 1] != table[i >> 2])
+ __builtin_abort ();
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump "known peeling = 3" "vect" } } */
+/* { dg-final { scan-tree-dump "not using broadcast for uniform gather offset
because of peeling or epilogue" "vect" } } */
+/* { dg-final { scan-tree-dump-not "computing cost for broadcast" "vect" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-peel-3.c
b/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-peel-3.c
new file mode 100644
index 00000000000..ac192d6cdfb
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-peel-3.c
@@ -0,0 +1,46 @@
+/* Peeling for alignment by an amount known only at run time: the store
+ goes through a pointer whose alignment is unknown, so the peel count is
+ computed at run time. The vector loop's start is then unknown at compile
+ time, so the broadcast must not be used. Call with every misalignment. */
+/* { dg-do run { target { x86_64-*-* i?86-*-* } } } */
+/* { dg-additional-options "-mtune=atom -mprefer-vector-width=128" } */
+
+#include "tree-vect.h"
+
+#define N 1024
+
+int table[N];
+int buf[N + 64] __attribute__((aligned (64)));
+
+void __attribute__((noipa))
+f (int *restrict out, int n)
+{
+ for (int i = 0; i < n; i++)
+ out[i] = table[i >> 2];
+}
+
+int
+main (void)
+{
+ check_vect ();
+
+ for (int i = 0; i < N; i++)
+ table[i] = i * 7 + 3;
+
+ for (int off = 0; off < 4; off++)
+ {
+ int *out = buf + off;
+ f (out, N - 8);
+#pragma GCC novector
+ for (int i = 0; i < N - 8; i++)
+ if (out[i] != table[i >> 2])
+ __builtin_abort ();
+ }
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump "Peeling for alignment will be applied" "vect"
} } */
+/* { dg-final { scan-tree-dump-not "known peeling = " "vect" } } */
+/* { dg-final { scan-tree-dump "not using broadcast for uniform gather offset
because of peeling or epilogue" "vect" } } */
+/* { dg-final { scan-tree-dump-not "computing cost for broadcast" "vect" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-run-1.c
b/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-run-1.c
new file mode 100644
index 00000000000..ab91ab5a4a7
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-run-1.c
@@ -0,0 +1,66 @@
+/* Uniform gather offsets with no peeling: the vector loop starts at i = 0,
+ so every chunk starts at a multiple of the vector length and the
+ broadcast is valid. Checks the result for each supported operator. */
+
+#include "tree-vect.h"
+
+#define N 256
+
+int table[N];
+int out[N] __attribute__((aligned (64)));
+
+void __attribute__((noipa))
+f_shr (int n)
+{
+ for (int i = 0; i < n; i++)
+ out[i] = table[i >> 2];
+}
+
+void __attribute__((noipa))
+f_div (int n)
+{
+ for (int i = 0; i < n; i++)
+ out[i] = table[i / 8];
+}
+
+void __attribute__((noipa))
+f_and_not (int n)
+{
+ for (int i = 0; i < n; i++)
+ out[i] = table[i & ~7];
+}
+
+void __attribute__((noipa))
+f_and (int n)
+{
+ for (int i = 0; i < n; i++)
+ out[i] = table[i & 8];
+}
+
+#define CHECK(EXPR) \
+ _Pragma ("GCC novector") \
+ for (int i = 0; i < N; i++) \
+ if (out[i] != table[EXPR]) \
+ __builtin_abort ();
+
+int
+main (void)
+{
+ check_vect ();
+
+ for (int i = 0; i < N; i++)
+ table[i] = i * 7 + 3;
+
+ f_shr (N);
+ CHECK (i >> 2)
+ f_div (N);
+ CHECK (i / 8)
+ f_and_not (N);
+ CHECK (i & ~7)
+ f_and (N);
+ CHECK (i & 8)
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump "computing cost for broadcast" "vect" { target
{ x86_64-*-* i?86-*-* } } } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-run-2.c
b/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-run-2.c
new file mode 100644
index 00000000000..fb449a97689
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-run-2.c
@@ -0,0 +1,37 @@
+/* Uniform gather offset with a non-zero start that is a multiple of the
+ vector length (i starts at 8). Every chunk still starts at a multiple
+ of the vector length, so the broadcast is valid. */
+
+#include "tree-vect.h"
+
+#define N 256
+
+int table[N];
+int out[N] __attribute__((aligned (64)));
+
+void __attribute__((noipa))
+f (int n)
+{
+ for (int i = 8; i < n; i++)
+ out[i] = table[i >> 2];
+}
+
+int
+main (void)
+{
+ check_vect ();
+
+ for (int i = 0; i < N; i++)
+ table[i] = i * 7 + 3;
+
+ f (N);
+
+#pragma GCC novector
+ for (int i = 8; i < N; i++)
+ if (out[i] != table[i >> 2])
+ __builtin_abort ();
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump "computing cost for broadcast" "vect" { target
{ x86_64-*-* i?86-*-* } } } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-run-3.c
b/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-run-3.c
new file mode 100644
index 00000000000..78f72819481
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-run-3.c
@@ -0,0 +1,63 @@
+/* Gather offsets that are not uniform per vector chunk: the start is not a
+ multiple of the vector length (i starts at 1), the AND mask has low bits
+ set, and the divisor is not a power of two. The broadcast must not be
+ used, and the result must be correct. */
+
+#include "tree-vect.h"
+
+#define N 256
+
+int table[N];
+int out[N] __attribute__((aligned (64)));
+
+void __attribute__((noipa))
+f_base1 (int n)
+{
+ for (int i = 1; i < n; i++)
+ out[i] = table[i >> 2];
+}
+
+void __attribute__((noipa))
+f_and_odd (int n)
+{
+ for (int i = 0; i < n; i++)
+ out[i] = table[i & 5];
+}
+
+void __attribute__((noipa))
+f_div5 (int n)
+{
+ for (int i = 0; i < n; i++)
+ out[i] = table[i / 5];
+}
+
+int
+main (void)
+{
+ check_vect ();
+
+ for (int i = 0; i < N; i++)
+ table[i] = i * 7 + 3;
+
+ f_base1 (N);
+#pragma GCC novector
+ for (int i = 1; i < N; i++)
+ if (out[i] != table[i >> 2])
+ __builtin_abort ();
+
+ f_and_odd (N);
+#pragma GCC novector
+ for (int i = 0; i < N; i++)
+ if (out[i] != table[i & 5])
+ __builtin_abort ();
+
+ f_div5 (N);
+#pragma GCC novector
+ for (int i = 0; i < N; i++)
+ if (out[i] != table[i / 5])
+ __builtin_abort ();
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump-not "computing cost for broadcast" "vect" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-slp-1.c
b/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-slp-1.c
new file mode 100644
index 00000000000..4f8b1bc716a
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-slp-1.c
@@ -0,0 +1,41 @@
+#include "tree-vect.h"
+
+#define N 64
+
+int table[16] = { 1, 2, 3, 4, 5, 6, 7, 8,
+ 9, 10, 11, 12, 13, 14, 15, 16 };
+int y[N * 2];
+
+/* Two gathers from the same base that form one SLP group. i & 4 and
+ i & 8 are each uniform per 4-lane chunk, but the group's vector
+ interleaves the offsets of both statements, so it must not be turned
+ into a broadcast. */
+
+void __attribute__((noipa))
+f (int *restrict y, int *restrict t, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ y[i * 2] = t[i & 4] + 1;
+ y[i * 2 + 1] = t[i & 8] + 2;
+ }
+}
+
+int
+main (void)
+{
+ check_vect ();
+
+ f (y, table, N);
+#pragma GCC novector
+ for (int i = 0; i < N; ++i)
+ if (y[i * 2] != table[i & 4] + 1
+ || y[i * 2 + 1] != table[i & 8] + 2)
+ __builtin_abort ();
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump "Loop contains only SLP stmts" "vect" } } */
+/* { dg-final { scan-tree-dump "loop vectorized" "vect" } } */
+/* { dg-final { scan-tree-dump-not "computing cost for broadcast" "vect" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-slp-2.c
b/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-slp-2.c
new file mode 100644
index 00000000000..ed7b6b4a3d5
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-gather-uniform-slp-2.c
@@ -0,0 +1,39 @@
+#include "tree-vect.h"
+
+#define N 64
+
+int table[16] = { 1, 2, 3, 4, 5, 6, 7, 8,
+ 9, 10, 11, 12, 13, 14, 15, 16 };
+int y[N];
+
+/* Two gathers with offsets i & 4 and (i + 1) & 4, i stepping by two.
+ They are not uniform (the step is two) and are not grouped; this
+ checks that the result is correct. */
+
+void __attribute__((noipa))
+f (int *restrict y, int *restrict t, int n)
+{
+ for (int i = 0; i < n; i += 2)
+ {
+ y[i] = t[i & 4];
+ y[i + 1] = t[(i + 1) & 4];
+ }
+}
+
+int
+main (void)
+{
+ check_vect ();
+
+ f (y, table, N);
+#pragma GCC novector
+ for (int i = 0; i < N; ++i)
+ if (y[i] != table[i & 4])
+ __builtin_abort ();
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump "Loop contains only SLP stmts" "vect" } } */
+/* { dg-final { scan-tree-dump "loop vectorized" "vect" } } */
+/* { dg-final { scan-tree-dump-not "computing cost for broadcast" "vect" } } */
diff --git a/gcc/tree-vect-data-refs.cc b/gcc/tree-vect-data-refs.cc
index 22fce2c927b..027d7f171f3 100644
--- a/gcc/tree-vect-data-refs.cc
+++ b/gcc/tree-vect-data-refs.cc
@@ -4704,6 +4704,137 @@ vect_describe_gather_scatter_call (stmt_vec_info
stmt_info,
info->memory_type = TREE_TYPE (DR_REF (dr));
}
+/* Return true if the gather load offset OFF is the same for all lanes of
+ each vector of N = CONST_NUNITS lanes in LOOP. This assumes every vector
+ starts at a scalar iteration S with S % N == 0; the caller must check that
+ against peeling and epilogue loops. N rather than the vectorization
+ factor is the chunk size: vector copy J covers the scalar iterations
+ S + J * N to S + J * N + N - 1, and the vectorization factor is a multiple
+ of N. If the offset is uniform, the gather can use a scalar load plus
+ broadcast instead of a full emulated gather.
+
+ For example, with i = {0, +, 1}, N = 4 and a load of a[i >> 2]:
+
+ chunk 0: i = {0,1,2,3} -> i >> 2 = {0,0,0,0}
+ chunk 1: i = {4,5,6,7} -> i >> 2 = {1,1,1,1}
+
+ OFF must be i OP k with k an INTEGER_CST and the SCEV of i must be
+ {B, +, 1} with B >= 0 and B % N == 0, so that each chunk is
+ m * N to m * N + N - 1. N is a power of two, N = 2^p. Then:
+ * i >> k is uniform iff N - 1 < 2^k.
+ * i / k is uniform iff k % N == 0. N - 1 < k is not enough for a
+ divisor that is not a power of two (N = 4, k = 5: chunk {4,5,6,7}
+ gives {0,1,1,1}). A signed i must not wrap: with -fwrapv it can
+ become negative, and division truncates toward zero, so
+ {-8,-7,-6,-5} / 4 gives {-2,-1,-1,-1}.
+ * i & k is uniform iff (k & (N - 1)) == 0. The low p bits of i vary
+ across the lanes of a chunk and the higher bits are fixed (N = 8,
+ chunk 5: i = 40 to 47, binary 101000 to 101111), so i & k is the same
+ on every lane iff k has none of the low p bits set. */
+
+bool
+vect_check_gather_load_offset_uniform (class loop *loop,
+ HOST_WIDE_INT const_nunits,
+ tree off)
+{
+ /* Check ensures that the offset is an SSA_NAME which is qualified for
+ uniform detection. */
+ if (!off || TREE_CODE (off) != SSA_NAME)
+ return false;
+ tree chrec1 = NULL_TREE, rhs1 = NULL_TREE, rhs2 = NULL_TREE;
+ gimple *def_stmt = SSA_NAME_DEF_STMT (off);
+ if (!is_gimple_assign (def_stmt))
+ return false;
+ tree_code op_code = ERROR_MARK;
+ op_code = gimple_assign_rhs_code (def_stmt);
+ if (op_code != BIT_AND_EXPR && op_code != TRUNC_DIV_EXPR
+ && op_code != RSHIFT_EXPR)
+ return false;
+ rhs1 = gimple_assign_rhs1 (def_stmt);
+ rhs2 = gimple_assign_rhs2 (def_stmt);
+ /* Strict check for operand 2 to be constant. */
+ if (TREE_CODE (rhs2) != INTEGER_CST)
+ return false;
+ /* OFF can be defined by a pattern stmt, whose operands need not be in
+ the IL. */
+ if (TREE_CODE (rhs1) != SSA_NAME
+ || SSA_NAME_IS_DEFAULT_DEF (rhs1)
+ || !gimple_bb (SSA_NAME_DEF_STMT (rhs1)))
+ return false;
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, vect_location,
+ "offset uniformity check at vector nunits=%wd\n",
+ const_nunits);
+ chrec1 = analyze_scalar_evolution
+ (gimple_bb (SSA_NAME_DEF_STMT (rhs1))->loop_father, rhs1);
+ chrec1 = instantiate_parameters (loop, chrec1);
+ if (!chrec1 || chrec1 == chrec_dont_know)
+ return false;
+ /* Strict check for operand 1 to be poly rec and
+ operand 2 to be constant. */
+ if (TREE_CODE (chrec1) != POLYNOMIAL_CHREC
+ || CHREC_VARIABLE (chrec1) != (unsigned) loop->num)
+ return false;
+ tree scev_base = CHREC_LEFT (chrec1);
+ tree scev_step = CHREC_RIGHT (chrec1);
+ /* Valid for {B, +, 1} induction pattern now.
+ TODO Extend to handle general case. */
+ if (TREE_CODE (scev_base) != INTEGER_CST
+ || TREE_CODE (scev_step) != INTEGER_CST
+ || !integer_onep (scev_step)
+ || tree_int_cst_sgn (scev_base) < 0)
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, vect_location,
+ "cannot determine uniformity"
+ " because of non-constant or negative base or non-one"
+ " step\n");
+ return false;
+ }
+ /* Convert SCEV components and operands to wide_int for uniformity analysis.
+ These values are used to determine if the gather load offset is uniform
+ across vector lanes based on the operation type (RSHIFT, TRUNC_DIV, or
+ BIT_AND) and the relationship between the vector number of lanes and the
+ operand. */
+ wide_int rhs_wi = wi::to_wide (rhs2);
+ unsigned prec = TYPE_PRECISION (TREE_TYPE (rhs2));
+ unsigned int base_prec = TYPE_PRECISION (TREE_TYPE (scev_base));
+ wide_int base_wi = wi::to_wide (scev_base);
+ wide_int vf_m1 = wi::uhwi (const_nunits - 1, prec);
+ wide_int vf_base = wi::uhwi (const_nunits, base_prec);
+ /* {B,+,1} has the {0,+,1} chunk shape iff B is a multiple of N.
+ All three operators need this, so reject once. */
+ if (!wi::eq_p (wi::umod_trunc (base_wi, vf_base), wi::zero (base_prec)))
+ return false;
+ bool is_uniform = false;
+ if (op_code == RSHIFT_EXPR)
+ {
+ wide_int pow2_k = wi::lshift (wi::one (prec), rhs_wi);
+ if (wi::ltu_p (vf_m1, pow2_k))
+ is_uniform = true;
+ }
+ else if (op_code == TRUNC_DIV_EXPR
+ && !(!TYPE_UNSIGNED (TREE_TYPE (rhs1))
+ && TYPE_OVERFLOW_WRAPS (TREE_TYPE (rhs1))))
+ {
+ wide_int vf_wi = wi::uhwi (const_nunits, prec);
+ if (wi::eq_p (wi::umod_trunc (rhs_wi, vf_wi), wi::zero (prec)))
+ is_uniform = true;
+ }
+ else if (op_code == BIT_AND_EXPR)
+ {
+ wide_int mask_m1 = wi::bit_and (rhs_wi, vf_m1);
+ if (wi::eq_p (mask_m1, wi::zero (prec)))
+ is_uniform = true;
+ }
+ if (is_uniform && dump_enabled_p ())
+ dump_printf_loc (MSG_OPTIMIZED_LOCATIONS, vect_location,
+ "gather offset is uniform across %wd vector lanes, "
+ "broadcast optimization possible\n",
+ const_nunits);
+ return is_uniform;
+}
+
/* Return true if a non-affine read or write in STMT_INFO is suitable for a
gather load or scatter store with VECTYPE. Describe the operation in *INFO
if so. If it is suitable and ELSVALS is nonzero store the supported else
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index 6e7e948ff1e..f6048d69e31 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -2108,6 +2108,7 @@ get_load_store_type (vec_info *vinfo, stmt_vec_info
stmt_info,
*n_perms = -1U;
*n_loads = -1U;
ls->subchain_p = false;
+ ls->gs_offset_uniform_p = false;
bool perm_ok = true;
poly_int64 vf = loop_vinfo ? LOOP_VINFO_VECT_FACTOR (loop_vinfo) : 1;
@@ -2220,6 +2221,30 @@ get_load_store_type (vec_info *vinfo, stmt_vec_info
stmt_info,
return false;
}
*memory_access_type = VMAT_GATHER_SCATTER_EMULATED;
+
+ /* A single-lane gather load whose offset is the same for all lanes
+ of each vector can use a scalar load plus broadcast. The
+ uniformity check assumes each vector starts at a scalar iteration
+ that is a multiple of the number of lanes. Peeling for alignment
+ (with or without masks) and epilogue loops shift the start, so
+ reject those. */
+ if (vls_type == VLS_LOAD
+ && SLP_TREE_LANES (slp_node) == 1
+ && vect_check_gather_load_offset_uniform
+ (loop, nunits.to_constant (),
+ vect_get_slp_scalar_def (offset_node, 0)))
+ {
+ if (LOOP_VINFO_PEELING_FOR_ALIGNMENT (loop_vinfo) != 0
+ || LOOP_VINFO_EPILOGUE_P (loop_vinfo))
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
+ "not using broadcast for uniform gather "
+ "offset because of peeling or epilogue\n");
+ }
+ else
+ ls->gs_offset_uniform_p = true;
+ }
}
}
else
@@ -11279,18 +11304,39 @@ vectorizable_load (vec_info *vinfo,
unsigned HOST_WIDE_INT const_nunits = nunits.to_constant ();
if (costing_p)
{
- /* For emulated gathers N offset vector element
- offset add is consumed by the load). */
- inside_cost = record_stmt_cost (cost_vec, 1, vec_deconstruct,
- slp_node, 0, vect_body);
- /* N scalar loads plus gathering them into a
- vector. */
- inside_cost
- = record_stmt_cost (cost_vec, const_nunits, scalar_load,
- slp_node, 0, vect_body);
- inside_cost
- = record_stmt_cost (cost_vec, 1, vec_construct,
- slp_node, 0, vect_body);
+ if (ls.gs_offset_uniform_p)
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, vect_location,
+ "computing cost for broadcast\n");
+ /* Broadcast optimization: 1 extract + 1 load
+ + 1 broadcast. */
+ inside_cost = record_stmt_cost (cost_vec, 1,
+ vec_to_scalar, slp_node,
+ 0, vect_body);
+ inside_cost += record_stmt_cost (cost_vec, 1, scalar_load,
+ slp_node, 0, vect_body);
+ inside_cost += record_stmt_cost (cost_vec, 1,
+ scalar_to_vec, slp_node,
+ 0, vect_body);
+ }
+ else
+ {
+
+ /* For emulated gathers N offset vector element
+ offset add is consumed by the load). */
+ inside_cost = record_stmt_cost (cost_vec, 1,
+ vec_deconstruct, slp_node,
+ 0, vect_body);
+ /* N scalar loads plus gathering them into a
+ vector. */
+ inside_cost
+ = record_stmt_cost (cost_vec, const_nunits, scalar_load,
+ slp_node, 0, vect_body);
+ inside_cost
+ = record_stmt_cost (cost_vec, 1, vec_construct,
+ slp_node, 0, vect_body);
+ }
continue;
}
tree offset_vectype = TREE_TYPE (vec_offsets[0]);
@@ -11310,23 +11356,33 @@ vectorizable_load (vec_info *vinfo,
tree ltype = build_aligned_type (TREE_TYPE (vectype), align);
for (unsigned k = 0; k < const_nunits; ++k)
{
- tree boff = size_binop (MULT_EXPR, TYPE_SIZE (idx_type),
- bitsize_int (k + elt_offset));
- tree idx = gimple_build (&stmts, BIT_FIELD_REF, idx_type,
- vec_offset, TYPE_SIZE (idx_type),
- boff);
- idx = gimple_convert (&stmts, sizetype, idx);
- idx = gimple_build (&stmts, MULT_EXPR, sizetype, idx, scale);
- tree ptr = gimple_build (&stmts, PLUS_EXPR,
- TREE_TYPE (dataref_ptr),
- dataref_ptr, idx);
- ptr = gimple_convert (&stmts, ptr_type_node, ptr);
- tree elt = make_ssa_name (TREE_TYPE (vectype));
- tree ref = build2 (MEM_REF, ltype, ptr,
- build_int_cst (ref_type, 0));
- new_stmt = gimple_build_assign (elt, ref);
- gimple_set_vuse (new_stmt, gimple_vuse (gsi_stmt (*gsi)));
- gimple_seq_add_stmt (&stmts, new_stmt);
+ tree elt;
+ /* If all lanes are uniform then generate broadcast otherwise
+ emulated gathers. */
+ if (ls.gs_offset_uniform_p && k > 0)
+ elt = (*ctor_elts)[0].value;
+ else
+ {
+
+ tree boff = size_binop (MULT_EXPR, TYPE_SIZE (idx_type),
+ bitsize_int (k + elt_offset));
+ tree idx = gimple_build (&stmts, BIT_FIELD_REF, idx_type,
+ vec_offset, TYPE_SIZE (idx_type),
+ boff);
+ idx = gimple_convert (&stmts, sizetype, idx);
+ idx = gimple_build (&stmts, MULT_EXPR, sizetype, idx,
+ scale);
+ tree ptr = gimple_build (&stmts, PLUS_EXPR,
+ TREE_TYPE (dataref_ptr),
+ dataref_ptr, idx);
+ ptr = gimple_convert (&stmts, ptr_type_node, ptr);
+ elt = make_ssa_name (TREE_TYPE (vectype));
+ tree ref = build2 (MEM_REF, ltype, ptr,
+ build_int_cst (ref_type, 0));
+ new_stmt = gimple_build_assign (elt, ref);
+ gimple_set_vuse (new_stmt, gimple_vuse (gsi_stmt (*gsi)));
+ gimple_seq_add_stmt (&stmts, new_stmt);
+ }
CONSTRUCTOR_APPEND_ELT (ctor_elts, NULL_TREE, elt);
}
gsi_insert_seq_before (gsi, stmts, GSI_SAME_STMT);
diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
index 750b46a97b9..fa486be793a 100644
--- a/gcc/tree-vectorizer.h
+++ b/gcc/tree-vectorizer.h
@@ -314,6 +314,9 @@ struct vect_load_store_data : vect_data {
unsigned n_loads; // SLP_TREE_LOAD_PERMUTATION
/* Whether the load permutation is consecutive and simple. */
bool subchain_p; // VMAT_STRIDED_SLP and VMAT_GATHER_SCATTER
+ /* Whether the gather offset is the same for all lanes of each vector,
+ so a scalar load plus broadcast can replace the gather. */
+ bool gs_offset_uniform_p; // VMAT_GATHER_SCATTER_EMULATED
};
struct match_elt_t {
@@ -2644,6 +2647,8 @@ extern bool vect_check_gather_scatter (stmt_vec_info,
tree,
vec<int> * = nullptr);
extern void vect_describe_gather_scatter_call (stmt_vec_info,
gather_scatter_info *);
+extern bool vect_check_gather_load_offset_uniform (class loop *,
+ HOST_WIDE_INT, tree);
extern opt_result vect_find_stmt_data_reference (loop_p, gimple *,
vec<data_reference_p> *,
vec<int> *, int);
--
2.34.1
