> -----邮件原件-----
> 发件人: Richard Biener <[email protected]>
> 发送时间: 2026年9月30日 20:31
> 收件人: Xiaomei Xue <[email protected]>
> 抄送: [email protected]; [email protected]; Kewen Lin
> <[email protected]>; Qingkuan Lai <[email protected]>; Feng Xue
> <[email protected]>
> 主题: Re: [PATCH 1/2] feat: Add -fnon-temporal-store option to enhance
> nontemporal store optimization
>
> On Mon, Sep 28, 2026 at 12:52 PM Xiaomei Xue <[email protected]>
> wrote:
> >
> >
> >
> > > -----邮件原件-----
> > > 发件人: Richard Biener <[email protected]>
> > > 发送时间: 2026年9月24日 18:51
> > > 收件人: Xiaomei Xue <[email protected]>
> > > 抄送: [email protected]; [email protected]; Kewen Lin
> > > <[email protected]>; Qingkuan Lai <[email protected]>; Feng Xue
> > > <[email protected]>
> > > 主题: Re: [PATCH 1/2] feat: Add -fnon-temporal-store option to enhance
> > > nontemporal store optimization
> > >
> > > On Thu, Sep 24, 2026 at 10:28 AM Xiaomei Xue <[email protected]>
> > > wrote:
> > > >
> > > >
> > > >
> > > > > -----邮件原件-----
> > > > > 发件人: Xiaomei Xue
> > > > > 发送时间: 2026年7月2日 20:24
> > > > > 收件人: 'Richard Biener' <[email protected]>
> > > > > 抄送: '[email protected]' <[email protected]>;
> > > > > '[email protected]' <[email protected]>; Kewen Lin
> > > > > <[email protected]>; Qingkuan Lai <[email protected]>; Feng
> > > > > Xue <[email protected]>
> > > > > 主题: 答复: [PATCH 1/2] feat: Add -fnon-temporal-store option to
> > > enhance
> > > > > nontemporal store optimization
> > > > >
> > > > >
> > > > >
> > > > > > -----邮件原件-----
> > > > > > 发件人: Richard Biener <[email protected]>
> > > > > > 发送时间: 2026年7月1日 17:55
> > > > > > 收件人: Xiaomei Xue <[email protected]>
> > > > > > 抄送: [email protected]; [email protected]; Kewen Lin
> > > > > > <[email protected]>; Qingkuan Lai <[email protected]>
> > > > > > 主题: Re: [PATCH 1/2] feat: Add -fnon-temporal-store option to
> > > > > > enhance nontemporal store optimization
> > > > > >
> > > > > > On Wed, Jun 17, 2026 at 9:15 AM Xiaomei Xue
> > > > > > <[email protected]>
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > Thanks for the previous suggestions. I have reworked the
> > > > > > nontemporal-store logic. Instead of cloning the aprefetch
> > > > > > pass, I am planning to split out the NT-store analysis and run
> > > > > > it from vect_enhance_data_refs_alignment. This lets the
> > > > > > vectorizer identify
> > > > > > > vector NT-store candidates before deciding
> > > > > > > peeling/versioning, so the
> > > > > > result can guide data-reference alignment handling.
> > > > > > >
> > > > > > > In this design, the analysis only records candidates in
> > > > > > dr_vec_info::nontemporal_store_candidate. It does not mark the
> > > > > > GIMPLE store as nontemporal during analysis.
> > > > > > >
> > > > > > > I still have two questions.
> > > > > > >
> > > > > > > The first one is about the source of data references used
> > > > > > > by
> > > > > > determine_loop_nest_reuse.
> > > > > > >
> > > > > > > The NT-store analysis needs DR and dependence information
> > > > > > > for the
> > > > > loop.
> > > > > > I considered two possible approaches:
> > > > > > >
> > > > > > > 1. reuse the DRs/DDRs already computed by the vectorizer;
> > > > > > > 2. recompute DRs/DDRs inside determine_loop_nest_reuse.
> > > > > > >
> > > > > > > Reusing the vectorizer’s existing analysis avoids
> > > > > > > duplicate data-reference
> > > > > > analysis and keeps the NT-store analysis consistent with the
> > > > > > memory references and dependences currently seen by the
> > > > > > vectorizer. The downside is that it requires extending some
> > > > > > > interfaces, passing the vectorizer DRs/DDRs and a
> > > > > > > DR-to-mem_ref
> > > > > > mapping into the reuse-distance analysis.
> > > > > > >
> > > > > > > Recomputing DRs/DDRs inside determine_loop_nest_reuse
> > > > > > > would keep
> > > > > > the interface changes smaller and make the reuse analysis more
> > > > > self-contained.
> > > > > > However, it duplicates analysis work, and the recomputed
> > > > > > DRs/DDRs might differ subtly from the ones already
> > > > > > > used by the vectorizer.
> > > > > > >
> > > > > > > My current preference is to reuse the vectorizer DRs/DDRs,
> > > > > > > and handle
> > > > > > any unmapped DR or incomplete information conservatively. Does
> > > > > > that sound like the right direction, or would recomputing
> > > > > > DRs/DDRs inside the reuse-distance analysis be more acceptable
> > > > > > > in order to keep the interface impact smaller?
> > > > > >
> > > > > > Re-using the DDRs is better. The "plumbing" between the data
> > > > > > structures is a bit ugly, but well.
> > > > > >
> > > > > > I see you do
> > > > > >
> > > > > > @@ -6793,6 +6920,8 @@ vect_supportable_dr_alignment (vec_info
> > > > > > *vinfo, dr_vec_info *dr_info,
> > > > > > return dr_aligned;
> > > > > > else if (dr_safe_speculative_read_required (stmt_info))
> > > > > > return dr_unaligned_unsupported;
> > > > > > + else if (dr_info->nontemporal_store_candidate)
> > > > > > + return dr_unaligned_unsupported;
> > > > > >
> > > > > > that effectively means that we disable vectorization if we
> > > > > > cannot align all marked DRs (or resort to versioning for mutual
> alignment).
> > > > > > Is that a good trade-off? I suspect having
> > > > > > -fnon-temporal-store as global flag results in very "undirected"
> > > > > > applying of NT stores, is that what people would actually do?
> > > > > > We should abstract from looking at single bechmarks that
> > > > > > improve by adding an additional flag here - how would we
> > > > > > ensure that enabling the flag by default does not regress
> > > > > > performance? An alternative might be a pragma/attribute in
> > > > > > the source on a loop or even a specific memory reference.
> > > > > >
> > > > >
> > > > > Yes, this needs to be fixed. If we cannot guarantee the
> > > > > alignment required by NT stores, we should not give up normal
> vectorization.
> > > > > The NT-store choice should only be made when the target can
> > > > > support it for the selected alignment; otherwise we should fall
> > > > > back to normal
> > > vector stores.
> > > > > This optimization is mainly intended for streaming loops whose
> > > > > performance is dominated by memory bandwidth, such as
> > > > > STREAM-like kernels(
> > > > > https://www.cs.virginia.edu/stream/FTP/Code/stream.c). In that
> > > > > benchmark setting the source cannot be changed, so adding a
> > > > > pragma or attribute is not an option; the only practical way to
> > > > > enable the optimization is through a compiler option. I agree
> > > > > that -fnon-temporal-store is a coarse global knob, so it should
> > > > > remain disabled by default. Users who enable it explicitly need
> > > > > to accept the
> > > possible cost from forced alignment, versioning, or peeling.
> > > > >
> > > > > > Do you have statistics on how many loops are
> > > > > > peeled/versioned[/not vectorized] with this flag on say SPEC CPU?
> > > > > >
> > > > >
> > > > > I have done an initial check on current trunk with -Ofast
> > > -fnon-temporal-store.
> > > > > So far, dr_info->nontemporal_store_candidate is set only in SPEC
> > > > > CPU 519.lbm_r and 538.imagick_r. In both cases the candidates
> > > > > are rejected by the current alignment check, so no NT stores are
> > > > > emitted. I will collect more detailed numbers for peeled /
> > > > > versioned / not-vectorized loops before the next version.
> > > > >
> > > > > > I'm not familiar with the aprefetch pass - does the analysis
> > > > > > you re-use really require a different data structure or could
> > > > > > some refactoring make it work with pure data_references / ddrs as
> well?
> > > > > >
> > > > >
> > > > > The analysis is still tied to the aprefetch pass because the
> > > > > reuse-distance computation relies on ref->prefetch_mod from the
> > > aprefetch data structures.
> > > > > Making it operate directly on data_references / DDRs would
> > > > > require a fairly large refactoring of that analysis. For this
> > > > > patch, I kept the change conservative and reused the existing
> > > > > structure as much as
> > > possible.
> > > > >
> > > > > > > The second question is about supporting NT stores under
> -fopenmp.
> > > > > > >
> > > > > > > For example:
> > > > > > >
> > > > > > > #pragma omp parallel for
> > > > > > > for (j = 0; j < loop_num; j++)
> > > > > > > c[j] = a[j] + b[j];
> > > > > > >
> > > > > > > From the memory-access pattern, such a loop may also be a
> > > > > > > candidate for
> > > > > > NT stores. However, after OpenMP lowering, the original loop
> > > > > > is outlined into an OpenMP worker function. At the later
> > > > > > NT-store analysis point, we see the outlined function and its
> > > > > > > local loop, and it is no longer straightforward to recover
> > > > > > > the original global
> > > > > > iteration count of the parallel for.
> > > > > > >
> > > > > > > I previously tried to infer the original iteration count
> > > > > > > from the OpenMP
> > > > > > work-sharing/runtime partitioning pattern, but that only
> > > > > > covers limited cases and depends quite heavily on the exact
> > > > > > lowered GIMPLE
> > > form.
> > > > > > >
> > > > > > > Another possibility would be to record, during OpenMP
> > > > > > > lowering, the
> > > > > > relationship between the outlined function and the
> > > > > > corresponding OMP loop iteration count. But that also has some
> issues:
> > > > > > >
> > > > > > > 1. a single cfun may contain multiple OMP loops, so
> > > > > > > storing a single trip
> > > > > > count in struct function is not accurate;
> > > > > > > 2. if the information is stored in struct loop, the loop
> > > > > > > may be transformed
> > > > > > between OpenMP lowering and the later NT-store analysis,
> > > > > > making the saved information stale;
> > > > > > > 3. maintaining a mapping from outlined functions back to
> > > > > > > original OMP
> > > > > > loop information would also need to account for later
> > > > > > optimizations that change the function body and loop structure.
> > > > > > >
> > > > > > > So I would like to ask whether there is a recommended way
> > > > > > > to obtain or
> > > > > > preserve the original OpenMP loop iteration count for such
> > > > > > -fopenmp cases. Or, at the NT-store analysis stage, should we
> > > > > > for now handle this conservatively based only on the loop
> > > > > > > currently present in the outlined function: if we cannot
> > > > > > > determine a
> > > > > > concrete loop iteration count, we simply do not trigger the
> > > > > > NT-store optimization?
> > > > > >
> > > > > > It's a good question, it might hint at that marking the actual
> > > > > > memory references as candidates is more feasible in this case.
> > > > > >
> > > >
> > > > Not sure I understand it well, do you mean marking the memory
> > > > references
> > > as NT-store candidates earlier, before OpenMP lowering? Or do you
> > > think this kind of case should be supported?
> > >
> > > Yes, before or during OMP lowering.
> > >
> > > As I read this whole work is to make the stream benchmark numbers
> > > look better and there is no anticipated other practical use. Is that
> > > correct?
> > >
> > > Richard.
> > >
> >
> >
> > Currently, STREAM shows the largest benefit in our tests. This work aims to
> extend the applicability of the existing NT-store optimization. Although GCC
> already supports this optimization, opportunities can be missed in some
> vectorized loops because the alignment requirements are not met. This change
> makes the vectorizer consider NT-store requirements when making alignment
> decisions, potentially benefiting streaming loops in users’ own applications
> where NT stores are appropriate.
>
> Sorry for asking stupid questions - but isn't this optimizing prefetching and
> forgoing optimizing vectorization? Currently prefetching runs after (optimal)
> vectorization. How does that cause "alignemnt requirements to be not met"?
> Neither vectorization nor prefetching perform high-level loop optimizations
> like blocking, of course, but you don't fix that either.
>
> Richard.
>
Consider the following test case:
#define STREAM_ARRAY_SIZE 100000000
typedef unsigned long long ssize_t;
typedef double DP;
void
tuned_STREAM_Add (DP* restrict a, DP* restrict b, DP* restrict c)
{
ssize_t j;
for (j = 0; j < STREAM_ARRAY_SIZE; j++)
c[j] = a[j] + b[j];
}
Here, `c[j]` is write-only within the loop and is a typical candidate for
non-temporal (NT) stores.
With `-O3 -fprefetch-loop-arrays -fdump-tree-aprefetch-details`, the store is
correctly marked as `{nt}` at the GIMPLE level. However, no NT instruction is
generated in the final assembly.
The current issue in GCC is that vectorization runs before NT marking. By the
time NT candidates are identified, vector width and alignment decisions have
already been made.
As a result, during RTL expansion (`expand_assignment`), the vectorized store
is treated as misaligned and cannot be lowered to NT instructions (e.g.,
MOVNTDQ), which require strict alignment. Instead, a regular store is emitted.
The purpose of this patch is to move NT analysis before vectorization, allowing
the vectorizer to identify NT store candidates and take NT alignment
requirements into account, so that vectorized NT store instructions can be
successfully generated.
Loop blocking is outside the scope of this patch.
> >
> > > >
> > > >
> > > > > > Richard.
> > > > > >
> > > > > > >
> > > > > > > Thanks for your time and comments.
> > > > > > >
> > > > > > >
> > > > > > > > -----邮件原件-----
> > > > > > > > 发件人: Richard Biener <[email protected]>
> > > > > > > > 发送时间: 2026年4月27日 17:06
> > > > > > > > 收件人: Xiaomei Xue <[email protected]>
> > > > > > > > 抄送: [email protected]; [email protected]; Kewen Lin
> > > > > > > > <[email protected]>; Qingkuan Lai <[email protected]>
> > > > > > > > 主题: Re: [PATCH 1/2] feat: Add -fnon-temporal-store option
> > > > > > > > to enhance nontemporal store optimization
> > > > > > > >
> > > > > > > > On Fri, Apr 24, 2026 at 11:33 AM Xiaomei Xue
> > > > > > > > <[email protected]>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > -----邮件原件-----
> > > > > > > > > > 发件人: Richard Biener <[email protected]>
> > > > > > > > > > 发送时间: 2026年4月24日 14:55
> > > > > > > > > > 收件人: Xiaomei Xue <[email protected]>
> > > > > > > > > > 抄送: [email protected]; Qingkuan Lai
> > > > > > > > > > <[email protected]>; Kewen Lin <[email protected]>;
> > > > > > > > > > Jakub Jelinek <[email protected]>
> > > > > > > > > > 主题: Re: [PATCH 1/2] feat: Add -fnon-temporal-store
> > > > > > > > > > option to enhance nontemporal store optimization
> > > > > > > > > >
> > > > > > > > > > On Tue, Dec 23, 2025 at 6:45 AM xuexiaomei
> > > > > > > > > > <[email protected]>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > From: "[email protected]" <[email protected]>
> > > > > > > > > > >
> > > > > > > > > > > 1. Clone the aprefetch pass. When nontemporal store
> > > > > > > > > > > optimization is
> > > > > > > > > > enabled,
> > > > > > > > > > > first perform the analysis for nontemporal stores
> > > > > > > > > > > in this pass, and
> > > > > > > > then
> > > > > > > > > > > handle data alignment in the vect pass.
> > > > > > > > > > > 2. Attempt to retrieve the original loop's trip
> > > > > > > > > > > count from within an
> > > > > > > > OpenMP
> > > > > > > > > > > outlined function.
> > > > > > > > > >
> > > > > > > > > > Few high-level comments below
> > > > > > > > > >
> > > > > > > > > > > gcc/ChangeLog:
> > > > > > > > > > >
> > > > > > > > > > > * common.opt: Add -fnon-temporal-store.
> > > > > > > > > > > * passes.def: Add pass_loop_prefetch before
> > > > > > > > pass_if_conversion.
> > > > > > > > > > > * tree-ssa-loop-niter.cc
> > > > > > > > > > > (get_oldest_def_with_type_conv): New.
> > > > > > > > > > > (get_omp_orig_loop_niters): New.
> > > > > > > > > > > (traverse_niters_expr_operands): New.
> > > > > > > > > > > (get_const_niters_from_omp_outlined): New.
> > > > > > > > > > > * tree-ssa-loop-niter.h
> > > > > > > > > > > (get_const_niters_from_omp_outlined): New.
> > > > > > > > > > > * tree-ssa-loop-prefetch.cc
> > > > > > > > > > > (nontemporal_store_p): Attempt to get the
> > > > > > > > > > > preferred SIMD mode
> > > > > > > > > > for
> > > > > > > > > > > scalar type.
> > > > > > > > > > > (mark_nontemporal_store): Add parameter to
> > > > > > > > > > > control
> > > > > > > > > > nontemporal
> > > > > > > > > > > store analysis.
> > > > > > > > > > > (mark_nontemporal_stores): Ditto.
> > > > > > > > > > > (loop_prefetch_arrays): Ditto.
> > > > > > > > > > > (tree_ssa_prefetch_arrays): Ditto.
> > > > > > > > > > > (pass_loop_prefetch::execute): Ditto.
> > > > > > > > > > > (determine_loop_nest_reuse): Attempt to
> > > > > > > > > > > obtain const niters from
> > > > > > > > > > omp
> > > > > > > > > > > outlined fn.
> > > > > > > > > > > (unset_nontemporal_move): New
> > > > > > > > > > > (class pass_loop_prefetch): Add clone method
> > > > > > > > > > > and set pass
> > > > > > > > param.
> > > > > > > > > > > * tree-vect-data-refs.cc
> > > (vect_supportable_dr_alignment):
> > > > > > > > > > > Enforce alignment for statements marked as
> > > nontemporal.
> > > > > > > > > > >
> > > > > > > > > > > Signed-off-by: [email protected]
> > > > > > > > > > > <[email protected]>
> > > > > > > > > > > ---
> > > > > > > > > > > gcc/common.opt | 4 +
> > > > > > > > > > > gcc/passes.def | 3 +-
> > > > > > > > > > > gcc/tree-ssa-loop-niter.cc | 151
> > > > > > > > > > ++++++++++++++++++++++++++++++++++
> > > > > > > > > > > gcc/tree-ssa-loop-niter.h | 1 +
> > > > > > > > > > > gcc/tree-ssa-loop-prefetch.cc | 109
> > > ++++++++++++++++++++----
> > > > > > > > > > > gcc/tree-vect-data-refs.cc | 6 ++
> > > > > > > > > > > 6 files changed, 258 insertions(+), 16 deletions(-)
> > > > > > > > > > >
> > > > > > > > > > > diff --git a/gcc/common.opt b/gcc/common.opt index
> > > > > > > > > > > 50de980615f..f6e9c8314a0 100644
> > > > > > > > > > > --- a/gcc/common.opt
> > > > > > > > > > > +++ b/gcc/common.opt
> > > > > > > > > > > @@ -2615,6 +2615,10 @@ fprefetch-loop-arrays Common
> > > > > > > > > > > Var(flag_prefetch_loop_arrays) Init(-1) Optimization
> > > > > > > > > > > Generate prefetch instructions, if available, for arrays
> > > > > > > > > > > in
> loops.
> > > > > > > > > > >
> > > > > > > > > > > +fnon-temporal-store Common
> > > > > > > > > > > +Var(flag_non_temporal_store) Init(0) Optimization
> > > > > > > > > > > +Generate nontemporal store instructions in loops
> > > if available.
> > > > > > > > > > > +
> > > > > > > > > > > fprofile
> > > > > > > > > > > Common Var(profile_flag) Enable basic program
> > > > > > > > > > > profiling code.
> > > > > > > > > > > diff --git a/gcc/passes.def b/gcc/passes.def index
> > > > > > > > > > > fac04cd86c7..140c707b217 100644
> > > > > > > > > > > --- a/gcc/passes.def
> > > > > > > > > > > +++ b/gcc/passes.def
> > > > > > > > > > > @@ -314,6 +314,7 @@ along with GCC; see the file
> COPYING3.
> > > > > > > > > > > If not
> > > > > > > > see
> > > > > > > > > > > NEXT_PASS (pass_parallelize_loops, false
> > > > > > > > > > > /* oacc_kernels_p
> > > > > > > > */);
> > > > > > > > > > > NEXT_PASS (pass_expand_omp_ssa);
> > > > > > > > > > > NEXT_PASS (pass_ch_vect);
> > > > > > > > > > > + NEXT_PASS (pass_loop_prefetch, true /*
> > > > > > > > > > > + nontemporal_store_analysis_p */);
> > > > > > > > > > > NEXT_PASS (pass_if_conversion);
> > > > > > > > > > > /* pass_vectorize must immediately follow
> > > > > > > > pass_if_conversion.
> > > > > > > > > > > Please do not add any other passes in
> between.
> > > > > > > > > > > */ @@
> > > > > > > > > > > -329,7 +330,7 @@ along with GCC; see the file COPYING3.
> > > > > > > > > > > If not
> > > > > see
> > > > > > > > > > > NEXT_PASS (pass_dse);
> > > > > > > > > > > POP_INSERT_PASSES ()
> > > > > > > > > > > NEXT_PASS (pass_slp_vectorize);
> > > > > > > > > > > - NEXT_PASS (pass_loop_prefetch);
> > > > > > > > > > > + NEXT_PASS (pass_loop_prefetch, false);
> > > > > > > > > > > /* Run IVOPTs after the last pass that
> > > > > > > > > > > uses data-reference
> > > > > > > > > > analysis
> > > > > > > > > > > as that doesn't handle TARGET_MEM_REFs.
> */
> > > > > > > > > > > NEXT_PASS (pass_iv_optimize); diff --git
> > > > > > > > > > > a/gcc/tree-ssa-loop-niter.cc
> > > > > > > > > > > b/gcc/tree-ssa-loop-niter.cc index
> > > > > > > > > > > 8aa52dc54b0..44aeb0d223e 100644
> > > > > > > > > > > --- a/gcc/tree-ssa-loop-niter.cc
> > > > > > > > > > > +++ b/gcc/tree-ssa-loop-niter.cc
> > > > > > > > > > > @@ -5183,6 +5183,157 @@
> > > > > > > > > > > estimated_stmt_executions_int (class loop
> > > > > > > > > > *loop)
> > > > > > > > > > > return snit < 0 ? -1 : snit; }
> > > > > > > > > > >
> > > > > > > > > > > +/* Extract x from type conversion. ie:
> > > > > > > > > > > + a=(uint64_t)b, we can use this function to get b from
> > > > > > > > > > > a.
> > > > > > > > > > > +*/
> > > > > > > > > > > +
> > > > > > > > > > > +static tree
> > > > > > > > > > > +get_oldest_def_with_type_conv (tree t) {
> > > > > > > > > > > + auto ssa = t;
> > > > > > > > > > > + while (ssa && TREE_CODE (ssa) == SSA_NAME)
> > > > > > > > > > > + {
> > > > > > > > > > > + auto g = SSA_NAME_DEF_STMT (ssa);
> > > > > > > > > > > + if (!is_gimple_assign (g))
> > > > > > > > > > > + break;
> > > > > > > > > > > + auto rhs_code = gimple_assign_rhs_code (g);
> > > > > > > > > > > + if (rhs_code != NOP_EXPR && rhs_code !=
> SSA_NAME)
> > > > > > > > > > > + break;
> > > > > > > > > > > + ssa = gimple_assign_rhs1 (g);
> > > > > > > > > > > + }
> > > > > > > > > > > + return ssa;
> > > > > > > > > > > +}
> > > > > > > > > > > +
> > > > > > > > > > > +/* Recursively traces the SSA variable OP to obtain
> > > > > > > > > > > +the original
> > > > > loop's
> > > > > > > > > > > + constant iteration count from an OpenMP outlined
> > > function.
> > > > > > > > > > > +Only
> > > > > > > > > > supports
> > > > > > > > > > > + cases where the trip count is an INTEGER_CST.
> > > > > > > > > > > + The trip count can only
> > > > > > > > > > be
> > > > > > > > > > > + determined when OpenMP static scheduling is used
> > > > > > > > > > > + without a specified
> > > > > > > > > > chunk
> > > > > > > > > > > + size. */
> > > > > > > > > > > +
> > > > > > > > > > > +static HOST_WIDE_INT get_omp_orig_loop_niters (tree
> > > > > > > > > > > +op) {
> > > > > > > > > > > + HOST_WIDE_INT niters = -1;
> > > > > > > > > > > + tree rhs1 = NULL_TREE, rhs2 = NULL_TREE, fndecl,
> > > > > > > > > > > +arg;
> > > > > > > > > > > + gimple *def;
> > > > > > > > > > > + gphi *phi;
> > > > > > > > > > > + unsigned num_args;
> > > > > > > > > > > + enum tree_code code;
> > > > > > > > > > > +
> > > > > > > > > > > + if (!op)
> > > > > > > > > > > + return niters;
> > > > > > > > > > > +
> > > > > > > > > > > + if (TREE_CODE (op) != SSA_NAME)
> > > > > > > > > > > + return niters;
> > > > > > > > > > > +
> > > > > > > > > > > + def = SSA_NAME_DEF_STMT (op); if (!def)
> > > > > > > > > > > + return niters;
> > > > > > > > > > > +
> > > > > > > > > > > + if (gimple_code (def) == GIMPLE_PHI)
> > > > > > > > > > > + {
> > > > > > > > > > > + /* Only handle PHI nodes in the outermost loop. */
> > > > > > > > > > > + if (gimple_bb (def)->loop_father !=
> > > > > current_loops->tree_root)
> > > > > > > > > > > + return niters;
> > > > > > > > > > > +
> > > > > > > > > > > + phi = as_a<gphi *> (def);
> > > > > > > > > > > + num_args = gimple_phi_num_args (phi);
> > > > > > > > > > > + if (num_args > 2)
> > > > > > > > > > > + return niters;
> > > > > > > > > > > +
> > > > > > > > > > > + /* Recursively trace PHI arguments. */
> > > > > > > > > > > + for (unsigned i = 0; i < num_args; i++)
> > > > > > > > > > > + {
> > > > > > > > > > > + arg = gimple_phi_arg_def (phi, i);
> > > > > > > > > > > + niters = get_omp_orig_loop_niters (arg);
> > > > > > > > > > > + if (niters != -1)
> > > > > > > > > > > + break;
> > > > > > > > > > > + }
> > > > > > > > > > > + return niters;
> > > > > > > > > > > + }
> > > > > > > > > > > +
> > > > > > > > > > > + if (!is_gimple_assign (def))
> > > > > > > > > > > + return niters;
> > > > > > > > > > > + code = gimple_assign_rhs_code (def);
> > > > > > > > > > > + rhs1 = gimple_assign_rhs1 (def);
> > > > > > > > > > > +
> > > > > > > > > > > + if (code == COND_EXPR)
> > > > > > > > > > > + return get_omp_orig_loop_niters
> > > > > > > > > > > + (gimple_assign_rhs2 (def));
> > > > > > > > > > > +
> > > > > > > > > > > + /* Recursively trace rhs1 if not a
> > > > > > > > > > > + divisianalyze_tree_opson/modulo or not
> > > > > > > > > > a
> > > > > > > > > > > + * constant. */
> > > > > > > > > > > + if (code != TRUNC_MOD_EXPR && code !=
> TRUNC_DIV_EXPR)
> > > > > > > > > > > + return get_omp_orig_loop_niters (rhs1); else
> > > > > > > > > > > + if (TREE_CODE
> > > > > > > > > > > + (rhs1) != INTEGER_CST)
> > > > > > > > > > > + return get_omp_orig_loop_niters (rhs1);
> > > > > > > > > > > +
> > > > > > > > > > > + /* Find the expression q = n / nthreads or tt = n %
> nthreads.
> > > > > > > > > > > + */
> > > > > > > > > > > + rhs2 = gimple_assign_rhs2 (def);
> > > > > > > > > > > + rhs2 = get_oldest_def_with_type_conv (rhs2); def
> > > > > > > > > > > + = SSA_NAME_DEF_STMT (rhs2); if (def && is_gimple_call
> (def))
> > > > > > > > > > > + {
> > > > > > > > > > > + fndecl = gimple_call_fndecl (def);
> > > > > > > > > > > + if (fndecl
> > > > > > > > > > > + && (DECL_FUNCTION_CODE (fndecl) ==
> > > > > > > > > > BUILT_IN_OMP_GET_NUM_THREADS
> > > > > > > > > > > + || DECL_FUNCTION_CODE (fndecl) ==
> > > > > > > > > > BUILT_IN_OMP_GET_THREAD_NUM)
> > > > > > > > > > > + && tree_fits_shwi_p (rhs1))
> > > > > > > > > > > + return tree_to_shwi (rhs1);
> > > > > > > > > > > + }
> > > > > > > > > > > + niters = get_omp_orig_loop_niters (rhs2);
> > > > > > > > > > > +
> > > > > > > > > > > + return niters;
> > > > > > > > > > > +}
> > > > > > > > > > > +
> > > > > > > > > > > +/* Recursively traverses all operands of
> > > > > > > > > > > +NITERS_EXPR to determine if
> > > > > > the
> > > > > > > > > > > + expression depends on the number of OpenMP
> > > > > > > > > > > + threads, where
> > > > > > > > > > NITERS_EXPR
> > > > > > > > > > > + is typically of the form (upper bound - lower
> > > > > > > > > > > + bound
> > > > > > > > > > > + + 1) for a thread's
> > > > > > > > > > task.
> > > > > > > > > > > + If a dependency is found, NITERS is updated with
> > > > > > > > > > > + the trip count of
> > > > > > > > the
> > > > > > > > > > > + original loop corresponding to the OpenMP
> > > > > > > > > > > + outlined
> > > function.
> > > > > > > > > > > + */
> > > > > > > > > > > +
> > > > > > > > > > > +static void
> > > > > > > > > > > +traverse_niters_expr_operands (HOST_WIDE_INT
> > > > > > > > > > > +*niters, tree
> > > > > > > > > > > +niters_expr) {
> > > > > > > > > > > + if (!niters_expr)
> > > > > > > > > > > + return;
> > > > > > > > > > > +
> > > > > > > > > > > + unsigned n = TREE_OPERAND_LENGTH (niters_expr);
> > > > > > > > > > > + if (n
> > > == 0)
> > > > > > > > > > > + {
> > > > > > > > > > > + *niters = get_omp_orig_loop_niters (niters_expr);
> > > > > > > > > > > + return;
> > > > > > > > > > > + }
> > > > > > > > > > > +
> > > > > > > > > > > + for (unsigned i = 0; i < n; i++)
> > > > > > > > > > > + {
> > > > > > > > > > > + tree op = TREE_OPERAND (niters_expr, i);
> > > > > > > > > > > + if (TREE_CODE (op) == INTEGER_CST)
> > > > > > > > > > > + continue;
> > > > > > > > > > > +
> > > > > > > > > > > + if (TREE_CODE (op) == NOP_EXPR)
> > > > > > > > > > > + op = TREE_OPERAND (op, 0);
> > > > > > > > > > > + traverse_niters_expr_operands (niters, op);
> > > > > > > > > > > + }
> > > > > > > > > > > +}
> > > > > > > > > > > +
> > > > > > > > > > > +/* Get the constant niters of the original loop
> > > > > > > > > > > +from an OpenMP
> > > > > > outlined
> > > > > > > > > > > + function. Only supports the case where niters is
> > > > > > > > > > > + an
> > > > > INTEGER_CST.
> > > > > > > > > > > +*/
> > > > > > > > > > > +
> > > > > > > > > > > +HOST_WIDE_INT
> > > > > > > > > > > +get_const_niters_from_omp_outlined (class loop
> > > > > > > > > > > +*loop)
> > > > > > > > > >
> > > > > > > > > > I think this is something we should preserve by other
> > > > > > > > > > means and not have to reverse-engineer it. OMP
> > > > > > > > > > lowering/expansion could store the original
> > > > > > > > > > (constant?) number of iterations of a loop in class
> > > > > > > > > > loop of the loop for
> > > > > > > > example.
> > > > > > > > > > Jakub?
> > > > > > > > > >
> > > > > > > > > > > +{
> > > > > > > > > > > + HOST_WIDE_INT niters = -1;
> > > > > > > > > > > + tree niters_expr = number_of_latch_executions
> > > > > > > > > > > +(loop);
> > > > > > > > > > > + if (TREE_CODE (niters_expr) == COND_EXPR)
> > > > > > > > > > > + niters_expr = TREE_OPERAND (niters_expr, 1);
> > > > > > > > > > > +
> > > > > > > > > > > + niters_expr = analyze_scalar_evolution (loop,
> > > > > > > > > > > + niters_expr); if (!niters_expr ||
> > > > > > > > > > > + chrec_contains_undetermined
> > > > > (niters_expr))
> > > > > > > > > > > + return niters;
> > > > > > > > > > > +
> > > > > > > > > > > + traverse_niters_expr_operands (&niters,
> > > > > > > > > > > +niters_expr);
> > > > > > > > > > > + return niters;
> > > > > > > > > > > +}
> > > > > > > > > > > +
> > > > > > > > > > > /* Sets NIT to the maximum number of executions of
> > > > > > > > > > > the latch of
> > > > > > the
> > > > > > > > > > > LOOP, plus one. If we have no reliable
> > > > > > > > > > > estimate, the function
> > > > > > > > returns
> > > > > > > > > > > false, otherwise returns true. */ diff --git
> > > > > > > > > > > a/gcc/tree-ssa-loop-niter.h
> > > > > > > > > > > b/gcc/tree-ssa-loop-niter.h index
> > > > > > > > > > > 88e47b9ab94..a564bacf941 100644
> > > > > > > > > > > --- a/gcc/tree-ssa-loop-niter.h
> > > > > > > > > > > +++ b/gcc/tree-ssa-loop-niter.h
> > > > > > > > > > > @@ -38,6 +38,7 @@ extern tree loop_niter_by_eval
> > > > > > > > > > > (class loop *, edge); extern tree
> > > > > > > > > > > find_loop_niter_by_eval (class loop *, edge *);
> > > > > > > > > > > extern bool estimated_loop_iterations (class loop *,
> > > > > > > > > > > widest_int *); extern HOST_WIDE_INT
> > > > > > > > > > > estimated_loop_iterations_int (class loop *);
> > > > > > > > > > > +extern HOST_WIDE_INT
> > > > > > > > > > > +get_const_niters_from_omp_outlined
> > > > > > > > > > > +(class loop *);
> > > > > > > > > > > extern bool max_loop_iterations (class loop *,
> > > > > > > > > > > widest_int *); extern HOST_WIDE_INT
> > > > > > > > > > > max_loop_iterations_int (class loop *); extern bool
> > > > > > > > > > > likely_max_loop_iterations (class loop *, widest_int
> > > > > > > > > > > *); diff --git a/gcc/tree-ssa-loop-prefetch.cc
> > > > > > > > > > > b/gcc/tree-ssa-loop-prefetch.cc index
> > > > > > > > > > > dad53bfb79b..fd83382905f
> > > > > > > > > > > 100644
> > > > > > > > > > > --- a/gcc/tree-ssa-loop-prefetch.cc
> > > > > > > > > > > +++ b/gcc/tree-ssa-loop-prefetch.cc
> > > > > > > > > > > @@ -48,6 +48,7 @@ along with GCC; see the file COPYING3.
> > > > > > > > > > > If not see #include "tree-data-ref.h"
> > > > > > > > > > > #include "diagnostic-core.h"
> > > > > > > > > > > #include "dbgcnt.h"
> > > > > > > > > > > +#include "tree-parloops.h"
> > > > > > > > > > >
> > > > > > > > > > > /* This pass inserts prefetch instructions to
> > > > > > > > > > > optimize cache usage
> > > > > > during
> > > > > > > > > > > accesses to arrays in loops. It processes loops
> > > > > > > > > > > sequentially
> > > > > and:
> > > > > > > > > > > @@ -1245,7 +1246,7 @@ issue_prefetches (struct
> > > > > > > > > > > mem_ref_group
> > > > > > > > *groups,
> > > > > > > > > > > can be used. */
> > > > > > > > > > >
> > > > > > > > > > > static bool
> > > > > > > > > > > -nontemporal_store_p (struct mem_ref *ref)
> > > > > > > > > > > +nontemporal_store_p (struct mem_ref *ref, bool
> > > > > > > > > > > +nontemporal_store_analysis_p)
> > > > > > > > > > > {
> > > > > > > > > > > machine_mode mode;
> > > > > > > > > > > enum insn_code code; @@ -1264,6 +1265,20 @@
> > > > > > > > > > > nontemporal_store_p (struct mem_ref
> > > > > > *ref)
> > > > > > > > > > > return false;
> > > > > > > > > > >
> > > > > > > > > > > code = optab_handler (storent_optab, mode);
> > > > > > > > > > > +
> > > > > > > > > > > + /* If there is no nontemporal store instruction
> > > > > > > > > > > + for the current scalar
> > > > > > > > > > mode,
> > > > > > > > > > > + then try to get the preferred SIMD (vector)
> > > > > > > > > > > + mode for this scalar
> > > > > > > > type
> > > > > > > > > > > + and check if there is a corresponding
> > > > > > > > > > > + nontemporal store instruction. */ if
> > > > > > > > > > > + (nontemporal_store_analysis_p && code ==
> > > > > > > > > > CODE_FOR_nothing
> > > > > > > > > > > + && is_a<scalar_mode> (mode))
> > > > > > > > > > > + {
> > > > > > > > > > > + scalar_mode smode = as_a<scalar_mode> (mode);
> > > > > > > > > > > + machine_mode simd_mode =
> > > > > > > > > > targetm.vectorize.preferred_simd_mode
> > > > > > > > > > > + (smode);
> > > > > > > > > > > +
> > > > > > > > > >
> > > > > > > > > > There might be other SIMD modes, and you don't know
> > > > > > > > > > whether the store will be vectorized or not. In the
> > > > > > > > > > description you say we'll expand to a non-NT store
> > > > > > > > > > anyway, if it doesn't end up supported, so why
> > > > > > > > bother checking here?
> > > > > > > > > >
> > > > > > > > > The goal of the preferred_simd_mode + nontemporal
> > > > > > > > > capability check is to
> > > > > > > > avoid being overly conservative when scalar NT stores are
> > > > > > > > unavailable but vector NT stores are still possible.
> > > > > > > > >
> > > > > > > > > For example:
> > > > > > > > > typedef unsigned char DP; void __attribute__
> > > > > > > > > ((noinline)) test () {
> > > > > > > > > ssize_t j;
> > > > > > > > > for (j = 0; j < 1000000; j++)
> > > > > > > > > {
> > > > > > > > > c[j] = a[j] + b[j];
> > > > > > > > > }
> > > > > > > > > }
> > > > > > > > > On i386, there is no scalar NT store support for byte
> > > > > > > > > types, so a scalar-only
> > > > > > > > check would reject NT stores.
> > > > > > > > > However, this loop can be vectorized (e.g. QImode →
> > > > > > > > > V16QImode), and the
> > > > > > > > resulting vector store may be lowered to an NT-capable
> > > > > > > > instruction if supported by the ISA. This keeps NT store
> > > > > > > > as a valid
> > > optimization path.
> > > > > > > > > In addition, marking scalar as NT-capable also serves as
> > > > > > > > > a hint for
> > > > > > > > vectorization alignment and helps guide generation of
> > > > > > > > streaming stores when profitable.
> > > > > > > > >
> > > > > > > > > > Iff, then I'd gate processing on the availability on
> > > > > > > > > > any storent expander, but then the targets default of
> > > > > > > > > > having the pass enabled or not should also work to that
> extent?
> > > > > > > > > >
> > > > > > > > > > > + if (VECTOR_MODE_P (simd_mode))
> > > > > > > > > > > + code = optab_handler (storent_optab, simd_mode);
> > > > > > > > > > > + }
> > > > > > > > > > > +
> > > > > > > > > > > return code != CODE_FOR_nothing; }
> > > > > > > > > > >
> > > > > > > > > > > @@ -1271,9 +1286,9 @@ nontemporal_store_p (struct
> > > > > > > > > > > mem_ref
> > > > > > *ref)
> > > > > > > > > > > and return true. Otherwise, we return false.
> > > > > > > > > > > */
> > > > > > > > > > >
> > > > > > > > > > > static bool
> > > > > > > > > > > -mark_nontemporal_store (struct mem_ref *ref)
> > > > > > > > > > > +mark_nontemporal_store (struct mem_ref *ref, bool
> > > > > > > > > > > +nontemporal_store_analysis_p)
> > > > > > > > > > > {
> > > > > > > > > > > - if (!nontemporal_store_p (ref))
> > > > > > > > > > > + if (!nontemporal_store_p (ref,
> > > > > > > > > > > + nontemporal_store_analysis_p))
> > > > > > > > > > > return false;
> > > > > > > > > > >
> > > > > > > > > > > if (dump_file && (dump_flags & TDF_DETAILS)) @@
> > > > > > > > > > > -1343,7
> > > > > > > > > > > +1358,8
> > > > > > > > > > @@
> > > > > > > > > > > may_use_storent_in_loop_p (class loop *loop)
> > > > > > > > > > > references in the loop. Returns whether we
> > > > > > > > > > > inserted any mfence call. */
> > > > > > > > > > >
> > > > > > > > > > > static bool
> > > > > > > > > > > -mark_nontemporal_stores (class loop *loop, struct
> > > > > > > > > > > mem_ref_group
> > > > > > > > > > > *groups)
> > > > > > > > > > > +mark_nontemporal_stores (class loop *loop, struct
> > > > > > > > > > > +mem_ref_group
> > > > > > > > > > *groups,
> > > > > > > > > > > + bool
> > > > > > > > > > > + nontemporal_store_analysis_p)
> > > > > > > > > >
> > > > > > > > > > I'd really separate NT store processing from
> > > > > > > > > > prefetching, you only seem to disable it half-way. Why?
> > > > > > > > >
> > > > > > > > > The reason is to keep the change minimal and reuse the
> > > > > > > > > existing prefetch
> > > > > > > > infrastructure. Do you suggest making a separated function for
> this?
> > > > > > > >
> > > > > > > > As indicated below ....
> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > {
> > > > > > > > > > > struct mem_ref *ref;
> > > > > > > > > > > bool any = false; @@ -1353,9 +1369,12 @@
> > > > > > > > > > > mark_nontemporal_stores (class loop *loop, struct
> > > > > > > > > > > mem_ref_group *groups)
> > > > > > > > > > >
> > > > > > > > > > > for (; groups; groups = groups->next)
> > > > > > > > > > > for (ref = groups->refs; ref; ref = ref->next)
> > > > > > > > > > > - any |= mark_nontemporal_store (ref);
> > > > > > > > > > > + any |= mark_nontemporal_store (ref,
> > > > > > > > > > > + nontemporal_store_analysis_p);
> > > > > > > > > > >
> > > > > > > > > > > - if (any && FENCE_FOLLOWING_MOVNT != NULL_TREE)
> > > > > > > > > > > + /* Avoid inserting excessive fence instructions,
> > > > > > > > > > > + insert only in the
> > > > > > > > > > aprefetch2
> > > > > > > > > > > + pass. */
> > > > > > > > > > > + if (!nontemporal_store_analysis_p
> > > > > > > > > > > + && any && FENCE_FOLLOWING_MOVNT !=
> NULL_TREE)
> > > > > > > > > > > {
> > > > > > > > > > > emit_mfence_after_loop (loop);
> > > > > > > > > > > return true;
> > > > > > > > > > > @@ -1653,6 +1672,10 @@ determine_loop_nest_reuse
> > > > > > > > > > > (class loop *loop, struct mem_ref_group *refs,
> > > > > > > > > > >
> > > > > > > > > > > aloop = vloops[i];
> > > > > > > > > > > vol = estimated_stmt_executions_int (aloop);
> > > > > > > > > > > +
> > > > > > > > > > > + if (vol == -1 && flag_openmp &&
> > > > > > > > > > > + parallelized_function_p
> > > > > > > > > > (cfun->decl))
> > > > > > > > > > > + vol = get_const_niters_from_omp_outlined
> > > > > > > > > > > + (aloop);
> > > > > > > > > >
> > > > > > > > > > I'll note that heuristics are based on cache size
> > > > > > > > > > params, but for OMP parallelized code that will be
> > > > > > > > > > off, since the cache sharing info across threads isn't
> > > > > > > > > > known. This at least deserves a
> > > > > > comment.
> > > > > > > > > >
> > > > > > > > > > > +
> > > > > > > > > > > if (vol == -1)
> > > > > > > > > > > vol = expected_loop_iterations (aloop);
> > > > > > > > > > > volume *= vol; @@ -1886,7 +1909,8 @@
> > > > > > > > > > > insn_to_prefetch_ratio_too_small_p
> > > > > > > > (unsigned
> > > > > > > > > > ninsns, unsigned prefetch_count,
> > > > > > > > > > > to update SSA for virtual operands and LC SSA
> > > > > > > > > > > for a split
> > > edge.
> > > > > > > > > > > */
> > > > > > > > > > >
> > > > > > > > > > > static bool
> > > > > > > > > > > -loop_prefetch_arrays (class loop *loop, bool
> > > > > > > > > > > &need_lc_ssa_update)
> > > > > > > > > > > +loop_prefetch_arrays (class loop *loop, bool
> > > &need_lc_ssa_update,
> > > > > > > > > > > + bool
> > > > > > > > > > > +nontemporal_store_analysis_p)
> > > > > > > > > > > {
> > > > > > > > > > > struct mem_ref_group *refs;
> > > > > > > > > > > unsigned ahead, ninsns, time, unroll_factor; @@
> > > > > > > > > > > -1927,7
> > > > > > > > > > > +1951,8 @@ loop_prefetch_arrays (class loop *loop,
> > > > > > > > > > > +bool
> > > > > > &need_lc_ssa_update)
> > > > > > > > > > > /* Give up prefetching if the number of memory
> > > > > > > > > > > references in
> > > > > the
> > > > > > > > > > > loop is not reasonable based on profitablity
> > > > > > > > > > > and compilation
> > > > > > time
> > > > > > > > > > > considerations. */
> > > > > > > > > > > - if (!mem_ref_count_reasonable_p (ninsns,
> > > > > > > > > > > mem_ref_count))
> > > > > > > > > > > + if (!nontemporal_store_analysis_p
> > > > > > > > > > > + && !mem_ref_count_reasonable_p (ninsns,
> > > > > > > > > > > + mem_ref_count))
> > > > > > > > > > > goto fail;
> > > > > > > > > > >
> > > > > > > > > > > /* Step 2: estimate the reuse effects. */ @@
> > > > > > > > > > > -1957,11
> > > > > > > > > > > +1982,18 @@ loop_prefetch_arrays (class loop *loop,
> > > > > > > > > > > +bool
> > > > > > > > > > > &need_lc_ssa_update)
> > > > > > > > > > >
> > > > > > > > > > > /* Prefetching is not likely to be profitable if
> > > > > > > > > > > the instruction to
> > > > > > > > prefetch
> > > > > > > > > > > ratio is too small. */
> > > > > > > > > > > - if (insn_to_prefetch_ratio_too_small_p (ninsns,
> > > > > > > > > > > prefetch_count,
> > > > > > > > > > > + if (!nontemporal_store_analysis_p &&
> > > > > > > > > > > + insn_to_prefetch_ratio_too_small_p (ninsns,
> > > > > > > > > > > + prefetch_count,
> > > > > > > > > > >
> unroll_factor))
> > > > > > > > > > > goto fail;
> > > > > > > > > > >
> > > > > > > > > > > - need_lc_ssa_update |= mark_nontemporal_stores
> > > > > > > > > > > (loop, refs);
> > > > > > > > > > > + need_lc_ssa_update
> > > > > > > > > > > + |= mark_nontemporal_stores (loop, refs,
> > > > > > > > > > > + nontemporal_store_analysis_p);
> > > > > > > > > > > +
> > > > > > > > > > > + /* With nontemporal store optimization enabled,
> > > > > > > > > > > + only the analysis phase
> > > > > > > > > > is
> > > > > > > > > > > + performed during the aprefetch1 pass. */ if
> > > > > > > > > > > + (nontemporal_store_analysis_p)
> > > > > > > > > > > + goto fail;
> > > > > > > > > > >
> > > > > > > > > > > /* Step 4: what to prefetch? */
> > > > > > > > > > > if (!schedule_prefetches (refs, unroll_factor,
> > > > > > > > > > > ahead)) @@
> > > > > > > > > > > -1983,10
> > > > > > > > > > > +2015,33 @@ fail:
> > > > > > > > > > > return unrolled;
> > > > > > > > > > > }
> > > > > > > > > > >
> > > > > > > > > > > +/* Cancel the nontemporal store marked in aprefetch1
> > > > > > > > > > > pass.
> > > > > > > > > > > +*/
> > > > > > > > > > > +
> > > > > > > > > > > +static void
> > > > > > > > > > > +unset_nontemporal_move (void) {
> > > > > > > > > > > + basic_block bb;
> > > > > > > > > > > + gimple_stmt_iterator bsi;
> > > > > > > > > > > + gimple *stmt;
> > > > > > > > > > > + FOR_EACH_BB_FN (bb, cfun)
> > > > > > > > > > > + {
> > > > > > > > > > > + for (bsi = gsi_start_bb (bb); !gsi_end_p
> > > > > > > > > > > +(bsi); gsi_next
> > > (&bsi))
> > > > > > > > > > > + {
> > > > > > > > > > > + stmt = gsi_stmt (bsi);
> > > > > > > > > > > + if (!is_gimple_assign (stmt))
> > > > > > > > > > > + continue;
> > > > > > > > > > > +
> > > > > > > > > > > + gassign *assign_stmt = as_a<gassign *> (stmt);
> > > > > > > > > > > + if (gimple_assign_nontemporal_move_p
> > > (assign_stmt))
> > > > > > > > > > > + gimple_assign_set_nontemporal_move
> > > > > > > > > > > + (assign_stmt,
> > > > > > false);
> > > > > > > > > > > + }
> > > > > > > > > > > + }
> > > > > > > > > > > +}
> > > > > > > > > > > +
> > > > > > > > > > > /* Issue prefetch instructions for array references in
> > > > > > > > > > > loops.
> > > > > > > > > > > */
> > > > > > > > > > >
> > > > > > > > > > > unsigned int
> > > > > > > > > > > -tree_ssa_prefetch_arrays (void)
> > > > > > > > > > > +tree_ssa_prefetch_arrays (bool
> > > > > > > > > > > +nontemporal_store_analysis_p)
> > > > > > > > > > > {
> > > > > > > > > > > bool unrolled = false;
> > > > > > > > > > > bool need_lc_ssa_update = false; @@ -2033,12
> > > > > > > > > > > +2088,21 @@ tree_ssa_prefetch_arrays (void)
> > > > > > > > > > > set_builtin_decl (BUILT_IN_PREFETCH, decl, false);
> > > > > > > > > > > }
> > > > > > > > > > >
> > > > > > > > > > > + /* In the aprefetch1 pass, stmts that are
> > > > > > > > > > > + successfully analyzed are
> > > > > > > > marked
> > > > > > > > > > > + with nontemporal store, but no fence is inserted.
> > > > > > > > > > > + The main
> > > > > > > > purpose is
> > > > > > > > > > > + to complete alignment operations in the vect pass.
> > > > > > > > > > > + In the
> > > > > > > > aprefetch2
> > > > > > > > > > > + pass, the nontemporal store flag is removed,
> > > > > > > > > > > + and the stmts
> > > > > are
> > > > > > > > > > > + re-analyzed under the condition of alignment.
> > > > > > > > > > > + */
> > > > > > > > > >
> > > > > > > > > > Hmm, but then, if you only make this to guide how
> > > > > > > > > > vectorization behaves, why not adjust the heuristics
> > > > > > > > > > in vect_enhance_data_refs_alignment? That also knows
> > > > > > > > > > write-only
> > > > > data.
> > > > > > > > > > It would seem to be a much more natural place to honor
> > > > > > > > > > a later request to
> > > > > > > > use NT stores.
> > > > > > > >
> > > > > > > > ... I'd rather have this not in a new pass but integrated
> > > > > > > > into the vectorizer alignment peeling/versioning heuristic.
> > > > > > > >
> > > > > > > > Richard.
> > > > > > > >
> > > > > > > > > > Richard.
> > > > > > > > > >
> > > > > > > > > > > + if (!nontemporal_store_analysis_p)
> > > > > > > > > > > + unset_nontemporal_move ();
> > > > > > > > > > > +
> > > > > > > > > > > for (auto loop : loops_list (cfun, LI_FROM_INNERMOST))
> > > > > > > > > > > {
> > > > > > > > > > > if (dump_file && (dump_flags & TDF_DETAILS))
> > > > > > > > > > > fprintf (dump_file, "Processing loop %d:\n",
> > > > > > > > > > > loop->num);
> > > > > > > > > > >
> > > > > > > > > > > - unrolled |= loop_prefetch_arrays (loop,
> > > > > need_lc_ssa_update);
> > > > > > > > > > > + unrolled |= loop_prefetch_arrays (loop,
> > > > > > > > > > > + need_lc_ssa_update,
> > > > > > > > > > > +
> > > > > > > > > > > + nontemporal_store_analysis_p);
> > > > > > > > > > >
> > > > > > > > > > > if (dump_file && (dump_flags & TDF_DETAILS))
> > > > > > > > > > > fprintf (dump_file, "\n\n"); @@ -2078,15
> > > > > > > > > > > +2142,30 @@ class pass_loop_prefetch : public
> > > > > > > > > > > gimple_opt_pass {
> > > > > > > > > > > public:
> > > > > > > > > > > pass_loop_prefetch (gcc::context *ctxt)
> > > > > > > > > > > - : gimple_opt_pass (pass_data_loop_prefetch, ctxt)
> > > > > > > > > > > + : gimple_opt_pass (pass_data_loop_prefetch, ctxt),
> > > > > > > > > > > + nontemporal_store_analysis_p (false)
> > > > > > > > > > > {}
> > > > > > > > > > >
> > > > > > > > > > > /* opt_pass methods: */
> > > > > > > > > > > + opt_pass *clone () { return new
> > > > > > > > > > > + pass_loop_prefetch (m_ctxt); }
> > > > > > > > > > > bool gate (function *) final override
> > > > > > > > > > > {
> > > > > > > > > > > - return flag_prefetch_loop_arrays > 0;
> > > > > > > > > > > + if (nontemporal_store_analysis_p)
> > > > > > > > > > > + return optimize >= 3 && flag_non_temporal_store
> > > > > > > > > > > + && (flag_tree_loop_vectorize ||
> > > > > > > > > > cfun->has_force_vectorize_loops);
> > > > > > > > > > > + else
> > > > > > > > > > > + return flag_prefetch_loop_arrays > 0
> > > > > > > > > > > + || (optimize >= 3 &&
> > > > > > > > > > > + flag_non_temporal_store);
> > > > > > > > > > > }
> > > > > > > > > > > unsigned int execute (function *) final override;
> > > > > > > > > > > + void set_pass_param (unsigned int n, bool param) {
> > > > > > > > > > > + gcc_assert (n == 0);
> > > > > > > > > > > + nontemporal_store_analysis_p = param; }
> > > > > > > > > > > +
> > > > > > > > > > > +private:
> > > > > > > > > > > + bool nontemporal_store_analysis_p;
> > > > > > > > > > >
> > > > > > > > > > > }; // class pass_loop_prefetch
> > > > > > > > > > >
> > > > > > > > > > > @@ -2110,7 +2189,7 @@ pass_loop_prefetch::execute
> > > > > > > > > > > (function
> > > > > > *fun)
> > > > > > > > > > > return 0;
> > > > > > > > > > > }
> > > > > > > > > > >
> > > > > > > > > > > - return tree_ssa_prefetch_arrays ();
> > > > > > > > > > > + return tree_ssa_prefetch_arrays
> > > > > > > > > > > + (nontemporal_store_analysis_p);
> > > > > > > > > > > }
> > > > > > > > > > >
> > > > > > > > > > > } // anon namespace diff --git
> > > > > > > > > > > a/gcc/tree-vect-data-refs.cc
> > > > > > > > > > > b/gcc/tree-vect-data-refs.cc index
> > > > > > > > > > > 89a9077af0d..afc86d15cd5
> > > > > > > > > > > 100644
> > > > > > > > > > > --- a/gcc/tree-vect-data-refs.cc
> > > > > > > > > > > +++ b/gcc/tree-vect-data-refs.cc
> > > > > > > > > > > @@ -6787,6 +6787,12 @@ vect_supportable_dr_alignment
> > > > > > > > > > > (vec_info *vinfo,
> > > > > > > > > > dr_vec_info *dr_info,
> > > > > > > > > > > else if (dr_safe_speculative_read_required (stmt_info))
> > > > > > > > > > > return dr_unaligned_unsupported;
> > > > > > > > > > >
> > > > > > > > > > > + /* When using the vmovnt instruction, the data
> > > > > > > > > > > + address must be aligned
> > > > > > > > > > to the
> > > > > > > > > > > + vector width. */ gassign *assign_stmt =
> > > > > > > > > > > + dyn_cast<gassign *> (stmt_info->stmt); if
> > > > > > > > > > > + (assign_stmt &&
> > > > > > gimple_assign_nontemporal_move_p (assign_stmt))
> > > > > > > > > > > + return dr_unaligned_unsupported;
> > > > > > > > > > > +
> > > > > > > > > > > if (loop_vinfo)
> > > > > > > > > > > {
> > > > > > > > > > > vect_loop = LOOP_VINFO_LOOP (loop_vinfo);
> > > > > > > > > > > --
> > > > > > > > > > > 2.22.0
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > >
> >
