On Mon, Sep 28, 2026 at 12:52 PM Xiaomei Xue <[email protected]> wrote: > > > > > -----邮件原件----- > > 发件人: Richard Biener <[email protected]> > > 发送时间: 2026年9月24日 18:51 > > 收件人: Xiaomei Xue <[email protected]> > > 抄送: [email protected]; [email protected]; Kewen Lin > > <[email protected]>; Qingkuan Lai <[email protected]>; Feng Xue > > <[email protected]> > > 主题: Re: [PATCH 1/2] feat: Add -fnon-temporal-store option to enhance > > nontemporal store optimization > > > > On Thu, Sep 24, 2026 at 10:28 AM Xiaomei Xue <[email protected]> > > wrote: > > > > > > > > > > > > > -----邮件原件----- > > > > 发件人: Xiaomei Xue > > > > 发送时间: 2026年7月2日 20:24 > > > > 收件人: 'Richard Biener' <[email protected]> > > > > 抄送: '[email protected]' <[email protected]>; > > > > '[email protected]' <[email protected]>; Kewen Lin > > > > <[email protected]>; Qingkuan Lai <[email protected]>; Feng Xue > > > > <[email protected]> > > > > 主题: 答复: [PATCH 1/2] feat: Add -fnon-temporal-store option to > > enhance > > > > nontemporal store optimization > > > > > > > > > > > > > > > > > -----邮件原件----- > > > > > 发件人: Richard Biener <[email protected]> > > > > > 发送时间: 2026年7月1日 17:55 > > > > > 收件人: Xiaomei Xue <[email protected]> > > > > > 抄送: [email protected]; [email protected]; Kewen Lin > > > > > <[email protected]>; Qingkuan Lai <[email protected]> > > > > > 主题: Re: [PATCH 1/2] feat: Add -fnon-temporal-store option to > > > > > enhance nontemporal store optimization > > > > > > > > > > On Wed, Jun 17, 2026 at 9:15 AM Xiaomei Xue <[email protected]> > > > > > wrote: > > > > > > > > > > > > Hi, > > > > > > > > > > > > Thanks for the previous suggestions. I have reworked the > > > > > nontemporal-store logic. Instead of cloning the aprefetch pass, I > > > > > am planning to split out the NT-store analysis and run it from > > > > > vect_enhance_data_refs_alignment. This lets the vectorizer > > > > > identify > > > > > > vector NT-store candidates before deciding peeling/versioning, > > > > > > so the > > > > > result can guide data-reference alignment handling. > > > > > > > > > > > > In this design, the analysis only records candidates in > > > > > dr_vec_info::nontemporal_store_candidate. It does not mark the > > > > > GIMPLE store as nontemporal during analysis. > > > > > > > > > > > > I still have two questions. > > > > > > > > > > > > The first one is about the source of data references used by > > > > > determine_loop_nest_reuse. > > > > > > > > > > > > The NT-store analysis needs DR and dependence information for > > > > > > the > > > > loop. > > > > > I considered two possible approaches: > > > > > > > > > > > > 1. reuse the DRs/DDRs already computed by the vectorizer; > > > > > > 2. recompute DRs/DDRs inside determine_loop_nest_reuse. > > > > > > > > > > > > Reusing the vectorizer’s existing analysis avoids duplicate > > > > > > data-reference > > > > > analysis and keeps the NT-store analysis consistent with the > > > > > memory references and dependences currently seen by the > > > > > vectorizer. The downside is that it requires extending some > > > > > > interfaces, passing the vectorizer DRs/DDRs and a > > > > > > DR-to-mem_ref > > > > > mapping into the reuse-distance analysis. > > > > > > > > > > > > Recomputing DRs/DDRs inside determine_loop_nest_reuse would > > > > > > keep > > > > > the interface changes smaller and make the reuse analysis more > > > > self-contained. > > > > > However, it duplicates analysis work, and the recomputed DRs/DDRs > > > > > might differ subtly from the ones already > > > > > > used by the vectorizer. > > > > > > > > > > > > My current preference is to reuse the vectorizer DRs/DDRs, and > > > > > > handle > > > > > any unmapped DR or incomplete information conservatively. Does > > > > > that sound like the right direction, or would recomputing DRs/DDRs > > > > > inside the reuse-distance analysis be more acceptable > > > > > > in order to keep the interface impact smaller? > > > > > > > > > > Re-using the DDRs is better. The "plumbing" between the data > > > > > structures is a bit ugly, but well. > > > > > > > > > > I see you do > > > > > > > > > > @@ -6793,6 +6920,8 @@ vect_supportable_dr_alignment (vec_info > > > > > *vinfo, dr_vec_info *dr_info, > > > > > return dr_aligned; > > > > > else if (dr_safe_speculative_read_required (stmt_info)) > > > > > return dr_unaligned_unsupported; > > > > > + else if (dr_info->nontemporal_store_candidate) > > > > > + return dr_unaligned_unsupported; > > > > > > > > > > that effectively means that we disable vectorization if we cannot > > > > > align all marked DRs (or resort to versioning for mutual alignment). > > > > > Is that a good trade-off? I suspect having -fnon-temporal-store > > > > > as global flag results in very "undirected" > > > > > applying of NT stores, is that > > > > > what people would actually do? We should abstract from looking at > > > > > single bechmarks that improve by adding an additional flag here - > > > > > how would we ensure that enabling the flag by default does not > > > > > regress performance? An alternative might be a pragma/attribute > > > > > in the source on a loop or even a specific memory reference. > > > > > > > > > > > > > Yes, this needs to be fixed. If we cannot guarantee the alignment > > > > required by NT stores, we should not give up normal vectorization. > > > > The NT-store choice should only be made when the target can support > > > > it for the selected alignment; otherwise we should fall back to normal > > vector stores. > > > > This optimization is mainly intended for streaming loops whose > > > > performance is dominated by memory bandwidth, such as STREAM-like > > > > kernels( https://www.cs.virginia.edu/stream/FTP/Code/stream.c). In > > > > that benchmark setting the source cannot be changed, so adding a > > > > pragma or attribute is not an option; the only practical way to > > > > enable the optimization is through a compiler option. I agree that > > > > -fnon-temporal-store is a coarse global knob, so it should remain > > > > disabled by default. Users who enable it explicitly need to accept the > > possible cost from forced alignment, versioning, or peeling. > > > > > > > > > Do you have statistics on how many loops are > > > > > peeled/versioned[/not vectorized] with this flag on say SPEC CPU? > > > > > > > > > > > > > I have done an initial check on current trunk with -Ofast > > -fnon-temporal-store. > > > > So far, dr_info->nontemporal_store_candidate is set only in SPEC CPU > > > > 519.lbm_r and 538.imagick_r. In both cases the candidates are > > > > rejected by the current alignment check, so no NT stores are > > > > emitted. I will collect more detailed numbers for peeled / versioned > > > > / not-vectorized loops before the next version. > > > > > > > > > I'm not familiar with the aprefetch pass - does the analysis you > > > > > re-use really require a different data structure or could some > > > > > refactoring make it work with pure data_references / ddrs as well? > > > > > > > > > > > > > The analysis is still tied to the aprefetch pass because the > > > > reuse-distance computation relies on ref->prefetch_mod from the > > aprefetch data structures. > > > > Making it operate directly on data_references / DDRs would require a > > > > fairly large refactoring of that analysis. For this patch, I kept > > > > the change conservative and reused the existing structure as much as > > possible. > > > > > > > > > > The second question is about supporting NT stores under -fopenmp. > > > > > > > > > > > > For example: > > > > > > > > > > > > #pragma omp parallel for > > > > > > for (j = 0; j < loop_num; j++) > > > > > > c[j] = a[j] + b[j]; > > > > > > > > > > > > From the memory-access pattern, such a loop may also be a > > > > > > candidate for > > > > > NT stores. However, after OpenMP lowering, the original loop is > > > > > outlined into an OpenMP worker function. At the later NT-store > > > > > analysis point, we see the outlined function and its > > > > > > local loop, and it is no longer straightforward to recover the > > > > > > original global > > > > > iteration count of the parallel for. > > > > > > > > > > > > I previously tried to infer the original iteration count from > > > > > > the OpenMP > > > > > work-sharing/runtime partitioning pattern, but that only covers > > > > > limited cases and depends quite heavily on the exact lowered GIMPLE > > form. > > > > > > > > > > > > Another possibility would be to record, during OpenMP > > > > > > lowering, the > > > > > relationship between the outlined function and the corresponding > > > > > OMP loop iteration count. But that also has some issues: > > > > > > > > > > > > 1. a single cfun may contain multiple OMP loops, so storing a > > > > > > single trip > > > > > count in struct function is not accurate; > > > > > > 2. if the information is stored in struct loop, the loop may > > > > > > be transformed > > > > > between OpenMP lowering and the later NT-store analysis, making > > > > > the saved information stale; > > > > > > 3. maintaining a mapping from outlined functions back to > > > > > > original OMP > > > > > loop information would also need to account for later > > > > > optimizations that change the function body and loop structure. > > > > > > > > > > > > So I would like to ask whether there is a recommended way to > > > > > > obtain or > > > > > preserve the original OpenMP loop iteration count for such > > > > > -fopenmp cases. Or, at the NT-store analysis stage, should we for > > > > > now handle this conservatively based only on the loop > > > > > > currently present in the outlined function: if we cannot > > > > > > determine a > > > > > concrete loop iteration count, we simply do not trigger the > > > > > NT-store optimization? > > > > > > > > > > It's a good question, it might hint at that marking the actual > > > > > memory references as candidates is more feasible in this case. > > > > > > > > > > > Not sure I understand it well, do you mean marking the memory references > > as NT-store candidates earlier, before OpenMP lowering? Or do you think this > > kind of case should be supported? > > > > Yes, before or during OMP lowering. > > > > As I read this whole work is to make the stream benchmark numbers look > > better and there is no anticipated other practical use. Is that correct? > > > > Richard. > > > > > Currently, STREAM shows the largest benefit in our tests. This work aims to > extend the applicability of the existing NT-store optimization. Although GCC > already supports this optimization, opportunities can be missed in some > vectorized loops because the alignment requirements are not met. This change > makes the vectorizer consider NT-store requirements when making alignment > decisions, potentially benefiting streaming loops in users’ own applications > where NT stores are appropriate.
Sorry for asking stupid questions - but isn't this optimizing prefetching and forgoing optimizing vectorization? Currently prefetching runs after (optimal) vectorization. How does that cause "alignemnt requirements to be not met"? Neither vectorization nor prefetching perform high-level loop optimizations like blocking, of course, but you don't fix that either. Richard. > > > > > > > > > > > > Richard. > > > > > > > > > > > > > > > > > Thanks for your time and comments. > > > > > > > > > > > > > > > > > > > -----邮件原件----- > > > > > > > 发件人: Richard Biener <[email protected]> > > > > > > > 发送时间: 2026年4月27日 17:06 > > > > > > > 收件人: Xiaomei Xue <[email protected]> > > > > > > > 抄送: [email protected]; [email protected]; Kewen Lin > > > > > > > <[email protected]>; Qingkuan Lai <[email protected]> > > > > > > > 主题: Re: [PATCH 1/2] feat: Add -fnon-temporal-store option to > > > > > > > enhance nontemporal store optimization > > > > > > > > > > > > > > On Fri, Apr 24, 2026 at 11:33 AM Xiaomei Xue > > > > > > > <[email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -----邮件原件----- > > > > > > > > > 发件人: Richard Biener <[email protected]> > > > > > > > > > 发送时间: 2026年4月24日 14:55 > > > > > > > > > 收件人: Xiaomei Xue <[email protected]> > > > > > > > > > 抄送: [email protected]; Qingkuan Lai > > > > > > > > > <[email protected]>; Kewen Lin <[email protected]>; > > > > > > > > > Jakub Jelinek <[email protected]> > > > > > > > > > 主题: Re: [PATCH 1/2] feat: Add -fnon-temporal-store option > > > > > > > > > to enhance nontemporal store optimization > > > > > > > > > > > > > > > > > > On Tue, Dec 23, 2025 at 6:45 AM xuexiaomei > > > > > > > > > <[email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > From: "[email protected]" <[email protected]> > > > > > > > > > > > > > > > > > > > > 1. Clone the aprefetch pass. When nontemporal store > > > > > > > > > > optimization is > > > > > > > > > enabled, > > > > > > > > > > first perform the analysis for nontemporal stores in > > > > > > > > > > this pass, and > > > > > > > then > > > > > > > > > > handle data alignment in the vect pass. > > > > > > > > > > 2. Attempt to retrieve the original loop's trip count > > > > > > > > > > from within an > > > > > > > OpenMP > > > > > > > > > > outlined function. > > > > > > > > > > > > > > > > > > Few high-level comments below > > > > > > > > > > > > > > > > > > > gcc/ChangeLog: > > > > > > > > > > > > > > > > > > > > * common.opt: Add -fnon-temporal-store. > > > > > > > > > > * passes.def: Add pass_loop_prefetch before > > > > > > > pass_if_conversion. > > > > > > > > > > * tree-ssa-loop-niter.cc > > > > > > > > > > (get_oldest_def_with_type_conv): New. > > > > > > > > > > (get_omp_orig_loop_niters): New. > > > > > > > > > > (traverse_niters_expr_operands): New. > > > > > > > > > > (get_const_niters_from_omp_outlined): New. > > > > > > > > > > * tree-ssa-loop-niter.h > > > > > > > > > > (get_const_niters_from_omp_outlined): New. > > > > > > > > > > * tree-ssa-loop-prefetch.cc > > > > > > > > > > (nontemporal_store_p): Attempt to get the > > > > > > > > > > preferred SIMD mode > > > > > > > > > for > > > > > > > > > > scalar type. > > > > > > > > > > (mark_nontemporal_store): Add parameter to > > > > > > > > > > control > > > > > > > > > nontemporal > > > > > > > > > > store analysis. > > > > > > > > > > (mark_nontemporal_stores): Ditto. > > > > > > > > > > (loop_prefetch_arrays): Ditto. > > > > > > > > > > (tree_ssa_prefetch_arrays): Ditto. > > > > > > > > > > (pass_loop_prefetch::execute): Ditto. > > > > > > > > > > (determine_loop_nest_reuse): Attempt to obtain > > > > > > > > > > const niters from > > > > > > > > > omp > > > > > > > > > > outlined fn. > > > > > > > > > > (unset_nontemporal_move): New > > > > > > > > > > (class pass_loop_prefetch): Add clone method and > > > > > > > > > > set pass > > > > > > > param. > > > > > > > > > > * tree-vect-data-refs.cc > > (vect_supportable_dr_alignment): > > > > > > > > > > Enforce alignment for statements marked as > > nontemporal. > > > > > > > > > > > > > > > > > > > > Signed-off-by: [email protected] <[email protected]> > > > > > > > > > > --- > > > > > > > > > > gcc/common.opt | 4 + > > > > > > > > > > gcc/passes.def | 3 +- > > > > > > > > > > gcc/tree-ssa-loop-niter.cc | 151 > > > > > > > > > ++++++++++++++++++++++++++++++++++ > > > > > > > > > > gcc/tree-ssa-loop-niter.h | 1 + > > > > > > > > > > gcc/tree-ssa-loop-prefetch.cc | 109 > > ++++++++++++++++++++---- > > > > > > > > > > gcc/tree-vect-data-refs.cc | 6 ++ > > > > > > > > > > 6 files changed, 258 insertions(+), 16 deletions(-) > > > > > > > > > > > > > > > > > > > > diff --git a/gcc/common.opt b/gcc/common.opt index > > > > > > > > > > 50de980615f..f6e9c8314a0 100644 > > > > > > > > > > --- a/gcc/common.opt > > > > > > > > > > +++ b/gcc/common.opt > > > > > > > > > > @@ -2615,6 +2615,10 @@ fprefetch-loop-arrays Common > > > > > > > > > > Var(flag_prefetch_loop_arrays) Init(-1) Optimization > > > > > > > > > > Generate prefetch instructions, if available, for arrays in > > > > > > > > > > loops. > > > > > > > > > > > > > > > > > > > > +fnon-temporal-store > > > > > > > > > > +Common Var(flag_non_temporal_store) Init(0) > > > > > > > > > > +Optimization Generate nontemporal store instructions in > > > > > > > > > > loops > > if available. > > > > > > > > > > + > > > > > > > > > > fprofile > > > > > > > > > > Common Var(profile_flag) Enable basic program > > > > > > > > > > profiling code. > > > > > > > > > > diff --git a/gcc/passes.def b/gcc/passes.def index > > > > > > > > > > fac04cd86c7..140c707b217 100644 > > > > > > > > > > --- a/gcc/passes.def > > > > > > > > > > +++ b/gcc/passes.def > > > > > > > > > > @@ -314,6 +314,7 @@ along with GCC; see the file COPYING3. > > > > > > > > > > If not > > > > > > > see > > > > > > > > > > NEXT_PASS (pass_parallelize_loops, false /* > > > > > > > > > > oacc_kernels_p > > > > > > > */); > > > > > > > > > > NEXT_PASS (pass_expand_omp_ssa); > > > > > > > > > > NEXT_PASS (pass_ch_vect); > > > > > > > > > > + NEXT_PASS (pass_loop_prefetch, true /* > > > > > > > > > > + nontemporal_store_analysis_p */); > > > > > > > > > > NEXT_PASS (pass_if_conversion); > > > > > > > > > > /* pass_vectorize must immediately follow > > > > > > > pass_if_conversion. > > > > > > > > > > Please do not add any other passes in between. > > > > > > > > > > */ @@ > > > > > > > > > > -329,7 +330,7 @@ along with GCC; see the file COPYING3. > > > > > > > > > > If not > > > > see > > > > > > > > > > NEXT_PASS (pass_dse); > > > > > > > > > > POP_INSERT_PASSES () > > > > > > > > > > NEXT_PASS (pass_slp_vectorize); > > > > > > > > > > - NEXT_PASS (pass_loop_prefetch); > > > > > > > > > > + NEXT_PASS (pass_loop_prefetch, false); > > > > > > > > > > /* Run IVOPTs after the last pass that uses > > > > > > > > > > data-reference > > > > > > > > > analysis > > > > > > > > > > as that doesn't handle TARGET_MEM_REFs. */ > > > > > > > > > > NEXT_PASS (pass_iv_optimize); diff --git > > > > > > > > > > a/gcc/tree-ssa-loop-niter.cc > > > > > > > > > > b/gcc/tree-ssa-loop-niter.cc index > > > > > > > > > > 8aa52dc54b0..44aeb0d223e 100644 > > > > > > > > > > --- a/gcc/tree-ssa-loop-niter.cc > > > > > > > > > > +++ b/gcc/tree-ssa-loop-niter.cc > > > > > > > > > > @@ -5183,6 +5183,157 @@ estimated_stmt_executions_int > > > > > > > > > > (class loop > > > > > > > > > *loop) > > > > > > > > > > return snit < 0 ? -1 : snit; } > > > > > > > > > > > > > > > > > > > > +/* Extract x from type conversion. ie: > > > > > > > > > > + a=(uint64_t)b, we can use this function to get b from a. > > > > > > > > > > +*/ > > > > > > > > > > + > > > > > > > > > > +static tree > > > > > > > > > > +get_oldest_def_with_type_conv (tree t) { > > > > > > > > > > + auto ssa = t; > > > > > > > > > > + while (ssa && TREE_CODE (ssa) == SSA_NAME) > > > > > > > > > > + { > > > > > > > > > > + auto g = SSA_NAME_DEF_STMT (ssa); > > > > > > > > > > + if (!is_gimple_assign (g)) > > > > > > > > > > + break; > > > > > > > > > > + auto rhs_code = gimple_assign_rhs_code (g); > > > > > > > > > > + if (rhs_code != NOP_EXPR && rhs_code != SSA_NAME) > > > > > > > > > > + break; > > > > > > > > > > + ssa = gimple_assign_rhs1 (g); > > > > > > > > > > + } > > > > > > > > > > + return ssa; > > > > > > > > > > +} > > > > > > > > > > + > > > > > > > > > > +/* Recursively traces the SSA variable OP to obtain the > > > > > > > > > > +original > > > > loop's > > > > > > > > > > + constant iteration count from an OpenMP outlined > > function. > > > > > > > > > > +Only > > > > > > > > > supports > > > > > > > > > > + cases where the trip count is an INTEGER_CST. The > > > > > > > > > > + trip count can only > > > > > > > > > be > > > > > > > > > > + determined when OpenMP static scheduling is used > > > > > > > > > > + without a specified > > > > > > > > > chunk > > > > > > > > > > + size. */ > > > > > > > > > > + > > > > > > > > > > +static HOST_WIDE_INT > > > > > > > > > > +get_omp_orig_loop_niters (tree op) { > > > > > > > > > > + HOST_WIDE_INT niters = -1; > > > > > > > > > > + tree rhs1 = NULL_TREE, rhs2 = NULL_TREE, fndecl, arg; > > > > > > > > > > + gimple *def; > > > > > > > > > > + gphi *phi; > > > > > > > > > > + unsigned num_args; > > > > > > > > > > + enum tree_code code; > > > > > > > > > > + > > > > > > > > > > + if (!op) > > > > > > > > > > + return niters; > > > > > > > > > > + > > > > > > > > > > + if (TREE_CODE (op) != SSA_NAME) > > > > > > > > > > + return niters; > > > > > > > > > > + > > > > > > > > > > + def = SSA_NAME_DEF_STMT (op); if (!def) > > > > > > > > > > + return niters; > > > > > > > > > > + > > > > > > > > > > + if (gimple_code (def) == GIMPLE_PHI) > > > > > > > > > > + { > > > > > > > > > > + /* Only handle PHI nodes in the outermost loop. */ > > > > > > > > > > + if (gimple_bb (def)->loop_father != > > > > current_loops->tree_root) > > > > > > > > > > + return niters; > > > > > > > > > > + > > > > > > > > > > + phi = as_a<gphi *> (def); > > > > > > > > > > + num_args = gimple_phi_num_args (phi); > > > > > > > > > > + if (num_args > 2) > > > > > > > > > > + return niters; > > > > > > > > > > + > > > > > > > > > > + /* Recursively trace PHI arguments. */ > > > > > > > > > > + for (unsigned i = 0; i < num_args; i++) > > > > > > > > > > + { > > > > > > > > > > + arg = gimple_phi_arg_def (phi, i); > > > > > > > > > > + niters = get_omp_orig_loop_niters (arg); > > > > > > > > > > + if (niters != -1) > > > > > > > > > > + break; > > > > > > > > > > + } > > > > > > > > > > + return niters; > > > > > > > > > > + } > > > > > > > > > > + > > > > > > > > > > + if (!is_gimple_assign (def)) > > > > > > > > > > + return niters; > > > > > > > > > > + code = gimple_assign_rhs_code (def); > > > > > > > > > > + rhs1 = gimple_assign_rhs1 (def); > > > > > > > > > > + > > > > > > > > > > + if (code == COND_EXPR) > > > > > > > > > > + return get_omp_orig_loop_niters (gimple_assign_rhs2 > > > > > > > > > > + (def)); > > > > > > > > > > + > > > > > > > > > > + /* Recursively trace rhs1 if not a > > > > > > > > > > + divisianalyze_tree_opson/modulo or not > > > > > > > > > a > > > > > > > > > > + * constant. */ > > > > > > > > > > + if (code != TRUNC_MOD_EXPR && code != TRUNC_DIV_EXPR) > > > > > > > > > > + return get_omp_orig_loop_niters (rhs1); else if > > > > > > > > > > + (TREE_CODE > > > > > > > > > > + (rhs1) != INTEGER_CST) > > > > > > > > > > + return get_omp_orig_loop_niters (rhs1); > > > > > > > > > > + > > > > > > > > > > + /* Find the expression q = n / nthreads or tt = n % > > > > > > > > > > nthreads. > > > > > > > > > > + */ > > > > > > > > > > + rhs2 = gimple_assign_rhs2 (def); > > > > > > > > > > + rhs2 = get_oldest_def_with_type_conv (rhs2); def = > > > > > > > > > > + SSA_NAME_DEF_STMT (rhs2); if (def && is_gimple_call > > > > > > > > > > (def)) > > > > > > > > > > + { > > > > > > > > > > + fndecl = gimple_call_fndecl (def); > > > > > > > > > > + if (fndecl > > > > > > > > > > + && (DECL_FUNCTION_CODE (fndecl) == > > > > > > > > > BUILT_IN_OMP_GET_NUM_THREADS > > > > > > > > > > + || DECL_FUNCTION_CODE (fndecl) == > > > > > > > > > BUILT_IN_OMP_GET_THREAD_NUM) > > > > > > > > > > + && tree_fits_shwi_p (rhs1)) > > > > > > > > > > + return tree_to_shwi (rhs1); > > > > > > > > > > + } > > > > > > > > > > + niters = get_omp_orig_loop_niters (rhs2); > > > > > > > > > > + > > > > > > > > > > + return niters; > > > > > > > > > > +} > > > > > > > > > > + > > > > > > > > > > +/* Recursively traverses all operands of NITERS_EXPR to > > > > > > > > > > +determine if > > > > > the > > > > > > > > > > + expression depends on the number of OpenMP threads, > > > > > > > > > > + where > > > > > > > > > NITERS_EXPR > > > > > > > > > > + is typically of the form (upper bound - lower bound > > > > > > > > > > + + 1) for a thread's > > > > > > > > > task. > > > > > > > > > > + If a dependency is found, NITERS is updated with the > > > > > > > > > > + trip count of > > > > > > > the > > > > > > > > > > + original loop corresponding to the OpenMP outlined > > function. > > > > > > > > > > + */ > > > > > > > > > > + > > > > > > > > > > +static void > > > > > > > > > > +traverse_niters_expr_operands (HOST_WIDE_INT *niters, > > > > > > > > > > +tree > > > > > > > > > > +niters_expr) { > > > > > > > > > > + if (!niters_expr) > > > > > > > > > > + return; > > > > > > > > > > + > > > > > > > > > > + unsigned n = TREE_OPERAND_LENGTH (niters_expr); if (n > > == 0) > > > > > > > > > > + { > > > > > > > > > > + *niters = get_omp_orig_loop_niters (niters_expr); > > > > > > > > > > + return; > > > > > > > > > > + } > > > > > > > > > > + > > > > > > > > > > + for (unsigned i = 0; i < n; i++) > > > > > > > > > > + { > > > > > > > > > > + tree op = TREE_OPERAND (niters_expr, i); > > > > > > > > > > + if (TREE_CODE (op) == INTEGER_CST) > > > > > > > > > > + continue; > > > > > > > > > > + > > > > > > > > > > + if (TREE_CODE (op) == NOP_EXPR) > > > > > > > > > > + op = TREE_OPERAND (op, 0); > > > > > > > > > > + traverse_niters_expr_operands (niters, op); > > > > > > > > > > + } > > > > > > > > > > +} > > > > > > > > > > + > > > > > > > > > > +/* Get the constant niters of the original loop from an > > > > > > > > > > +OpenMP > > > > > outlined > > > > > > > > > > + function. Only supports the case where niters is an > > > > INTEGER_CST. > > > > > > > > > > +*/ > > > > > > > > > > + > > > > > > > > > > +HOST_WIDE_INT > > > > > > > > > > +get_const_niters_from_omp_outlined (class loop *loop) > > > > > > > > > > > > > > > > > > I think this is something we should preserve by other > > > > > > > > > means and not have to reverse-engineer it. OMP > > > > > > > > > lowering/expansion could store the original > > > > > > > > > (constant?) number of iterations of a loop in class loop > > > > > > > > > of the loop for > > > > > > > example. > > > > > > > > > Jakub? > > > > > > > > > > > > > > > > > > > +{ > > > > > > > > > > + HOST_WIDE_INT niters = -1; > > > > > > > > > > + tree niters_expr = number_of_latch_executions (loop); > > > > > > > > > > + if (TREE_CODE (niters_expr) == COND_EXPR) > > > > > > > > > > + niters_expr = TREE_OPERAND (niters_expr, 1); > > > > > > > > > > + > > > > > > > > > > + niters_expr = analyze_scalar_evolution (loop, > > > > > > > > > > + niters_expr); if (!niters_expr || > > > > > > > > > > + chrec_contains_undetermined > > > > (niters_expr)) > > > > > > > > > > + return niters; > > > > > > > > > > + > > > > > > > > > > + traverse_niters_expr_operands (&niters, niters_expr); > > > > > > > > > > + return niters; > > > > > > > > > > +} > > > > > > > > > > + > > > > > > > > > > /* Sets NIT to the maximum number of executions of the > > > > > > > > > > latch of > > > > > the > > > > > > > > > > LOOP, plus one. If we have no reliable estimate, > > > > > > > > > > the function > > > > > > > returns > > > > > > > > > > false, otherwise returns true. */ diff --git > > > > > > > > > > a/gcc/tree-ssa-loop-niter.h b/gcc/tree-ssa-loop-niter.h > > > > > > > > > > index > > > > > > > > > > 88e47b9ab94..a564bacf941 100644 > > > > > > > > > > --- a/gcc/tree-ssa-loop-niter.h > > > > > > > > > > +++ b/gcc/tree-ssa-loop-niter.h > > > > > > > > > > @@ -38,6 +38,7 @@ extern tree loop_niter_by_eval (class > > > > > > > > > > loop *, edge); extern tree find_loop_niter_by_eval > > > > > > > > > > (class loop *, edge *); extern bool > > > > > > > > > > estimated_loop_iterations (class loop *, widest_int *); > > > > > > > > > > extern HOST_WIDE_INT estimated_loop_iterations_int > > > > > > > > > > (class loop *); > > > > > > > > > > +extern HOST_WIDE_INT get_const_niters_from_omp_outlined > > > > > > > > > > +(class loop *); > > > > > > > > > > extern bool max_loop_iterations (class loop *, > > > > > > > > > > widest_int *); extern HOST_WIDE_INT > > > > > > > > > > max_loop_iterations_int (class loop *); extern bool > > > > > > > > > > likely_max_loop_iterations (class loop *, widest_int *); > > > > > > > > > > diff --git a/gcc/tree-ssa-loop-prefetch.cc > > > > > > > > > > b/gcc/tree-ssa-loop-prefetch.cc index > > > > > > > > > > dad53bfb79b..fd83382905f > > > > > > > > > > 100644 > > > > > > > > > > --- a/gcc/tree-ssa-loop-prefetch.cc > > > > > > > > > > +++ b/gcc/tree-ssa-loop-prefetch.cc > > > > > > > > > > @@ -48,6 +48,7 @@ along with GCC; see the file COPYING3. > > > > > > > > > > If not see #include "tree-data-ref.h" > > > > > > > > > > #include "diagnostic-core.h" > > > > > > > > > > #include "dbgcnt.h" > > > > > > > > > > +#include "tree-parloops.h" > > > > > > > > > > > > > > > > > > > > /* This pass inserts prefetch instructions to optimize > > > > > > > > > > cache usage > > > > > during > > > > > > > > > > accesses to arrays in loops. It processes loops > > > > > > > > > > sequentially > > > > and: > > > > > > > > > > @@ -1245,7 +1246,7 @@ issue_prefetches (struct > > > > > > > > > > mem_ref_group > > > > > > > *groups, > > > > > > > > > > can be used. */ > > > > > > > > > > > > > > > > > > > > static bool > > > > > > > > > > -nontemporal_store_p (struct mem_ref *ref) > > > > > > > > > > +nontemporal_store_p (struct mem_ref *ref, bool > > > > > > > > > > +nontemporal_store_analysis_p) > > > > > > > > > > { > > > > > > > > > > machine_mode mode; > > > > > > > > > > enum insn_code code; > > > > > > > > > > @@ -1264,6 +1265,20 @@ nontemporal_store_p (struct > > > > > > > > > > mem_ref > > > > > *ref) > > > > > > > > > > return false; > > > > > > > > > > > > > > > > > > > > code = optab_handler (storent_optab, mode); > > > > > > > > > > + > > > > > > > > > > + /* If there is no nontemporal store instruction for > > > > > > > > > > + the current scalar > > > > > > > > > mode, > > > > > > > > > > + then try to get the preferred SIMD (vector) mode > > > > > > > > > > + for this scalar > > > > > > > type > > > > > > > > > > + and check if there is a corresponding nontemporal > > > > > > > > > > + store instruction. */ if > > > > > > > > > > + (nontemporal_store_analysis_p && code == > > > > > > > > > CODE_FOR_nothing > > > > > > > > > > + && is_a<scalar_mode> (mode)) > > > > > > > > > > + { > > > > > > > > > > + scalar_mode smode = as_a<scalar_mode> (mode); > > > > > > > > > > + machine_mode simd_mode = > > > > > > > > > targetm.vectorize.preferred_simd_mode > > > > > > > > > > + (smode); > > > > > > > > > > + > > > > > > > > > > > > > > > > > > There might be other SIMD modes, and you don't know > > > > > > > > > whether the store will be vectorized or not. In the > > > > > > > > > description you say we'll expand to a non-NT store anyway, > > > > > > > > > if it doesn't end up supported, so why > > > > > > > bother checking here? > > > > > > > > > > > > > > > > > The goal of the preferred_simd_mode + nontemporal capability > > > > > > > > check is to > > > > > > > avoid being overly conservative when scalar NT stores are > > > > > > > unavailable but vector NT stores are still possible. > > > > > > > > > > > > > > > > For example: > > > > > > > > typedef unsigned char DP; > > > > > > > > void __attribute__ ((noinline)) test () { > > > > > > > > ssize_t j; > > > > > > > > for (j = 0; j < 1000000; j++) > > > > > > > > { > > > > > > > > c[j] = a[j] + b[j]; > > > > > > > > } > > > > > > > > } > > > > > > > > On i386, there is no scalar NT store support for byte types, > > > > > > > > so a scalar-only > > > > > > > check would reject NT stores. > > > > > > > > However, this loop can be vectorized (e.g. QImode → > > > > > > > > V16QImode), and the > > > > > > > resulting vector store may be lowered to an NT-capable > > > > > > > instruction if supported by the ISA. This keeps NT store as a > > > > > > > valid > > optimization path. > > > > > > > > In addition, marking scalar as NT-capable also serves as a > > > > > > > > hint for > > > > > > > vectorization alignment and helps guide generation of > > > > > > > streaming stores when profitable. > > > > > > > > > > > > > > > > > Iff, then I'd gate processing on the availability on any > > > > > > > > > storent expander, but then the targets default of having > > > > > > > > > the pass enabled or not should also work to that extent? > > > > > > > > > > > > > > > > > > > + if (VECTOR_MODE_P (simd_mode)) > > > > > > > > > > + code = optab_handler (storent_optab, simd_mode); > > > > > > > > > > + } > > > > > > > > > > + > > > > > > > > > > return code != CODE_FOR_nothing; } > > > > > > > > > > > > > > > > > > > > @@ -1271,9 +1286,9 @@ nontemporal_store_p (struct > > > > > > > > > > mem_ref > > > > > *ref) > > > > > > > > > > and return true. Otherwise, we return false. */ > > > > > > > > > > > > > > > > > > > > static bool > > > > > > > > > > -mark_nontemporal_store (struct mem_ref *ref) > > > > > > > > > > +mark_nontemporal_store (struct mem_ref *ref, bool > > > > > > > > > > +nontemporal_store_analysis_p) > > > > > > > > > > { > > > > > > > > > > - if (!nontemporal_store_p (ref)) > > > > > > > > > > + if (!nontemporal_store_p (ref, > > > > > > > > > > + nontemporal_store_analysis_p)) > > > > > > > > > > return false; > > > > > > > > > > > > > > > > > > > > if (dump_file && (dump_flags & TDF_DETAILS)) @@ > > > > > > > > > > -1343,7 > > > > > > > > > > +1358,8 > > > > > > > > > @@ > > > > > > > > > > may_use_storent_in_loop_p (class loop *loop) > > > > > > > > > > references in the loop. Returns whether we inserted > > > > > > > > > > any mfence call. */ > > > > > > > > > > > > > > > > > > > > static bool > > > > > > > > > > -mark_nontemporal_stores (class loop *loop, struct > > > > > > > > > > mem_ref_group > > > > > > > > > > *groups) > > > > > > > > > > +mark_nontemporal_stores (class loop *loop, struct > > > > > > > > > > +mem_ref_group > > > > > > > > > *groups, > > > > > > > > > > + bool > > > > > > > > > > + nontemporal_store_analysis_p) > > > > > > > > > > > > > > > > > > I'd really separate NT store processing from prefetching, > > > > > > > > > you only seem to disable it half-way. Why? > > > > > > > > > > > > > > > > The reason is to keep the change minimal and reuse the > > > > > > > > existing prefetch > > > > > > > infrastructure. Do you suggest making a separated function for > > > > > > > this? > > > > > > > > > > > > > > As indicated below .... > > > > > > > > > > > > > > > > > > > > > > > > > > { > > > > > > > > > > struct mem_ref *ref; > > > > > > > > > > bool any = false; > > > > > > > > > > @@ -1353,9 +1369,12 @@ mark_nontemporal_stores (class > > > > > > > > > > loop *loop, struct mem_ref_group *groups) > > > > > > > > > > > > > > > > > > > > for (; groups; groups = groups->next) > > > > > > > > > > for (ref = groups->refs; ref; ref = ref->next) > > > > > > > > > > - any |= mark_nontemporal_store (ref); > > > > > > > > > > + any |= mark_nontemporal_store (ref, > > > > > > > > > > + nontemporal_store_analysis_p); > > > > > > > > > > > > > > > > > > > > - if (any && FENCE_FOLLOWING_MOVNT != NULL_TREE) > > > > > > > > > > + /* Avoid inserting excessive fence instructions, > > > > > > > > > > + insert only in the > > > > > > > > > aprefetch2 > > > > > > > > > > + pass. */ > > > > > > > > > > + if (!nontemporal_store_analysis_p > > > > > > > > > > + && any && FENCE_FOLLOWING_MOVNT != NULL_TREE) > > > > > > > > > > { > > > > > > > > > > emit_mfence_after_loop (loop); > > > > > > > > > > return true; > > > > > > > > > > @@ -1653,6 +1672,10 @@ determine_loop_nest_reuse (class > > > > > > > > > > loop *loop, struct mem_ref_group *refs, > > > > > > > > > > > > > > > > > > > > aloop = vloops[i]; > > > > > > > > > > vol = estimated_stmt_executions_int (aloop); > > > > > > > > > > + > > > > > > > > > > + if (vol == -1 && flag_openmp && > > > > > > > > > > + parallelized_function_p > > > > > > > > > (cfun->decl)) > > > > > > > > > > + vol = get_const_niters_from_omp_outlined > > > > > > > > > > + (aloop); > > > > > > > > > > > > > > > > > > I'll note that heuristics are based on cache size params, > > > > > > > > > but for OMP parallelized code that will be off, since the > > > > > > > > > cache sharing info across threads isn't known. This at > > > > > > > > > least deserves a > > > > > comment. > > > > > > > > > > > > > > > > > > > + > > > > > > > > > > if (vol == -1) > > > > > > > > > > vol = expected_loop_iterations (aloop); > > > > > > > > > > volume *= vol; > > > > > > > > > > @@ -1886,7 +1909,8 @@ insn_to_prefetch_ratio_too_small_p > > > > > > > (unsigned > > > > > > > > > ninsns, unsigned prefetch_count, > > > > > > > > > > to update SSA for virtual operands and LC SSA for a > > > > > > > > > > split > > edge. > > > > > > > > > > */ > > > > > > > > > > > > > > > > > > > > static bool > > > > > > > > > > -loop_prefetch_arrays (class loop *loop, bool > > > > > > > > > > &need_lc_ssa_update) > > > > > > > > > > +loop_prefetch_arrays (class loop *loop, bool > > &need_lc_ssa_update, > > > > > > > > > > + bool > > > > > > > > > > +nontemporal_store_analysis_p) > > > > > > > > > > { > > > > > > > > > > struct mem_ref_group *refs; > > > > > > > > > > unsigned ahead, ninsns, time, unroll_factor; @@ > > > > > > > > > > -1927,7 > > > > > > > > > > +1951,8 @@ loop_prefetch_arrays (class loop *loop, bool > > > > > &need_lc_ssa_update) > > > > > > > > > > /* Give up prefetching if the number of memory > > > > > > > > > > references in > > > > the > > > > > > > > > > loop is not reasonable based on profitablity and > > > > > > > > > > compilation > > > > > time > > > > > > > > > > considerations. */ > > > > > > > > > > - if (!mem_ref_count_reasonable_p (ninsns, > > > > > > > > > > mem_ref_count)) > > > > > > > > > > + if (!nontemporal_store_analysis_p > > > > > > > > > > + && !mem_ref_count_reasonable_p (ninsns, > > > > > > > > > > + mem_ref_count)) > > > > > > > > > > goto fail; > > > > > > > > > > > > > > > > > > > > /* Step 2: estimate the reuse effects. */ @@ > > > > > > > > > > -1957,11 > > > > > > > > > > +1982,18 @@ loop_prefetch_arrays (class loop *loop, bool > > > > > > > > > > &need_lc_ssa_update) > > > > > > > > > > > > > > > > > > > > /* Prefetching is not likely to be profitable if the > > > > > > > > > > instruction to > > > > > > > prefetch > > > > > > > > > > ratio is too small. */ > > > > > > > > > > - if (insn_to_prefetch_ratio_too_small_p (ninsns, > > > > > > > > > > prefetch_count, > > > > > > > > > > + if (!nontemporal_store_analysis_p && > > > > > > > > > > + insn_to_prefetch_ratio_too_small_p (ninsns, > > > > > > > > > > + prefetch_count, > > > > > > > > > > unroll_factor)) > > > > > > > > > > goto fail; > > > > > > > > > > > > > > > > > > > > - need_lc_ssa_update |= mark_nontemporal_stores (loop, > > > > > > > > > > refs); > > > > > > > > > > + need_lc_ssa_update > > > > > > > > > > + |= mark_nontemporal_stores (loop, refs, > > > > > > > > > > + nontemporal_store_analysis_p); > > > > > > > > > > + > > > > > > > > > > + /* With nontemporal store optimization enabled, only > > > > > > > > > > + the analysis phase > > > > > > > > > is > > > > > > > > > > + performed during the aprefetch1 pass. */ if > > > > > > > > > > + (nontemporal_store_analysis_p) > > > > > > > > > > + goto fail; > > > > > > > > > > > > > > > > > > > > /* Step 4: what to prefetch? */ > > > > > > > > > > if (!schedule_prefetches (refs, unroll_factor, > > > > > > > > > > ahead)) @@ > > > > > > > > > > -1983,10 > > > > > > > > > > +2015,33 @@ fail: > > > > > > > > > > return unrolled; > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > +/* Cancel the nontemporal store marked in aprefetch1 pass. > > > > > > > > > > +*/ > > > > > > > > > > + > > > > > > > > > > +static void > > > > > > > > > > +unset_nontemporal_move (void) { > > > > > > > > > > + basic_block bb; > > > > > > > > > > + gimple_stmt_iterator bsi; > > > > > > > > > > + gimple *stmt; > > > > > > > > > > + FOR_EACH_BB_FN (bb, cfun) > > > > > > > > > > + { > > > > > > > > > > + for (bsi = gsi_start_bb (bb); !gsi_end_p (bsi); > > > > > > > > > > gsi_next > > (&bsi)) > > > > > > > > > > + { > > > > > > > > > > + stmt = gsi_stmt (bsi); > > > > > > > > > > + if (!is_gimple_assign (stmt)) > > > > > > > > > > + continue; > > > > > > > > > > + > > > > > > > > > > + gassign *assign_stmt = as_a<gassign *> (stmt); > > > > > > > > > > + if (gimple_assign_nontemporal_move_p > > (assign_stmt)) > > > > > > > > > > + gimple_assign_set_nontemporal_move > > > > > > > > > > + (assign_stmt, > > > > > false); > > > > > > > > > > + } > > > > > > > > > > + } > > > > > > > > > > +} > > > > > > > > > > + > > > > > > > > > > /* Issue prefetch instructions for array references in > > > > > > > > > > loops. > > > > > > > > > > */ > > > > > > > > > > > > > > > > > > > > unsigned int > > > > > > > > > > -tree_ssa_prefetch_arrays (void) > > > > > > > > > > +tree_ssa_prefetch_arrays (bool > > > > > > > > > > +nontemporal_store_analysis_p) > > > > > > > > > > { > > > > > > > > > > bool unrolled = false; > > > > > > > > > > bool need_lc_ssa_update = false; @@ -2033,12 +2088,21 > > > > > > > > > > @@ tree_ssa_prefetch_arrays (void) > > > > > > > > > > set_builtin_decl (BUILT_IN_PREFETCH, decl, false); > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > + /* In the aprefetch1 pass, stmts that are > > > > > > > > > > + successfully analyzed are > > > > > > > marked > > > > > > > > > > + with nontemporal store, but no fence is inserted. > > > > > > > > > > + The main > > > > > > > purpose is > > > > > > > > > > + to complete alignment operations in the vect pass. > > > > > > > > > > + In the > > > > > > > aprefetch2 > > > > > > > > > > + pass, the nontemporal store flag is removed, and > > > > > > > > > > + the stmts > > > > are > > > > > > > > > > + re-analyzed under the condition of alignment. */ > > > > > > > > > > > > > > > > > > Hmm, but then, if you only make this to guide how > > > > > > > > > vectorization behaves, why not adjust the heuristics in > > > > > > > > > vect_enhance_data_refs_alignment? That also knows > > > > > > > > > write-only > > > > data. > > > > > > > > > It would seem to be a much more natural place to honor a > > > > > > > > > later request to > > > > > > > use NT stores. > > > > > > > > > > > > > > ... I'd rather have this not in a new pass but integrated into > > > > > > > the vectorizer alignment peeling/versioning heuristic. > > > > > > > > > > > > > > Richard. > > > > > > > > > > > > > > > > Richard. > > > > > > > > > > > > > > > > > > > + if (!nontemporal_store_analysis_p) > > > > > > > > > > + unset_nontemporal_move (); > > > > > > > > > > + > > > > > > > > > > for (auto loop : loops_list (cfun, LI_FROM_INNERMOST)) > > > > > > > > > > { > > > > > > > > > > if (dump_file && (dump_flags & TDF_DETAILS)) > > > > > > > > > > fprintf (dump_file, "Processing loop %d:\n", > > > > > > > > > > loop->num); > > > > > > > > > > > > > > > > > > > > - unrolled |= loop_prefetch_arrays (loop, > > > > need_lc_ssa_update); > > > > > > > > > > + unrolled |= loop_prefetch_arrays (loop, > > > > > > > > > > + need_lc_ssa_update, > > > > > > > > > > + > > > > > > > > > > + nontemporal_store_analysis_p); > > > > > > > > > > > > > > > > > > > > if (dump_file && (dump_flags & TDF_DETAILS)) > > > > > > > > > > fprintf (dump_file, "\n\n"); @@ -2078,15 > > > > > > > > > > +2142,30 @@ class pass_loop_prefetch : public > > > > > > > > > > gimple_opt_pass { > > > > > > > > > > public: > > > > > > > > > > pass_loop_prefetch (gcc::context *ctxt) > > > > > > > > > > - : gimple_opt_pass (pass_data_loop_prefetch, ctxt) > > > > > > > > > > + : gimple_opt_pass (pass_data_loop_prefetch, ctxt), > > > > > > > > > > + nontemporal_store_analysis_p (false) > > > > > > > > > > {} > > > > > > > > > > > > > > > > > > > > /* opt_pass methods: */ > > > > > > > > > > + opt_pass *clone () { return new pass_loop_prefetch > > > > > > > > > > + (m_ctxt); } > > > > > > > > > > bool gate (function *) final override > > > > > > > > > > { > > > > > > > > > > - return flag_prefetch_loop_arrays > 0; > > > > > > > > > > + if (nontemporal_store_analysis_p) > > > > > > > > > > + return optimize >= 3 && flag_non_temporal_store > > > > > > > > > > + && (flag_tree_loop_vectorize || > > > > > > > > > cfun->has_force_vectorize_loops); > > > > > > > > > > + else > > > > > > > > > > + return flag_prefetch_loop_arrays > 0 > > > > > > > > > > + || (optimize >= 3 && > > > > > > > > > > + flag_non_temporal_store); > > > > > > > > > > } > > > > > > > > > > unsigned int execute (function *) final override; > > > > > > > > > > + void set_pass_param (unsigned int n, bool param) { > > > > > > > > > > + gcc_assert (n == 0); > > > > > > > > > > + nontemporal_store_analysis_p = param; } > > > > > > > > > > + > > > > > > > > > > +private: > > > > > > > > > > + bool nontemporal_store_analysis_p; > > > > > > > > > > > > > > > > > > > > }; // class pass_loop_prefetch > > > > > > > > > > > > > > > > > > > > @@ -2110,7 +2189,7 @@ pass_loop_prefetch::execute > > > > > > > > > > (function > > > > > *fun) > > > > > > > > > > return 0; > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > - return tree_ssa_prefetch_arrays (); > > > > > > > > > > + return tree_ssa_prefetch_arrays > > > > > > > > > > + (nontemporal_store_analysis_p); > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > } // anon namespace > > > > > > > > > > diff --git a/gcc/tree-vect-data-refs.cc > > > > > > > > > > b/gcc/tree-vect-data-refs.cc index > > > > > > > > > > 89a9077af0d..afc86d15cd5 > > > > > > > > > > 100644 > > > > > > > > > > --- a/gcc/tree-vect-data-refs.cc > > > > > > > > > > +++ b/gcc/tree-vect-data-refs.cc > > > > > > > > > > @@ -6787,6 +6787,12 @@ vect_supportable_dr_alignment > > > > > > > > > > (vec_info *vinfo, > > > > > > > > > dr_vec_info *dr_info, > > > > > > > > > > else if (dr_safe_speculative_read_required (stmt_info)) > > > > > > > > > > return dr_unaligned_unsupported; > > > > > > > > > > > > > > > > > > > > + /* When using the vmovnt instruction, the data > > > > > > > > > > + address must be aligned > > > > > > > > > to the > > > > > > > > > > + vector width. */ > > > > > > > > > > + gassign *assign_stmt = dyn_cast<gassign *> > > > > > > > > > > + (stmt_info->stmt); if (assign_stmt && > > > > > gimple_assign_nontemporal_move_p (assign_stmt)) > > > > > > > > > > + return dr_unaligned_unsupported; > > > > > > > > > > + > > > > > > > > > > if (loop_vinfo) > > > > > > > > > > { > > > > > > > > > > vect_loop = LOOP_VINFO_LOOP (loop_vinfo); > > > > > > > > > > -- > > > > > > > > > > 2.22.0 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
