On Fri, 5 Jun 2026, Tamar Christina wrote:
> NOTE: This branch is still very much WIP. It has not been cleaned up enough
> to be upstreamable yet, but I am sending it now to give Richi an idea of the
> design direction.
>
> This should apply to any reasonably recent trunk. I am currently based on
> g:c8ca8e647caf945dc4694107884cf0f4eba5da2f.
>
> The branch builds and runs, but it does not bootstrap. I have only tested
> incremental builds.
>
> Some parts are more worked out than others, since I have not reached all of
> them yet. I have tried to write down the intended direction as well, so that
> you can give me "the look", Richi :)
>
> I *also* want to point out that the AArch64 code here is just illustrative and
> not how we will implement these. Work is ongoing to replace our intrinsics
> with
> ones that generate new overloaded builtins, to allow legality checks to be
> cleaner and also not need the type shims between the internal vectorizer
> represenations and intrinsics. i.e. atm we need a type shim between
>
> int32x4_t and a signed type made form V4SI. This will be handled by adding
> to the build-ins internal overloads with the correct types.
>
> ### Introduction
>
> The main intended purpose of this work is threefold:
>
> 1. Support re-optimizing intrinsics and vector code in SLP that was created by
> earlier vectorization passes. For example, two back-to-back loops could be
> completely unrolled and vectorized during loop vectorization, but would not
> then be optimized together. The idea is to let SLP discovery re-analyze
> the
> vector code as a single entity. This makes the long-standing question of
> whether BB SLP or loop vectorization should run first less important, since
> we can re-vectorize the generated vector code.
>
> 2. Support Arm SVE2's Hybrid Vector Length Agnostic (HVLA) programming model.
> The intention is to transform code written with intrinsics for older
> AdvSIMD
> code into VLA code, so that we can use longer vector lengths.
>
> 3. Support workloads, particularly scientific-computing workloads, in which
> users manually create vector types that are longer than the target supports
> using GCC vector extensions. The expectation is that the compiler can then
> decompose those operations into supported operations. Clang can do this
> for
> some examples, although its approach seems quite limited and appears to
> work
> only when the operations are constructors of PHI nodes.
I hope to replace "optimized" vector lowering with re-vectorization,
possibly only keep lowering to scalar operations.
> To solve these problems, the approach I have taken in GCC is to model vector
> types as possible base "scalar" types. The main advantage is that the current
> compiler workflows can stay the same. Fundamentally, this does not change
> most
> of the vectorizer, apart from analysis and the usual extensions to the code
> generators.
>
> This also means that the code remains subject to cost modelling and to other
> features we have added over the years, such as epilogue masks.
>
> As an example:
>
> ```c
> #include <arm_neon.h>
>
> void foo (uint8_t *a, uint8_t *b, uint8_t *c, int n)
> {
> for (int i = 0; i < n; i += 16, a += 16, b += 16, c += 16)
> {
> uint8x8_t av1 = vld1_u8 (a);
> uint8x8_t av2 = vld1_u8 (a + 8);
> uint8x8_t bv1 = vld1_u8 (b);
> uint8x8_t bv2 = vld1_u8 (b + 8);
> vst1_u8 (c, vadd_u8 (av1, bv1));
> vst1_u8 (c + 8, vadd_u8 (av2, bv2));
> }
> }
> ```
>
> At the moment, `-O3` generates:
>
> ```asm
> .L3:
> ldr d29, [x0, x4]
> ldr d28, [x1, x4]
> ldr d31, [x7, x4]
> ldr d30, [x6, x4]
> add v28.8b, v29.8b, v28.8b
> add v30.8b, v31.8b, v30.8b
> str d28, [x2, x4]
> str d30, [x5, x4]
> add x4, x4, 16
> cmp w3, w4
> bgt .L3
> ```
>
> This underuses the available load bandwidth. Ideally these would be Q-sized
> loads and one ADD, i.e. SLP would combine them.
>
> With the branch, `-O3 -march=armv8-a` generates:
>
> ```asm
> .L5:
> ldr q31, [x0, x4]
> ldr q30, [x1, x4]
> add v30.16b, v31.16b, v30.16b
> str q30, [x2, x4]
> add x4, x4, 16
> cmp x4, x7
> bne .L5
> ```
>
> `-O3 -march=armv8-a+sve` generates:
>
> ```asm
> .L5:
> ldr q31, [x0, x3]
> ldr q30, [x1, x3]
> add v30.16b, v31.16b, v30.16b
> str q30, [x2, x3]
> add x3, x3, 16
> cmp x5, x3
> bne .L5
> lsl w3, w4, 4
> cmp w6, w3
> beq .L1
> .L4:
> sub w6, w6, w3
> ubfiz x3, x3, 3, 32
> whilelo p7.b, xzr, x6
> ld1b z29.b, p7/z, [x0, x3]
> ld1b z28.b, p7/z, [x1, x3]
> add z28.b, z28.b, z29.b
> st1b z28.b, p7, [x2, x3]
> .L1:
> ```
>
> `-O3 -march=armv8-a+sve -mautovec-preference=sve-only` generates:
>
> ```asm
> .L4:
> ld1b z31.b, p7/z, [x0, x4]
> ld1b z30.b, p7/z, [x1, x4]
> add z30.b, z30.b, z31.b
> st1b z30.b, p7, [x2, x4]
> add x4, x4, x5
> whilelo p7.b, x4, x3
> b.any .L4
> ```
>
> ### Representation
>
> There is no change to the SLP trees themselves. The vector operation is added
> to the node with N copies. `print_slp` will be updated to show that these are
> vector lanes rather than individual scalar operations. This is a
> representational thing since SLP internal nodes are vectorized from the
> representation and not the scalar elements, but it makes everything line up.
>
> The group size for the operation is then set to the number of units in the
> incoming vector. For memory accesses, `dr_group_size` is increased too. At
> the moment, and probably for the first version, I only plan to support linear
> loads and load-lane operations. Recognizing the other load types requires a
> target hook and time to fill in the DRs.
I think we need to nail this at least. So for a v4si add you end up
with 4 times the vector add in SLP_TREE_SCALAR_STMTS, one stmt
per resulting vector lane. I see how this is the least problematic
representation, but is it a good one? In most cases we do not
absolutely need an entry in SLP_TREE_SCALAR_STMTS. The other
"least" problematic representation would be to have no
SLP_TREE_SCALAR_STMTS. I think I originally suggested to forfeit
the one lane, one SLP_TREE_SCALAR_STMTS entry invariant so you'd
have a single vector stmt entry in SLP_TREE_SCALAR_STMTS.
We do currently have (redudnant) SLP_TREE_LANES () that is most
of the time equal to 'group size' which is currently most of the
time equal to SLP_TREE_SCALAR_STMTS.length (). This is somewhat
of a partial transition worth completing with this work.
BB external "vectors" currently populate SLP_TREE_VEC_DEFS, not
SLP_TREE_SCALAR_STMTS, they are vect_external and facing SLP
we always have a VEC_PERM node. That would be a very minimal
case of "vector SLP" to handle in the new representation.
DRs group-size adjusting should be done based on scalar elements,
we do eventually need to track gaps between two vector loads and
code that advances the number by 1 + GAP needs to be adjusted to
advance by <size> + GAP. For group element 0 we overload
DR_GROUP_SIZE, for elts > 0 it's basically unused, so we'd want
a DR_GROUP_ELT_SIZE I guess.
+ /* We should probably adjust group size, but for now lets hack. */
+ unsigned HOST_WIDE_INT vect_nunits = 0;
+ auto vscalar_stmt = STMT_VINFO_STMT (stmt_info);
+ tree vscalar_type = TREE_TYPE (gimple_get_lhs (vscalar_stmt));
+ if (VECTOR_TYPE_P (vscalar_type)
+ && TYPE_VECTOR_SUBPARTS (vscalar_type).is_constant (&vect_nunits))
+ dr_group_size *= vect_nunits;
DR_GROUP_SIZE should be the size in elements. I'll note the above
suggests that you currently do not (want to?) handle mixed type
scalar stmts in an SLP node? Like re-vectorizing V2SI + SI
to predicated tail V4SI?
> For BB SLP, the current code starts a seed whenever it finds a store with a
> vector type that the target does not support. This then fails analysis
> because
> no vector type can be found. The plan is to make vector type selection return
> the largest supported type for the given inner mode. During `vectorizable_*`
> analysis, we would then avoid doing any other check beyond whether the target
> supports the new mode for the given operation. To match the incoming VF, we
> would increase the required unroll factor so that we get the same effective
> VF.
>
> Decreasing the VF is not possible at the moment. For example:
>
> ```c
> #include <arm_neon.h>
>
> void foo (uint8_t *a, uint8_t *b, uint8_t *c, int n)
> {
> for (int i = 0; i < n; i += 32, a += 32, b += 32, c += 32)
> {
> uint8x16_t av1 = vld1q_u8 (a);
> uint8x16_t av2 = vld1q_u8 (a + 16);
> uint8x16_t bv1 = vld1q_u8 (b);
> uint8x16_t bv2 = vld1q_u8 (b + 16);
> vst1q_u8 (c, vaddq_u8 (av1, bv1));
> vst1q_u8 (c + 16, vaddq_u8 (av2, bv2));
> }
> }
> ```
>
> cannot collapse down to:
>
> ```asm
> .L5:
> ldr q31, [x0, x4]
> ldr q30, [x1, x4]
> add v30.16b, v31.16b, v30.16b
> str q30, [x2, x4]
> add x4, x4, 16
> cmp x4, x7
> bne .L5
> ```
>
> This would require fractional VF support; in other words, it would require
> support for re-rolling the loop. With the current design, that should work
> naturally once we add fractional VF support.
>
> ### Areas that need the most changes
>
> These are the areas that I think need the most work. The branch is currently
> structured so that the number-of-units adjustment is done in each callee
> rather
> than when building the SLP representation. This is purely temporary, and
> there
> are `#if 1`s around the temporary code so that I can remove it later.
>
> This lets me change the functions one at a time instead of having everything
> ICE because asserts are hit in areas that I have not reached yet.
>
> ### Vector Operations
>
> There are two types of vector operations that we want to support here.
>
> The first type uses a normal GIMPLE representation. In the examples above,
> `vadd` is lowered as a normal `PLUS_EXPR`, so testing and transforming it is
> straightforward.
>
> In many other cases, however, we see target intrinsics. To vectorize these, I
> have added a new `vectorizable_vect_call` that uses a target hook,
> `targetm.vectorize.translate_builtin_call`, to fill in vectorization
> information about the call. In particular, the hook must be able to tell the
> vectorizer whether we are going from an unmasked mode to a masked mode,
> whether
> there is a builtin with masking support that we can use, and where the mask
> argument appears in the argument list.
So with all the fancy stuff below I think we should start with a core
set that IMO should tackle basic vector operations, loads and stores
and those BB external vector data inputs.
Vector bools are indeed special but also important, it looks like what
you do is reasonable.
Given there's overlap (I'll cross-review that now) with SLP scalar
stmts and DR represnetation in the predicated tail vectorization
series getting this part correct will avoid future churn.
I see "target hook" mentioned too many times ;)
> ### Vector Booleans
>
> Normally, the vectorizer has to detect booleans in order to differentiate data
> operations from control operations. When vectorizing existing vector code,
> that has already been done for us. However, we still need to be able to
> expand
> the lanes and, if necessary, convert between boolean representation models.
>
> For example, when changing AdvSIMD code into SVE code, the boolean
> representation has to change because the underlying truth type is different.
>
> To fix this and support boolean operations, there is a new pattern,
> `vect_recog_vect_bool_pattern`, that runs only on boolean vector comparisons.
> It transforms them using a pattern as a scalar boolean. That is, it creates:
>
> ```c
> cmp ? -1 : 0
> ```
>
> from a vector comparison, with the type of the operation set to the truth type
> of the original code. This allows us to vectorize these as if they were
> normal
> scalar comparisons.
>
> For example:
>
> ```c
> #include <arm_neon.h>
>
> void foo (uint8_t *a, uint8_t *b, uint8_t *c, int n)
> {
> for (int i = 0; i < n; i += 8, a += 8, b += 8, c += 8)
> {
> uint8x8_t av1 = vld1_u8 (a);
> uint8x8_t bv1 = vld1_u8 (b);
> vst1_u8 (c, vcgt_u8 (av1, bv1));
> }
> }
> ```
>
> compiled with `-march=armv9-a+sve -O3` gives:
>
> ```asm
> .L5:
> ldr q31, [x1, x3]
> ldr q30, [x0, x3]
> cmhi v30.16b, v30.16b, v31.16b
> str q30, [x2, x3]
> add x3, x3, 16
> cmp x3, x5
> bne .L5
> lsl w3, w4, 4
> cmp w6, w3
> beq .L1
> .L4:
> sub w6, w6, w3
> ptrue p6.b, all
> ubfiz x3, x3, 3, 32
> whilelo p7.b, xzr, x6
> ld1b z28.b, p7/z, [x0, x3]
> ld1b z27.b, p7/z, [x1, x3]
> cmplo p15.b, p6/z, z27.b, z28.b
> mov z29.b, p15/z, #-1
> st1b z29.b, p7, [x2, x3]
> .L1:
> ret
> .p2align 2,,3
> .L6:
> mov x4, 0
> .p2align 5,,15
> .L3:
> ldr d26, [x0, x4]
> ldr d25, [x1, x4]
> cmhi v25.8b, v26.8b, v25.8b
> str d25, [x2, x4]
> add x4, x4, 8
> cmp w3, w4
> bgt .L3
> ret
> .L7:
> mov w3, 0
> b .L4
> ```
>
> The vectorizer is now able to handle the conversion between different boolean
> modes while preserving the lane values of the original code.
>
> ### Vector Patterns
>
> The incoming code may use vector operations to perform masking through
> manually
> if-converted code. For example:
>
> ```c
> #include <arm_neon.h>
> #include <arm_sve.h>
>
> void foo (uint8_t *a, uint8_t *b, uint8_t *c, int n)
> {
> for (int i = 0; i < n; i += 8, a += 8, b += 8, c += 8)
> {
> uint8x8_t av1 = vld1_u8 (a);
> uint8x8_t bv1 = vld1_u8 (b);
> uint8x8_t cmp = vcgt_u8 (bv1, vdup_n_u8 (0));
> uint8x8_t add = vadd_u8 (bv1, vdup_n_u8 (1));
> uint8x8_t sel = vbsl_u8 (cmp, add, bv1);
> vst1_u8 (a, sel);
> }
> }
> ```
>
> This is a manually if-converted conditional add. We can re-vectorize it
> as-is,
> but what we really want is to optimize it as a masked operation.
>
> Since the operations are opaque to the vectorizer, there is a new target hook,
> `targetm.vectorize.recog_patterns_builtin_call`, and a pattern,
> `vect_recog_vect_scalar_pattern`, that allow the backend to define which
> sequence of operations should become a vectorizer pattern.
How much of this can we handle with SLP patterns? I think it would be
useful that we can recognize target vector builtins as the generic
per-lane operations they are, so we do not have to dispatch to target
code all the time?
> I have tried to avoid leaking internal vectorizer abstractions here, such as
> pattern definitions. The hook simply asks for the expected output type and
> the
> operations that make up the pattern, returning the new operation to put in
> place.
>
> I have not fully worked out this hook yet, or the previous one, so the design
> is still in flux.
>
> With the hook, the example above is vectorized as:
>
> ```asm
> mov z30.b, #1
> .L4:
> ld1b z31.b, p7/z, [x1, x2]
> cmpne p6.b, p7/z, z31.b, #0
> add z31.b, p6/m, z31.b, z30.b
> st1b z31.b, p7, [x0, x2]
> incb x2
> whilelo p7.b, x2, x3
> b.any .L4
> ```
>
> There will also be default implementations for patterns that we can detect
> from
> generic code, such as `VEC_PERM_EXPR`.
>
> ### If-Conversion
>
> Some vector code manually creates vector control-flow statements using
> reduction operations:
>
> ```c
> <bool> == vector reduction chain
> if (<bool>)
> add ...
> ```
>
> Ideally, we would if-convert these back down using the right values as masks.
> For instance, if the boolean above was created by:
>
> ```c
> a = b `cmp` d
> e = smaxv (a)
> if (e != 0)
> add ...
> ```
>
> we should be able to transform that into:
>
> ```c
> a = b `cmp` d
> cond_add (a, ...)
> ```
>
> The idea is to use `targetm.vectorize.recog_patterns_builtin_call`, if
> possible, to do this as well.
Likewise, SLP patterns?
> This is not currently implemented.
>
> ### Live Values
>
> Scalar live-out values are easy to support, for example for reductions.
> However, if the live value is a vector, we cannot currently do anything.
True. With loop vect we'd see a scalar lane extract eventually, but only
outside of the (re-)vectorization scope. With BB vectorization the
situation might be better, but we'd have to include the lane extract
as SLP graph entry then.
I'd like to see testcases where it matters.
> For example, if the function returns `a` and `a` is an `int32x4_t`, then we
> are
> stuck because we are required to return a 128-bit vector without changing the
> order or values of the lanes.
>
> In some cases, we can still handle this depending on the operation. For
> example, if the value is a simple load and accumulate then we can, since HVLA
> adds new instructions, including loads, permutes, stores, and reductions, that
> work on 128-bit element sizes. Effectively, every "element" is an AdvSIMD
> vector, and there are operations that work on those containers, such as:
>
> https://dougallj.github.io/asil/doc/addqv_z_p_z_16.html
>
> I am not entirely sure how to model this yet. Richard S chose not to
> introduce
> new modes here, but instead to represent the "element" and result modes using
> existing modes. For `ADDQV`, this is a reduction from element mode, say
> `VNx16QI`, into a normal vector mode, also say `VNx16QI`.
>
> This representation may introduce ambiguity in GIMPLE, but given that these
> operations are essentially opaque to GIMPLE, perhaps that is OK. I could use
> a
> property on the SLP node to distinguish them in the vectorizer.
>
> ### Reductions, Including Accumulator Splits
>
> Reductions have to make the same distinction as live values between reductions
> to scalars and reductions to vectors, then use the appropriate instructions.
> When we do accumulator splits, for example, the live reduction on the live
> value needs to be updated as well.
>
> The initial values for reductions also need either to be splats or sequences
> with a recognizable pattern, so that when we extend the VF we can extend the
> pattern.
>
> Alternatively, we must be able to find a neutral value to insert in the new
> lanes that will not affect the reduction, such as inserting zeros for
> addition.
A vector reduction inside the loop basically looks like N scalar
reductions, thus a SLP reduction, only the epilog ties them together.
I'll note that BB vectorization reduction does not chose to represent
the reduction step but keeps it as "root" stmts in the graph entry.
A proper SLP representation for a lane reduction would be needed to
do better I think. Apart from adding "hacks".
> ### Inductions
>
> Inductions have the same issue as reductions and permutes, but I expect them
> to
> be much less common. Most people still use scalar inductions with their
> vector
> code. I therefore do not think I will spend much time supporting vector
> inductions in the first version, unless most of the support falls out
> naturally
> from the reduction and permute work.
>
> ### Unpacking, Matching `_hi` and `_lo`
>
> Transforming from an ISA that has `_hi`/`_lo` unpacking support to one that
> has
> `_odd`/`_even` would require us to recognize the `_hi`/`_lo` operations and
> treat them as the same operation. The loop was effectively unrolled for this
> anyway. This would be similar to `TWO_OPERANDS` nodes, and probably just an
> extension of them.
>
> This would allow us to re-synthesize new `_hi`/`_lo` operations when
> increasing
> the VF. If changing ISA, we would also need to be able to pack the operation
> again.
>
> This is still a TODO. I think I can make it work, but I am not sure whether
> it
> belongs in v1.
Definitely not in [1/n] ;) IMO this is sth similar as the SLP permute
optimization phase, but we'd eliminate branches in the SLP graph by
merging nodes and permuting only to undo that then in a different way.
We currently do not have any case where we branch/merge the SLP graph
and the number of lanes difffer for incoming/outgoing edges btw.,
there's code that gets confused by this (also see - multiple vector
sizes in a loop, also see "max_nunits" we track and how we compute
the unroll factor based on that).
>
> ### Permutes
>
> Permutes will be a challenge. Extending an existing short permute to a longer
> permute, or to VLA, can only happen if the permute represents some kind of
> regular pattern, such as `REVERSE` or `ZIP`. We will therefore be limited in
> what we can transform.
>
> ### Alias Analysis
>
> I think alias analysis should work naturally once I can describe the DR
> correctly. I have listed it here because I still have a TODO to check this.
When we change the VF in loop vectorization (aka, VF != 1), yes. I think
this should mostly just work.
> ---
>
> Anyway, this is a long email. What do you think, Richi? I still have enough
> time to pivot. This is 1 of 3 large vectorizer patches that I have planned
> and
> am working on for GCC 17.
I say thanks for tackling this. The SLP/DR representation issues
are the most interesting bits, most of the easy cases should just
work.
Now I'll pivot to the predicated tail series parts that touch SLP.
Richard.
>
>
> gcc/ChangeLog:
>
> * config/aarch64/aarch64-builtins.cc (aarch64_eq_type_p):
> (aarch64_lookup_fndecl_by_name):
> (aarch64_find_builtin_fndecl_1):
> (aarch64_find_builtin_fndecl):
> (aarch64_internal_translate_builtin_call):
> * config/aarch64/aarch64-protos.h:
> * config/aarch64/aarch64-sve-builtins.cc
> (lookup_registered_function_by_name):
> (canonicalize_function_name):
> (lookup_fndecl_by_name):
> * config/aarch64/aarch64-sve-builtins.h (lookup_fndecl_by_name):
> * config/aarch64/aarch64-sve2.md (*cond_add_bsl<mode>):
> * config/aarch64/aarch64.cc (aarch64_internal_translate_builtin_call):
> (aarch64_autovectorize_translate_builtin_call):
> (TARGET_VECTORIZE_TRANSLATE_BUILTIN_CALL):
> * doc/tm.texi:
> * doc/tm.texi.in:
> * target.def:
> * tree-phinodes.cc (ideal_phi_node_len):
> * tree-phinodes.h (make_phi_node):
> * tree-vect-data-refs.cc (vect_analyze_group_access_1):
> (vect_analyze_data_ref_access):
> * tree-vect-loop.cc (vectorizable_reduction):
> * tree-vect-patterns.cc (vect_recog_vect_bool_pattern):
> (vect_recog_vect_scalar_pattern):
> * tree-vect-slp.cc (vect_build_slp_tree):
> (vect_build_slp_tree_2):
> (vect_build_slp_instance):
> (vect_analyze_slp_instance):
> (vect_slp_check_for_roots):
> (vect_transform_slp_perm_load_1):
> (vect_schedule_slp):
> * tree-vect-stmts.cc (get_load_store_type):
> (vectorizable_operation):
> (vectorizable_store):
> (vectorizable_load):
> (vect_is_simple_cond):
> (vect_shim_call_arg):
> (vectorizable_vect_call):
> (vect_analyze_stmt):
> (vect_transform_stmt):
> (get_related_vectype_for_scalar_type):
> (vect_get_vector_types_for_stmt):
> (vect_gen_len):
> * tree-vectorizer.h (enum stmt_vec_info_type):
>
> ---
> diff --git a/gcc/config/aarch64/aarch64-builtins.cc
> b/gcc/config/aarch64/aarch64-builtins.cc
> index
> b9cc2bf404f612a9688db4ce2f6a042b148ac938..54d7d2e918f613fa449ab268b84581d7ada483f3
> 100644
> --- a/gcc/config/aarch64/aarch64-builtins.cc
> +++ b/gcc/config/aarch64/aarch64-builtins.cc
> @@ -29,6 +29,7 @@
> #include "rtl.h"
> #include "tree.h"
> #include "gimple.h"
> +#include "cfgloop.h"
> #include "ssa.h"
> #include "memmodel.h"
> #include "tm_p.h"
> @@ -42,6 +43,7 @@
> #include "expr.h"
> #include "langhooks.h"
> #include "gimple-iterator.h"
> +#include "tree-vectorizer.h"
> #include "case-cfn-macros.h"
> #include "regs.h"
> #include "emit-rtl.h"
> @@ -5591,6 +5593,201 @@ aarch64_general_gimple_fold_builtin (unsigned int
> fcode, gcall *stmt,
> return new_stmt;
> }
>
> +/* Intrinsics type have a different type decorator than normal types. So
> + compare signess and mode instead. */
> +static bool
> +aarch64_eq_type_p (tree ty0, tree ty1)
> +{
> + if (TYPE_UNSIGNED (ty0) != TYPE_UNSIGNED (ty1))
> + return false;
> +
> + return TYPE_MODE (ty0) == TYPE_MODE (ty1);
> +}
> +
> +static tree aarch64_lookup_fndecl_by_name (const char *);
> +
> +static tree
> +aarch64_find_builtin_fndecl_1 (tree fndecl, vec<tree> &types,
> + tree req_ret_ty,
> + tree req_arg0_ty, tree req_arg1_ty,
> + int *maskopno = NULL)
> +{
> + tree t = fndecl;
> + do
> + {
> + if (TREE_CODE (t) != FUNCTION_DECL)
> + break;
> +
> + tree fntype = TREE_TYPE (t);
> + tree ret_ty = TREE_TYPE (fntype);
> + tree args = TYPE_ARG_TYPES (fntype);
> +
> + if (!TREE_CHAIN (args))
> + break;
> +
> + tree arg0_ty = TREE_VALUE (args);
> + tree arg1_ty = TREE_VALUE (TREE_CHAIN (args));
> +
> + if (aarch64_eq_type_p (ret_ty, req_ret_ty)
> + && aarch64_eq_type_p (arg0_ty, req_arg0_ty)
> + && aarch64_eq_type_p (arg1_ty, req_arg1_ty))
> + {
> + types.create (3);
> + types.quick_push (ret_ty);
> + types.quick_push (arg0_ty);
> + types.quick_push (arg1_ty);
> + return t;
> + }
> +
> + /* Check if it could be masked, very wrong and unsafe but good enough.
> */
> + if (maskopno && VECTOR_BOOLEAN_TYPE_P (arg0_ty))
> + {
> + *maskopno = 0;
> + tree arg0_ty = TREE_VALUE (TREE_CHAIN (args));
> + tree arg1_ty = TREE_VALUE (TREE_CHAIN (TREE_CHAIN (args)));
> + if (aarch64_eq_type_p (ret_ty, req_ret_ty)
> + && aarch64_eq_type_p (arg0_ty, req_arg0_ty)
> + && aarch64_eq_type_p (arg1_ty, req_arg1_ty))
> + {
> + types.create (4);
> + types.quick_push (ret_ty);
> + types.quick_push (TREE_VALUE (args));
> + types.quick_push (arg0_ty);
> + types.quick_push (arg1_ty);
> + return t;
> + }
> + }
> +
> + if (t == fndecl)
> + t = DECL_CHAIN (fndecl);
> + else
> + t = DECL_CHAIN (t);
> + }
> + while (t);
> +
> + return NULL;
> +}
> +
> +static tree
> +aarch64_find_builtin_fndecl (tree fndecl, tree req_ret_ty,
> + tree req_arg0_ty, tree req_arg1_ty,
> + int *maskopno, vec<tree> &types,
> + const char *fallback_name = nullptr)
> +{
> + tree res = aarch64_find_builtin_fndecl_1 (fndecl, types,
> + req_ret_ty, req_arg0_ty,
> + req_arg1_ty);
> + if (res)
> + return res;
> +
> + if (fallback_name)
> + {
> + tree t = aarch64_lookup_fndecl_by_name (fallback_name);
> + gcc_assert (t);
> + *maskopno = -1;
> + res = aarch64_find_builtin_fndecl_1 (t, types, req_ret_ty, req_arg0_ty,
> + req_arg1_ty, maskopno);
> + return res;
> + }
> +
> + return NULL_TREE;
> +}
> +
> +static tree
> +aarch64_lookup_fndecl_by_name (const char *name)
> +{
> + tree id = get_identifier (name);
> + if (!id)
> + return NULL_TREE;
> +
> + extern tree lookup_name (tree);
> + tree decl = lookup_name (id);
> + if (!decl || TREE_CODE (decl) != FUNCTION_DECL)
> + return aarch64_sve::lookup_fndecl_by_name (name);
> +
> + return decl;
> +}
> +
> +/* Implement TARGET_VECTORIZE_TRANSLATE_BUILTIN_CALL. */
> +gcall *
> +aarch64_internal_translate_builtin_call (vec_info *vinfo,
> + slp_tree node,
> + stmt_vec_info stmt_info,
> + int *maskopno,
> + vec<tree> &types,
> + tree *vectype_out)
> +{
> + gcall *stmt = dyn_cast <gcall *> (STMT_VINFO_STMT (stmt_info));
> + gcc_assert (stmt);
> + tree fndecl = gimple_call_fndecl (stmt);
> + unsigned int code = DECL_MD_FUNCTION_CODE (fndecl);
> + unsigned int subcode = code >> AARCH64_BUILTIN_SHIFT;
> +
> + /* There's nothing to do for SVE at this point in time, so bail out. */
> + if ((code & AARCH64_BUILTIN_CLASS) == AARCH64_BUILTIN_SVE)
> + return NULL;
> +
> + switch (subcode)
> + {
> + BUILTIN_VSDQ_I (BINOP_SSU, suqadd, 0, DEFAULT)
> + BUILTIN_VSDQ_I (BINOP_UUS, usqadd, 0, DEFAULT)
> + {
> + tree arg0 = gimple_call_arg (stmt, 0);
> + tree arg1 = gimple_call_arg (stmt, 1);
> + tree ty0 = get_vectype_for_scalar_type (vinfo, TREE_TYPE (arg0));
> + tree ty1 = get_vectype_for_scalar_type (vinfo, TREE_TYPE (arg1));
> + if (!ty0 || !ty1)
> + return NULL;
> +
> + /* For usqadd/suqadd the result type is the vectorized type of arg0. */
> + tree new_fndecl = aarch64_find_builtin_fndecl (fndecl, ty0, ty0, ty1,
> + maskopno, types,
> + "svsqadd_u8_x");
> + if (!new_fndecl)
> + return NULL;
> +
> + /* This is type incorrect, but we don't have the arguments yet. The
> + vectorizer will take care of it. */
> + gcall *new_call;
> + if (*maskopno == -1)
> + new_call = gimple_build_call (new_fndecl, 2, NULL, NULL);
> + else
> + new_call = gimple_build_call (new_fndecl, 3, NULL, NULL, NULL);
> + *vectype_out = ty0;
> + return new_call;
> + }
> + BUILTIN_VDQQH (BSL_P, simd_bsl, 0, DEFAULT)
> + VAR2 (BSL_P, simd_bsl,0, DEFAULT, di, v2di)
> + BUILTIN_VSDQ_I_DI (BSL_U, simd_bsl, 0, DEFAULT)
> + BUILTIN_VALLDIF (BSL_S, simd_bsl, 0, QUIET)
> + {
> + tree arg0 = gimple_call_arg (stmt, 0);
> + tree arg1 = gimple_call_arg (stmt, 1);
> + tree ty0 = get_vectype_for_scalar_type (vinfo, TREE_TYPE (arg0));
> + tree ty1 = get_vectype_for_scalar_type (vinfo, TREE_TYPE (arg1));
> + if (!ty0 || !ty1)
> + return NULL;
> +
> + /* For usqadd/suqadd the result type is the vectorized type of arg0. */
> + tree new_fndecl = aarch64_find_builtin_fndecl (fndecl, ty1, ty0, ty1,
> + maskopno, types,
> + "svbsl_u8");
> + if (!new_fndecl)
> + return NULL;
> +
> + types.safe_push (types[1]);
> + /* BSL does not support masks. */
> + gcc_assert (*maskopno == -1);
> + *vectype_out = ty0;
> + return gimple_build_call (new_fndecl, 3, NULL, NULL, NULL);
> + }
> + default:
> + return NULL;
> + }
> +
> + return NULL;
> +}
> +
> void
> aarch64_atomic_assign_expand_fenv (tree *hold, tree *clear, tree *update)
> {
> diff --git a/gcc/config/aarch64/aarch64-protos.h
> b/gcc/config/aarch64/aarch64-protos.h
> index
> 513b556398fac7c733de47c27587c3aaa07cbcf3..fb7062b3d75902e1298b9d64846daf69b418fbb7
> 100644
> --- a/gcc/config/aarch64/aarch64-protos.h
> +++ b/gcc/config/aarch64/aarch64-protos.h
> @@ -1185,6 +1185,7 @@ namespace aarch64_sve {
> void handle_arm_sme_h (bool);
> void handle_arm_neon_sve_bridge_h (bool);
> tree builtin_decl (unsigned, bool);
> + tree lookup_fndecl_by_name (const char *);
> bool builtin_type_p (const_tree);
> bool builtin_type_p (const_tree, unsigned int *, unsigned int *);
> const char *mangle_builtin_type (const_tree);
> diff --git a/gcc/config/aarch64/aarch64-sve-builtins.cc
> b/gcc/config/aarch64/aarch64-sve-builtins.cc
> index
> cd78713440f61ac18146f0be49a43a204dbb3ae1..fe5e267f087e1b2b62885695785e9065bee4d85a
> 100644
> --- a/gcc/config/aarch64/aarch64-sve-builtins.cc
> +++ b/gcc/config/aarch64/aarch64-sve-builtins.cc
> @@ -1063,6 +1063,45 @@ static GTY(()) hash_table<registered_function_hasher>
> *function_table;
> are IDENTIFIER_NODEs. */
> static GTY(()) hash_map<tree, registered_function *> *overload_names[2];
>
> +/* Return the decl of the registered SVE function called NAME, or null if
> + no such function has been registered. */
> +static tree
> +lookup_registered_function_by_name (const char *name)
> +{
> + unsigned int length = vec_safe_length (registered_functions);
> + for (unsigned int i = 0; i < length; ++i)
> + {
> + registered_function *rfn = (*registered_functions)[i];
> + if (!rfn || TREE_CODE (rfn->decl) != FUNCTION_DECL)
> + continue;
> +
> + tree decl_name = DECL_NAME (rfn->decl);
> + if (decl_name && strcmp (IDENTIFIER_POINTER (decl_name), name) == 0)
> + return rfn->decl;
> + }
> +
> + return NULL_TREE;
> +}
> +
> +/* Return the ACLE spelling that corresponds to NAME. */
> +static const char *
> +canonicalize_function_name (const char *name)
> +{
> + if (startswith (name, "__builtin_aarch64_"))
> + return name + strlen ("__builtin_aarch64_");
> +
> + if (startswith (name, "__builtin_sve_"))
> + return name + strlen ("__builtin_sve_");
> +
> + if (startswith (name, "__builtin_")
> + && (startswith (name + strlen ("__builtin_"), "sv")
> + || startswith (name + strlen ("__builtin_"), "arm_")
> + || startswith (name + strlen ("__builtin_"), "__arm_")))
> + return name + strlen ("__builtin_");
> +
> + return name;
> +}
> +
> /* Record that TYPE is an ABI-defined SVE type that contains NUM_ZR SVE
> vectors
> and NUM_PR SVE predicates. MANGLED_NAME, if nonnull, is the ABI-defined
> mangling of the type. mangling of the type. ACLE_NAME is the <arm_sve.h>
> @@ -4939,6 +4978,29 @@ builtin_decl (unsigned int code, bool)
> return (*registered_functions)[code]->decl;
> }
>
> +/* Return the decl for the SVE builtin called NAME, if one exists. Accept
> + both the ACLE spelling and the common builtin-prefixed aliases. */
> +tree
> +lookup_fndecl_by_name (const char *name)
> +{
> + const char *canonical_name = canonicalize_function_name (name);
> + tree decl = lookup_registered_function_by_name (canonical_name);
> + if (decl)
> + return decl;
> +
> + /* Internal callers sometimes need SVE decls even when arm_sve.h was not
> + included in the source. Materialize them lazily in that case. */
> + if (!function_table && (lang_GNU_C () || lang_GNU_CXX ()))
> + {
> + handle_arm_sve_h (false);
> + decl = lookup_registered_function_by_name (canonical_name);
> + if (decl)
> + return decl;
> + }
> +
> + return NULL_TREE;
> +}
> +
> /* Implement #pragma GCC aarch64 "arm_sme.h". */
> void
> handle_arm_sme_h (bool function_nulls_p)
> diff --git a/gcc/config/aarch64/aarch64-sve-builtins.h
> b/gcc/config/aarch64/aarch64-sve-builtins.h
> index
> f9e8c4cd7293a379e3334fecb846121cf88e0ebf..d4123dbe8b3c0a60f1b1a6b1a398d05989987c23
> 100644
> --- a/gcc/config/aarch64/aarch64-sve-builtins.h
> +++ b/gcc/config/aarch64/aarch64-sve-builtins.h
> @@ -856,6 +856,7 @@ bool vector_cst_all_same (tree, unsigned int);
> bool is_ptrue (tree, unsigned int);
> bool is_pfalse (tree);
> const function_instance *lookup_fndecl (tree);
> +tree lookup_fndecl_by_name (const char *);
>
> /* Try to find a mode with the given mode_suffix_info fields. Return the
> mode on success or MODE_none on failure. */
> diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc
> index
> 5a859e12b1a55e70284469e3e9c175eee849e18d..a9f3af387ddce6b86370992a4dc9717df5f7e82c
> 100644
> --- a/gcc/config/aarch64/aarch64.cc
> +++ b/gcc/config/aarch64/aarch64.cc
> @@ -103,6 +103,7 @@
> #include "aarch64-sched-dispatch.h"
> #include "aarch64-json-tunings-printer.h"
> #include "aarch64-json-tunings-parser.h"
> +#include "aarch64-builtins.h"
>
> /* This file should be included last. */
> #include "target-def.h"
> @@ -23868,6 +23869,24 @@ aarch64_autovectorize_vector_modes (vector_modes
> *modes, bool)
> return flags;
> }
>
> +/* This is just temporary as I don't have an entry point into the new
> intrinsics
> + framework yet. */
> +extern gcall *aarch64_internal_translate_builtin_call (vec_info *, slp_tree,
> + stmt_vec_info, int*,
> + vec<tree> &types, tree *);;
> +
> +/* Implement TARGET_VECTORIZE_TRANSLATE_BUILTIN_CALL. */
> +static gcall *
> +aarch64_autovectorize_translate_builtin_call (vec_info *vinfo,
> + slp_tree node,
> + stmt_vec_info stmt_info,
> + int *maskopno, vec<tree> &types,
> + tree *vectype_out)
> +{
> + return aarch64_internal_translate_builtin_call (vinfo, node, stmt_info,
> + maskopno, types, vectype_out);
> +}
> +
> /* Implement TARGET_CONVERT_TO_TYPE. Convert EXPR to TYPE. */
>
> static tree
> @@ -34098,6 +34117,10 @@ aarch64_libgcc_floating_mode_supported_p
> #define TARGET_VECTORIZE_AUTOVECTORIZE_VECTOR_MODES \
> aarch64_autovectorize_vector_modes
>
> +#undef TARGET_VECTORIZE_TRANSLATE_BUILTIN_CALL
> +#define TARGET_VECTORIZE_TRANSLATE_BUILTIN_CALL \
> + aarch64_autovectorize_translate_builtin_call
> +
> #undef TARGET_CONVERT_TO_TYPE
> #define TARGET_CONVERT_TO_TYPE aarch64_convert_to_type
>
> diff --git a/gcc/doc/tm.texi b/gcc/doc/tm.texi
> index
> 1fbf43c6c57799372752f581528e122689a33f15..4f48737bb916fefd51dd738a235679af1b4f693c
> 100644
> --- a/gcc/doc/tm.texi
> +++ b/gcc/doc/tm.texi
> @@ -6537,6 +6537,20 @@ current cost model is for the scalar version of a loop
> or block; otherwise
> it is for the vector version.
> @end deftypefn
>
> +@deftypefn {Target Hook} {gcall *} TARGET_VECTORIZE_TRANSLATE_BUILTIN_CALL
> (vec_info *@var{vinfo}, struct _slp_tree *@var{node}, class _stmt_vec_info
> *@var{stmt_info}, int *@var{maskopno}, vec<tree> @var{&types}, tree
> *@var{vectype_out})
> +This hook returns true if masked operation @var{ifn} (really of
> +type @code{internal_fn}) should be considered more expensive to use than
> +implementing the same operation without masking. GCC can then try to use
> +unconditional operations instead with extra selects.
> +@end deftypefn
> +
> +@deftypefn {Target Hook} {gimple *}
> TARGET_VECTORIZE_RECOG_PATTERNS_BUILTIN_CALL (vec_info *@var{vinfo}, struct
> _slp_tree *@var{node}, class _stmt_vec_info *@var{stmt_info}, vec<gimple>
> @var{&chain}, tree *@var{vectype_out})
> +This hook returns true if masked operation @var{ifn} (really of
> +type @code{internal_fn}) should be considered more expensive to use than
> +implementing the same operation without masking. GCC can then try to use
> +unconditional operations instead with extra selects.
> +@end deftypefn
> +
> @deftypefn {Target Hook} tree TARGET_VECTORIZE_BUILTIN_GATHER (const_tree
> @var{mem_vectype}, const_tree @var{index_type}, int @var{scale})
> Target builtin that implements vector gather operation. @var{mem_vectype}
> is the vector type of the load and @var{index_type} is scalar type of
> diff --git a/gcc/doc/tm.texi.in b/gcc/doc/tm.texi.in
> index
> 6ecbeda09ca9b6473f12f78f8805ced44907f146..bc6d60d2ec47e62a31596bd96f3d106ae65254fb
> 100644
> --- a/gcc/doc/tm.texi.in
> +++ b/gcc/doc/tm.texi.in
> @@ -4341,6 +4341,10 @@ address; but often a machine-dependent strategy can
> generate better code.
>
> @hook TARGET_VECTORIZE_CREATE_COSTS
>
> +@hook TARGET_VECTORIZE_TRANSLATE_BUILTIN_CALL
> +
> +@hook TARGET_VECTORIZE_RECOG_PATTERNS_BUILTIN_CALL
> +
> @hook TARGET_VECTORIZE_BUILTIN_GATHER
>
> @hook TARGET_VECTORIZE_BUILTIN_SCATTER
> diff --git a/gcc/target.def b/gcc/target.def
> index
> 25a94559e198aef2c3e842f02aa5414c5193c4a6..3c9a2aba29691b80ab502b057e9e7244e56758a4
> 100644
> --- a/gcc/target.def
> +++ b/gcc/target.def
> @@ -2098,6 +2098,33 @@ stores.",
> (const_tree vectype, const_tree index_type, int scale),
> NULL)
>
> +/* Function to say how to transform a gimple call. If transformation is
> + possible return the new gcall, otherwise NULL. */
> +DEFHOOK
> +(translate_builtin_call,
> + "This hook returns true if masked operation @var{ifn} (really of\n\
> +type @code{internal_fn}) should be considered more expensive to use than\n\
> +implementing the same operation without masking. GCC can then try to use\n\
> +unconditional operations instead with extra selects.",
> + gcall *,
> + (vec_info *vinfo, struct _slp_tree *node, class _stmt_vec_info *stmt_info,
> + int *maskopno, vec<tree> &types, tree *vectype_out),
> + NULL)
> +
> +/* Function to recognize patterns during vectorization of gimple calls. If a
> + pattern is matched then return the new pattern and other statements it
> + consumes, otherwise NULL. */
> +DEFHOOK
> +(recog_patterns_builtin_call,
> + "This hook returns true if masked operation @var{ifn} (really of\n\
> +type @code{internal_fn}) should be considered more expensive to use than\n\
> +implementing the same operation without masking. GCC can then try to use\n\
> +unconditional operations instead with extra selects.",
> + gimple *,
> + (vec_info *vinfo, class _stmt_vec_info *stmt_info,
> + vec<gimple *> &chain, tree *vectype_out),
> + NULL)
> +
> /* Target function to initialize the cost model for a loop or block. */
> DEFHOOK
> (create_costs,
> diff --git a/gcc/tree-phinodes.cc b/gcc/tree-phinodes.cc
> index
> e0663d96b7d4b96a1a48e5bfb9fef1329e526ae7..d97c6148d2f303bf9280dbbf19a3610133d36f8c
> 100644
> --- a/gcc/tree-phinodes.cc
> +++ b/gcc/tree-phinodes.cc
> @@ -165,7 +165,7 @@ ideal_phi_node_len (int len)
>
> /* Return a PHI node with LEN argument slots for variable VAR. */
>
> -static gphi *
> +gphi *
> make_phi_node (tree var, int len)
> {
> gphi *phi;
> diff --git a/gcc/tree-phinodes.h b/gcc/tree-phinodes.h
> index
> b6fa492cb248f4d0e1a7838d0b34da6ae3a108fd..08741bb4076d9459454120ead335e52e49953352
> 100644
> --- a/gcc/tree-phinodes.h
> +++ b/gcc/tree-phinodes.h
> @@ -23,6 +23,7 @@ along with GCC; see the file COPYING3. If not see
> extern void phinodes_print_statistics (void);
> extern void reserve_phi_args_for_new_edge (basic_block);
> extern gphi *create_phi_node (tree, basic_block);
> +gphi *make_phi_node (tree, int);
> extern void add_phi_arg (gphi *, tree, edge, location_t);
> extern void remove_phi_args (edge);
> extern void remove_phi_node (gimple_stmt_iterator *, bool);
> diff --git a/gcc/tree-vect-data-refs.cc b/gcc/tree-vect-data-refs.cc
> index
> 0e57e1068d6a1906d1d8e9edb761d5dd4b7da758..eefd5ad2f91e0292cda52354bf70b406a62adbcf
> 100644
> --- a/gcc/tree-vect-data-refs.cc
> +++ b/gcc/tree-vect-data-refs.cc
> @@ -3176,6 +3176,17 @@ vect_analyze_group_access_1 (vec_info *vinfo,
> dr_vec_info *dr_info)
> data_reference *dr = dr_info->dr;
> tree step = DR_STEP (dr);
> tree scalar_type = TREE_TYPE (DR_REF (dr));
> + unsigned long nunits = 0;
> + /* TODO: Add? */
> + if (VECTOR_TYPE_P (scalar_type)
> + && !TYPE_VECTOR_SUBPARTS (scalar_type).is_constant (&nunits))
> + {
> + if (dump_enabled_p ())
> + dump_printf_loc (MSG_NOTE, vect_location,
> + "Variable length vectors not supported yet: %T\n",
> + step);
> + return false;
> + }
> HOST_WIDE_INT type_size = TREE_INT_CST_LOW (TYPE_SIZE_UNIT (scalar_type));
> stmt_vec_info stmt_info = dr_info->stmt;
> loop_vec_info loop_vinfo = dyn_cast <loop_vec_info> (vinfo);
> @@ -3207,6 +3218,8 @@ vect_analyze_group_access_1 (vec_info *vinfo,
> dr_vec_info *dr_info)
> }
> groupsize = absu_hwi (dr_step) / type_size;
> }
> + else if (bb_vinfo && nunits > 0)
> + groupsize = nunits;
> else
> groupsize = 0;
>
> @@ -3463,6 +3476,29 @@ vect_analyze_data_ref_access (vec_info *vinfo,
> dr_vec_info *dr_info)
> }
> }
>
> + /* Allow loads and stores with zero steps and vector types in BB
> + vectorization. TODO: This needs support for checking the type of
> access
> + so e.g. LOAD_LANES, Gathers etc. */
> + if (!loop_vinfo && integer_zerop (step) && VECTOR_TYPE_P (scalar_type))
> + {
> + if (!TYPE_VECTOR_SUBPARTS (scalar_type).is_constant ())
> + {
> + if (dump_enabled_p ())
> + dump_printf_loc (MSG_NOTE, vect_location,
> + "Variable length vectors not supported yet: %T\n",
> + step);
> + return false;
> + }
> + DR_GROUP_FIRST_ELEMENT (stmt_info) = NULL;
> + DR_GROUP_NEXT_ELEMENT (stmt_info) = NULL;
> +
> + if (dump_enabled_p () && 0)
> + dump_printf_loc (MSG_NOTE, vect_location,
> + "Allowing vector memory access in BB\n");
> +
> + return true;
> + }
> +
> if (loop && nested_in_vect_loop_p (loop, stmt_info))
> {
> /* Interleaved accesses are not yet supported within outer-loop
> @@ -3485,9 +3521,21 @@ vect_analyze_data_ref_access (vec_info *vinfo,
> dr_vec_info *dr_info)
> if (TREE_CODE (step) == INTEGER_CST)
> {
> HOST_WIDE_INT dr_step = TREE_INT_CST_LOW (step);
> - if (!tree_int_cst_compare (step, TYPE_SIZE_UNIT (scalar_type))
> - || (dr_step < 0
> - && !compare_tree_int (TYPE_SIZE_UNIT (scalar_type), -dr_step)))
> + bool sequantial = false;
> + poly_int64 step_p, type_size_p;
> + if (poly_int_tree_p (step, &step_p)
> + && poly_int_tree_p (TYPE_SIZE_UNIT (scalar_type), &type_size_p))
> + sequantial
> + = known_eq (step_p, type_size_p)
> + || (known_lt (step_p, 0)
> + && !known_eq (-step_p, type_size_p));
> + else
> + sequantial
> + = !tree_int_cst_compare (step, TYPE_SIZE_UNIT (scalar_type))
> + || (dr_step < 0
> + && !compare_tree_int (TYPE_SIZE_UNIT (scalar_type), -dr_step));
> +
> + if (sequantial)
> {
> /* Mark that it is not interleaving. */
> DR_GROUP_FIRST_ELEMENT (stmt_info) = NULL;
> diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
> index
> 51e7f05100de7ab31994f3b4a30f5c5be8d8afdd..7dee95951eef95622c9554c1bbe01c3e02cd7894
> 100644
> --- a/gcc/tree-vect-loop.cc
> +++ b/gcc/tree-vect-loop.cc
> @@ -7184,12 +7184,17 @@ vectorizable_reduction (loop_vec_info loop_vinfo,
> gcc_unreachable ();
> bool lane_reducing = lane_reducing_op_p (op.code);
>
> - if (!POINTER_TYPE_P (op.type) && !INTEGRAL_TYPE_P (op.type)
> - && !SCALAR_FLOAT_TYPE_P (op.type))
> + tree scalar_type = op.type;
> + if (VECTOR_TYPE_P (scalar_type))
> + scalar_type = TREE_TYPE (scalar_type);
> +
> + if (!POINTER_TYPE_P (scalar_type)
> + && !INTEGRAL_TYPE_P (scalar_type)
> + && !SCALAR_FLOAT_TYPE_P (scalar_type))
> return false;
>
> /* Do not try to vectorize bit-precision reductions. */
> - if (!type_has_mode_precision_p (op.type)
> + if (!type_has_mode_precision_p (scalar_type)
> && op.code != BIT_AND_EXPR
> && op.code != BIT_IOR_EXPR
> && op.code != BIT_XOR_EXPR)
> @@ -7444,7 +7449,7 @@ vectorizable_reduction (loop_vec_info loop_vinfo,
> if (!reduc_chain
> && known_eq (LOOP_VINFO_VECT_FACTOR (loop_vinfo), 1u))
> ;
> - else if (needs_fold_left_reduction_p (op.type, orig_code))
> + else if (needs_fold_left_reduction_p (scalar_type, orig_code))
> {
> /* When vectorizing a reduction chain w/o SLP the reduction PHI
> is not directly used in stmt. */
> @@ -7573,7 +7578,7 @@ vectorizable_reduction (loop_vec_info loop_vinfo,
> else if (reduction_type == COND_REDUCTION)
> {
> int scalar_precision
> - = GET_MODE_PRECISION (SCALAR_TYPE_MODE (op.type));
> + = GET_MODE_PRECISION (SCALAR_TYPE_MODE (scalar_type));
> cr_index_scalar_type = make_unsigned_type (scalar_precision);
> cr_index_vector_type = get_same_sized_vectype (cr_index_scalar_type,
> vectype_out);
> diff --git a/gcc/tree-vect-patterns.cc b/gcc/tree-vect-patterns.cc
> index
> 5f5e3fb6201da379460c46e9d7ff1d01625d0998..ca806f7d8d94333fc4301b8a8e0da5ad0a4fad0b
> 100644
> --- a/gcc/tree-vect-patterns.cc
> +++ b/gcc/tree-vect-patterns.cc
> @@ -6243,6 +6243,140 @@ vect_recog_bool_pattern (vec_info *vinfo,
> return NULL;
> }
>
> +static gimple *
> +vect_recog_vect_bool_pattern (vec_info *vinfo,
> + stmt_vec_info stmt_vinfo, tree *type_out)
> +{
> + gimple *last_stmt = STMT_VINFO_STMT (stmt_vinfo);
> + enum tree_code rhs_code;
> + tree var, lhs, rhs, vectype;
> + gimple *pattern_stmt;
> +
> + if (!is_gimple_assign (last_stmt))
> + return NULL;
> +
> + var = gimple_assign_rhs1 (last_stmt);
> + lhs = gimple_assign_lhs (last_stmt);
> + rhs_code = gimple_assign_rhs_code (last_stmt);
> +
> + if (rhs_code == VIEW_CONVERT_EXPR)
> + var = TREE_OPERAND (var, 0);
> +
> + if (!VECTOR_BOOLEAN_TYPE_P (TREE_TYPE (var)))
> + return NULL;
> +
> + if (rhs_code != VEC_COND_EXPR)
> + return NULL;
> +
> + tree op0 = gimple_assign_rhs2 (last_stmt);
> + tree op1 = gimple_assign_rhs3 (last_stmt);
> + if (!integer_zerop (op1)
> + || !integer_all_onesp (op0))
> + return NULL;
> +
> + /* Find the comparison and make it part of the pattern we'll make. */
> + auto cmp_info = vinfo->lookup_def (var);
> + if (!cmp_info)
> + return NULL;
> +
> + gimple *cmp_stmt = STMT_VINFO_STMT (cmp_info);
> + if (!is_gimple_assign (cmp_stmt)
> + || TREE_CODE_CLASS (gimple_assign_rhs_code (cmp_stmt)) !=
> tcc_comparison)
> + return NULL;
> +
> + vectype = get_vectype_for_scalar_type (vinfo, TREE_TYPE (lhs));
> + if (vectype == NULL_TREE)
> + return NULL;
> +
> + /* Build a scalar type for the boolean result that when
> + vectorized matches the vector type of the result in
> + size and number of elements. */
> + unsigned prec
> + = vector_element_size (tree_to_poly_uint64 (TYPE_SIZE (vectype)),
> + TYPE_VECTOR_SUBPARTS (vectype));
> +
> + tree type
> + = build_nonstandard_integer_type (prec, TYPE_UNSIGNED (TREE_TYPE (var)));
> +
> + tree new_vectype = get_mask_type_for_scalar_type (vinfo, type);
> + if (new_vectype == NULL_TREE)
> + return NULL;
> +
> + /* copy the scalar statement to get a new UID. */
> + tree lhs_cmp
> + = vect_recog_temp_ssa_var (TREE_TYPE (gimple_get_lhs (cmp_stmt)), NULL);
> + gimple *copy_stmt
> + = gimple_build_assign (lhs_cmp, gimple_assign_rhs_code (cmp_stmt),
> + gimple_assign_rhs1 (cmp_stmt),
> + gimple_assign_rhs2 (cmp_stmt));
> + append_pattern_def_seq (vinfo, stmt_vinfo, copy_stmt, new_vectype, type);
> +
> + tree lhs_ivar = vect_recog_temp_ssa_var (type, NULL);
> + pattern_stmt = gimple_build_assign (lhs_ivar, COND_EXPR, lhs_cmp,
> + build_all_ones_cst (type),
> + build_zero_cst (type));
> +
> + *type_out = vectype;
> + vect_pattern_detected ("vect_recog_vect_bool_pattern", last_stmt);
> +
> + return pattern_stmt;
> +}
> +
> +static gimple *
> +vect_recog_vect_scalar_pattern (vec_info *vinfo,
> + stmt_vec_info stmt_vinfo, tree *type_out)
> +{
> + gimple *last_stmt = STMT_VINFO_STMT (stmt_vinfo);
> + enum tree_code rhs_code;
> + tree var, lhs, rhs, vectype;
> + gimple *pattern_stmt;
> + loop_vec_info loop_vinfo = dyn_cast <loop_vec_info> (vinfo);
> + class loop *loop = loop_vinfo ? LOOP_VINFO_LOOP (loop_vinfo) : NULL;
> +
> +
> + lhs = gimple_get_lhs (last_stmt);
> + if (!lhs || !VECTOR_TYPE_P (TREE_TYPE (lhs)))
> + return NULL;
> +
> + if (targetm.vectorize.recog_patterns_builtin_call)
> + {
> + auto_vec<gimple *> chain;
> + gimple *res
> + = targetm.vectorize.recog_patterns_builtin_call (vinfo, stmt_vinfo,
> + chain, type_out);
> + vect_pattern_detected ("vect_recog_vect_scalar_pattern", last_stmt);
> + }
> +
> +#if 1
> + /* First pattern is reductions, if we have a vector reduction then adjust
> + any constant lanes if we can. TOOD: Maybe this should force HVLA? */
> + if (loop && is_a <gphi *> (last_stmt))
> + {
> + /* Check the header edge. */
> + gphi *phi = as_a <gphi *>(last_stmt);
> + tree init = PHI_ARG_DEF_FROM_EDGE (phi, loop_preheader_edge (loop));
> + if ((var = ssa_uniform_vector_p (init)))
> + {
> + tree type = TREE_TYPE (lhs);
> + vectype = get_vectype_for_scalar_type (vinfo, type);
> + if (vectype == NULL_TREE)
> + return NULL;
> +
> + tree lhs_ivar = vect_recog_temp_ssa_var (type, NULL);
> + gphi *res = make_phi_node (lhs_ivar, 2);
> + tree step = PHI_ARG_DEF_FROM_EDGE (phi, loop_latch_edge (loop));
> + add_phi_arg (res, init, loop_preheader_edge (loop), UNKNOWN_LOCATION);
> + add_phi_arg (res, step, loop_latch_edge (loop), UNKNOWN_LOCATION);
> + pattern_stmt = res;
> +
> + *type_out = vectype;
> + vect_pattern_detected ("vect_recog_vect_scalar_pattern", last_stmt);
> + }
> + }
> +#endif
> +
> + return NULL;
> +}
>
> /* Function vect_recog_mask_conversion_pattern
>
> @@ -7467,6 +7601,8 @@ static vect_recog_func vect_vect_recog_func_ptrs[] = {
> { vect_recog_sat_trunc_pattern, "sat_trunc" },
> { vect_recog_gcond_pattern, "gcond" },
> { vect_recog_bool_pattern, "bool" },
> + { vect_recog_vect_bool_pattern, "vect_bool" },
> + { vect_recog_vect_scalar_pattern, "vect_scalar" },
> /* This must come before mask conversion, and includes the parts
> of mask conversion that are needed for gather and scatter
> internal functions. */
> diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc
> index
> 075e93f04a97cddd6bc4b7f82485f08ee22c953b..c1bf905f54786c3622a155f777e6889a635506d4
> 100644
> --- a/gcc/tree-vect-slp.cc
> +++ b/gcc/tree-vect-slp.cc
> @@ -1950,7 +1950,7 @@ vect_build_slp_tree (vec_info *vinfo,
> for (i = 0; i < group_size; ++i)
> if (!matches[i])
> break;
> - gcc_assert (i < group_size);
> + //gcc_assert (i < group_size);
> }
> memcpy (res->failed, matches, sizeof (bool) * group_size);
> }
> @@ -2107,6 +2107,9 @@ vect_build_slp_tree_2 (vec_info *vinfo, slp_tree node,
> &vectype))
> return NULL;
>
> + if (known_gt (group_size, this_max_nunits))
> + return NULL;
> +
> /* If the SLP node is a load, terminate the recursion unless masked. */
> if (STMT_VINFO_DATA_REF (stmt_info)
> && DR_IS_READ (STMT_VINFO_DATA_REF (stmt_info)))
> @@ -2133,6 +2136,7 @@ vect_build_slp_tree_2 (vec_info *vinfo, slp_tree node,
> FOR_EACH_VEC_ELT (SLP_TREE_SCALAR_STMTS (node), j, load_info)
> {
> int load_place;
> + tree type;
> if (! load_info)
> {
> if (STMT_VINFO_GROUPED_ACCESS (stmt_info))
> @@ -2140,6 +2144,11 @@ vect_build_slp_tree_2 (vec_info *vinfo, slp_tree node,
> else
> load_place = 0;
> }
> + /* ?? Not all vector loads are linear, model this. */
> + else if ((type = TREE_TYPE (gimple_get_lhs (STMT_VINFO_STMT
> (load_info))))
> + && VECTOR_TYPE_P (type)
> + && TYPE_VECTOR_SUBPARTS (type).is_constant ())
> + load_place = j;
> else if (STMT_VINFO_GROUPED_ACCESS (stmt_info))
> load_place = vect_get_place_in_interleaving_chain
> (load_info, first_stmt_info);
> @@ -3141,7 +3150,7 @@ fail:
> n_vector_builds++;
> }
> }
> - if (all_uniform_p
> + if (all_uniform_p && false
> || n_vector_builds > 1
> || (n_vector_builds == children.length ()
> && is_a <gphi *> (stmt_info->stmt)))
> @@ -4282,7 +4291,9 @@ vect_build_slp_instance (vec_info *vinfo,
> /* While we arrive here even with slp_inst_kind_store we should only
> for group_size == 1. The code to split store groups is only in
> vect_analyze_slp_instance now. */
> +#if 0
> gcc_assert (kind != slp_inst_kind_store || group_size == 1);
> +#endif
>
> /* Free the allocated memory. */
> scalar_stmts.release ();
> @@ -5014,7 +5025,16 @@ vect_analyze_slp_instance (vec_info *vinfo,
> stmt_vec_info next_info = stmt_info;
> while (next_info)
> {
> - scalar_stmts.quick_push (vect_stmt_to_vectorize (next_info));
> + scalar_stmts.safe_push (vect_stmt_to_vectorize (next_info));
> +#if 1
> + /* If we're a vector type, insert dummy nodes. */
> + unsigned HOST_WIDE_INT vect_nunits = 0;
> + tree vectype = TREE_TYPE (gimple_get_lhs (STMT_VINFO_STMT
> (next_info)));
> + if (VECTOR_TYPE_P (vectype)
> + && TYPE_VECTOR_SUBPARTS (vectype).is_constant (&vect_nunits))
> + for (auto i = 0; i < vect_nunits - 1UL; i++)
> + scalar_stmts.safe_push (next_info);
> +#endif
> next_info = DR_GROUP_NEXT_ELEMENT (next_info);
> }
>
> @@ -5038,6 +5058,13 @@ vect_analyze_slp_instance (vec_info *vinfo,
>
> /* Build the tree for the SLP instance. */
> unsigned int group_size = scalar_stmts.length ();
> +#if 0
> + /* If we're a vector type, adjust group size by nunits. */
> + unsigned HOST_WIDE_INT vect_nunits = 0;
> + tree vectype = TREE_TYPE (gimple_get_lhs (STMT_VINFO_STMT
> (scalar_stmts[0])));
> + if (TYPE_VECTOR_SUBPARTS (vectype).is_constant (&vect_nunits))
> + group_size *= vect_nunits;
> +#endif
> bool *matches = XALLOCAVEC (bool, group_size);
> poly_uint64 max_nunits = 1;
> unsigned tree_size = 0;
> @@ -9780,6 +9807,7 @@ vect_slp_check_for_roots (bb_vec_info bb_vinfo)
> enum tree_code code = gimple_assign_rhs_code (assign);
> use_operand_p use_p;
> gimple *use_stmt;
> + optab op;
> if (code == CONSTRUCTOR)
> {
> if (!VECTOR_TYPE_P (TREE_TYPE (rhs))
> @@ -9995,6 +10023,24 @@ vect_slp_check_for_roots (bb_vec_info bb_vinfo)
> }
> }
> }
> +#if 1
> + else if (gimple_vdef (assign)
> + && VECTOR_TYPE_P (TREE_TYPE (rhs))
> + && ((op = optab_for_tree_code (code, TREE_TYPE (rhs),
> optab_default)) == unknown_optab
> + || !can_implement_p (op, TYPE_MODE (TREE_TYPE (rhs)))))
> + {
> + vec<stmt_vec_info> roots = vNULL;
> + auto stmt_vinfo = bb_vinfo->lookup_stmt (assign);
> + roots.safe_push (stmt_vinfo);
> + vec<stmt_vec_info> stmts;
> + auto num_entries = TYPE_VECTOR_SUBPARTS (TREE_TYPE (rhs)).to_constant
> () / 2;
> + stmts.create (num_entries);
> + for (int i = 0; i < num_entries; i++)
> + stmts.quick_push (stmt_vinfo);
> + bb_vinfo->roots.safe_push (slp_root (slp_inst_kind_store,
> + stmts, roots));
> + }
> +#endif
> }
> }
>
> @@ -10984,6 +11030,14 @@ vect_transform_slp_perm_load_1 (vec_info *vinfo,
> slp_tree node,
> dr_group_size = DR_GROUP_SIZE (stmt_info);
> }
>
> + /* We should probably adjust group size, but for now lets hack. */
> + unsigned HOST_WIDE_INT vect_nunits = 0;
> + auto vscalar_stmt = STMT_VINFO_STMT (stmt_info);
> + tree vscalar_type = TREE_TYPE (gimple_get_lhs (vscalar_stmt));
> + if (VECTOR_TYPE_P (vscalar_type)
> + && TYPE_VECTOR_SUBPARTS (vscalar_type).is_constant (&vect_nunits))
> + dr_group_size *= vect_nunits;
> +
> mode = TYPE_MODE (vectype);
> poly_uint64 nunits = TYPE_VECTOR_SUBPARTS (vectype);
> unsigned int nstmts = vect_get_num_copies (vinfo, node);
> @@ -12431,6 +12485,8 @@ vect_schedule_slp (vec_info *vinfo, const
> vec<slp_instance> &slp_instances)
> break;
>
> store_info = vect_orig_stmt (store_info);
> + if (!store_info)
> + continue;
> /* Free the attached stmt_vec_info and remove the stmt. */
> vinfo->remove_stmt (store_info);
>
> diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
> index
> 7687c107a6afc54741cb5e3f5756bcb65196b519..3cd8e4dfba4b3bf3173753a08cb214493b802e13
> 100644
> --- a/gcc/tree-vect-stmts.cc
> +++ b/gcc/tree-vect-stmts.cc
> @@ -2108,9 +2108,11 @@ get_load_store_type (vec_info *vinfo, stmt_vec_info
> stmt_info,
> bool perm_ok = true;
> poly_int64 vf = loop_vinfo ? LOOP_VINFO_VECT_FACTOR (loop_vinfo) : 1;
>
> +#if 1
> if (SLP_TREE_LOAD_PERMUTATION (slp_node).exists ())
> perm_ok = vect_transform_slp_perm_load (vinfo, slp_node, vNULL, NULL,
> vf, true, n_perms, n_loads);
> +#endif
>
> if (STMT_VINFO_GROUPED_ACCESS (stmt_info))
> {
> @@ -2728,8 +2730,18 @@ get_load_store_type (vec_info *vinfo, stmt_vec_info
> stmt_info,
> if (k > maxk)
> maxk = k;
> tree vectype = SLP_TREE_VECTYPE (slp_node);
> +
> + /* We should probably adjust group size, but for now lets hack. */
> + unsigned HOST_WIDE_INT vect_nunits = 0;
> + auto group_size = DR_GROUP_SIZE (group_info);
> + auto vscalar_stmt = STMT_VINFO_STMT (group_info);
> + tree vscalar_type = TREE_TYPE (gimple_get_lhs (vscalar_stmt));
> + if (VECTOR_TYPE_P (vscalar_type)
> + && TYPE_VECTOR_SUBPARTS (vscalar_type).is_constant (&vect_nunits))
> + group_size *= vect_nunits;
> +
> if (!TYPE_VECTOR_SUBPARTS (vectype).is_constant (&nunits)
> - || maxk >= (DR_GROUP_SIZE (group_info) & ~(nunits - 1)))
> + || maxk >= (group_size & ~(nunits - 1)))
> {
> if (dump_enabled_p ())
> dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
> @@ -6618,13 +6630,17 @@ vectorizable_operation (vec_info *vinfo,
> }
>
> scalar_dest = gimple_assign_lhs (stmt);
> + tree scalar_type = TREE_TYPE (scalar_dest);
> + if (VECTOR_TYPE_P (scalar_type))
> + scalar_type = TREE_TYPE (scalar_type);
> +
> vectype_out = SLP_TREE_VECTYPE (slp_node);
>
> /* Most operations cannot handle bit-precision types without extra
> truncations. */
> bool mask_op_p = VECTOR_BOOLEAN_TYPE_P (vectype_out);
> if (!mask_op_p
> - && !type_has_mode_precision_p (TREE_TYPE (scalar_dest))
> + && !type_has_mode_precision_p (scalar_type)
> /* Exception are bitwise binary operations. */
> && code != BIT_IOR_EXPR
> && code != BIT_XOR_EXPR
> @@ -6658,7 +6674,7 @@ vectorizable_operation (vec_info *vinfo,
> type. */
> if (VECT_SCALAR_BOOLEAN_TYPE_P (TREE_TYPE (op0)))
> {
> - if (!VECT_SCALAR_BOOLEAN_TYPE_P (TREE_TYPE (scalar_dest)))
> + if (!VECT_SCALAR_BOOLEAN_TYPE_P (scalar_type))
> {
> if (dump_enabled_p ())
> dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
> @@ -8297,6 +8313,14 @@ vectorizable_store (vec_info *vinfo,
> group_size = 1;
> }
>
> + /* We should probably adjust group size, but for now lets hack. */
> + unsigned HOST_WIDE_INT vect_nunits = 0;
> + auto vscalar_stmt = STMT_VINFO_STMT (first_stmt_info);
> + tree vscalar_type = TREE_TYPE (gimple_get_lhs (vscalar_stmt));
> + if (VECTOR_TYPE_P (vscalar_type)
> + && TYPE_VECTOR_SUBPARTS (vscalar_type).is_constant (&vect_nunits))
> + group_size *= vect_nunits;
> +
> if (STMT_VINFO_SIMD_LANE_ACCESS_P (stmt_info) > 1 && cost_vec)
> {
> if (!check_scan_store (vinfo, stmt_info, vectype, rhs_dt, slp_node,
> @@ -9887,6 +9911,9 @@ vectorizable_load (vec_info *vinfo,
> padding bits in the type that might leak otherwise.
> Refer to PR115336. */
> tree scalar_type = TREE_TYPE (scalar_dest);
> + if (VECTOR_TYPE_P (scalar_type))
> + scalar_type = TREE_TYPE (scalar_type);
> +
> bool type_mode_padding_p
> = TYPE_PRECISION (scalar_type) < GET_MODE_PRECISION (GET_MODE_INNER
> (mode));
>
> @@ -10509,6 +10536,7 @@ vectorizable_load (vec_info *vinfo,
> first_stmt_info = stmt_info;
> group_size = 1;
> }
> +
> /* For SLP vectorization we directly vectorize a subchain
> without permutation. */
> if (! SLP_TREE_LOAD_PERMUTATION (slp_node).exists ())
> @@ -10522,6 +10550,14 @@ vectorizable_load (vec_info *vinfo,
> first_dr_info = STMT_VINFO_DR_INFO (first_stmt_info);
> group_gap_adj = 0;
>
> + /* We should probably adjust group size, but for now lets hack. */
> + unsigned HOST_WIDE_INT vect_nunits = 0;
> + auto vscalar_stmt = STMT_VINFO_STMT (first_stmt_info);
> + tree vscalar_type = TREE_TYPE (gimple_get_lhs (vscalar_stmt));
> + if (VECTOR_TYPE_P (vscalar_type)
> + && TYPE_VECTOR_SUBPARTS (vscalar_type).is_constant (&vect_nunits))
> + group_size *= vect_nunits;
> +
> /* VEC_NUM is the number of vect stmts to be created for this group.
> */
> grouped_load = false;
> /* If an SLP permutation is from N elements to N elements,
> @@ -12015,7 +12051,8 @@ vect_is_simple_cond (tree cond, vec_info *vinfo,
>
> /* Mask case. */
> if (TREE_CODE (cond) == SSA_NAME
> - && VECT_SCALAR_BOOLEAN_TYPE_P (TREE_TYPE (cond)))
> + && (VECT_SCALAR_BOOLEAN_TYPE_P (TREE_TYPE (cond))
> + || VECTOR_BOOLEAN_TYPE_P (TREE_TYPE (cond))))
> {
> if (!vect_is_simple_use (vinfo, slp_node, 0, &cond,
> &slp_op, &dts[0], comp_vectype)
> @@ -13209,6 +13246,226 @@ vectorizable_early_exit (loop_vec_info loop_vinfo,
> stmt_vec_info stmt_info,
> return true;
> }
>
> +/* Intrinsics and vector types may not match even though the underlying modes
> + are the same. This is particularly true of masks and SVE modes. As such
> + convert between the two. */
> +static void
> +vect_shim_call_arg (vec<tree> &types, vec_info *vinfo, stmt_vec_info
> stmt_info,
> + gimple_stmt_iterator *gsi, gcall* call, int argno,
> + tree newarg)
> +{
> + tree ty_dest = types[argno+1];
> + tree ty_src = TREE_TYPE (newarg);
> +
> + if (ty_src != ty_dest)
> + {
> + tree lhs = vect_create_destination_var (newarg, ty_dest);
> + tree op = build1 (VIEW_CONVERT_EXPR, ty_dest, newarg);
> + gimple *tmp = gimple_build_assign (lhs, VIEW_CONVERT_EXPR, op);
> + lhs = make_ssa_name (lhs, tmp);
> + gimple_assign_set_lhs (tmp, lhs);
> + vect_finish_stmt_generation (vinfo, stmt_info, tmp, gsi);
> + newarg = lhs;
> +
> + if (dump_enabled_p ())
> + dump_printf_loc (MSG_NOTE, vect_location, "inserted type shim between:
> %T"
> + " and %T due to mismatch between generic type and vector"
> + " call.\n", ty_dest, ty_src);
> + }
> +
> + gimple_call_set_arg (call, argno, newarg);
> +}
> +
> +bool
> +vectorizable_vect_call (vec_info *vinfo,
> + stmt_vec_info stmt_info, gimple_stmt_iterator *gsi,
> + slp_tree slp_node,
> + stmt_vector_for_cost *cost_vec)
> +{
> + enum vect_def_type dt[5]
> + = { vect_unknown_def_type, vect_unknown_def_type, vect_unknown_def_type,
> + vect_unknown_def_type, vect_unknown_def_type };
> + tree vectypes[ARRAY_SIZE (dt)] = {};
> + slp_tree slp_op[ARRAY_SIZE (dt)] = {};
> + tree op;
> +
> + /* If the target hook isn't configured there's nothing to do. */
> + if (!targetm.vectorize.translate_builtin_call)
> + return false;
> +
> + if (!STMT_VINFO_RELEVANT_P (stmt_info))
> + return false;
> +
> + if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_internal_def
> + && cost_vec)
> + return false;
> +
> + /* Is STMT_INFO a vectorizable vector call? */
> + gcall *stmt = dyn_cast <gcall *> (STMT_VINFO_STMT (stmt_info));
> + if (!stmt || !gimple_call_builtin_p (stmt))
> + return false;
> +
> + /* It must be a built in with a vector type as an argument. */
> + bool has_vect_arg = false;
> + int nargs = gimple_call_num_args (stmt);
> +
> + /* Check if it has more arguments than we support at the moment. */
> + if (nargs > 5)
> + {
> + if (dump_enabled_p ())
> + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
> + "vector function call has too many arguments.\n");
> + return false;
> + }
> +
> + for (int i = 0; i < nargs; i++)
> + {
> + has_vect_arg = has_vect_arg
> + || VECTOR_TYPE_P (TREE_TYPE (gimple_call_arg (stmt, i)));
> +
> + if (!vect_is_simple_use (vinfo, slp_node,
> + i, &op, &slp_op[i], &dt[i], &vectypes[i]))
> + {
> + if (dump_enabled_p ())
> + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
> + "use not simple.\n");
> + return false;
> + }
> + }
> +
> + if (!has_vect_arg)
> + {
> + if (dump_enabled_p ())
> + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
> + "function is not a vectorizable vector function.\n");
> + return false;
> + }
> +
> + /* We only handle functions that do not read or clobber memory. */
> + if (gimple_vuse (stmt))
> + {
> + if (dump_enabled_p ())
> + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
> + "function reads from or writes to memory.\n");
> + return false;
> + }
> +
> + tree vectype_out;
> + int maskopno = -1;
> + auto_vec<tree> types;
> + gcall *call
> + = targetm.vectorize.translate_builtin_call (vinfo, slp_node, stmt_info,
> + &maskopno, types, &vectype_out);
> +
> + if (!call)
> + {
> + if (dump_enabled_p ())
> + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
> + "target did not have implementation for %G.\n",
> + (gimple*)stmt);
> + return false;
> + }
> +
> + loop_vec_info loop_vinfo = dyn_cast <loop_vec_info> (vinfo);
> + vec_loop_masks *masks = (loop_vinfo ? &LOOP_VINFO_MASKS (loop_vinfo) :
> NULL);
> + vec_loop_lens *lens = (loop_vinfo ? &LOOP_VINFO_LENS (loop_vinfo) : NULL);
> + bool masked_loop_p = loop_vinfo && LOOP_VINFO_FULLY_MASKED_P (loop_vinfo);
> + bool len_loop_p = loop_vinfo && LOOP_VINFO_FULLY_WITH_LENGTH_P
> (loop_vinfo);
> + unsigned int vect_nargs = nargs;
> +
> + if (maskopno != -1)
> + {
> + if (!loop_vinfo) // || (!masked_loop_p && !len_loop_p))
> + {
> + if (dump_enabled_p ())
> + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
> + "target translation requires predication but "
> + "masks not supported: %G.\n",
> + (gimple*)stmt);
> + return false;
> + }
> + vect_nargs += 1;
> + }
> +
> + if (cost_vec) /* transformation not required. */
> + {
> + SLP_TREE_TYPE (slp_node) = vector_call_vec_info_type;
> + DUMP_VECT_SCOPE ("vectorizable_vect_call");
> + vect_model_simple_cost (vinfo, 1, slp_node, cost_vec);
> + return true;
> + }
> +
> + /* Transform. */
> +
> + if (dump_enabled_p ())
> + dump_printf_loc (MSG_NOTE, vect_location, "transform vector call.\n");
> +
> + /* Resolve arguments. */
> + auto_vec<vec<tree> > vec_defs (nargs);
> + vect_get_slp_defs (vinfo, slp_node, &vec_defs);
> +
> + /* Handle def. */
> + tree scalar_dest = gimple_call_lhs (stmt);
> + tree vec_dest = vect_create_destination_var (scalar_dest, vectype_out);
> + tree new_temp = make_ssa_name (vec_dest, stmt);
> + gimple_call_set_lhs (call, new_temp);
> + gimple_call_set_nothrow (call, true);
> +
> + /* Now fill in the new values. Intrinsics types and vector types may not
> + match, so convert the types. */
> + int shift = 0;
> + for (int i = 0; i < nargs; i++)
> + {
> + if (maskopno == i)
> + {
> + tree mask;
> + if (masked_loop_p)
> + {
> + /* ?? Handle decomposition here, can that happen??. */
> + unsigned int vec_num = 1; //vec_oprnds0.length ();
> + mask = vect_get_loop_mask (loop_vinfo, gsi, masks, vec_num,
> + vectype_out, 0);
> + }
> + else
> + {
> + tree mask_vectype = truth_type_for (vectype_out);
> + mask = vect_build_all_ones_mask (loop_vinfo, stmt_info,
> + mask_vectype);
> + }
> + vect_shim_call_arg (types, vinfo, stmt_info, gsi, call, i, mask);
> + shift++;
> + }
> + tree arg = vec_defs[i][0];
> + vect_shim_call_arg (types, vinfo, stmt_info, gsi, call, i+shift, arg);
> + }
> +
> + gimple *res = call;
> + /* Check if the result types match, if not, shim them. */
> + if (vectype_out != types[0])
> + {
> + /* First update the call. */
> + tree lhs = vect_create_destination_var (scalar_dest, types[0]);
> + tree new_lhs = make_ssa_name (lhs, NULL);
> + gimple_call_set_lhs (call, new_lhs);
> + vect_finish_stmt_generation (vinfo, stmt_info, call, gsi);
> + tree op = build1 (VIEW_CONVERT_EXPR, vectype_out, new_lhs);
> + gimple *tmp = gimple_build_assign (vec_dest, VIEW_CONVERT_EXPR, op);
> + gimple_assign_set_lhs (tmp, new_temp);
> + vect_finish_stmt_generation (vinfo, stmt_info, tmp, gsi);
> + res = tmp;
> +
> + if (dump_enabled_p ())
> + dump_printf_loc (MSG_NOTE, vect_location, "inserted type shim between:
> %T"
> + " and %T due to mismatch between generic type and vector"
> + " call.\n", types[0], vectype_out);
> + }
> + else
> + vect_finish_stmt_generation (vinfo, stmt_info, call, gsi);
> +
> + slp_node->push_vec_def (res);
> + return true;
> +}
> +
> /* If SLP_NODE is nonnull, return true if vectorizable_live_operation
> can handle all live statements in the node. Otherwise return true
> if STMT_INFO is not live or if vectorizable_live_operation can handle it.
> @@ -13338,6 +13595,7 @@ vect_analyze_stmt (vec_info *vinfo,
> || vectorizable_shift (vinfo, stmt_info, NULL, node, cost_vec)
> || vectorizable_condition (vinfo, stmt_info, NULL, node, cost_vec)
> || vectorizable_comparison (vinfo, stmt_info, NULL, node, cost_vec)
> + || vectorizable_vect_call (vinfo, stmt_info, NULL, node, cost_vec)
> || (bb_vinfo
> && vectorizable_phi (bb_vinfo, stmt_info, node, cost_vec))
> || (is_a <loop_vec_info> (vinfo)
> @@ -13454,6 +13712,10 @@ vect_transform_stmt (vec_info *vinfo,
> done = vectorizable_call (vinfo, stmt_info, gsi, slp_node, NULL);
> break;
>
> + case vector_call_vec_info_type:
> + done = vectorizable_vect_call (vinfo, stmt_info, gsi, slp_node, NULL);
> + break;
> +
> case call_simd_clone_vec_info_type:
> done = vectorizable_simd_clone_call (vinfo, stmt_info, gsi,
> slp_node, NULL);
> @@ -13572,6 +13834,10 @@ get_related_vectype_for_scalar_type (machine_mode
> prevailing_mode,
> machine_mode simd_mode;
> tree vectype;
>
> + /* For vector types, get the inner most type. */
> + if (VECTOR_TYPE_P (scalar_type))
> + scalar_type = TREE_TYPE (scalar_type);
> +
> if ((!INTEGRAL_TYPE_P (scalar_type)
> && !POINTER_TYPE_P (scalar_type)
> && !SCALAR_FLOAT_TYPE_P (scalar_type))
> @@ -14801,6 +15067,10 @@ vect_get_vector_types_for_stmt (vec_info *vinfo,
> stmt_vec_info stmt_info,
> else
> scalar_type = TREE_TYPE (gimple_get_lhs (stmt));
>
> + /* If scalar_type is a vector type, get the innner typ. */
> + if (VECTOR_TYPE_P (scalar_type))
> + scalar_type = TREE_TYPE (scalar_type);
> +
> if (dump_enabled_p ())
> {
> if (group_size)
> @@ -14901,4 +15171,3 @@ vect_gen_len (tree len, tree start_index, tree
> end_index, tree len_limit)
>
> return stmts;
> }
> -
> diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
> index
> 35e531c11c178839bcf3218e58ad7521662d3b42..1bba7702b628cc4012848e1d887be03fe3d2533d
> 100644
> --- a/gcc/tree-vectorizer.h
> +++ b/gcc/tree-vectorizer.h
> @@ -229,6 +229,7 @@ enum stmt_vec_info_type {
> shift_vec_info_type,
> op_vec_info_type,
> call_vec_info_type,
> + vector_call_vec_info_type,
> call_simd_clone_vec_info_type,
> assignment_vec_info_type,
> condition_vec_info_type,
>
>
>
--
Richard Biener <[email protected]>
SUSE Software Solutions Germany GmbH,
Frankenstrasse 146, 90461 Nuernberg, Germany;
GF: Jochen Jaser, Andrew McDonald, Werner Knoblich; (HRB 36809, AG Nuernberg)
