[PATCH][vect][WIP]: teach the vectorizer to be able to vectorize already vectorized code [PR119187]

[Tamar Christina]

NOTE: This branch is still very much WIP.  It has not been cleaned up enough
to be upstreamable yet, but I am sending it now to give Richi an idea of the
design direction.

This should apply to any reasonably recent trunk.  I am currently based on
g:c8ca8e647caf945dc4694107884cf0f4eba5da2f.

The branch builds and runs, but it does not bootstrap.  I have only tested
incremental builds.

Some parts are more worked out than others, since I have not reached all of
them yet.  I have tried to write down the intended direction as well, so that
you can give me "the look", Richi :)

I *also* want to point out that the AArch64 code here is just illustrative and
not how we will implement these.  Work is ongoing to replace our intrinsics with
ones that generate new overloaded builtins, to allow legality checks to be
cleaner and also not need the type shims between the internal vectorizer
represenations and intrinsics. i.e. atm we need a type shim between

int32x4_t and a signed type made form V4SI.  This will be handled by adding
to the build-ins internal overloads with the correct types.

### Introduction

The main intended purpose of this work is threefold:

1. Support re-optimizing intrinsics and vector code in SLP that was created by
   earlier vectorization passes.  For example, two back-to-back loops could be
   completely unrolled and vectorized during loop vectorization, but would not
   then be optimized together.  The idea is to let SLP discovery re-analyze the
   vector code as a single entity.  This makes the long-standing question of
   whether BB SLP or loop vectorization should run first less important, since
   we can re-vectorize the generated vector code.

2. Support Arm SVE2's Hybrid Vector Length Agnostic (HVLA) programming model.
   The intention is to transform code written with intrinsics for older AdvSIMD
   code into VLA code, so that we can use longer vector lengths.

3. Support workloads, particularly scientific-computing workloads, in which
   users manually create vector types that are longer than the target supports
   using GCC vector extensions.  The expectation is that the compiler can then
   decompose those operations into supported operations.  Clang can do this for
   some examples, although its approach seems quite limited and appears to work
   only when the operations are constructors of PHI nodes.

To solve these problems, the approach I have taken in GCC is to model vector
types as possible base "scalar" types.  The main advantage is that the current
compiler workflows can stay the same.  Fundamentally, this does not change most
of the vectorizer, apart from analysis and the usual extensions to the code
generators.

This also means that the code remains subject to cost modelling and to other
features we have added over the years, such as epilogue masks.

As an example:

```c
#include <arm_neon.h>

void foo (uint8_t *a, uint8_t *b, uint8_t *c, int n)
{
  for (int i = 0; i < n; i += 16, a += 16, b += 16, c += 16)
    {
      uint8x8_t av1 = vld1_u8 (a);
      uint8x8_t av2 = vld1_u8 (a + 8);
      uint8x8_t bv1 = vld1_u8 (b);
      uint8x8_t bv2 = vld1_u8 (b + 8);
      vst1_u8 (c, vadd_u8 (av1, bv1));
      vst1_u8 (c + 8, vadd_u8 (av2, bv2));
    }
}
```

At the moment, `-O3` generates:

```asm
.L3:
        ldr     d29, [x0, x4]
        ldr     d28, [x1, x4]
        ldr     d31, [x7, x4]
        ldr     d30, [x6, x4]
        add     v28.8b, v29.8b, v28.8b
        add     v30.8b, v31.8b, v30.8b
        str     d28, [x2, x4]
        str     d30, [x5, x4]
        add     x4, x4, 16
        cmp     w3, w4
        bgt     .L3
```

This underuses the available load bandwidth.  Ideally these would be Q-sized
loads and one ADD, i.e. SLP would combine them.

With the branch, `-O3 -march=armv8-a` generates:

```asm
.L5:
        ldr     q31, [x0, x4]
        ldr     q30, [x1, x4]
        add     v30.16b, v31.16b, v30.16b
        str     q30, [x2, x4]
        add     x4, x4, 16
        cmp     x4, x7
        bne     .L5
```

`-O3 -march=armv8-a+sve` generates:

```asm
.L5:
        ldr     q31, [x0, x3]
        ldr     q30, [x1, x3]
        add     v30.16b, v31.16b, v30.16b
        str     q30, [x2, x3]
        add     x3, x3, 16
        cmp     x5, x3
        bne     .L5
        lsl     w3, w4, 4
        cmp     w6, w3
        beq     .L1
.L4:
        sub     w6, w6, w3
        ubfiz   x3, x3, 3, 32
        whilelo p7.b, xzr, x6
        ld1b    z29.b, p7/z, [x0, x3]
        ld1b    z28.b, p7/z, [x1, x3]
        add     z28.b, z28.b, z29.b
        st1b    z28.b, p7, [x2, x3]
.L1:
```

`-O3 -march=armv8-a+sve -mautovec-preference=sve-only` generates:

```asm
.L4:
        ld1b    z31.b, p7/z, [x0, x4]
        ld1b    z30.b, p7/z, [x1, x4]
        add     z30.b, z30.b, z31.b
        st1b    z30.b, p7, [x2, x4]
        add     x4, x4, x5
        whilelo p7.b, x4, x3
        b.any   .L4
```

### Representation

There is no change to the SLP trees themselves.  The vector operation is added
to the node with N copies.  `print_slp` will be updated to show that these are
vector lanes rather than individual scalar operations.  This is a
representational thing since SLP internal nodes are vectorized from the
representation and not the scalar elements,  but it makes everything line up.

The group size for the operation is then set to the number of units in the
incoming vector.  For memory accesses, `dr_group_size` is increased too.  At
the moment, and probably for the first version, I only plan to support linear
loads and load-lane operations.  Recognizing the other load types requires a
target hook and time to fill in the DRs.

For BB SLP, the current code starts a seed whenever it finds a store with a
vector type that the target does not support.  This then fails analysis because
no vector type can be found.  The plan is to make vector type selection return
the largest supported type for the given inner mode.  During `vectorizable_*`
analysis, we would then avoid doing any other check beyond whether the target
supports the new mode for the given operation.  To match the incoming VF, we
would increase the required unroll factor so that we get the same effective VF.

Decreasing the VF is not possible at the moment.  For example:

```c
#include <arm_neon.h>

void foo (uint8_t *a, uint8_t *b, uint8_t *c, int n)
{
  for (int i = 0; i < n; i += 32, a += 32, b += 32, c += 32)
    {
      uint8x16_t av1 = vld1q_u8 (a);
      uint8x16_t av2 = vld1q_u8 (a + 16);
      uint8x16_t bv1 = vld1q_u8 (b);
      uint8x16_t bv2 = vld1q_u8 (b + 16);
      vst1q_u8 (c, vaddq_u8 (av1, bv1));
      vst1q_u8 (c + 16, vaddq_u8 (av2, bv2));
    }
}
```

cannot collapse down to:

```asm
.L5:
        ldr     q31, [x0, x4]
        ldr     q30, [x1, x4]
        add     v30.16b, v31.16b, v30.16b
        str     q30, [x2, x4]
        add     x4, x4, 16
        cmp     x4, x7
        bne     .L5
```

This would require fractional VF support; in other words, it would require
support for re-rolling the loop.  With the current design, that should work
naturally once we add fractional VF support.

### Areas that need the most changes

These are the areas that I think need the most work.  The branch is currently
structured so that the number-of-units adjustment is done in each callee rather
than when building the SLP representation.  This is purely temporary, and there
are `#if 1`s around the temporary code so that I can remove it later.

This lets me change the functions one at a time instead of having everything
ICE because asserts are hit in areas that I have not reached yet.

### Vector Operations

There are two types of vector operations that we want to support here.

The first type uses a normal GIMPLE representation.  In the examples above,
`vadd` is lowered as a normal `PLUS_EXPR`, so testing and transforming it is
straightforward.

In many other cases, however, we see target intrinsics.  To vectorize these, I
have added a new `vectorizable_vect_call` that uses a target hook,
`targetm.vectorize.translate_builtin_call`, to fill in vectorization
information about the call.  In particular, the hook must be able to tell the
vectorizer whether we are going from an unmasked mode to a masked mode, whether
there is a builtin with masking support that we can use, and where the mask
argument appears in the argument list.

### Vector Booleans

Normally, the vectorizer has to detect booleans in order to differentiate data
operations from control operations.  When vectorizing existing vector code,
that has already been done for us.  However, we still need to be able to expand
the lanes and, if necessary, convert between boolean representation models.

For example, when changing AdvSIMD code into SVE code, the boolean
representation has to change because the underlying truth type is different.

To fix this and support boolean operations, there is a new pattern,
`vect_recog_vect_bool_pattern`, that runs only on boolean vector comparisons.
It transforms them using a pattern as a scalar boolean.  That is, it creates:

```c
cmp ? -1 : 0
```

from a vector comparison, with the type of the operation set to the truth type
of the original code.  This allows us to vectorize these as if they were normal
scalar comparisons.

For example:

```c
#include <arm_neon.h>

void foo (uint8_t *a, uint8_t *b, uint8_t *c, int n)
{
  for (int i = 0; i < n; i += 8, a += 8, b += 8, c += 8)
    {
      uint8x8_t av1 = vld1_u8 (a);
      uint8x8_t bv1 = vld1_u8 (b);
      vst1_u8 (c, vcgt_u8 (av1, bv1));
    }
}
```

compiled with `-march=armv9-a+sve -O3` gives:

```asm
.L5:
        ldr     q31, [x1, x3]
        ldr     q30, [x0, x3]
        cmhi    v30.16b, v30.16b, v31.16b
        str     q30, [x2, x3]
        add     x3, x3, 16
        cmp     x3, x5
        bne     .L5
        lsl     w3, w4, 4
        cmp     w6, w3
        beq     .L1
.L4:
        sub     w6, w6, w3
        ptrue   p6.b, all
        ubfiz   x3, x3, 3, 32
        whilelo p7.b, xzr, x6
        ld1b    z28.b, p7/z, [x0, x3]
        ld1b    z27.b, p7/z, [x1, x3]
        cmplo   p15.b, p6/z, z27.b, z28.b
        mov     z29.b, p15/z, #-1
        st1b    z29.b, p7, [x2, x3]
.L1:
        ret
        .p2align 2,,3
.L6:
        mov     x4, 0
        .p2align 5,,15
.L3:
        ldr     d26, [x0, x4]
        ldr     d25, [x1, x4]
        cmhi    v25.8b, v26.8b, v25.8b
        str     d25, [x2, x4]
        add     x4, x4, 8
        cmp     w3, w4
        bgt     .L3
        ret
.L7:
        mov     w3, 0
        b       .L4
```

The vectorizer is now able to handle the conversion between different boolean
modes while preserving the lane values of the original code.

### Vector Patterns

The incoming code may use vector operations to perform masking through manually
if-converted code.  For example:

```c
#include <arm_neon.h>
#include <arm_sve.h>

void foo (uint8_t *a, uint8_t *b, uint8_t *c, int n)
{
  for (int i = 0; i < n; i += 8, a += 8, b += 8, c += 8)
    {
      uint8x8_t av1 = vld1_u8 (a);
      uint8x8_t bv1 = vld1_u8 (b);
      uint8x8_t cmp = vcgt_u8 (bv1, vdup_n_u8 (0));
      uint8x8_t add = vadd_u8 (bv1, vdup_n_u8 (1));
      uint8x8_t sel = vbsl_u8 (cmp, add, bv1);
      vst1_u8 (a, sel);
    }
}
```

This is a manually if-converted conditional add.  We can re-vectorize it as-is,
but what we really want is to optimize it as a masked operation.

Since the operations are opaque to the vectorizer, there is a new target hook,
`targetm.vectorize.recog_patterns_builtin_call`, and a pattern,
`vect_recog_vect_scalar_pattern`, that allow the backend to define which
sequence of operations should become a vectorizer pattern.

I have tried to avoid leaking internal vectorizer abstractions here, such as
pattern definitions.  The hook simply asks for the expected output type and the
operations that make up the pattern, returning the new operation to put in
place.

I have not fully worked out this hook yet, or the previous one, so the design
is still in flux.

With the hook, the example above is vectorized as:

```asm
        mov     z30.b, #1
.L4:
        ld1b    z31.b, p7/z, [x1, x2]
        cmpne   p6.b, p7/z, z31.b, #0
        add     z31.b, p6/m, z31.b, z30.b
        st1b    z31.b, p7, [x0, x2]
        incb    x2
        whilelo p7.b, x2, x3
        b.any   .L4
```

There will also be default implementations for patterns that we can detect from
generic code, such as `VEC_PERM_EXPR`.

### If-Conversion

Some vector code manually creates vector control-flow statements using
reduction operations:

```c
<bool> == vector reduction chain
if (<bool>)
  add ...
```

Ideally, we would if-convert these back down using the right values as masks.
For instance, if the boolean above was created by:

```c
a = b `cmp` d
e = smaxv (a)
if (e != 0)
  add ...
```

we should be able to transform that into:

```c
a = b `cmp` d
cond_add (a, ...)
```

The idea is to use `targetm.vectorize.recog_patterns_builtin_call`, if
possible, to do this as well.

This is not currently implemented.

### Live Values

Scalar live-out values are easy to support, for example for reductions.
However, if the live value is a vector, we cannot currently do anything.

For example, if the function returns `a` and `a` is an `int32x4_t`, then we are
stuck because we are required to return a 128-bit vector without changing the
order or values of the lanes.

In some cases, we can still handle this depending on the operation.  For
example, if the value is a simple load and accumulate then we can, since HVLA
adds new instructions, including loads, permutes, stores, and reductions, that
work on 128-bit element sizes.  Effectively, every "element" is an AdvSIMD
vector, and there are operations that work on those containers, such as:

https://dougallj.github.io/asil/doc/addqv_z_p_z_16.html

I am not entirely sure how to model this yet.  Richard S chose not to introduce
new modes here, but instead to represent the "element" and result modes using
existing modes.  For `ADDQV`, this is a reduction from element mode, say
`VNx16QI`, into a normal vector mode, also say `VNx16QI`.

This representation may introduce ambiguity in GIMPLE, but given that these
operations are essentially opaque to GIMPLE, perhaps that is OK.  I could use a
property on the SLP node to distinguish them in the vectorizer.

### Reductions, Including Accumulator Splits

Reductions have to make the same distinction as live values between reductions
to scalars and reductions to vectors, then use the appropriate instructions.
When we do accumulator splits, for example, the live reduction on the live
value needs to be updated as well.

The initial values for reductions also need either to be splats or sequences
with a recognizable pattern, so that when we extend the VF we can extend the
pattern.

Alternatively, we must be able to find a neutral value to insert in the new
lanes that will not affect the reduction, such as inserting zeros for addition.

### Inductions

Inductions have the same issue as reductions and permutes, but I expect them to
be much less common.  Most people still use scalar inductions with their vector
code.  I therefore do not think I will spend much time supporting vector
inductions in the first version, unless most of the support falls out naturally
from the reduction and permute work.

### Unpacking, Matching `_hi` and `_lo`

Transforming from an ISA that has `_hi`/`_lo` unpacking support to one that has
`_odd`/`_even` would require us to recognize the `_hi`/`_lo` operations and
treat them as the same operation.  The loop was effectively unrolled for this
anyway.  This would be similar to `TWO_OPERANDS` nodes, and probably just an
extension of them.

This would allow us to re-synthesize new `_hi`/`_lo` operations when increasing
the VF.  If changing ISA, we would also need to be able to pack the operation
again.

This is still a TODO.  I think I can make it work, but I am not sure whether it
belongs in v1.

### Permutes

Permutes will be a challenge.  Extending an existing short permute to a longer
permute, or to VLA, can only happen if the permute represents some kind of
regular pattern, such as `REVERSE` or `ZIP`.  We will therefore be limited in
what we can transform.

### Alias Analysis

I think alias analysis should work naturally once I can describe the DR
correctly.  I have listed it here because I still have a TODO to check this.

---

Anyway, this is a long email.  What do you think, Richi?  I still have enough
time to pivot.  This is 1 of 3 large vectorizer patches that I have planned and
am working on for GCC 17.


gcc/ChangeLog:

        * config/aarch64/aarch64-builtins.cc (aarch64_eq_type_p):
        (aarch64_lookup_fndecl_by_name):
        (aarch64_find_builtin_fndecl_1):
        (aarch64_find_builtin_fndecl):
        (aarch64_internal_translate_builtin_call):
        * config/aarch64/aarch64-protos.h:
        * config/aarch64/aarch64-sve-builtins.cc 
(lookup_registered_function_by_name):
        (canonicalize_function_name):
        (lookup_fndecl_by_name):
        * config/aarch64/aarch64-sve-builtins.h (lookup_fndecl_by_name):
        * config/aarch64/aarch64-sve2.md (*cond_add_bsl<mode>):
        * config/aarch64/aarch64.cc (aarch64_internal_translate_builtin_call):
        (aarch64_autovectorize_translate_builtin_call):
        (TARGET_VECTORIZE_TRANSLATE_BUILTIN_CALL):
        * doc/tm.texi:
        * doc/tm.texi.in:
        * target.def:
        * tree-phinodes.cc (ideal_phi_node_len):
        * tree-phinodes.h (make_phi_node):
        * tree-vect-data-refs.cc (vect_analyze_group_access_1):
        (vect_analyze_data_ref_access):
        * tree-vect-loop.cc (vectorizable_reduction):
        * tree-vect-patterns.cc (vect_recog_vect_bool_pattern):
        (vect_recog_vect_scalar_pattern):
        * tree-vect-slp.cc (vect_build_slp_tree):
        (vect_build_slp_tree_2):
        (vect_build_slp_instance):
        (vect_analyze_slp_instance):
        (vect_slp_check_for_roots):
        (vect_transform_slp_perm_load_1):
        (vect_schedule_slp):
        * tree-vect-stmts.cc (get_load_store_type):
        (vectorizable_operation):
        (vectorizable_store):
        (vectorizable_load):
        (vect_is_simple_cond):
        (vect_shim_call_arg):
        (vectorizable_vect_call):
        (vect_analyze_stmt):
        (vect_transform_stmt):
        (get_related_vectype_for_scalar_type):
        (vect_get_vector_types_for_stmt):
        (vect_gen_len):
        * tree-vectorizer.h (enum stmt_vec_info_type):

---
diff --git a/gcc/config/aarch64/aarch64-builtins.cc 
b/gcc/config/aarch64/aarch64-builtins.cc
index 
b9cc2bf404f612a9688db4ce2f6a042b148ac938..54d7d2e918f613fa449ab268b84581d7ada483f3
 100644
--- a/gcc/config/aarch64/aarch64-builtins.cc
+++ b/gcc/config/aarch64/aarch64-builtins.cc
@@ -29,6 +29,7 @@
 #include "rtl.h"
 #include "tree.h"
 #include "gimple.h"
+#include "cfgloop.h"
 #include "ssa.h"
 #include "memmodel.h"
 #include "tm_p.h"
@@ -42,6 +43,7 @@
 #include "expr.h"
 #include "langhooks.h"
 #include "gimple-iterator.h"
+#include "tree-vectorizer.h"
 #include "case-cfn-macros.h"
 #include "regs.h"
 #include "emit-rtl.h"
@@ -5591,6 +5593,201 @@ aarch64_general_gimple_fold_builtin (unsigned int 
fcode, gcall *stmt,
   return new_stmt;
 }
 
+/* Intrinsics type have a different type decorator than normal types.  So
+   compare signess and mode instead.  */
+static bool
+aarch64_eq_type_p (tree ty0, tree ty1)
+{
+  if (TYPE_UNSIGNED (ty0) != TYPE_UNSIGNED (ty1))
+    return false;
+
+  return TYPE_MODE (ty0) == TYPE_MODE (ty1);
+}
+
+static tree aarch64_lookup_fndecl_by_name (const char *);
+
+static tree
+aarch64_find_builtin_fndecl_1 (tree fndecl, vec<tree> &types,
+                              tree req_ret_ty,
+                              tree req_arg0_ty, tree req_arg1_ty,
+                              int *maskopno = NULL)
+{
+  tree t = fndecl;
+  do
+    {
+      if (TREE_CODE (t) != FUNCTION_DECL)
+        break;
+
+      tree fntype = TREE_TYPE (t);
+      tree ret_ty = TREE_TYPE (fntype);
+      tree args = TYPE_ARG_TYPES (fntype);
+
+      if (!TREE_CHAIN (args))
+       break;
+
+      tree arg0_ty = TREE_VALUE (args);
+      tree arg1_ty = TREE_VALUE (TREE_CHAIN (args));
+
+      if (aarch64_eq_type_p (ret_ty, req_ret_ty)
+         && aarch64_eq_type_p (arg0_ty, req_arg0_ty)
+         && aarch64_eq_type_p (arg1_ty, req_arg1_ty))
+       {
+         types.create (3);
+         types.quick_push (ret_ty);
+         types.quick_push (arg0_ty);
+         types.quick_push (arg1_ty);
+         return t;
+       }
+
+      /* Check if it could be masked, very wrong and unsafe but good enough.  
*/
+      if (maskopno && VECTOR_BOOLEAN_TYPE_P (arg0_ty))
+       {
+         *maskopno = 0;
+         tree arg0_ty = TREE_VALUE (TREE_CHAIN (args));
+         tree arg1_ty = TREE_VALUE (TREE_CHAIN (TREE_CHAIN (args)));
+         if (aarch64_eq_type_p (ret_ty, req_ret_ty)
+             && aarch64_eq_type_p (arg0_ty, req_arg0_ty)
+             && aarch64_eq_type_p (arg1_ty, req_arg1_ty))
+           {
+             types.create (4);
+             types.quick_push (ret_ty);
+             types.quick_push (TREE_VALUE (args));
+             types.quick_push (arg0_ty);
+             types.quick_push (arg1_ty);
+             return t;
+           }
+       }
+
+      if (t == fndecl)
+       t = DECL_CHAIN (fndecl);
+      else
+       t = DECL_CHAIN (t);
+    }
+  while (t);
+
+  return NULL;
+}
+
+static tree
+aarch64_find_builtin_fndecl (tree fndecl, tree req_ret_ty,
+                            tree req_arg0_ty, tree req_arg1_ty,
+                            int *maskopno, vec<tree> &types,
+                            const char *fallback_name = nullptr)
+{
+  tree res = aarch64_find_builtin_fndecl_1 (fndecl, types,
+                                           req_ret_ty, req_arg0_ty,
+                                           req_arg1_ty);
+  if (res)
+    return res;
+
+  if (fallback_name)
+    {
+      tree t = aarch64_lookup_fndecl_by_name (fallback_name);
+      gcc_assert (t);
+      *maskopno = -1;
+      res = aarch64_find_builtin_fndecl_1 (t, types, req_ret_ty, req_arg0_ty,
+                                          req_arg1_ty, maskopno);
+      return res;
+    }
+
+  return NULL_TREE;
+}
+
+static tree
+aarch64_lookup_fndecl_by_name (const char *name)
+{
+  tree id = get_identifier (name);
+  if (!id)
+    return NULL_TREE;
+
+  extern tree lookup_name (tree);
+  tree decl = lookup_name (id);
+  if (!decl || TREE_CODE (decl) != FUNCTION_DECL)
+    return aarch64_sve::lookup_fndecl_by_name (name);
+
+  return decl;
+}
+
+/* Implement TARGET_VECTORIZE_TRANSLATE_BUILTIN_CALL.  */
+gcall *
+aarch64_internal_translate_builtin_call (vec_info *vinfo,
+                                        slp_tree node,
+                                        stmt_vec_info stmt_info,
+                                        int *maskopno,
+                                        vec<tree> &types,
+                                        tree *vectype_out)
+{
+  gcall *stmt = dyn_cast <gcall *> (STMT_VINFO_STMT (stmt_info));
+  gcc_assert (stmt);
+  tree fndecl = gimple_call_fndecl (stmt);
+  unsigned int code = DECL_MD_FUNCTION_CODE (fndecl);
+  unsigned int subcode = code >> AARCH64_BUILTIN_SHIFT;
+
+  /* There's nothing to do for SVE at this point in time, so bail out.  */
+  if ((code & AARCH64_BUILTIN_CLASS) == AARCH64_BUILTIN_SVE)
+    return NULL;
+
+  switch (subcode)
+    {
+    BUILTIN_VSDQ_I (BINOP_SSU, suqadd, 0, DEFAULT)
+    BUILTIN_VSDQ_I (BINOP_UUS, usqadd, 0, DEFAULT)
+      {
+       tree arg0 = gimple_call_arg (stmt, 0);
+       tree arg1 = gimple_call_arg (stmt, 1);
+       tree ty0 = get_vectype_for_scalar_type (vinfo, TREE_TYPE (arg0));
+       tree ty1 = get_vectype_for_scalar_type (vinfo, TREE_TYPE (arg1));
+       if (!ty0 || !ty1)
+         return NULL;
+
+       /* For usqadd/suqadd the result type is the vectorized type of arg0.  */
+       tree new_fndecl = aarch64_find_builtin_fndecl (fndecl, ty0, ty0, ty1,
+                                                      maskopno, types,
+                                                      "svsqadd_u8_x");
+       if (!new_fndecl)
+         return NULL;
+
+       /* This is type incorrect, but we don't have the arguments yet.  The
+          vectorizer will take care of it.  */
+       gcall *new_call;
+       if (*maskopno == -1)
+         new_call = gimple_build_call (new_fndecl, 2, NULL, NULL);
+       else
+         new_call = gimple_build_call (new_fndecl, 3, NULL, NULL, NULL);
+       *vectype_out = ty0;
+       return new_call;
+      }
+    BUILTIN_VDQQH (BSL_P, simd_bsl, 0, DEFAULT)
+    VAR2 (BSL_P, simd_bsl,0, DEFAULT, di, v2di)
+    BUILTIN_VSDQ_I_DI (BSL_U, simd_bsl, 0, DEFAULT)
+    BUILTIN_VALLDIF (BSL_S, simd_bsl, 0, QUIET)
+      {
+       tree arg0 = gimple_call_arg (stmt, 0);
+       tree arg1 = gimple_call_arg (stmt, 1);
+       tree ty0 = get_vectype_for_scalar_type (vinfo, TREE_TYPE (arg0));
+       tree ty1 = get_vectype_for_scalar_type (vinfo, TREE_TYPE (arg1));
+       if (!ty0 || !ty1)
+         return NULL;
+
+       /* For usqadd/suqadd the result type is the vectorized type of arg0.  */
+       tree new_fndecl = aarch64_find_builtin_fndecl (fndecl, ty1, ty0, ty1,
+                                                      maskopno, types,
+                                                      "svbsl_u8");
+       if (!new_fndecl)
+         return NULL;
+
+       types.safe_push (types[1]);
+       /* BSL does not support masks.  */
+       gcc_assert (*maskopno == -1);
+       *vectype_out = ty0;
+       return gimple_build_call (new_fndecl, 3, NULL, NULL, NULL);
+      }
+    default:
+      return NULL;
+    }
+
+  return NULL;
+}
+
 void
 aarch64_atomic_assign_expand_fenv (tree *hold, tree *clear, tree *update)
 {
diff --git a/gcc/config/aarch64/aarch64-protos.h 
b/gcc/config/aarch64/aarch64-protos.h
index 
513b556398fac7c733de47c27587c3aaa07cbcf3..fb7062b3d75902e1298b9d64846daf69b418fbb7
 100644
--- a/gcc/config/aarch64/aarch64-protos.h
+++ b/gcc/config/aarch64/aarch64-protos.h
@@ -1185,6 +1185,7 @@ namespace aarch64_sve {
   void handle_arm_sme_h (bool);
   void handle_arm_neon_sve_bridge_h (bool);
   tree builtin_decl (unsigned, bool);
+  tree lookup_fndecl_by_name (const char *);
   bool builtin_type_p (const_tree);
   bool builtin_type_p (const_tree, unsigned int *, unsigned int *);
   const char *mangle_builtin_type (const_tree);
diff --git a/gcc/config/aarch64/aarch64-sve-builtins.cc 
b/gcc/config/aarch64/aarch64-sve-builtins.cc
index 
cd78713440f61ac18146f0be49a43a204dbb3ae1..fe5e267f087e1b2b62885695785e9065bee4d85a
 100644
--- a/gcc/config/aarch64/aarch64-sve-builtins.cc
+++ b/gcc/config/aarch64/aarch64-sve-builtins.cc
@@ -1063,6 +1063,45 @@ static GTY(()) hash_table<registered_function_hasher> 
*function_table;
    are IDENTIFIER_NODEs.  */
 static GTY(()) hash_map<tree, registered_function *> *overload_names[2];
 
+/* Return the decl of the registered SVE function called NAME, or null if
+   no such function has been registered.  */
+static tree
+lookup_registered_function_by_name (const char *name)
+{
+  unsigned int length = vec_safe_length (registered_functions);
+  for (unsigned int i = 0; i < length; ++i)
+    {
+      registered_function *rfn = (*registered_functions)[i];
+      if (!rfn || TREE_CODE (rfn->decl) != FUNCTION_DECL)
+       continue;
+
+      tree decl_name = DECL_NAME (rfn->decl);
+      if (decl_name && strcmp (IDENTIFIER_POINTER (decl_name), name) == 0)
+       return rfn->decl;
+    }
+
+  return NULL_TREE;
+}
+
+/* Return the ACLE spelling that corresponds to NAME.  */
+static const char *
+canonicalize_function_name (const char *name)
+{
+  if (startswith (name, "__builtin_aarch64_"))
+    return name + strlen ("__builtin_aarch64_");
+
+  if (startswith (name, "__builtin_sve_"))
+    return name + strlen ("__builtin_sve_");
+
+  if (startswith (name, "__builtin_")
+      && (startswith (name + strlen ("__builtin_"), "sv")
+         || startswith (name + strlen ("__builtin_"), "arm_")
+         || startswith (name + strlen ("__builtin_"), "__arm_")))
+    return name + strlen ("__builtin_");
+
+  return name;
+}
+
 /* Record that TYPE is an ABI-defined SVE type that contains NUM_ZR SVE vectors
    and NUM_PR SVE predicates.  MANGLED_NAME, if nonnull, is the ABI-defined
    mangling of the type.  mangling of the type.  ACLE_NAME is the <arm_sve.h>
@@ -4939,6 +4978,29 @@ builtin_decl (unsigned int code, bool)
   return (*registered_functions)[code]->decl;
 }
 
+/* Return the decl for the SVE builtin called NAME, if one exists.  Accept
+   both the ACLE spelling and the common builtin-prefixed aliases.  */
+tree
+lookup_fndecl_by_name (const char *name)
+{
+  const char *canonical_name = canonicalize_function_name (name);
+  tree decl = lookup_registered_function_by_name (canonical_name);
+  if (decl)
+    return decl;
+
+  /* Internal callers sometimes need SVE decls even when arm_sve.h was not
+     included in the source.  Materialize them lazily in that case.  */
+  if (!function_table && (lang_GNU_C () || lang_GNU_CXX ()))
+    {
+      handle_arm_sve_h (false);
+      decl = lookup_registered_function_by_name (canonical_name);
+      if (decl)
+       return decl;
+    }
+
+  return NULL_TREE;
+}
+
 /* Implement #pragma GCC aarch64 "arm_sme.h".  */
 void
 handle_arm_sme_h (bool function_nulls_p)
diff --git a/gcc/config/aarch64/aarch64-sve-builtins.h 
b/gcc/config/aarch64/aarch64-sve-builtins.h
index 
f9e8c4cd7293a379e3334fecb846121cf88e0ebf..d4123dbe8b3c0a60f1b1a6b1a398d05989987c23
 100644
--- a/gcc/config/aarch64/aarch64-sve-builtins.h
+++ b/gcc/config/aarch64/aarch64-sve-builtins.h
@@ -856,6 +856,7 @@ bool vector_cst_all_same (tree, unsigned int);
 bool is_ptrue (tree, unsigned int);
 bool is_pfalse (tree);
 const function_instance *lookup_fndecl (tree);
+tree lookup_fndecl_by_name (const char *);
 
 /* Try to find a mode with the given mode_suffix_info fields.  Return the
    mode on success or MODE_none on failure.  */
diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc
index 
5a859e12b1a55e70284469e3e9c175eee849e18d..a9f3af387ddce6b86370992a4dc9717df5f7e82c
 100644
--- a/gcc/config/aarch64/aarch64.cc
+++ b/gcc/config/aarch64/aarch64.cc
@@ -103,6 +103,7 @@
 #include "aarch64-sched-dispatch.h"
 #include "aarch64-json-tunings-printer.h"
 #include "aarch64-json-tunings-parser.h"
+#include "aarch64-builtins.h"
 
 /* This file should be included last.  */
 #include "target-def.h"
@@ -23868,6 +23869,24 @@ aarch64_autovectorize_vector_modes (vector_modes 
*modes, bool)
   return flags;
 }
 
+/* This is just temporary as I don't have an entry point into the new 
intrinsics
+   framework yet.  */
+extern gcall *aarch64_internal_translate_builtin_call (vec_info *, slp_tree,
+                                               stmt_vec_info, int*,
+                                               vec<tree> &types, tree *);;
+
+/* Implement TARGET_VECTORIZE_TRANSLATE_BUILTIN_CALL.  */
+static gcall *
+aarch64_autovectorize_translate_builtin_call (vec_info *vinfo,
+                                             slp_tree node,
+                                             stmt_vec_info stmt_info,
+                                             int *maskopno, vec<tree> &types,
+                                             tree *vectype_out)
+{
+  return aarch64_internal_translate_builtin_call (vinfo, node, stmt_info,
+                                                 maskopno, types, vectype_out);
+}
+
 /* Implement TARGET_CONVERT_TO_TYPE.  Convert EXPR to TYPE.  */
 
 static tree
@@ -34098,6 +34117,10 @@ aarch64_libgcc_floating_mode_supported_p
 #define TARGET_VECTORIZE_AUTOVECTORIZE_VECTOR_MODES \
   aarch64_autovectorize_vector_modes
 
+#undef TARGET_VECTORIZE_TRANSLATE_BUILTIN_CALL
+#define TARGET_VECTORIZE_TRANSLATE_BUILTIN_CALL \
+  aarch64_autovectorize_translate_builtin_call
+
 #undef TARGET_CONVERT_TO_TYPE
 #define TARGET_CONVERT_TO_TYPE aarch64_convert_to_type
 
diff --git a/gcc/doc/tm.texi b/gcc/doc/tm.texi
index 
1fbf43c6c57799372752f581528e122689a33f15..4f48737bb916fefd51dd738a235679af1b4f693c
 100644
--- a/gcc/doc/tm.texi
+++ b/gcc/doc/tm.texi
@@ -6537,6 +6537,20 @@ current cost model is for the scalar version of a loop 
or block; otherwise
 it is for the vector version.
 @end deftypefn
 
+@deftypefn {Target Hook} {gcall *} TARGET_VECTORIZE_TRANSLATE_BUILTIN_CALL 
(vec_info *@var{vinfo}, struct _slp_tree *@var{node}, class _stmt_vec_info 
*@var{stmt_info}, int *@var{maskopno}, vec<tree> @var{&types}, tree 
*@var{vectype_out})
+This hook returns true if masked operation @var{ifn} (really of
+type @code{internal_fn}) should be considered more expensive to use than
+implementing the same operation without masking.  GCC can then try to use
+unconditional operations instead with extra selects.
+@end deftypefn
+
+@deftypefn {Target Hook} {gimple *} 
TARGET_VECTORIZE_RECOG_PATTERNS_BUILTIN_CALL (vec_info *@var{vinfo}, struct 
_slp_tree *@var{node}, class _stmt_vec_info *@var{stmt_info}, vec<gimple> 
@var{&chain}, tree *@var{vectype_out})
+This hook returns true if masked operation @var{ifn} (really of
+type @code{internal_fn}) should be considered more expensive to use than
+implementing the same operation without masking.  GCC can then try to use
+unconditional operations instead with extra selects.
+@end deftypefn
+
 @deftypefn {Target Hook} tree TARGET_VECTORIZE_BUILTIN_GATHER (const_tree 
@var{mem_vectype}, const_tree @var{index_type}, int @var{scale})
 Target builtin that implements vector gather operation.  @var{mem_vectype}
 is the vector type of the load and @var{index_type} is scalar type of
diff --git a/gcc/doc/tm.texi.in b/gcc/doc/tm.texi.in
index 
6ecbeda09ca9b6473f12f78f8805ced44907f146..bc6d60d2ec47e62a31596bd96f3d106ae65254fb
 100644
--- a/gcc/doc/tm.texi.in
+++ b/gcc/doc/tm.texi.in
@@ -4341,6 +4341,10 @@ address;  but often a machine-dependent strategy can 
generate better code.
 
 @hook TARGET_VECTORIZE_CREATE_COSTS
 
+@hook TARGET_VECTORIZE_TRANSLATE_BUILTIN_CALL
+
+@hook TARGET_VECTORIZE_RECOG_PATTERNS_BUILTIN_CALL
+
 @hook TARGET_VECTORIZE_BUILTIN_GATHER
 
 @hook TARGET_VECTORIZE_BUILTIN_SCATTER
diff --git a/gcc/target.def b/gcc/target.def
index 
25a94559e198aef2c3e842f02aa5414c5193c4a6..3c9a2aba29691b80ab502b057e9e7244e56758a4
 100644
--- a/gcc/target.def
+++ b/gcc/target.def
@@ -2098,6 +2098,33 @@ stores.",
  (const_tree vectype, const_tree index_type, int scale),
  NULL)
 
+/* Function to say how to transform a gimple call.  If transformation is
+   possible return the new gcall, otherwise NULL.  */
+DEFHOOK
+(translate_builtin_call,
+ "This hook returns true if masked operation @var{ifn} (really of\n\
+type @code{internal_fn}) should be considered more expensive to use than\n\
+implementing the same operation without masking.  GCC can then try to use\n\
+unconditional operations instead with extra selects.",
+ gcall *,
+ (vec_info *vinfo, struct _slp_tree *node, class _stmt_vec_info *stmt_info,
+  int *maskopno, vec<tree> &types, tree *vectype_out),
+ NULL)
+
+/* Function to recognize patterns during vectorization of gimple calls.  If a
+   pattern is matched then return the new pattern and other statements it
+   consumes, otherwise NULL.  */
+DEFHOOK
+(recog_patterns_builtin_call,
+ "This hook returns true if masked operation @var{ifn} (really of\n\
+type @code{internal_fn}) should be considered more expensive to use than\n\
+implementing the same operation without masking.  GCC can then try to use\n\
+unconditional operations instead with extra selects.",
+ gimple *,
+ (vec_info *vinfo, class _stmt_vec_info *stmt_info,
+  vec<gimple *> &chain, tree *vectype_out),
+ NULL)
+
 /* Target function to initialize the cost model for a loop or block.  */
 DEFHOOK
 (create_costs,
diff --git a/gcc/tree-phinodes.cc b/gcc/tree-phinodes.cc
index 
e0663d96b7d4b96a1a48e5bfb9fef1329e526ae7..d97c6148d2f303bf9280dbbf19a3610133d36f8c
 100644
--- a/gcc/tree-phinodes.cc
+++ b/gcc/tree-phinodes.cc
@@ -165,7 +165,7 @@ ideal_phi_node_len (int len)
 
 /* Return a PHI node with LEN argument slots for variable VAR.  */
 
-static gphi *
+gphi *
 make_phi_node (tree var, int len)
 {
   gphi *phi;
diff --git a/gcc/tree-phinodes.h b/gcc/tree-phinodes.h
index 
b6fa492cb248f4d0e1a7838d0b34da6ae3a108fd..08741bb4076d9459454120ead335e52e49953352
 100644
--- a/gcc/tree-phinodes.h
+++ b/gcc/tree-phinodes.h
@@ -23,6 +23,7 @@ along with GCC; see the file COPYING3.  If not see
 extern void phinodes_print_statistics (void);
 extern void reserve_phi_args_for_new_edge (basic_block);
 extern gphi *create_phi_node (tree, basic_block);
+gphi *make_phi_node (tree, int);
 extern void add_phi_arg (gphi *, tree, edge, location_t);
 extern void remove_phi_args (edge);
 extern void remove_phi_node (gimple_stmt_iterator *, bool);
diff --git a/gcc/tree-vect-data-refs.cc b/gcc/tree-vect-data-refs.cc
index 
0e57e1068d6a1906d1d8e9edb761d5dd4b7da758..eefd5ad2f91e0292cda52354bf70b406a62adbcf
 100644
--- a/gcc/tree-vect-data-refs.cc
+++ b/gcc/tree-vect-data-refs.cc
@@ -3176,6 +3176,17 @@ vect_analyze_group_access_1 (vec_info *vinfo, 
dr_vec_info *dr_info)
   data_reference *dr = dr_info->dr;
   tree step = DR_STEP (dr);
   tree scalar_type = TREE_TYPE (DR_REF (dr));
+  unsigned long nunits = 0;
+  /* TODO: Add? */
+  if (VECTOR_TYPE_P (scalar_type)
+      && !TYPE_VECTOR_SUBPARTS (scalar_type).is_constant (&nunits))
+    {
+      if (dump_enabled_p ())
+       dump_printf_loc (MSG_NOTE, vect_location,
+                       "Variable length vectors not supported yet: %T\n",
+                       step);
+         return false;
+    }
   HOST_WIDE_INT type_size = TREE_INT_CST_LOW (TYPE_SIZE_UNIT (scalar_type));
   stmt_vec_info stmt_info = dr_info->stmt;
   loop_vec_info loop_vinfo = dyn_cast <loop_vec_info> (vinfo);
@@ -3207,6 +3218,8 @@ vect_analyze_group_access_1 (vec_info *vinfo, dr_vec_info 
*dr_info)
        }
       groupsize = absu_hwi (dr_step) / type_size;
     }
+  else if (bb_vinfo && nunits > 0)
+    groupsize = nunits;
   else
     groupsize = 0;
 
@@ -3463,6 +3476,29 @@ vect_analyze_data_ref_access (vec_info *vinfo, 
dr_vec_info *dr_info)
        }
     }
 
+  /* Allow loads and stores with zero steps and vector types in BB
+     vectorization.  TODO:  This needs support for checking the type of access
+     so e.g. LOAD_LANES, Gathers etc.  */
+  if (!loop_vinfo && integer_zerop (step) && VECTOR_TYPE_P (scalar_type))
+    {
+      if (!TYPE_VECTOR_SUBPARTS (scalar_type).is_constant ())
+       {
+         if (dump_enabled_p ())
+           dump_printf_loc (MSG_NOTE, vect_location,
+                       "Variable length vectors not supported yet: %T\n",
+                       step);
+         return false;
+       }
+      DR_GROUP_FIRST_ELEMENT (stmt_info) = NULL;
+      DR_GROUP_NEXT_ELEMENT (stmt_info) = NULL;
+
+      if (dump_enabled_p () && 0)
+       dump_printf_loc (MSG_NOTE, vect_location,
+                       "Allowing vector memory access in BB\n");
+
+      return true;
+    }
+
   if (loop && nested_in_vect_loop_p (loop, stmt_info))
     {
       /* Interleaved accesses are not yet supported within outer-loop
@@ -3485,9 +3521,21 @@ vect_analyze_data_ref_access (vec_info *vinfo, 
dr_vec_info *dr_info)
   if (TREE_CODE (step) == INTEGER_CST)
     {
       HOST_WIDE_INT dr_step = TREE_INT_CST_LOW (step);
-      if (!tree_int_cst_compare (step, TYPE_SIZE_UNIT (scalar_type))
-         || (dr_step < 0
-             && !compare_tree_int (TYPE_SIZE_UNIT (scalar_type), -dr_step)))
+      bool sequantial = false;
+      poly_int64 step_p, type_size_p;
+      if (poly_int_tree_p (step, &step_p)
+         && poly_int_tree_p (TYPE_SIZE_UNIT (scalar_type), &type_size_p))
+       sequantial
+         = known_eq (step_p, type_size_p)
+           || (known_lt (step_p, 0)
+               && !known_eq (-step_p, type_size_p));
+      else
+       sequantial
+         = !tree_int_cst_compare (step, TYPE_SIZE_UNIT (scalar_type))
+           || (dr_step < 0
+               && !compare_tree_int (TYPE_SIZE_UNIT (scalar_type), -dr_step));
+
+      if (sequantial)
        {
          /* Mark that it is not interleaving.  */
          DR_GROUP_FIRST_ELEMENT (stmt_info) = NULL;
diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
index 
51e7f05100de7ab31994f3b4a30f5c5be8d8afdd..7dee95951eef95622c9554c1bbe01c3e02cd7894
 100644
--- a/gcc/tree-vect-loop.cc
+++ b/gcc/tree-vect-loop.cc
@@ -7184,12 +7184,17 @@ vectorizable_reduction (loop_vec_info loop_vinfo,
     gcc_unreachable ();
   bool lane_reducing = lane_reducing_op_p (op.code);
 
-  if (!POINTER_TYPE_P (op.type) && !INTEGRAL_TYPE_P (op.type)
-      && !SCALAR_FLOAT_TYPE_P (op.type))
+  tree scalar_type = op.type;
+  if (VECTOR_TYPE_P (scalar_type))
+    scalar_type = TREE_TYPE (scalar_type);
+
+  if (!POINTER_TYPE_P (scalar_type)
+      && !INTEGRAL_TYPE_P (scalar_type)
+      && !SCALAR_FLOAT_TYPE_P (scalar_type))
     return false;
 
   /* Do not try to vectorize bit-precision reductions.  */
-  if (!type_has_mode_precision_p (op.type)
+  if (!type_has_mode_precision_p (scalar_type)
       && op.code != BIT_AND_EXPR
       && op.code != BIT_IOR_EXPR
       && op.code != BIT_XOR_EXPR)
@@ -7444,7 +7449,7 @@ vectorizable_reduction (loop_vec_info loop_vinfo,
       if (!reduc_chain
          && known_eq (LOOP_VINFO_VECT_FACTOR (loop_vinfo), 1u))
        ;
-      else if (needs_fold_left_reduction_p (op.type, orig_code))
+      else if (needs_fold_left_reduction_p (scalar_type, orig_code))
        {
          /* When vectorizing a reduction chain w/o SLP the reduction PHI
             is not directly used in stmt.  */
@@ -7573,7 +7578,7 @@ vectorizable_reduction (loop_vec_info loop_vinfo,
   else if (reduction_type == COND_REDUCTION)
     {
       int scalar_precision
-       = GET_MODE_PRECISION (SCALAR_TYPE_MODE (op.type));
+       = GET_MODE_PRECISION (SCALAR_TYPE_MODE (scalar_type));
       cr_index_scalar_type = make_unsigned_type (scalar_precision);
       cr_index_vector_type = get_same_sized_vectype (cr_index_scalar_type,
                                                vectype_out);
diff --git a/gcc/tree-vect-patterns.cc b/gcc/tree-vect-patterns.cc
index 
5f5e3fb6201da379460c46e9d7ff1d01625d0998..ca806f7d8d94333fc4301b8a8e0da5ad0a4fad0b
 100644
--- a/gcc/tree-vect-patterns.cc
+++ b/gcc/tree-vect-patterns.cc
@@ -6243,6 +6243,140 @@ vect_recog_bool_pattern (vec_info *vinfo,
     return NULL;
 }
 
+static gimple *
+vect_recog_vect_bool_pattern (vec_info *vinfo,
+                             stmt_vec_info stmt_vinfo, tree *type_out)
+{
+  gimple *last_stmt = STMT_VINFO_STMT (stmt_vinfo);
+  enum tree_code rhs_code;
+  tree var, lhs, rhs, vectype;
+  gimple *pattern_stmt;
+
+  if (!is_gimple_assign (last_stmt))
+    return NULL;
+
+  var = gimple_assign_rhs1 (last_stmt);
+  lhs = gimple_assign_lhs (last_stmt);
+  rhs_code = gimple_assign_rhs_code (last_stmt);
+
+  if (rhs_code == VIEW_CONVERT_EXPR)
+    var = TREE_OPERAND (var, 0);
+
+  if (!VECTOR_BOOLEAN_TYPE_P (TREE_TYPE (var)))
+    return NULL;
+
+  if (rhs_code != VEC_COND_EXPR)
+    return NULL;
+
+  tree op0 = gimple_assign_rhs2 (last_stmt);
+  tree op1 = gimple_assign_rhs3 (last_stmt);
+  if (!integer_zerop (op1)
+      || !integer_all_onesp (op0))
+    return NULL;
+
+  /* Find the comparison and make it part of the pattern we'll make.  */
+  auto cmp_info = vinfo->lookup_def (var);
+  if (!cmp_info)
+    return NULL;
+
+  gimple *cmp_stmt = STMT_VINFO_STMT (cmp_info);
+  if (!is_gimple_assign (cmp_stmt)
+      || TREE_CODE_CLASS (gimple_assign_rhs_code (cmp_stmt)) != tcc_comparison)
+    return NULL;
+
+  vectype = get_vectype_for_scalar_type (vinfo, TREE_TYPE (lhs));
+  if (vectype == NULL_TREE)
+    return NULL;
+
+  /* Build a scalar type for the boolean result that when
+     vectorized matches the vector type of the result in
+      size and number of elements.  */
+  unsigned prec
+    = vector_element_size (tree_to_poly_uint64 (TYPE_SIZE (vectype)),
+                          TYPE_VECTOR_SUBPARTS (vectype));
+
+  tree type
+    = build_nonstandard_integer_type (prec, TYPE_UNSIGNED (TREE_TYPE (var)));
+
+  tree new_vectype = get_mask_type_for_scalar_type (vinfo, type);
+  if (new_vectype == NULL_TREE)
+    return NULL;
+
+  /* copy the scalar statement to get a new UID.  */
+  tree lhs_cmp
+    = vect_recog_temp_ssa_var (TREE_TYPE (gimple_get_lhs (cmp_stmt)), NULL);
+  gimple *copy_stmt
+     = gimple_build_assign (lhs_cmp, gimple_assign_rhs_code (cmp_stmt),
+                           gimple_assign_rhs1 (cmp_stmt),
+                           gimple_assign_rhs2 (cmp_stmt));
+  append_pattern_def_seq (vinfo, stmt_vinfo, copy_stmt, new_vectype, type);
+
+  tree lhs_ivar = vect_recog_temp_ssa_var (type, NULL);
+  pattern_stmt = gimple_build_assign (lhs_ivar, COND_EXPR, lhs_cmp,
+                                     build_all_ones_cst (type),
+                                     build_zero_cst (type));
+
+  *type_out = vectype;
+  vect_pattern_detected ("vect_recog_vect_bool_pattern", last_stmt);
+
+  return pattern_stmt;
+}
+
+static gimple *
+vect_recog_vect_scalar_pattern (vec_info *vinfo,
+                               stmt_vec_info stmt_vinfo, tree *type_out)
+{
+  gimple *last_stmt = STMT_VINFO_STMT (stmt_vinfo);
+  enum tree_code rhs_code;
+  tree var, lhs, rhs, vectype;
+  gimple *pattern_stmt;
+  loop_vec_info loop_vinfo = dyn_cast <loop_vec_info> (vinfo);
+  class loop *loop = loop_vinfo ? LOOP_VINFO_LOOP (loop_vinfo) : NULL;
+
+
+  lhs = gimple_get_lhs (last_stmt);
+  if (!lhs || !VECTOR_TYPE_P (TREE_TYPE (lhs)))
+    return NULL;
+
+  if (targetm.vectorize.recog_patterns_builtin_call)
+    {
+      auto_vec<gimple *> chain;
+      gimple *res
+       = targetm.vectorize.recog_patterns_builtin_call (vinfo, stmt_vinfo,
+                                                        chain, type_out);
+      vect_pattern_detected ("vect_recog_vect_scalar_pattern", last_stmt);
+    }
+
+#if 1
+  /* First pattern is reductions, if we have a vector reduction then adjust
+     any constant lanes if we can.  TOOD: Maybe this should force HVLA? */
+  if (loop && is_a <gphi *> (last_stmt))
+    {
+      /* Check the header edge.  */
+      gphi *phi = as_a <gphi *>(last_stmt);
+      tree init = PHI_ARG_DEF_FROM_EDGE (phi, loop_preheader_edge (loop));
+      if ((var = ssa_uniform_vector_p (init)))
+       {
+         tree type = TREE_TYPE (lhs);
+         vectype = get_vectype_for_scalar_type (vinfo, type);
+         if (vectype == NULL_TREE)
+           return NULL;
+
+         tree lhs_ivar = vect_recog_temp_ssa_var (type, NULL);
+         gphi *res = make_phi_node (lhs_ivar, 2);
+         tree step = PHI_ARG_DEF_FROM_EDGE (phi, loop_latch_edge (loop));
+         add_phi_arg (res, init, loop_preheader_edge (loop), UNKNOWN_LOCATION);
+         add_phi_arg (res, step, loop_latch_edge (loop), UNKNOWN_LOCATION);
+         pattern_stmt = res;
+
+         *type_out = vectype;
+         vect_pattern_detected ("vect_recog_vect_scalar_pattern", last_stmt);
+       }
+    }
+#endif
+
+  return NULL;
+}
 
 /* Function vect_recog_mask_conversion_pattern
 
@@ -7467,6 +7601,8 @@ static vect_recog_func vect_vect_recog_func_ptrs[] = {
   { vect_recog_sat_trunc_pattern, "sat_trunc" },
   { vect_recog_gcond_pattern, "gcond" },
   { vect_recog_bool_pattern, "bool" },
+  { vect_recog_vect_bool_pattern, "vect_bool" },
+  { vect_recog_vect_scalar_pattern, "vect_scalar" },
   /* This must come before mask conversion, and includes the parts
      of mask conversion that are needed for gather and scatter
      internal functions.  */
diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc
index 
075e93f04a97cddd6bc4b7f82485f08ee22c953b..c1bf905f54786c3622a155f777e6889a635506d4
 100644
--- a/gcc/tree-vect-slp.cc
+++ b/gcc/tree-vect-slp.cc
@@ -1950,7 +1950,7 @@ vect_build_slp_tree (vec_info *vinfo,
          for (i = 0; i < group_size; ++i)
            if (!matches[i])
              break;
-         gcc_assert (i < group_size);
+         //gcc_assert (i < group_size);
        }
       memcpy (res->failed, matches, sizeof (bool) * group_size);
     }
@@ -2107,6 +2107,9 @@ vect_build_slp_tree_2 (vec_info *vinfo, slp_tree node,
                              &vectype))
     return NULL;
 
+  if (known_gt (group_size, this_max_nunits))
+    return NULL;
+
   /* If the SLP node is a load, terminate the recursion unless masked.  */
   if (STMT_VINFO_DATA_REF (stmt_info)
       && DR_IS_READ (STMT_VINFO_DATA_REF (stmt_info)))
@@ -2133,6 +2136,7 @@ vect_build_slp_tree_2 (vec_info *vinfo, slp_tree node,
          FOR_EACH_VEC_ELT (SLP_TREE_SCALAR_STMTS (node), j, load_info)
            {
              int load_place;
+             tree type;
              if (! load_info)
                {
                  if (STMT_VINFO_GROUPED_ACCESS (stmt_info))
@@ -2140,6 +2144,11 @@ vect_build_slp_tree_2 (vec_info *vinfo, slp_tree node,
                  else
                    load_place = 0;
                }
+             /* ?? Not all vector loads are linear, model this.  */
+             else if ((type = TREE_TYPE (gimple_get_lhs (STMT_VINFO_STMT 
(load_info))))
+                      && VECTOR_TYPE_P (type)
+                      && TYPE_VECTOR_SUBPARTS (type).is_constant ())
+               load_place = j;
              else if (STMT_VINFO_GROUPED_ACCESS (stmt_info))
                load_place = vect_get_place_in_interleaving_chain
                    (load_info, first_stmt_info);
@@ -3141,7 +3150,7 @@ fail:
                n_vector_builds++;
            }
        }
-      if (all_uniform_p
+      if (all_uniform_p && false
          || n_vector_builds > 1
          || (n_vector_builds == children.length ()
              && is_a <gphi *> (stmt_info->stmt)))
@@ -4282,7 +4291,9 @@ vect_build_slp_instance (vec_info *vinfo,
   /* While we arrive here even with slp_inst_kind_store we should only
      for group_size == 1.  The code to split store groups is only in
      vect_analyze_slp_instance now.  */
+#if 0
   gcc_assert (kind != slp_inst_kind_store || group_size == 1);
+#endif
 
   /* Free the allocated memory.  */
   scalar_stmts.release ();
@@ -5014,7 +5025,16 @@ vect_analyze_slp_instance (vec_info *vinfo,
   stmt_vec_info next_info = stmt_info;
   while (next_info)
     {
-      scalar_stmts.quick_push (vect_stmt_to_vectorize (next_info));
+      scalar_stmts.safe_push (vect_stmt_to_vectorize (next_info));
+#if 1
+      /* If we're a vector type, insert dummy nodes.  */
+      unsigned HOST_WIDE_INT vect_nunits = 0;
+      tree vectype = TREE_TYPE (gimple_get_lhs (STMT_VINFO_STMT (next_info)));
+      if (VECTOR_TYPE_P (vectype)
+         && TYPE_VECTOR_SUBPARTS (vectype).is_constant (&vect_nunits))
+       for (auto i = 0; i < vect_nunits - 1UL; i++)
+         scalar_stmts.safe_push (next_info);
+#endif
       next_info = DR_GROUP_NEXT_ELEMENT (next_info);
     }
 
@@ -5038,6 +5058,13 @@ vect_analyze_slp_instance (vec_info *vinfo,
 
   /* Build the tree for the SLP instance.  */
   unsigned int group_size = scalar_stmts.length ();
+#if 0
+  /* If we're a vector type, adjust group size by nunits.  */
+  unsigned HOST_WIDE_INT vect_nunits = 0;
+  tree vectype = TREE_TYPE (gimple_get_lhs (STMT_VINFO_STMT 
(scalar_stmts[0])));
+  if (TYPE_VECTOR_SUBPARTS (vectype).is_constant (&vect_nunits))
+    group_size *= vect_nunits;
+#endif
   bool *matches = XALLOCAVEC (bool, group_size);
   poly_uint64 max_nunits = 1;
   unsigned tree_size = 0;
@@ -9780,6 +9807,7 @@ vect_slp_check_for_roots (bb_vec_info bb_vinfo)
       enum tree_code code = gimple_assign_rhs_code (assign);
       use_operand_p use_p;
       gimple *use_stmt;
+      optab op;
       if (code == CONSTRUCTOR)
        {
          if (!VECTOR_TYPE_P (TREE_TYPE (rhs))
@@ -9995,6 +10023,24 @@ vect_slp_check_for_roots (bb_vec_info bb_vinfo)
                }
            }
        }
+#if 1
+       else if (gimple_vdef (assign)
+              && VECTOR_TYPE_P (TREE_TYPE (rhs))
+              && ((op = optab_for_tree_code (code, TREE_TYPE (rhs), 
optab_default)) == unknown_optab
+                  || !can_implement_p (op, TYPE_MODE (TREE_TYPE (rhs)))))
+       {
+         vec<stmt_vec_info> roots = vNULL;
+         auto stmt_vinfo = bb_vinfo->lookup_stmt (assign);
+         roots.safe_push (stmt_vinfo);
+         vec<stmt_vec_info> stmts;
+         auto num_entries = TYPE_VECTOR_SUBPARTS (TREE_TYPE (rhs)).to_constant 
() / 2;
+         stmts.create (num_entries);
+         for (int i = 0; i < num_entries; i++)
+           stmts.quick_push (stmt_vinfo);
+         bb_vinfo->roots.safe_push (slp_root (slp_inst_kind_store,
+                                              stmts, roots));
+       }
+#endif
     }
 }
 
@@ -10984,6 +11030,14 @@ vect_transform_slp_perm_load_1 (vec_info *vinfo, 
slp_tree node,
       dr_group_size = DR_GROUP_SIZE (stmt_info);
     }
 
+  /* We should probably adjust group size, but for now lets hack.  */
+  unsigned HOST_WIDE_INT vect_nunits = 0;
+  auto vscalar_stmt = STMT_VINFO_STMT (stmt_info);
+  tree vscalar_type = TREE_TYPE (gimple_get_lhs (vscalar_stmt));
+  if (VECTOR_TYPE_P (vscalar_type)
+      && TYPE_VECTOR_SUBPARTS (vscalar_type).is_constant (&vect_nunits))
+    dr_group_size *= vect_nunits;
+
   mode = TYPE_MODE (vectype);
   poly_uint64 nunits = TYPE_VECTOR_SUBPARTS (vectype);
   unsigned int nstmts = vect_get_num_copies (vinfo, node);
@@ -12431,6 +12485,8 @@ vect_schedule_slp (vec_info *vinfo, const 
vec<slp_instance> &slp_instances)
            break;
 
          store_info = vect_orig_stmt (store_info);
+         if (!store_info)
+           continue;
          /* Free the attached stmt_vec_info and remove the stmt.  */
          vinfo->remove_stmt (store_info);
 
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index 
7687c107a6afc54741cb5e3f5756bcb65196b519..3cd8e4dfba4b3bf3173753a08cb214493b802e13
 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -2108,9 +2108,11 @@ get_load_store_type (vec_info  *vinfo, stmt_vec_info 
stmt_info,
   bool perm_ok = true;
   poly_int64 vf = loop_vinfo ? LOOP_VINFO_VECT_FACTOR (loop_vinfo) : 1;
 
+#if 1
   if (SLP_TREE_LOAD_PERMUTATION (slp_node).exists ())
     perm_ok = vect_transform_slp_perm_load (vinfo, slp_node, vNULL, NULL,
                                            vf, true, n_perms, n_loads);
+#endif
 
   if (STMT_VINFO_GROUPED_ACCESS (stmt_info))
     {
@@ -2728,8 +2730,18 @@ get_load_store_type (vec_info  *vinfo, stmt_vec_info 
stmt_info,
            if (k > maxk)
              maxk = k;
          tree vectype = SLP_TREE_VECTYPE (slp_node);
+
+         /* We should probably adjust group size, but for now lets hack.  */
+         unsigned HOST_WIDE_INT vect_nunits = 0;
+         auto group_size = DR_GROUP_SIZE (group_info);
+         auto vscalar_stmt = STMT_VINFO_STMT (group_info);
+         tree vscalar_type = TREE_TYPE (gimple_get_lhs (vscalar_stmt));
+         if (VECTOR_TYPE_P (vscalar_type)
+             && TYPE_VECTOR_SUBPARTS (vscalar_type).is_constant (&vect_nunits))
+           group_size *= vect_nunits;
+
          if (!TYPE_VECTOR_SUBPARTS (vectype).is_constant (&nunits)
-             || maxk >= (DR_GROUP_SIZE (group_info) & ~(nunits - 1)))
+             || maxk >= (group_size & ~(nunits - 1)))
            {
              if (dump_enabled_p ())
                dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
@@ -6618,13 +6630,17 @@ vectorizable_operation (vec_info *vinfo,
     }
 
   scalar_dest = gimple_assign_lhs (stmt);
+  tree scalar_type = TREE_TYPE (scalar_dest);
+  if (VECTOR_TYPE_P (scalar_type))
+    scalar_type = TREE_TYPE (scalar_type);
+
   vectype_out = SLP_TREE_VECTYPE (slp_node);
 
   /* Most operations cannot handle bit-precision types without extra
      truncations.  */
   bool mask_op_p = VECTOR_BOOLEAN_TYPE_P (vectype_out);
   if (!mask_op_p
-      && !type_has_mode_precision_p (TREE_TYPE (scalar_dest))
+      && !type_has_mode_precision_p (scalar_type)
       /* Exception are bitwise binary operations.  */
       && code != BIT_IOR_EXPR
       && code != BIT_XOR_EXPR
@@ -6658,7 +6674,7 @@ vectorizable_operation (vec_info *vinfo,
         type.  */
       if (VECT_SCALAR_BOOLEAN_TYPE_P (TREE_TYPE (op0)))
        {
-         if (!VECT_SCALAR_BOOLEAN_TYPE_P (TREE_TYPE (scalar_dest)))
+         if (!VECT_SCALAR_BOOLEAN_TYPE_P (scalar_type))
            {
              if (dump_enabled_p ())
                dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
@@ -8297,6 +8313,14 @@ vectorizable_store (vec_info *vinfo,
       group_size = 1;
     }
 
+    /* We should probably adjust group size, but for now lets hack.  */
+    unsigned HOST_WIDE_INT vect_nunits = 0;
+    auto vscalar_stmt = STMT_VINFO_STMT (first_stmt_info);
+    tree vscalar_type = TREE_TYPE (gimple_get_lhs (vscalar_stmt));
+    if (VECTOR_TYPE_P (vscalar_type)
+       && TYPE_VECTOR_SUBPARTS (vscalar_type).is_constant (&vect_nunits))
+      group_size *= vect_nunits;
+
   if (STMT_VINFO_SIMD_LANE_ACCESS_P (stmt_info) > 1 && cost_vec)
     {
       if (!check_scan_store (vinfo, stmt_info, vectype, rhs_dt, slp_node,
@@ -9887,6 +9911,9 @@ vectorizable_load (vec_info *vinfo,
      padding bits in the type that might leak otherwise.
      Refer to PR115336.  */
   tree scalar_type = TREE_TYPE (scalar_dest);
+  if (VECTOR_TYPE_P (scalar_type))
+    scalar_type = TREE_TYPE (scalar_type);
+
   bool type_mode_padding_p
     = TYPE_PRECISION (scalar_type) < GET_MODE_PRECISION (GET_MODE_INNER 
(mode));
 
@@ -10509,6 +10536,7 @@ vectorizable_load (vec_info *vinfo,
          first_stmt_info = stmt_info;
          group_size = 1;
        }
+
       /* For SLP vectorization we directly vectorize a subchain
          without permutation.  */
       if (! SLP_TREE_LOAD_PERMUTATION (slp_node).exists ())
@@ -10522,6 +10550,14 @@ vectorizable_load (vec_info *vinfo,
       first_dr_info = STMT_VINFO_DR_INFO (first_stmt_info);
       group_gap_adj = 0;
 
+      /* We should probably adjust group size, but for now lets hack.  */
+      unsigned HOST_WIDE_INT vect_nunits = 0;
+      auto vscalar_stmt = STMT_VINFO_STMT (first_stmt_info);
+      tree vscalar_type = TREE_TYPE (gimple_get_lhs (vscalar_stmt));
+      if (VECTOR_TYPE_P (vscalar_type)
+         && TYPE_VECTOR_SUBPARTS (vscalar_type).is_constant (&vect_nunits))
+       group_size *= vect_nunits;
+
       /* VEC_NUM is the number of vect stmts to be created for this group.  */
       grouped_load = false;
       /* If an SLP permutation is from N elements to N elements,
@@ -12015,7 +12051,8 @@ vect_is_simple_cond (tree cond, vec_info *vinfo,
 
   /* Mask case.  */
   if (TREE_CODE (cond) == SSA_NAME
-      && VECT_SCALAR_BOOLEAN_TYPE_P (TREE_TYPE (cond)))
+      && (VECT_SCALAR_BOOLEAN_TYPE_P (TREE_TYPE (cond))
+         || VECTOR_BOOLEAN_TYPE_P (TREE_TYPE (cond))))
     {
       if (!vect_is_simple_use (vinfo, slp_node, 0, &cond,
                               &slp_op, &dts[0], comp_vectype)
@@ -13209,6 +13246,226 @@ vectorizable_early_exit (loop_vec_info loop_vinfo, 
stmt_vec_info stmt_info,
   return true;
 }
 
+/* Intrinsics and vector types may not match even though the underlying modes
+   are the same.  This is particularly true of masks and SVE modes.  As such
+   convert between the two.  */
+static void
+vect_shim_call_arg (vec<tree> &types, vec_info *vinfo, stmt_vec_info stmt_info,
+                   gimple_stmt_iterator *gsi, gcall* call, int argno,
+                   tree newarg)
+{
+  tree ty_dest = types[argno+1];
+  tree ty_src = TREE_TYPE (newarg);
+
+  if (ty_src != ty_dest)
+    {
+      tree lhs = vect_create_destination_var (newarg, ty_dest);
+      tree op = build1 (VIEW_CONVERT_EXPR, ty_dest, newarg);
+      gimple *tmp = gimple_build_assign (lhs, VIEW_CONVERT_EXPR, op);
+      lhs = make_ssa_name (lhs, tmp);
+      gimple_assign_set_lhs (tmp, lhs);
+      vect_finish_stmt_generation (vinfo, stmt_info, tmp, gsi);
+      newarg = lhs;
+
+    if (dump_enabled_p ())
+      dump_printf_loc (MSG_NOTE, vect_location, "inserted type shim between: 
%T"
+                      " and %T due to mismatch between generic type and vector"
+                      " call.\n", ty_dest, ty_src);
+    }
+
+  gimple_call_set_arg (call, argno, newarg);
+}
+
+bool
+vectorizable_vect_call (vec_info *vinfo,
+                       stmt_vec_info stmt_info, gimple_stmt_iterator *gsi,
+                       slp_tree slp_node,
+                       stmt_vector_for_cost *cost_vec)
+{
+  enum vect_def_type dt[5]
+    = { vect_unknown_def_type, vect_unknown_def_type, vect_unknown_def_type,
+       vect_unknown_def_type, vect_unknown_def_type };
+  tree vectypes[ARRAY_SIZE (dt)] = {};
+  slp_tree slp_op[ARRAY_SIZE (dt)] = {};
+  tree op;
+
+  /* If the target hook isn't configured there's nothing to do.  */
+  if (!targetm.vectorize.translate_builtin_call)
+    return false;
+
+  if (!STMT_VINFO_RELEVANT_P (stmt_info))
+    return false;
+
+  if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_internal_def
+      && cost_vec)
+    return false;
+
+  /* Is STMT_INFO a vectorizable vector call?   */
+  gcall *stmt = dyn_cast <gcall *> (STMT_VINFO_STMT (stmt_info));
+  if (!stmt || !gimple_call_builtin_p (stmt))
+    return false;
+
+  /* It must be a built in with a vector type as an argument. */
+  bool has_vect_arg = false;
+  int nargs = gimple_call_num_args (stmt);
+
+  /* Check if it has more arguments than we support at the moment.  */
+  if (nargs > 5)
+    {
+      if (dump_enabled_p ())
+       dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
+                        "vector function call has too many arguments.\n");
+      return false;
+  }
+
+  for (int i = 0; i < nargs; i++)
+    {
+      has_vect_arg = has_vect_arg
+                    || VECTOR_TYPE_P (TREE_TYPE (gimple_call_arg (stmt, i)));
+
+      if (!vect_is_simple_use (vinfo, slp_node,
+                              i, &op, &slp_op[i], &dt[i], &vectypes[i]))
+       {
+         if (dump_enabled_p ())
+           dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
+                            "use not simple.\n");
+         return false;
+       }
+    }
+
+  if (!has_vect_arg)
+    {
+      if (dump_enabled_p ())
+       dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
+                        "function is not a vectorizable vector function.\n");
+      return false;
+    }
+
+  /* We only handle functions that do not read or clobber memory.  */
+  if (gimple_vuse (stmt))
+    {
+      if (dump_enabled_p ())
+       dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
+                        "function reads from or writes to memory.\n");
+      return false;
+    }
+
+  tree vectype_out;
+  int maskopno = -1;
+  auto_vec<tree> types;
+  gcall *call
+    = targetm.vectorize.translate_builtin_call (vinfo, slp_node, stmt_info,
+                                               &maskopno, types, &vectype_out);
+
+  if (!call)
+    {
+      if (dump_enabled_p ())
+       dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
+                        "target did not have implementation for %G.\n",
+                        (gimple*)stmt);
+      return false;
+    }
+
+  loop_vec_info loop_vinfo = dyn_cast <loop_vec_info> (vinfo);
+  vec_loop_masks *masks = (loop_vinfo ? &LOOP_VINFO_MASKS (loop_vinfo) : NULL);
+  vec_loop_lens *lens = (loop_vinfo ? &LOOP_VINFO_LENS (loop_vinfo) : NULL);
+  bool masked_loop_p = loop_vinfo && LOOP_VINFO_FULLY_MASKED_P (loop_vinfo);
+  bool len_loop_p = loop_vinfo && LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo);
+  unsigned int vect_nargs = nargs;
+
+  if (maskopno != -1)
+    {
+      if (!loop_vinfo) // || (!masked_loop_p && !len_loop_p))
+       {
+         if (dump_enabled_p ())
+           dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
+                            "target translation requires predication but "
+                            "masks not supported: %G.\n",
+                            (gimple*)stmt);
+         return false;
+       }
+      vect_nargs += 1;
+    }
+
+  if (cost_vec) /* transformation not required.  */
+    {
+      SLP_TREE_TYPE (slp_node) = vector_call_vec_info_type;
+      DUMP_VECT_SCOPE ("vectorizable_vect_call");
+      vect_model_simple_cost (vinfo, 1, slp_node, cost_vec);
+      return true;
+    }
+
+  /* Transform.  */
+
+  if (dump_enabled_p ())
+    dump_printf_loc (MSG_NOTE, vect_location, "transform vector call.\n");
+
+  /* Resolve arguments.  */
+  auto_vec<vec<tree> > vec_defs (nargs);
+  vect_get_slp_defs (vinfo, slp_node, &vec_defs);
+
+  /* Handle def.  */
+  tree scalar_dest = gimple_call_lhs (stmt);
+  tree vec_dest = vect_create_destination_var (scalar_dest, vectype_out);
+  tree new_temp = make_ssa_name (vec_dest, stmt);
+  gimple_call_set_lhs (call, new_temp);
+  gimple_call_set_nothrow (call, true);
+
+  /* Now fill in the new values.  Intrinsics types and vector types may not
+     match, so convert the types.  */
+  int shift = 0;
+  for (int i = 0; i < nargs; i++)
+    {
+      if (maskopno == i)
+       {
+         tree mask;
+         if (masked_loop_p)
+           {
+             /* ?? Handle decomposition here, can that happen??.  */
+             unsigned int vec_num = 1; //vec_oprnds0.length ();
+             mask = vect_get_loop_mask (loop_vinfo, gsi, masks, vec_num,
+                                        vectype_out, 0);
+           }
+         else
+           {
+             tree mask_vectype = truth_type_for (vectype_out);
+             mask = vect_build_all_ones_mask (loop_vinfo, stmt_info,
+                                              mask_vectype);
+           }
+         vect_shim_call_arg (types, vinfo, stmt_info, gsi, call, i, mask);
+         shift++;
+       }
+      tree arg = vec_defs[i][0];
+      vect_shim_call_arg (types, vinfo, stmt_info, gsi, call, i+shift, arg);
+    }
+
+  gimple *res = call;
+  /* Check if the result types match, if not, shim them.  */
+  if (vectype_out != types[0])
+    {
+      /* First update the call.  */
+      tree lhs = vect_create_destination_var (scalar_dest, types[0]);
+      tree new_lhs = make_ssa_name (lhs, NULL);
+      gimple_call_set_lhs (call, new_lhs);
+      vect_finish_stmt_generation (vinfo, stmt_info, call, gsi);
+      tree op = build1 (VIEW_CONVERT_EXPR, vectype_out, new_lhs);
+      gimple *tmp = gimple_build_assign (vec_dest, VIEW_CONVERT_EXPR, op);
+      gimple_assign_set_lhs (tmp, new_temp);
+      vect_finish_stmt_generation (vinfo, stmt_info, tmp, gsi);
+      res = tmp; 
+
+      if (dump_enabled_p ())
+       dump_printf_loc (MSG_NOTE, vect_location, "inserted type shim between: 
%T"
+                      " and %T due to mismatch between generic type and vector"
+                      " call.\n", types[0], vectype_out);
+    }
+  else
+    vect_finish_stmt_generation (vinfo, stmt_info, call, gsi);
+
+  slp_node->push_vec_def (res);
+  return true;
+}
+
 /* If SLP_NODE is nonnull, return true if vectorizable_live_operation
    can handle all live statements in the node.  Otherwise return true
    if STMT_INFO is not live or if vectorizable_live_operation can handle it.
@@ -13338,6 +13595,7 @@ vect_analyze_stmt (vec_info *vinfo,
          || vectorizable_shift (vinfo, stmt_info, NULL, node, cost_vec)
          || vectorizable_condition (vinfo, stmt_info, NULL, node, cost_vec)
          || vectorizable_comparison (vinfo, stmt_info, NULL, node, cost_vec)
+         || vectorizable_vect_call (vinfo, stmt_info, NULL, node, cost_vec)
          || (bb_vinfo
              && vectorizable_phi (bb_vinfo, stmt_info, node, cost_vec))
          || (is_a <loop_vec_info> (vinfo)
@@ -13454,6 +13712,10 @@ vect_transform_stmt (vec_info *vinfo,
       done = vectorizable_call (vinfo, stmt_info, gsi, slp_node, NULL);
       break;
 
+    case vector_call_vec_info_type:
+      done = vectorizable_vect_call (vinfo, stmt_info, gsi, slp_node, NULL);
+      break;
+
     case call_simd_clone_vec_info_type:
       done = vectorizable_simd_clone_call (vinfo, stmt_info, gsi,
                                           slp_node, NULL);
@@ -13572,6 +13834,10 @@ get_related_vectype_for_scalar_type (machine_mode 
prevailing_mode,
   machine_mode simd_mode;
   tree vectype;
 
+  /* For vector types, get the inner most type.  */
+  if (VECTOR_TYPE_P (scalar_type))
+    scalar_type = TREE_TYPE (scalar_type);
+
   if ((!INTEGRAL_TYPE_P (scalar_type)
        && !POINTER_TYPE_P (scalar_type)
        && !SCALAR_FLOAT_TYPE_P (scalar_type))
@@ -14801,6 +15067,10 @@ vect_get_vector_types_for_stmt (vec_info *vinfo, 
stmt_vec_info stmt_info,
       else
        scalar_type = TREE_TYPE (gimple_get_lhs (stmt));
 
+      /* If scalar_type is a vector type, get the innner typ.  */
+      if (VECTOR_TYPE_P (scalar_type))
+       scalar_type = TREE_TYPE (scalar_type);
+
       if (dump_enabled_p ())
        {
          if (group_size)
@@ -14901,4 +15171,3 @@ vect_gen_len (tree len, tree start_index, tree 
end_index, tree len_limit)
 
   return stmts;
 }
-
diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
index 
35e531c11c178839bcf3218e58ad7521662d3b42..1bba7702b628cc4012848e1d887be03fe3d2533d
 100644
--- a/gcc/tree-vectorizer.h
+++ b/gcc/tree-vectorizer.h
@@ -229,6 +229,7 @@ enum stmt_vec_info_type {
   shift_vec_info_type,
   op_vec_info_type,
   call_vec_info_type,
+  vector_call_vec_info_type,
   call_simd_clone_vec_info_type,
   assignment_vec_info_type,
   condition_vec_info_type,


-- 

diff --git a/gcc/config/aarch64/aarch64-builtins.cc b/gcc/config/aarch64/aarch64-builtins.cc
index b9cc2bf404f612a9688db4ce2f6a042b148ac938..54d7d2e918f613fa449ab268b84581d7ada483f3 100644
--- a/gcc/config/aarch64/aarch64-builtins.cc
+++ b/gcc/config/aarch64/aarch64-builtins.cc
@@ -29,6 +29,7 @@
 #include "rtl.h"
 #include "tree.h"
 #include "gimple.h"
+#include "cfgloop.h"
 #include "ssa.h"
 #include "memmodel.h"
 #include "tm_p.h"
@@ -42,6 +43,7 @@
 #include "expr.h"
 #include "langhooks.h"
 #include "gimple-iterator.h"
+#include "tree-vectorizer.h"
 #include "case-cfn-macros.h"
 #include "regs.h"
 #include "emit-rtl.h"
@@ -5591,6 +5593,201 @@ aarch64_general_gimple_fold_builtin (unsigned int fcode, gcall *stmt,
   return new_stmt;
 }
 
+/* Intrinsics type have a different type decorator than normal types.  So
+   compare signess and mode instead.  */
+static bool
+aarch64_eq_type_p (tree ty0, tree ty1)
+{
+  if (TYPE_UNSIGNED (ty0) != TYPE_UNSIGNED (ty1))
+    return false;
+
+  return TYPE_MODE (ty0) == TYPE_MODE (ty1);
+}
+
+static tree aarch64_lookup_fndecl_by_name (const char *);
+
+static tree
+aarch64_find_builtin_fndecl_1 (tree fndecl, vec<tree> &types,
+			       tree req_ret_ty,
+			       tree req_arg0_ty, tree req_arg1_ty,
+			       int *maskopno = NULL)
+{
+  tree t = fndecl;
+  do
+    {
+      if (TREE_CODE (t) != FUNCTION_DECL)
+        break;
+
+      tree fntype = TREE_TYPE (t);
+      tree ret_ty = TREE_TYPE (fntype);
+      tree args = TYPE_ARG_TYPES (fntype);
+
+      if (!TREE_CHAIN (args))
+	break;
+
+      tree arg0_ty = TREE_VALUE (args);
+      tree arg1_ty = TREE_VALUE (TREE_CHAIN (args));
+
+      if (aarch64_eq_type_p (ret_ty, req_ret_ty)
+	  && aarch64_eq_type_p (arg0_ty, req_arg0_ty)
+	  && aarch64_eq_type_p (arg1_ty, req_arg1_ty))
+	{
+	  types.create (3);
+	  types.quick_push (ret_ty);
+	  types.quick_push (arg0_ty);
+	  types.quick_push (arg1_ty);
+	  return t;
+	}
+
+      /* Check if it could be masked, very wrong and unsafe but good enough.  */
+      if (maskopno && VECTOR_BOOLEAN_TYPE_P (arg0_ty))
+	{
+	  *maskopno = 0;
+	  tree arg0_ty = TREE_VALUE (TREE_CHAIN (args));
+	  tree arg1_ty = TREE_VALUE (TREE_CHAIN (TREE_CHAIN (args)));
+	  if (aarch64_eq_type_p (ret_ty, req_ret_ty)
+	      && aarch64_eq_type_p (arg0_ty, req_arg0_ty)
+	      && aarch64_eq_type_p (arg1_ty, req_arg1_ty))
+	    {
+	      types.create (4);
+	      types.quick_push (ret_ty);
+	      types.quick_push (TREE_VALUE (args));
+	      types.quick_push (arg0_ty);
+	      types.quick_push (arg1_ty);
+	      return t;
+	    }
+	}
+
+      if (t == fndecl)
+	t = DECL_CHAIN (fndecl);
+      else
+	t = DECL_CHAIN (t);
+    }
+  while (t);
+
+  return NULL;
+}
+
+static tree
+aarch64_find_builtin_fndecl (tree fndecl, tree req_ret_ty,
+			     tree req_arg0_ty, tree req_arg1_ty,
+			     int *maskopno, vec<tree> &types,
+			     const char *fallback_name = nullptr)
+{
+  tree res = aarch64_find_builtin_fndecl_1 (fndecl, types,
+					    req_ret_ty, req_arg0_ty,
+					    req_arg1_ty);
+  if (res)
+    return res;
+
+  if (fallback_name)
+    {
+      tree t = aarch64_lookup_fndecl_by_name (fallback_name);
+      gcc_assert (t);
+      *maskopno = -1;
+      res = aarch64_find_builtin_fndecl_1 (t, types, req_ret_ty, req_arg0_ty,
+					   req_arg1_ty, maskopno);
+      return res;
+    }
+
+  return NULL_TREE;
+}
+
+static tree
+aarch64_lookup_fndecl_by_name (const char *name)
+{
+  tree id = get_identifier (name);
+  if (!id)
+    return NULL_TREE;
+
+  extern tree lookup_name (tree);
+  tree decl = lookup_name (id);
+  if (!decl || TREE_CODE (decl) != FUNCTION_DECL)
+    return aarch64_sve::lookup_fndecl_by_name (name);
+
+  return decl;
+}
+
+/* Implement TARGET_VECTORIZE_TRANSLATE_BUILTIN_CALL.  */
+gcall *
+aarch64_internal_translate_builtin_call (vec_info *vinfo,
+					 slp_tree node,
+					 stmt_vec_info stmt_info,
+					 int *maskopno,
+					 vec<tree> &types,
+					 tree *vectype_out)
+{
+  gcall *stmt = dyn_cast <gcall *> (STMT_VINFO_STMT (stmt_info));
+  gcc_assert (stmt);
+  tree fndecl = gimple_call_fndecl (stmt);
+  unsigned int code = DECL_MD_FUNCTION_CODE (fndecl);
+  unsigned int subcode = code >> AARCH64_BUILTIN_SHIFT;
+
+  /* There's nothing to do for SVE at this point in time, so bail out.  */
+  if ((code & AARCH64_BUILTIN_CLASS) == AARCH64_BUILTIN_SVE)
+    return NULL;
+
+  switch (subcode)
+    {
+    BUILTIN_VSDQ_I (BINOP_SSU, suqadd, 0, DEFAULT)
+    BUILTIN_VSDQ_I (BINOP_UUS, usqadd, 0, DEFAULT)
+      {
+	tree arg0 = gimple_call_arg (stmt, 0);
+	tree arg1 = gimple_call_arg (stmt, 1);
+	tree ty0 = get_vectype_for_scalar_type (vinfo, TREE_TYPE (arg0));
+	tree ty1 = get_vectype_for_scalar_type (vinfo, TREE_TYPE (arg1));
+	if (!ty0 || !ty1)
+	  return NULL;
+
+	/* For usqadd/suqadd the result type is the vectorized type of arg0.  */
+	tree new_fndecl = aarch64_find_builtin_fndecl (fndecl, ty0, ty0, ty1,
+						       maskopno, types,
+						       "svsqadd_u8_x");
+	if (!new_fndecl)
+	  return NULL;
+
+	/* This is type incorrect, but we don't have the arguments yet.  The
+	   vectorizer will take care of it.  */
+	gcall *new_call;
+	if (*maskopno == -1)
+	  new_call = gimple_build_call (new_fndecl, 2, NULL, NULL);
+	else
+	  new_call = gimple_build_call (new_fndecl, 3, NULL, NULL, NULL);
+	*vectype_out = ty0;
+	return new_call;
+      }
+    BUILTIN_VDQQH (BSL_P, simd_bsl, 0, DEFAULT)
+    VAR2 (BSL_P, simd_bsl,0, DEFAULT, di, v2di)
+    BUILTIN_VSDQ_I_DI (BSL_U, simd_bsl, 0, DEFAULT)
+    BUILTIN_VALLDIF (BSL_S, simd_bsl, 0, QUIET)
+      {
+	tree arg0 = gimple_call_arg (stmt, 0);
+	tree arg1 = gimple_call_arg (stmt, 1);
+	tree ty0 = get_vectype_for_scalar_type (vinfo, TREE_TYPE (arg0));
+	tree ty1 = get_vectype_for_scalar_type (vinfo, TREE_TYPE (arg1));
+	if (!ty0 || !ty1)
+	  return NULL;
+
+	/* For usqadd/suqadd the result type is the vectorized type of arg0.  */
+	tree new_fndecl = aarch64_find_builtin_fndecl (fndecl, ty1, ty0, ty1,
+						       maskopno, types,
+						       "svbsl_u8");
+	if (!new_fndecl)
+	  return NULL;
+
+	types.safe_push (types[1]);
+	/* BSL does not support masks.  */
+	gcc_assert (*maskopno == -1);
+	*vectype_out = ty0;
+	return gimple_build_call (new_fndecl, 3, NULL, NULL, NULL);
+      }
+    default:
+      return NULL;
+    }
+
+  return NULL;
+}
+
 void
 aarch64_atomic_assign_expand_fenv (tree *hold, tree *clear, tree *update)
 {
diff --git a/gcc/config/aarch64/aarch64-protos.h b/gcc/config/aarch64/aarch64-protos.h
index 513b556398fac7c733de47c27587c3aaa07cbcf3..fb7062b3d75902e1298b9d64846daf69b418fbb7 100644
--- a/gcc/config/aarch64/aarch64-protos.h
+++ b/gcc/config/aarch64/aarch64-protos.h
@@ -1185,6 +1185,7 @@ namespace aarch64_sve {
   void handle_arm_sme_h (bool);
   void handle_arm_neon_sve_bridge_h (bool);
   tree builtin_decl (unsigned, bool);
+  tree lookup_fndecl_by_name (const char *);
   bool builtin_type_p (const_tree);
   bool builtin_type_p (const_tree, unsigned int *, unsigned int *);
   const char *mangle_builtin_type (const_tree);
diff --git a/gcc/config/aarch64/aarch64-sve-builtins.cc b/gcc/config/aarch64/aarch64-sve-builtins.cc
index cd78713440f61ac18146f0be49a43a204dbb3ae1..fe5e267f087e1b2b62885695785e9065bee4d85a 100644
--- a/gcc/config/aarch64/aarch64-sve-builtins.cc
+++ b/gcc/config/aarch64/aarch64-sve-builtins.cc
@@ -1063,6 +1063,45 @@ static GTY(()) hash_table<registered_function_hasher> *function_table;
    are IDENTIFIER_NODEs.  */
 static GTY(()) hash_map<tree, registered_function *> *overload_names[2];
 
+/* Return the decl of the registered SVE function called NAME, or null if
+   no such function has been registered.  */
+static tree
+lookup_registered_function_by_name (const char *name)
+{
+  unsigned int length = vec_safe_length (registered_functions);
+  for (unsigned int i = 0; i < length; ++i)
+    {
+      registered_function *rfn = (*registered_functions)[i];
+      if (!rfn || TREE_CODE (rfn->decl) != FUNCTION_DECL)
+	continue;
+
+      tree decl_name = DECL_NAME (rfn->decl);
+      if (decl_name && strcmp (IDENTIFIER_POINTER (decl_name), name) == 0)
+	return rfn->decl;
+    }
+
+  return NULL_TREE;
+}
+
+/* Return the ACLE spelling that corresponds to NAME.  */
+static const char *
+canonicalize_function_name (const char *name)
+{
+  if (startswith (name, "__builtin_aarch64_"))
+    return name + strlen ("__builtin_aarch64_");
+
+  if (startswith (name, "__builtin_sve_"))
+    return name + strlen ("__builtin_sve_");
+
+  if (startswith (name, "__builtin_")
+      && (startswith (name + strlen ("__builtin_"), "sv")
+	  || startswith (name + strlen ("__builtin_"), "arm_")
+	  || startswith (name + strlen ("__builtin_"), "__arm_")))
+    return name + strlen ("__builtin_");
+
+  return name;
+}
+
 /* Record that TYPE is an ABI-defined SVE type that contains NUM_ZR SVE vectors
    and NUM_PR SVE predicates.  MANGLED_NAME, if nonnull, is the ABI-defined
    mangling of the type.  mangling of the type.  ACLE_NAME is the <arm_sve.h>
@@ -4939,6 +4978,29 @@ builtin_decl (unsigned int code, bool)
   return (*registered_functions)[code]->decl;
 }
 
+/* Return the decl for the SVE builtin called NAME, if one exists.  Accept
+   both the ACLE spelling and the common builtin-prefixed aliases.  */
+tree
+lookup_fndecl_by_name (const char *name)
+{
+  const char *canonical_name = canonicalize_function_name (name);
+  tree decl = lookup_registered_function_by_name (canonical_name);
+  if (decl)
+    return decl;
+
+  /* Internal callers sometimes need SVE decls even when arm_sve.h was not
+     included in the source.  Materialize them lazily in that case.  */
+  if (!function_table && (lang_GNU_C () || lang_GNU_CXX ()))
+    {
+      handle_arm_sve_h (false);
+      decl = lookup_registered_function_by_name (canonical_name);
+      if (decl)
+	return decl;
+    }
+
+  return NULL_TREE;
+}
+
 /* Implement #pragma GCC aarch64 "arm_sme.h".  */
 void
 handle_arm_sme_h (bool function_nulls_p)
diff --git a/gcc/config/aarch64/aarch64-sve-builtins.h b/gcc/config/aarch64/aarch64-sve-builtins.h
index f9e8c4cd7293a379e3334fecb846121cf88e0ebf..d4123dbe8b3c0a60f1b1a6b1a398d05989987c23 100644
--- a/gcc/config/aarch64/aarch64-sve-builtins.h
+++ b/gcc/config/aarch64/aarch64-sve-builtins.h
@@ -856,6 +856,7 @@ bool vector_cst_all_same (tree, unsigned int);
 bool is_ptrue (tree, unsigned int);
 bool is_pfalse (tree);
 const function_instance *lookup_fndecl (tree);
+tree lookup_fndecl_by_name (const char *);
 
 /* Try to find a mode with the given mode_suffix_info fields.  Return the
    mode on success or MODE_none on failure.  */
diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc
index 5a859e12b1a55e70284469e3e9c175eee849e18d..a9f3af387ddce6b86370992a4dc9717df5f7e82c 100644
--- a/gcc/config/aarch64/aarch64.cc
+++ b/gcc/config/aarch64/aarch64.cc
@@ -103,6 +103,7 @@
 #include "aarch64-sched-dispatch.h"
 #include "aarch64-json-tunings-printer.h"
 #include "aarch64-json-tunings-parser.h"
+#include "aarch64-builtins.h"
 
 /* This file should be included last.  */
 #include "target-def.h"
@@ -23868,6 +23869,24 @@ aarch64_autovectorize_vector_modes (vector_modes *modes, bool)
   return flags;
 }
 
+/* This is just temporary as I don't have an entry point into the new intrinsics
+   framework yet.  */
+extern gcall *aarch64_internal_translate_builtin_call (vec_info *, slp_tree,
+						stmt_vec_info, int*,
+						vec<tree> &types, tree *);;
+
+/* Implement TARGET_VECTORIZE_TRANSLATE_BUILTIN_CALL.  */
+static gcall *
+aarch64_autovectorize_translate_builtin_call (vec_info *vinfo,
+					      slp_tree node,
+					      stmt_vec_info stmt_info,
+					      int *maskopno, vec<tree> &types,
+					      tree *vectype_out)
+{
+  return aarch64_internal_translate_builtin_call (vinfo, node, stmt_info,
+						  maskopno, types, vectype_out);
+}
+
 /* Implement TARGET_CONVERT_TO_TYPE.  Convert EXPR to TYPE.  */
 
 static tree
@@ -34098,6 +34117,10 @@ aarch64_libgcc_floating_mode_supported_p
 #define TARGET_VECTORIZE_AUTOVECTORIZE_VECTOR_MODES \
   aarch64_autovectorize_vector_modes
 
+#undef TARGET_VECTORIZE_TRANSLATE_BUILTIN_CALL
+#define TARGET_VECTORIZE_TRANSLATE_BUILTIN_CALL \
+  aarch64_autovectorize_translate_builtin_call
+
 #undef TARGET_CONVERT_TO_TYPE
 #define TARGET_CONVERT_TO_TYPE aarch64_convert_to_type
 
diff --git a/gcc/doc/tm.texi b/gcc/doc/tm.texi
index 1fbf43c6c57799372752f581528e122689a33f15..4f48737bb916fefd51dd738a235679af1b4f693c 100644
--- a/gcc/doc/tm.texi
+++ b/gcc/doc/tm.texi
@@ -6537,6 +6537,20 @@ current cost model is for the scalar version of a loop or block; otherwise
 it is for the vector version.
 @end deftypefn
 
+@deftypefn {Target Hook} {gcall *} TARGET_VECTORIZE_TRANSLATE_BUILTIN_CALL (vec_info *@var{vinfo}, struct _slp_tree *@var{node}, class _stmt_vec_info *@var{stmt_info}, int *@var{maskopno}, vec<tree> @var{&types}, tree *@var{vectype_out})
+This hook returns true if masked operation @var{ifn} (really of
+type @code{internal_fn}) should be considered more expensive to use than
+implementing the same operation without masking.  GCC can then try to use
+unconditional operations instead with extra selects.
+@end deftypefn
+
+@deftypefn {Target Hook} {gimple *} TARGET_VECTORIZE_RECOG_PATTERNS_BUILTIN_CALL (vec_info *@var{vinfo}, struct _slp_tree *@var{node}, class _stmt_vec_info *@var{stmt_info}, vec<gimple> @var{&chain}, tree *@var{vectype_out})
+This hook returns true if masked operation @var{ifn} (really of
+type @code{internal_fn}) should be considered more expensive to use than
+implementing the same operation without masking.  GCC can then try to use
+unconditional operations instead with extra selects.
+@end deftypefn
+
 @deftypefn {Target Hook} tree TARGET_VECTORIZE_BUILTIN_GATHER (const_tree @var{mem_vectype}, const_tree @var{index_type}, int @var{scale})
 Target builtin that implements vector gather operation.  @var{mem_vectype}
 is the vector type of the load and @var{index_type} is scalar type of
diff --git a/gcc/doc/tm.texi.in b/gcc/doc/tm.texi.in
index 6ecbeda09ca9b6473f12f78f8805ced44907f146..bc6d60d2ec47e62a31596bd96f3d106ae65254fb 100644
--- a/gcc/doc/tm.texi.in
+++ b/gcc/doc/tm.texi.in
@@ -4341,6 +4341,10 @@ address;  but often a machine-dependent strategy can generate better code.
 
 @hook TARGET_VECTORIZE_CREATE_COSTS
 
+@hook TARGET_VECTORIZE_TRANSLATE_BUILTIN_CALL
+
+@hook TARGET_VECTORIZE_RECOG_PATTERNS_BUILTIN_CALL
+
 @hook TARGET_VECTORIZE_BUILTIN_GATHER
 
 @hook TARGET_VECTORIZE_BUILTIN_SCATTER
diff --git a/gcc/target.def b/gcc/target.def
index 25a94559e198aef2c3e842f02aa5414c5193c4a6..3c9a2aba29691b80ab502b057e9e7244e56758a4 100644
--- a/gcc/target.def
+++ b/gcc/target.def
@@ -2098,6 +2098,33 @@ stores.",
  (const_tree vectype, const_tree index_type, int scale),
  NULL)
 
+/* Function to say how to transform a gimple call.  If transformation is
+   possible return the new gcall, otherwise NULL.  */
+DEFHOOK
+(translate_builtin_call,
+ "This hook returns true if masked operation @var{ifn} (really of\n\
+type @code{internal_fn}) should be considered more expensive to use than\n\
+implementing the same operation without masking.  GCC can then try to use\n\
+unconditional operations instead with extra selects.",
+ gcall *,
+ (vec_info *vinfo, struct _slp_tree *node, class _stmt_vec_info *stmt_info,
+  int *maskopno, vec<tree> &types, tree *vectype_out),
+ NULL)
+
+/* Function to recognize patterns during vectorization of gimple calls.  If a
+   pattern is matched then return the new pattern and other statements it
+   consumes, otherwise NULL.  */
+DEFHOOK
+(recog_patterns_builtin_call,
+ "This hook returns true if masked operation @var{ifn} (really of\n\
+type @code{internal_fn}) should be considered more expensive to use than\n\
+implementing the same operation without masking.  GCC can then try to use\n\
+unconditional operations instead with extra selects.",
+ gimple *,
+ (vec_info *vinfo, class _stmt_vec_info *stmt_info,
+  vec<gimple *> &chain, tree *vectype_out),
+ NULL)
+
 /* Target function to initialize the cost model for a loop or block.  */
 DEFHOOK
 (create_costs,
diff --git a/gcc/tree-phinodes.cc b/gcc/tree-phinodes.cc
index e0663d96b7d4b96a1a48e5bfb9fef1329e526ae7..d97c6148d2f303bf9280dbbf19a3610133d36f8c 100644
--- a/gcc/tree-phinodes.cc
+++ b/gcc/tree-phinodes.cc
@@ -165,7 +165,7 @@ ideal_phi_node_len (int len)
 
 /* Return a PHI node with LEN argument slots for variable VAR.  */
 
-static gphi *
+gphi *
 make_phi_node (tree var, int len)
 {
   gphi *phi;
diff --git a/gcc/tree-phinodes.h b/gcc/tree-phinodes.h
index b6fa492cb248f4d0e1a7838d0b34da6ae3a108fd..08741bb4076d9459454120ead335e52e49953352 100644
--- a/gcc/tree-phinodes.h
+++ b/gcc/tree-phinodes.h
@@ -23,6 +23,7 @@ along with GCC; see the file COPYING3.  If not see
 extern void phinodes_print_statistics (void);
 extern void reserve_phi_args_for_new_edge (basic_block);
 extern gphi *create_phi_node (tree, basic_block);
+gphi *make_phi_node (tree, int);
 extern void add_phi_arg (gphi *, tree, edge, location_t);
 extern void remove_phi_args (edge);
 extern void remove_phi_node (gimple_stmt_iterator *, bool);
diff --git a/gcc/tree-vect-data-refs.cc b/gcc/tree-vect-data-refs.cc
index 0e57e1068d6a1906d1d8e9edb761d5dd4b7da758..eefd5ad2f91e0292cda52354bf70b406a62adbcf 100644
--- a/gcc/tree-vect-data-refs.cc
+++ b/gcc/tree-vect-data-refs.cc
@@ -3176,6 +3176,17 @@ vect_analyze_group_access_1 (vec_info *vinfo, dr_vec_info *dr_info)
   data_reference *dr = dr_info->dr;
   tree step = DR_STEP (dr);
   tree scalar_type = TREE_TYPE (DR_REF (dr));
+  unsigned long nunits = 0;
+  /* TODO: Add? */
+  if (VECTOR_TYPE_P (scalar_type)
+      && !TYPE_VECTOR_SUBPARTS (scalar_type).is_constant (&nunits))
+    {
+      if (dump_enabled_p ())
+	dump_printf_loc (MSG_NOTE, vect_location,
+			"Variable length vectors not supported yet: %T\n",
+			step);
+	  return false;
+    }
   HOST_WIDE_INT type_size = TREE_INT_CST_LOW (TYPE_SIZE_UNIT (scalar_type));
   stmt_vec_info stmt_info = dr_info->stmt;
   loop_vec_info loop_vinfo = dyn_cast <loop_vec_info> (vinfo);
@@ -3207,6 +3218,8 @@ vect_analyze_group_access_1 (vec_info *vinfo, dr_vec_info *dr_info)
 	}
       groupsize = absu_hwi (dr_step) / type_size;
     }
+  else if (bb_vinfo && nunits > 0)
+    groupsize = nunits;
   else
     groupsize = 0;
 
@@ -3463,6 +3476,29 @@ vect_analyze_data_ref_access (vec_info *vinfo, dr_vec_info *dr_info)
 	}
     }
 
+  /* Allow loads and stores with zero steps and vector types in BB
+     vectorization.  TODO:  This needs support for checking the type of access
+     so e.g. LOAD_LANES, Gathers etc.  */
+  if (!loop_vinfo && integer_zerop (step) && VECTOR_TYPE_P (scalar_type))
+    {
+      if (!TYPE_VECTOR_SUBPARTS (scalar_type).is_constant ())
+	{
+	  if (dump_enabled_p ())
+	    dump_printf_loc (MSG_NOTE, vect_location,
+			"Variable length vectors not supported yet: %T\n",
+			step);
+	  return false;
+	}
+      DR_GROUP_FIRST_ELEMENT (stmt_info) = NULL;
+      DR_GROUP_NEXT_ELEMENT (stmt_info) = NULL;
+
+      if (dump_enabled_p () && 0)
+	dump_printf_loc (MSG_NOTE, vect_location,
+			"Allowing vector memory access in BB\n");
+
+      return true;
+    }
+
   if (loop && nested_in_vect_loop_p (loop, stmt_info))
     {
       /* Interleaved accesses are not yet supported within outer-loop
@@ -3485,9 +3521,21 @@ vect_analyze_data_ref_access (vec_info *vinfo, dr_vec_info *dr_info)
   if (TREE_CODE (step) == INTEGER_CST)
     {
       HOST_WIDE_INT dr_step = TREE_INT_CST_LOW (step);
-      if (!tree_int_cst_compare (step, TYPE_SIZE_UNIT (scalar_type))
-	  || (dr_step < 0
-	      && !compare_tree_int (TYPE_SIZE_UNIT (scalar_type), -dr_step)))
+      bool sequantial = false;
+      poly_int64 step_p, type_size_p;
+      if (poly_int_tree_p (step, &step_p)
+	  && poly_int_tree_p (TYPE_SIZE_UNIT (scalar_type), &type_size_p))
+	sequantial
+	  = known_eq (step_p, type_size_p)
+	    || (known_lt (step_p, 0)
+		&& !known_eq (-step_p, type_size_p));
+      else
+	sequantial
+	  = !tree_int_cst_compare (step, TYPE_SIZE_UNIT (scalar_type))
+	    || (dr_step < 0
+		&& !compare_tree_int (TYPE_SIZE_UNIT (scalar_type), -dr_step));
+
+      if (sequantial)
 	{
 	  /* Mark that it is not interleaving.  */
 	  DR_GROUP_FIRST_ELEMENT (stmt_info) = NULL;
diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
index 51e7f05100de7ab31994f3b4a30f5c5be8d8afdd..7dee95951eef95622c9554c1bbe01c3e02cd7894 100644
--- a/gcc/tree-vect-loop.cc
+++ b/gcc/tree-vect-loop.cc
@@ -7184,12 +7184,17 @@ vectorizable_reduction (loop_vec_info loop_vinfo,
     gcc_unreachable ();
   bool lane_reducing = lane_reducing_op_p (op.code);
 
-  if (!POINTER_TYPE_P (op.type) && !INTEGRAL_TYPE_P (op.type)
-      && !SCALAR_FLOAT_TYPE_P (op.type))
+  tree scalar_type = op.type;
+  if (VECTOR_TYPE_P (scalar_type))
+    scalar_type = TREE_TYPE (scalar_type);
+
+  if (!POINTER_TYPE_P (scalar_type)
+      && !INTEGRAL_TYPE_P (scalar_type)
+      && !SCALAR_FLOAT_TYPE_P (scalar_type))
     return false;
 
   /* Do not try to vectorize bit-precision reductions.  */
-  if (!type_has_mode_precision_p (op.type)
+  if (!type_has_mode_precision_p (scalar_type)
       && op.code != BIT_AND_EXPR
       && op.code != BIT_IOR_EXPR
       && op.code != BIT_XOR_EXPR)
@@ -7444,7 +7449,7 @@ vectorizable_reduction (loop_vec_info loop_vinfo,
       if (!reduc_chain
 	  && known_eq (LOOP_VINFO_VECT_FACTOR (loop_vinfo), 1u))
 	;
-      else if (needs_fold_left_reduction_p (op.type, orig_code))
+      else if (needs_fold_left_reduction_p (scalar_type, orig_code))
 	{
 	  /* When vectorizing a reduction chain w/o SLP the reduction PHI
 	     is not directly used in stmt.  */
@@ -7573,7 +7578,7 @@ vectorizable_reduction (loop_vec_info loop_vinfo,
   else if (reduction_type == COND_REDUCTION)
     {
       int scalar_precision
-	= GET_MODE_PRECISION (SCALAR_TYPE_MODE (op.type));
+	= GET_MODE_PRECISION (SCALAR_TYPE_MODE (scalar_type));
       cr_index_scalar_type = make_unsigned_type (scalar_precision);
       cr_index_vector_type = get_same_sized_vectype (cr_index_scalar_type,
 						vectype_out);
diff --git a/gcc/tree-vect-patterns.cc b/gcc/tree-vect-patterns.cc
index 5f5e3fb6201da379460c46e9d7ff1d01625d0998..ca806f7d8d94333fc4301b8a8e0da5ad0a4fad0b 100644
--- a/gcc/tree-vect-patterns.cc
+++ b/gcc/tree-vect-patterns.cc
@@ -6243,6 +6243,140 @@ vect_recog_bool_pattern (vec_info *vinfo,
     return NULL;
 }
 
+static gimple *
+vect_recog_vect_bool_pattern (vec_info *vinfo,
+			      stmt_vec_info stmt_vinfo, tree *type_out)
+{
+  gimple *last_stmt = STMT_VINFO_STMT (stmt_vinfo);
+  enum tree_code rhs_code;
+  tree var, lhs, rhs, vectype;
+  gimple *pattern_stmt;
+
+  if (!is_gimple_assign (last_stmt))
+    return NULL;
+
+  var = gimple_assign_rhs1 (last_stmt);
+  lhs = gimple_assign_lhs (last_stmt);
+  rhs_code = gimple_assign_rhs_code (last_stmt);
+
+  if (rhs_code == VIEW_CONVERT_EXPR)
+    var = TREE_OPERAND (var, 0);
+
+  if (!VECTOR_BOOLEAN_TYPE_P (TREE_TYPE (var)))
+    return NULL;
+
+  if (rhs_code != VEC_COND_EXPR)
+    return NULL;
+
+  tree op0 = gimple_assign_rhs2 (last_stmt);
+  tree op1 = gimple_assign_rhs3 (last_stmt);
+  if (!integer_zerop (op1)
+      || !integer_all_onesp (op0))
+    return NULL;
+
+  /* Find the comparison and make it part of the pattern we'll make.  */
+  auto cmp_info = vinfo->lookup_def (var);
+  if (!cmp_info)
+    return NULL;
+
+  gimple *cmp_stmt = STMT_VINFO_STMT (cmp_info);
+  if (!is_gimple_assign (cmp_stmt)
+      || TREE_CODE_CLASS (gimple_assign_rhs_code (cmp_stmt)) != tcc_comparison)
+    return NULL;
+
+  vectype = get_vectype_for_scalar_type (vinfo, TREE_TYPE (lhs));
+  if (vectype == NULL_TREE)
+    return NULL;
+
+  /* Build a scalar type for the boolean result that when
+     vectorized matches the vector type of the result in
+      size and number of elements.  */
+  unsigned prec
+    = vector_element_size (tree_to_poly_uint64 (TYPE_SIZE (vectype)),
+			   TYPE_VECTOR_SUBPARTS (vectype));
+
+  tree type
+    = build_nonstandard_integer_type (prec, TYPE_UNSIGNED (TREE_TYPE (var)));
+
+  tree new_vectype = get_mask_type_for_scalar_type (vinfo, type);
+  if (new_vectype == NULL_TREE)
+    return NULL;
+
+  /* copy the scalar statement to get a new UID.  */
+  tree lhs_cmp
+    = vect_recog_temp_ssa_var (TREE_TYPE (gimple_get_lhs (cmp_stmt)), NULL);
+  gimple *copy_stmt
+     = gimple_build_assign (lhs_cmp, gimple_assign_rhs_code (cmp_stmt),
+			    gimple_assign_rhs1 (cmp_stmt),
+			    gimple_assign_rhs2 (cmp_stmt));
+  append_pattern_def_seq (vinfo, stmt_vinfo, copy_stmt, new_vectype, type);
+
+  tree lhs_ivar = vect_recog_temp_ssa_var (type, NULL);
+  pattern_stmt = gimple_build_assign (lhs_ivar, COND_EXPR, lhs_cmp,
+				      build_all_ones_cst (type),
+				      build_zero_cst (type));
+
+  *type_out = vectype;
+  vect_pattern_detected ("vect_recog_vect_bool_pattern", last_stmt);
+
+  return pattern_stmt;
+}
+
+static gimple *
+vect_recog_vect_scalar_pattern (vec_info *vinfo,
+				stmt_vec_info stmt_vinfo, tree *type_out)
+{
+  gimple *last_stmt = STMT_VINFO_STMT (stmt_vinfo);
+  enum tree_code rhs_code;
+  tree var, lhs, rhs, vectype;
+  gimple *pattern_stmt;
+  loop_vec_info loop_vinfo = dyn_cast <loop_vec_info> (vinfo);
+  class loop *loop = loop_vinfo ? LOOP_VINFO_LOOP (loop_vinfo) : NULL;
+
+
+  lhs = gimple_get_lhs (last_stmt);
+  if (!lhs || !VECTOR_TYPE_P (TREE_TYPE (lhs)))
+    return NULL;
+
+  if (targetm.vectorize.recog_patterns_builtin_call)
+    {
+      auto_vec<gimple *> chain;
+      gimple *res
+	= targetm.vectorize.recog_patterns_builtin_call (vinfo, stmt_vinfo,
+							 chain, type_out);
+      vect_pattern_detected ("vect_recog_vect_scalar_pattern", last_stmt);
+    }
+
+#if 1
+  /* First pattern is reductions, if we have a vector reduction then adjust
+     any constant lanes if we can.  TOOD: Maybe this should force HVLA? */
+  if (loop && is_a <gphi *> (last_stmt))
+    {
+      /* Check the header edge.  */
+      gphi *phi = as_a <gphi *>(last_stmt);
+      tree init = PHI_ARG_DEF_FROM_EDGE (phi, loop_preheader_edge (loop));
+      if ((var = ssa_uniform_vector_p (init)))
+	{
+	  tree type = TREE_TYPE (lhs);
+	  vectype = get_vectype_for_scalar_type (vinfo, type);
+	  if (vectype == NULL_TREE)
+	    return NULL;
+
+	  tree lhs_ivar = vect_recog_temp_ssa_var (type, NULL);
+	  gphi *res = make_phi_node (lhs_ivar, 2);
+	  tree step = PHI_ARG_DEF_FROM_EDGE (phi, loop_latch_edge (loop));
+	  add_phi_arg (res, init, loop_preheader_edge (loop), UNKNOWN_LOCATION);
+	  add_phi_arg (res, step, loop_latch_edge (loop), UNKNOWN_LOCATION);
+	  pattern_stmt = res;
+
+	  *type_out = vectype;
+	  vect_pattern_detected ("vect_recog_vect_scalar_pattern", last_stmt);
+	}
+    }
+#endif
+
+  return NULL;
+}
 
 /* Function vect_recog_mask_conversion_pattern
 
@@ -7467,6 +7601,8 @@ static vect_recog_func vect_vect_recog_func_ptrs[] = {
   { vect_recog_sat_trunc_pattern, "sat_trunc" },
   { vect_recog_gcond_pattern, "gcond" },
   { vect_recog_bool_pattern, "bool" },
+  { vect_recog_vect_bool_pattern, "vect_bool" },
+  { vect_recog_vect_scalar_pattern, "vect_scalar" },
   /* This must come before mask conversion, and includes the parts
      of mask conversion that are needed for gather and scatter
      internal functions.  */
diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc
index 075e93f04a97cddd6bc4b7f82485f08ee22c953b..c1bf905f54786c3622a155f777e6889a635506d4 100644
--- a/gcc/tree-vect-slp.cc
+++ b/gcc/tree-vect-slp.cc
@@ -1950,7 +1950,7 @@ vect_build_slp_tree (vec_info *vinfo,
 	  for (i = 0; i < group_size; ++i)
 	    if (!matches[i])
 	      break;
-	  gcc_assert (i < group_size);
+	  //gcc_assert (i < group_size);
 	}
       memcpy (res->failed, matches, sizeof (bool) * group_size);
     }
@@ -2107,6 +2107,9 @@ vect_build_slp_tree_2 (vec_info *vinfo, slp_tree node,
 			      &vectype))
     return NULL;
 
+  if (known_gt (group_size, this_max_nunits))
+    return NULL;
+
   /* If the SLP node is a load, terminate the recursion unless masked.  */
   if (STMT_VINFO_DATA_REF (stmt_info)
       && DR_IS_READ (STMT_VINFO_DATA_REF (stmt_info)))
@@ -2133,6 +2136,7 @@ vect_build_slp_tree_2 (vec_info *vinfo, slp_tree node,
 	  FOR_EACH_VEC_ELT (SLP_TREE_SCALAR_STMTS (node), j, load_info)
 	    {
 	      int load_place;
+	      tree type;
 	      if (! load_info)
 		{
 		  if (STMT_VINFO_GROUPED_ACCESS (stmt_info))
@@ -2140,6 +2144,11 @@ vect_build_slp_tree_2 (vec_info *vinfo, slp_tree node,
 		  else
 		    load_place = 0;
 		}
+	      /* ?? Not all vector loads are linear, model this.  */
+	      else if ((type = TREE_TYPE (gimple_get_lhs (STMT_VINFO_STMT (load_info))))
+		       && VECTOR_TYPE_P (type)
+		       && TYPE_VECTOR_SUBPARTS (type).is_constant ())
+		load_place = j;
 	      else if (STMT_VINFO_GROUPED_ACCESS (stmt_info))
 		load_place = vect_get_place_in_interleaving_chain
 		    (load_info, first_stmt_info);
@@ -3141,7 +3150,7 @@ fail:
 		n_vector_builds++;
 	    }
 	}
-      if (all_uniform_p
+      if (all_uniform_p && false
 	  || n_vector_builds > 1
 	  || (n_vector_builds == children.length ()
 	      && is_a <gphi *> (stmt_info->stmt)))
@@ -4282,7 +4291,9 @@ vect_build_slp_instance (vec_info *vinfo,
   /* While we arrive here even with slp_inst_kind_store we should only
      for group_size == 1.  The code to split store groups is only in
      vect_analyze_slp_instance now.  */
+#if 0
   gcc_assert (kind != slp_inst_kind_store || group_size == 1);
+#endif
 
   /* Free the allocated memory.  */
   scalar_stmts.release ();
@@ -5014,7 +5025,16 @@ vect_analyze_slp_instance (vec_info *vinfo,
   stmt_vec_info next_info = stmt_info;
   while (next_info)
     {
-      scalar_stmts.quick_push (vect_stmt_to_vectorize (next_info));
+      scalar_stmts.safe_push (vect_stmt_to_vectorize (next_info));
+#if 1
+      /* If we're a vector type, insert dummy nodes.  */
+      unsigned HOST_WIDE_INT vect_nunits = 0;
+      tree vectype = TREE_TYPE (gimple_get_lhs (STMT_VINFO_STMT (next_info)));
+      if (VECTOR_TYPE_P (vectype)
+	  && TYPE_VECTOR_SUBPARTS (vectype).is_constant (&vect_nunits))
+	for (auto i = 0; i < vect_nunits - 1UL; i++)
+	  scalar_stmts.safe_push (next_info);
+#endif
       next_info = DR_GROUP_NEXT_ELEMENT (next_info);
     }
 
@@ -5038,6 +5058,13 @@ vect_analyze_slp_instance (vec_info *vinfo,
 
   /* Build the tree for the SLP instance.  */
   unsigned int group_size = scalar_stmts.length ();
+#if 0
+  /* If we're a vector type, adjust group size by nunits.  */
+  unsigned HOST_WIDE_INT vect_nunits = 0;
+  tree vectype = TREE_TYPE (gimple_get_lhs (STMT_VINFO_STMT (scalar_stmts[0])));
+  if (TYPE_VECTOR_SUBPARTS (vectype).is_constant (&vect_nunits))
+    group_size *= vect_nunits;
+#endif
   bool *matches = XALLOCAVEC (bool, group_size);
   poly_uint64 max_nunits = 1;
   unsigned tree_size = 0;
@@ -9780,6 +9807,7 @@ vect_slp_check_for_roots (bb_vec_info bb_vinfo)
       enum tree_code code = gimple_assign_rhs_code (assign);
       use_operand_p use_p;
       gimple *use_stmt;
+      optab op;
       if (code == CONSTRUCTOR)
 	{
 	  if (!VECTOR_TYPE_P (TREE_TYPE (rhs))
@@ -9995,6 +10023,24 @@ vect_slp_check_for_roots (bb_vec_info bb_vinfo)
 		}
 	    }
 	}
+#if 1
+	else if (gimple_vdef (assign)
+	       && VECTOR_TYPE_P (TREE_TYPE (rhs))
+	       && ((op = optab_for_tree_code (code, TREE_TYPE (rhs), optab_default)) == unknown_optab
+		   || !can_implement_p (op, TYPE_MODE (TREE_TYPE (rhs)))))
+	{
+	  vec<stmt_vec_info> roots = vNULL;
+	  auto stmt_vinfo = bb_vinfo->lookup_stmt (assign);
+	  roots.safe_push (stmt_vinfo);
+	  vec<stmt_vec_info> stmts;
+	  auto num_entries = TYPE_VECTOR_SUBPARTS (TREE_TYPE (rhs)).to_constant () / 2;
+	  stmts.create (num_entries);
+	  for (int i = 0; i < num_entries; i++)
+	    stmts.quick_push (stmt_vinfo);
+	  bb_vinfo->roots.safe_push (slp_root (slp_inst_kind_store,
+					       stmts, roots));
+	}
+#endif
     }
 }
 
@@ -10984,6 +11030,14 @@ vect_transform_slp_perm_load_1 (vec_info *vinfo, slp_tree node,
       dr_group_size = DR_GROUP_SIZE (stmt_info);
     }
 
+  /* We should probably adjust group size, but for now lets hack.  */
+  unsigned HOST_WIDE_INT vect_nunits = 0;
+  auto vscalar_stmt = STMT_VINFO_STMT (stmt_info);
+  tree vscalar_type = TREE_TYPE (gimple_get_lhs (vscalar_stmt));
+  if (VECTOR_TYPE_P (vscalar_type)
+      && TYPE_VECTOR_SUBPARTS (vscalar_type).is_constant (&vect_nunits))
+    dr_group_size *= vect_nunits;
+
   mode = TYPE_MODE (vectype);
   poly_uint64 nunits = TYPE_VECTOR_SUBPARTS (vectype);
   unsigned int nstmts = vect_get_num_copies (vinfo, node);
@@ -12431,6 +12485,8 @@ vect_schedule_slp (vec_info *vinfo, const vec<slp_instance> &slp_instances)
 	    break;
 
 	  store_info = vect_orig_stmt (store_info);
+	  if (!store_info)
+	    continue;
 	  /* Free the attached stmt_vec_info and remove the stmt.  */
 	  vinfo->remove_stmt (store_info);
 
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index 7687c107a6afc54741cb5e3f5756bcb65196b519..3cd8e4dfba4b3bf3173753a08cb214493b802e13 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -2108,9 +2108,11 @@ get_load_store_type (vec_info  *vinfo, stmt_vec_info stmt_info,
   bool perm_ok = true;
   poly_int64 vf = loop_vinfo ? LOOP_VINFO_VECT_FACTOR (loop_vinfo) : 1;
 
+#if 1
   if (SLP_TREE_LOAD_PERMUTATION (slp_node).exists ())
     perm_ok = vect_transform_slp_perm_load (vinfo, slp_node, vNULL, NULL,
 					    vf, true, n_perms, n_loads);
+#endif
 
   if (STMT_VINFO_GROUPED_ACCESS (stmt_info))
     {
@@ -2728,8 +2730,18 @@ get_load_store_type (vec_info  *vinfo, stmt_vec_info stmt_info,
 	    if (k > maxk)
 	      maxk = k;
 	  tree vectype = SLP_TREE_VECTYPE (slp_node);
+
+	  /* We should probably adjust group size, but for now lets hack.  */
+	  unsigned HOST_WIDE_INT vect_nunits = 0;
+	  auto group_size = DR_GROUP_SIZE (group_info);
+	  auto vscalar_stmt = STMT_VINFO_STMT (group_info);
+	  tree vscalar_type = TREE_TYPE (gimple_get_lhs (vscalar_stmt));
+	  if (VECTOR_TYPE_P (vscalar_type)
+	      && TYPE_VECTOR_SUBPARTS (vscalar_type).is_constant (&vect_nunits))
+	    group_size *= vect_nunits;
+
 	  if (!TYPE_VECTOR_SUBPARTS (vectype).is_constant (&nunits)
-	      || maxk >= (DR_GROUP_SIZE (group_info) & ~(nunits - 1)))
+	      || maxk >= (group_size & ~(nunits - 1)))
 	    {
 	      if (dump_enabled_p ())
 		dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
@@ -6618,13 +6630,17 @@ vectorizable_operation (vec_info *vinfo,
     }
 
   scalar_dest = gimple_assign_lhs (stmt);
+  tree scalar_type = TREE_TYPE (scalar_dest);
+  if (VECTOR_TYPE_P (scalar_type))
+    scalar_type = TREE_TYPE (scalar_type);
+
   vectype_out = SLP_TREE_VECTYPE (slp_node);
 
   /* Most operations cannot handle bit-precision types without extra
      truncations.  */
   bool mask_op_p = VECTOR_BOOLEAN_TYPE_P (vectype_out);
   if (!mask_op_p
-      && !type_has_mode_precision_p (TREE_TYPE (scalar_dest))
+      && !type_has_mode_precision_p (scalar_type)
       /* Exception are bitwise binary operations.  */
       && code != BIT_IOR_EXPR
       && code != BIT_XOR_EXPR
@@ -6658,7 +6674,7 @@ vectorizable_operation (vec_info *vinfo,
 	 type.  */
       if (VECT_SCALAR_BOOLEAN_TYPE_P (TREE_TYPE (op0)))
 	{
-	  if (!VECT_SCALAR_BOOLEAN_TYPE_P (TREE_TYPE (scalar_dest)))
+	  if (!VECT_SCALAR_BOOLEAN_TYPE_P (scalar_type))
 	    {
 	      if (dump_enabled_p ())
 		dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
@@ -8297,6 +8313,14 @@ vectorizable_store (vec_info *vinfo,
       group_size = 1;
     }
 
+    /* We should probably adjust group size, but for now lets hack.  */
+    unsigned HOST_WIDE_INT vect_nunits = 0;
+    auto vscalar_stmt = STMT_VINFO_STMT (first_stmt_info);
+    tree vscalar_type = TREE_TYPE (gimple_get_lhs (vscalar_stmt));
+    if (VECTOR_TYPE_P (vscalar_type)
+	&& TYPE_VECTOR_SUBPARTS (vscalar_type).is_constant (&vect_nunits))
+      group_size *= vect_nunits;
+
   if (STMT_VINFO_SIMD_LANE_ACCESS_P (stmt_info) > 1 && cost_vec)
     {
       if (!check_scan_store (vinfo, stmt_info, vectype, rhs_dt, slp_node,
@@ -9887,6 +9911,9 @@ vectorizable_load (vec_info *vinfo,
      padding bits in the type that might leak otherwise.
      Refer to PR115336.  */
   tree scalar_type = TREE_TYPE (scalar_dest);
+  if (VECTOR_TYPE_P (scalar_type))
+    scalar_type = TREE_TYPE (scalar_type);
+
   bool type_mode_padding_p
     = TYPE_PRECISION (scalar_type) < GET_MODE_PRECISION (GET_MODE_INNER (mode));
 
@@ -10509,6 +10536,7 @@ vectorizable_load (vec_info *vinfo,
 	  first_stmt_info = stmt_info;
 	  group_size = 1;
 	}
+
       /* For SLP vectorization we directly vectorize a subchain
          without permutation.  */
       if (! SLP_TREE_LOAD_PERMUTATION (slp_node).exists ())
@@ -10522,6 +10550,14 @@ vectorizable_load (vec_info *vinfo,
       first_dr_info = STMT_VINFO_DR_INFO (first_stmt_info);
       group_gap_adj = 0;
 
+      /* We should probably adjust group size, but for now lets hack.  */
+      unsigned HOST_WIDE_INT vect_nunits = 0;
+      auto vscalar_stmt = STMT_VINFO_STMT (first_stmt_info);
+      tree vscalar_type = TREE_TYPE (gimple_get_lhs (vscalar_stmt));
+      if (VECTOR_TYPE_P (vscalar_type)
+	  && TYPE_VECTOR_SUBPARTS (vscalar_type).is_constant (&vect_nunits))
+	group_size *= vect_nunits;
+
       /* VEC_NUM is the number of vect stmts to be created for this group.  */
       grouped_load = false;
       /* If an SLP permutation is from N elements to N elements,
@@ -12015,7 +12051,8 @@ vect_is_simple_cond (tree cond, vec_info *vinfo,
 
   /* Mask case.  */
   if (TREE_CODE (cond) == SSA_NAME
-      && VECT_SCALAR_BOOLEAN_TYPE_P (TREE_TYPE (cond)))
+      && (VECT_SCALAR_BOOLEAN_TYPE_P (TREE_TYPE (cond))
+	  || VECTOR_BOOLEAN_TYPE_P (TREE_TYPE (cond))))
     {
       if (!vect_is_simple_use (vinfo, slp_node, 0, &cond,
 			       &slp_op, &dts[0], comp_vectype)
@@ -13209,6 +13246,226 @@ vectorizable_early_exit (loop_vec_info loop_vinfo, stmt_vec_info stmt_info,
   return true;
 }
 
+/* Intrinsics and vector types may not match even though the underlying modes
+   are the same.  This is particularly true of masks and SVE modes.  As such
+   convert between the two.  */
+static void
+vect_shim_call_arg (vec<tree> &types, vec_info *vinfo, stmt_vec_info stmt_info,
+		    gimple_stmt_iterator *gsi, gcall* call, int argno,
+		    tree newarg)
+{
+  tree ty_dest = types[argno+1];
+  tree ty_src = TREE_TYPE (newarg);
+
+  if (ty_src != ty_dest)
+    {
+      tree lhs = vect_create_destination_var (newarg, ty_dest);
+      tree op = build1 (VIEW_CONVERT_EXPR, ty_dest, newarg);
+      gimple *tmp = gimple_build_assign (lhs, VIEW_CONVERT_EXPR, op);
+      lhs = make_ssa_name (lhs, tmp);
+      gimple_assign_set_lhs (tmp, lhs);
+      vect_finish_stmt_generation (vinfo, stmt_info, tmp, gsi);
+      newarg = lhs;
+
+    if (dump_enabled_p ())
+      dump_printf_loc (MSG_NOTE, vect_location, "inserted type shim between: %T"
+		       " and %T due to mismatch between generic type and vector"
+		       " call.\n", ty_dest, ty_src);
+    }
+
+  gimple_call_set_arg (call, argno, newarg);
+}
+
+bool
+vectorizable_vect_call (vec_info *vinfo,
+			stmt_vec_info stmt_info, gimple_stmt_iterator *gsi,
+			slp_tree slp_node,
+			stmt_vector_for_cost *cost_vec)
+{
+  enum vect_def_type dt[5]
+    = { vect_unknown_def_type, vect_unknown_def_type, vect_unknown_def_type,
+	vect_unknown_def_type, vect_unknown_def_type };
+  tree vectypes[ARRAY_SIZE (dt)] = {};
+  slp_tree slp_op[ARRAY_SIZE (dt)] = {};
+  tree op;
+
+  /* If the target hook isn't configured there's nothing to do.  */
+  if (!targetm.vectorize.translate_builtin_call)
+    return false;
+
+  if (!STMT_VINFO_RELEVANT_P (stmt_info))
+    return false;
+
+  if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_internal_def
+      && cost_vec)
+    return false;
+
+  /* Is STMT_INFO a vectorizable vector call?   */
+  gcall *stmt = dyn_cast <gcall *> (STMT_VINFO_STMT (stmt_info));
+  if (!stmt || !gimple_call_builtin_p (stmt))
+    return false;
+
+  /* It must be a built in with a vector type as an argument. */
+  bool has_vect_arg = false;
+  int nargs = gimple_call_num_args (stmt);
+
+  /* Check if it has more arguments than we support at the moment.  */
+  if (nargs > 5)
+    {
+      if (dump_enabled_p ())
+	dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
+			 "vector function call has too many arguments.\n");
+      return false;
+  }
+
+  for (int i = 0; i < nargs; i++)
+    {
+      has_vect_arg = has_vect_arg
+		     || VECTOR_TYPE_P (TREE_TYPE (gimple_call_arg (stmt, i)));
+
+      if (!vect_is_simple_use (vinfo, slp_node,
+			       i, &op, &slp_op[i], &dt[i], &vectypes[i]))
+	{
+	  if (dump_enabled_p ())
+	    dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
+			     "use not simple.\n");
+	  return false;
+	}
+    }
+
+  if (!has_vect_arg)
+    {
+      if (dump_enabled_p ())
+	dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
+			 "function is not a vectorizable vector function.\n");
+      return false;
+    }
+
+  /* We only handle functions that do not read or clobber memory.  */
+  if (gimple_vuse (stmt))
+    {
+      if (dump_enabled_p ())
+	dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
+			 "function reads from or writes to memory.\n");
+      return false;
+    }
+
+  tree vectype_out;
+  int maskopno = -1;
+  auto_vec<tree> types;
+  gcall *call
+    = targetm.vectorize.translate_builtin_call (vinfo, slp_node, stmt_info,
+						&maskopno, types, &vectype_out);
+
+  if (!call)
+    {
+      if (dump_enabled_p ())
+	dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
+			 "target did not have implementation for %G.\n",
+			 (gimple*)stmt);
+      return false;
+    }
+
+  loop_vec_info loop_vinfo = dyn_cast <loop_vec_info> (vinfo);
+  vec_loop_masks *masks = (loop_vinfo ? &LOOP_VINFO_MASKS (loop_vinfo) : NULL);
+  vec_loop_lens *lens = (loop_vinfo ? &LOOP_VINFO_LENS (loop_vinfo) : NULL);
+  bool masked_loop_p = loop_vinfo && LOOP_VINFO_FULLY_MASKED_P (loop_vinfo);
+  bool len_loop_p = loop_vinfo && LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo);
+  unsigned int vect_nargs = nargs;
+
+  if (maskopno != -1)
+    {
+      if (!loop_vinfo) // || (!masked_loop_p && !len_loop_p))
+	{
+	  if (dump_enabled_p ())
+	    dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
+			     "target translation requires predication but "
+			     "masks not supported: %G.\n",
+			     (gimple*)stmt);
+	  return false;
+	}
+      vect_nargs += 1;
+    }
+
+  if (cost_vec) /* transformation not required.  */
+    {
+      SLP_TREE_TYPE (slp_node) = vector_call_vec_info_type;
+      DUMP_VECT_SCOPE ("vectorizable_vect_call");
+      vect_model_simple_cost (vinfo, 1, slp_node, cost_vec);
+      return true;
+    }
+
+  /* Transform.  */
+
+  if (dump_enabled_p ())
+    dump_printf_loc (MSG_NOTE, vect_location, "transform vector call.\n");
+
+  /* Resolve arguments.  */
+  auto_vec<vec<tree> > vec_defs (nargs);
+  vect_get_slp_defs (vinfo, slp_node, &vec_defs);
+
+  /* Handle def.  */
+  tree scalar_dest = gimple_call_lhs (stmt);
+  tree vec_dest = vect_create_destination_var (scalar_dest, vectype_out);
+  tree new_temp = make_ssa_name (vec_dest, stmt);
+  gimple_call_set_lhs (call, new_temp);
+  gimple_call_set_nothrow (call, true);
+
+  /* Now fill in the new values.  Intrinsics types and vector types may not
+     match, so convert the types.  */
+  int shift = 0;
+  for (int i = 0; i < nargs; i++)
+    {
+      if (maskopno == i)
+	{
+	  tree mask;
+	  if (masked_loop_p)
+	    {
+	      /* ?? Handle decomposition here, can that happen??.  */
+	      unsigned int vec_num = 1; //vec_oprnds0.length ();
+	      mask = vect_get_loop_mask (loop_vinfo, gsi, masks, vec_num,
+					 vectype_out, 0);
+	    }
+	  else
+	    {
+	      tree mask_vectype = truth_type_for (vectype_out);
+	      mask = vect_build_all_ones_mask (loop_vinfo, stmt_info,
+					       mask_vectype);
+	    }
+	  vect_shim_call_arg (types, vinfo, stmt_info, gsi, call, i, mask);
+	  shift++;
+	}
+      tree arg = vec_defs[i][0];
+      vect_shim_call_arg (types, vinfo, stmt_info, gsi, call, i+shift, arg);
+    }
+
+  gimple *res = call;
+  /* Check if the result types match, if not, shim them.  */
+  if (vectype_out != types[0])
+    {
+      /* First update the call.  */
+      tree lhs = vect_create_destination_var (scalar_dest, types[0]);
+      tree new_lhs = make_ssa_name (lhs, NULL);
+      gimple_call_set_lhs (call, new_lhs);
+      vect_finish_stmt_generation (vinfo, stmt_info, call, gsi);
+      tree op = build1 (VIEW_CONVERT_EXPR, vectype_out, new_lhs);
+      gimple *tmp = gimple_build_assign (vec_dest, VIEW_CONVERT_EXPR, op);
+      gimple_assign_set_lhs (tmp, new_temp);
+      vect_finish_stmt_generation (vinfo, stmt_info, tmp, gsi);
+      res = tmp; 
+
+      if (dump_enabled_p ())
+	dump_printf_loc (MSG_NOTE, vect_location, "inserted type shim between: %T"
+		       " and %T due to mismatch between generic type and vector"
+		       " call.\n", types[0], vectype_out);
+    }
+  else
+    vect_finish_stmt_generation (vinfo, stmt_info, call, gsi);
+
+  slp_node->push_vec_def (res);
+  return true;
+}
+
 /* If SLP_NODE is nonnull, return true if vectorizable_live_operation
    can handle all live statements in the node.  Otherwise return true
    if STMT_INFO is not live or if vectorizable_live_operation can handle it.
@@ -13338,6 +13595,7 @@ vect_analyze_stmt (vec_info *vinfo,
 	  || vectorizable_shift (vinfo, stmt_info, NULL, node, cost_vec)
 	  || vectorizable_condition (vinfo, stmt_info, NULL, node, cost_vec)
 	  || vectorizable_comparison (vinfo, stmt_info, NULL, node, cost_vec)
+	  || vectorizable_vect_call (vinfo, stmt_info, NULL, node, cost_vec)
 	  || (bb_vinfo
 	      && vectorizable_phi (bb_vinfo, stmt_info, node, cost_vec))
 	  || (is_a <loop_vec_info> (vinfo)
@@ -13454,6 +13712,10 @@ vect_transform_stmt (vec_info *vinfo,
       done = vectorizable_call (vinfo, stmt_info, gsi, slp_node, NULL);
       break;
 
+    case vector_call_vec_info_type:
+      done = vectorizable_vect_call (vinfo, stmt_info, gsi, slp_node, NULL);
+      break;
+
     case call_simd_clone_vec_info_type:
       done = vectorizable_simd_clone_call (vinfo, stmt_info, gsi,
 					   slp_node, NULL);
@@ -13572,6 +13834,10 @@ get_related_vectype_for_scalar_type (machine_mode prevailing_mode,
   machine_mode simd_mode;
   tree vectype;
 
+  /* For vector types, get the inner most type.  */
+  if (VECTOR_TYPE_P (scalar_type))
+    scalar_type = TREE_TYPE (scalar_type);
+
   if ((!INTEGRAL_TYPE_P (scalar_type)
        && !POINTER_TYPE_P (scalar_type)
        && !SCALAR_FLOAT_TYPE_P (scalar_type))
@@ -14801,6 +15067,10 @@ vect_get_vector_types_for_stmt (vec_info *vinfo, stmt_vec_info stmt_info,
       else
 	scalar_type = TREE_TYPE (gimple_get_lhs (stmt));
 
+      /* If scalar_type is a vector type, get the innner typ.  */
+      if (VECTOR_TYPE_P (scalar_type))
+	scalar_type = TREE_TYPE (scalar_type);
+
       if (dump_enabled_p ())
 	{
 	  if (group_size)
@@ -14901,4 +15171,3 @@ vect_gen_len (tree len, tree start_index, tree end_index, tree len_limit)
 
   return stmts;
 }
-
diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
index 35e531c11c178839bcf3218e58ad7521662d3b42..1bba7702b628cc4012848e1d887be03fe3d2533d 100644
--- a/gcc/tree-vectorizer.h
+++ b/gcc/tree-vectorizer.h
@@ -229,6 +229,7 @@ enum stmt_vec_info_type {
   shift_vec_info_type,
   op_vec_info_type,
   call_vec_info_type,
+  vector_call_vec_info_type,
   call_simd_clone_vec_info_type,
   assignment_vec_info_type,
   condition_vec_info_type,