This applies a default 2:1 scalar/vector unit ratio to integer vector
body costs, which helps the cost model better match the actual hardware
throughput.
Take an int64 vadd at VLEN=128 for example: 4 scalar units and 2 vector
units can both process 4 elements per cycle. Without this ratio, both
vector and scalar stmts just get a cost of 1, which ends up making
vectorization look too cheap.
Skip applying this ratio for:
- reduction stmts
- gather/scatter stmts
Since this is just an RFC patch, a few things need comments:
1. The ratio is hardcoded to 2 for now.
- Should this come from the CPU model, tune info, or a param
instead? I'm leaving it as is for future work.
- How should we handle non-integer ratios like 3/2 ?
2. I'm not entirely sure how to handle some regression tests yet,
especially the vx tests. Here are what I do for now:
update the gpr2vr/vr2gpr:
- vx-5-[i|u]64.c
update the asm checks:
- vx-6-i64.c
add -fno-vect-cost-model and skip some -mrvv-max-lmul options:
- vcompress-avlprop-1.c
For the other cost-sensitive tests, add -fno-vect-cost-model,
I think these are safe:
- pr120378-[1|2|3|4].c
- pr112988-1.c
- gather_load_64-[3|5].c
- scatter_store_32-2.c
- scatter_store_64-[3|5].c
- pr112599-2.c
- pr113607.c
gcc/ChangeLog:
* config/riscv/riscv-vector-costs.cc (scalar_over_vector_unit_ratio):
New.
(costs::add_stmt_cost): Scale non-reduction, non-gather/scatter
integer vector body costs.
gcc/testsuite/ChangeLog:
* gcc.target/riscv/rvv/autovec/pr120378-1.c: Add
-fno-vect-cost-model.
* gcc.target/riscv/rvv/autovec/pr120378-2.c: Ditto.
* gcc.target/riscv/rvv/autovec/pr120378-3.c: Ditto.
* gcc.target/riscv/rvv/autovec/pr120378-4.c: Ditto.
* gcc.target/riscv/rvv/autovec/gather-scatter/gather_load_64-3.c:
Ditto.
* gcc.target/riscv/rvv/autovec/gather-scatter/gather_load_64-5.c:
Ditto.
* gcc.target/riscv/rvv/autovec/gather-scatter/scatter_store_32-2.c:
Ditto.
* gcc.target/riscv/rvv/autovec/gather-scatter/scatter_store_64-3.c:
Ditto.
* gcc.target/riscv/rvv/autovec/gather-scatter/scatter_store_64-5.c:
Ditto.
* gcc.target/riscv/rvv/autovec/pr112599-2.c: Ditto.
* gcc.target/riscv/rvv/autovec/pr113607.c: Ditto.
* gcc.target/riscv/rvv/vsetvl/pr112988-1.c: Ditto.
* gcc.target/riscv/rvv/autovec/vx_vf/vx-5-i64.c: Update costs.
* gcc.target/riscv/rvv/autovec/vx_vf/vx-5-u64.c: Ditto.
* gcc.target/riscv/rvv/autovec/vx_vf/vx-6-i64.c: Update asm checks.
* gcc.target/riscv/rvv/autovec/binop/vcompress-avlprop-1.c: Add
-fno-vect-cost-model and skip some -mrvv-max-lmul.
Signed-off-by: Zhongyao Chen <[email protected]>
---
gcc/config/riscv/riscv-vector-costs.cc | 17 ++++++++
.../rvv/autovec/binop/vcompress-avlprop-1.c | 4 +-
.../autovec/gather-scatter/gather_load_64-3.c | 2 +-
.../autovec/gather-scatter/gather_load_64-5.c | 2 +-
.../gather-scatter/scatter_store_32-2.c | 2 +-
.../gather-scatter/scatter_store_64-3.c | 2 +-
.../gather-scatter/scatter_store_64-5.c | 2 +-
.../gcc.target/riscv/rvv/autovec/pr112599-2.c | 2 +-
.../gcc.target/riscv/rvv/autovec/pr113607.c | 2 +-
.../gcc.target/riscv/rvv/autovec/pr120378-1.c | 2 +-
.../gcc.target/riscv/rvv/autovec/pr120378-2.c | 2 +-
.../gcc.target/riscv/rvv/autovec/pr120378-3.c | 2 +-
.../gcc.target/riscv/rvv/autovec/pr120378-4.c | 2 +-
.../riscv/rvv/autovec/vx_vf/vx-5-i64.c | 2 +-
.../riscv/rvv/autovec/vx_vf/vx-5-u64.c | 2 +-
.../riscv/rvv/autovec/vx_vf/vx-6-i64.c | 43 ++++++++++++-------
.../gcc.target/riscv/rvv/vsetvl/pr112988-1.c | 2 +-
17 files changed, 61 insertions(+), 31 deletions(-)
diff --git a/gcc/config/riscv/riscv-vector-costs.cc
b/gcc/config/riscv/riscv-vector-costs.cc
index bfb8a0ca63b..96eef3d6bfb 100644
--- a/gcc/config/riscv/riscv-vector-costs.cc
+++ b/gcc/config/riscv/riscv-vector-costs.cc
@@ -49,6 +49,10 @@ along with GCC; see the file COPYING3. If not see
namespace riscv_vector {
+/* Default scalar-over-vector unit ratio for vector cost scaling. This may
+ later come from the CPU model, tune info, or a user-visible interface. */
+static constexpr unsigned int scalar_over_vector_unit_ratio = 2;
+
/* Dynamic LMUL philosophy - Local linear-scan SSA live range based analysis
determine LMUL
@@ -1617,6 +1621,19 @@ costs::add_stmt_cost (int count, vect_cost_for_stmt kind,
stmt_cost = adjust_stmt_cost (kind, loop_vinfo, stmt_info, node, vectype,
stmt_cost);
+ /* Scale integer vector body costs by a scalar/vector ratio to better match
+ actual hardware throughput, avoid making vectorization too cheap. */
+ if (!costing_for_scalar ()
+ && loop_vinfo
+ && where == vect_body
+ && kind == vector_stmt
+ && !is_reduction (stmt_info, node)
+ && !(stmt_info && STMT_VINFO_GATHER_SCATTER_P (stmt_info))
+ && !(node && mat_gather_scatter_p (SLP_TREE_MEMORY_ACCESS_TYPE (node)))
+ && vectype
+ && VECTOR_INTEGER_TYPE_P (vectype))
+ stmt_cost *= scalar_over_vector_unit_ratio;
+
return record_stmt_cost (stmt_info, where, count * stmt_cost);
}
diff --git
a/gcc/testsuite/gcc.target/riscv/rvv/autovec/binop/vcompress-avlprop-1.c
b/gcc/testsuite/gcc.target/riscv/rvv/autovec/binop/vcompress-avlprop-1.c
index de86e904f93..1abfbe99a55 100644
--- a/gcc/testsuite/gcc.target/riscv/rvv/autovec/binop/vcompress-avlprop-1.c
+++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/binop/vcompress-avlprop-1.c
@@ -1,7 +1,7 @@
/* { dg-do compile } */
-/* { dg-options "-march=rv64gcv_zvl512b -mabi=lp64d -O3 -mrvv-vector-bits=zvl
-fno-schedule-insns -fno-schedule-insns2" } */
+/* { dg-options "-march=rv64gcv_zvl512b -mabi=lp64d -O3 -mrvv-vector-bits=zvl
-fno-schedule-insns -fno-schedule-insns2 -fno-vect-cost-model" } */
/* { dg-final { check-function-bodies "**" "" } } */
-/* { dg-skip-if "" { *-*-* } { "-mrvv-max-lmul=dynamic" } } */
+/* { dg-skip-if "" { *-*-* } { "-mrvv-max-lmul=dynamic" "-mrvv-max-lmul=m2"
"-mrvv-max-lmul=m4" "-mrvv-max-lmul=m8" } } */
#define MAX 10
struct s { struct s *n; } *p;
diff --git
a/gcc/testsuite/gcc.target/riscv/rvv/autovec/gather-scatter/gather_load_64-3.c
b/gcc/testsuite/gcc.target/riscv/rvv/autovec/gather-scatter/gather_load_64-3.c
index b23603a76e6..4a26068c941 100644
---
a/gcc/testsuite/gcc.target/riscv/rvv/autovec/gather-scatter/gather_load_64-3.c
+++
b/gcc/testsuite/gcc.target/riscv/rvv/autovec/gather-scatter/gather_load_64-3.c
@@ -1,5 +1,5 @@
/* { dg-do compile } */
-/* { dg-additional-options "-march=rv64gcv_zvfh -mabi=lp64d
-fdump-tree-vect-details" } */
+/* { dg-additional-options "-march=rv64gcv_zvfh -mabi=lp64d
-fdump-tree-vect-details -fno-vect-cost-model" } */
#include <stdint-gcc.h>
diff --git
a/gcc/testsuite/gcc.target/riscv/rvv/autovec/gather-scatter/gather_load_64-5.c
b/gcc/testsuite/gcc.target/riscv/rvv/autovec/gather-scatter/gather_load_64-5.c
index 19ff214c062..469c370a57b 100644
---
a/gcc/testsuite/gcc.target/riscv/rvv/autovec/gather-scatter/gather_load_64-5.c
+++
b/gcc/testsuite/gcc.target/riscv/rvv/autovec/gather-scatter/gather_load_64-5.c
@@ -1,5 +1,5 @@
/* { dg-do compile } */
-/* { dg-additional-options "-march=rv64gcv_zvfh -mabi=lp64d
-fdump-tree-vect-details" } */
+/* { dg-additional-options "-march=rv64gcv_zvfh -mabi=lp64d
-fdump-tree-vect-details -fno-vect-cost-model" } */
#include <stdint-gcc.h>
diff --git
a/gcc/testsuite/gcc.target/riscv/rvv/autovec/gather-scatter/scatter_store_32-2.c
b/gcc/testsuite/gcc.target/riscv/rvv/autovec/gather-scatter/scatter_store_32-2.c
index 35e97012469..fb0908ec5a2 100644
---
a/gcc/testsuite/gcc.target/riscv/rvv/autovec/gather-scatter/scatter_store_32-2.c
+++
b/gcc/testsuite/gcc.target/riscv/rvv/autovec/gather-scatter/scatter_store_32-2.c
@@ -1,5 +1,5 @@
/* { dg-do compile } */
-/* { dg-additional-options "-march=rv32gcv_zvfh -mabi=ilp32d
-fdump-tree-vect-details" } */
+/* { dg-additional-options "-march=rv32gcv_zvfh -mabi=ilp32d
-fdump-tree-vect-details -fno-vect-cost-model" } */
#include <stdint-gcc.h>
diff --git
a/gcc/testsuite/gcc.target/riscv/rvv/autovec/gather-scatter/scatter_store_64-3.c
b/gcc/testsuite/gcc.target/riscv/rvv/autovec/gather-scatter/scatter_store_64-3.c
index 47e8ce3e186..a2e665f49ce 100644
---
a/gcc/testsuite/gcc.target/riscv/rvv/autovec/gather-scatter/scatter_store_64-3.c
+++
b/gcc/testsuite/gcc.target/riscv/rvv/autovec/gather-scatter/scatter_store_64-3.c
@@ -1,5 +1,5 @@
/* { dg-do compile } */
-/* { dg-additional-options "-march=rv64gcv_zvfh -mabi=lp64d
-fdump-tree-vect-details" } */
+/* { dg-additional-options "-march=rv64gcv_zvfh -mabi=lp64d
-fdump-tree-vect-details -fno-vect-cost-model" } */
#include <stdint-gcc.h>
diff --git
a/gcc/testsuite/gcc.target/riscv/rvv/autovec/gather-scatter/scatter_store_64-5.c
b/gcc/testsuite/gcc.target/riscv/rvv/autovec/gather-scatter/scatter_store_64-5.c
index cc0b2fe15c4..c3e017a558e 100644
---
a/gcc/testsuite/gcc.target/riscv/rvv/autovec/gather-scatter/scatter_store_64-5.c
+++
b/gcc/testsuite/gcc.target/riscv/rvv/autovec/gather-scatter/scatter_store_64-5.c
@@ -1,5 +1,5 @@
/* { dg-do compile } */
-/* { dg-additional-options "-march=rv64gcv_zvfh -mabi=lp64d
-fdump-tree-vect-details" } */
+/* { dg-additional-options "-march=rv64gcv_zvfh -mabi=lp64d
-fdump-tree-vect-details -fno-vect-cost-model" } */
#include <stdint-gcc.h>
diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr112599-2.c
b/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr112599-2.c
index 79d87196bf7..967fe659961 100644
--- a/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr112599-2.c
+++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr112599-2.c
@@ -1,5 +1,5 @@
/* { dg-do compile } */
-/* { dg-options "-march=rv64gcv_zvl1024b -mabi=lp64d -O3" } */
+/* { dg-options "-march=rv64gcv_zvl1024b -mabi=lp64d -O3 -fno-vect-cost-model"
} */
struct s { struct s *n; } *p;
struct s ss;
diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr113607.c
b/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr113607.c
index 70a93665497..dc8b85a7a1c 100644
--- a/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr113607.c
+++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr113607.c
@@ -1,5 +1,5 @@
/* { dg-do compile } */
-/* { dg-options "-O3 -march=rv64gcv -mabi=lp64d -fdump-tree-optimized" } */
+/* { dg-options "-O3 -march=rv64gcv -mabi=lp64d -fdump-tree-optimized
-fno-vect-cost-model" } */
struct {
signed b;
diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr120378-1.c
b/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr120378-1.c
index 54f25e0ceaa..ba7103cdc5f 100644
--- a/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr120378-1.c
+++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr120378-1.c
@@ -1,5 +1,5 @@
/* { dg-do compile } */
-/* { dg-options "-march=rv64gcv -mabi=lp64d -O3 -fdump-tree-optimized" } */
+/* { dg-options "-march=rv64gcv -mabi=lp64d -O3 -fno-vect-cost-model
-fdump-tree-optimized" } */
#include <stdint.h>
diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr120378-2.c
b/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr120378-2.c
index 9a880e0f67b..66bcade1957 100644
--- a/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr120378-2.c
+++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr120378-2.c
@@ -1,5 +1,5 @@
/* { dg-do compile } */
-/* { dg-options "-march=rv64gcv -mabi=lp64d -O3" } */
+/* { dg-options "-march=rv64gcv -mabi=lp64d -O3 -fno-vect-cost-model" } */
#include <stdint.h>
diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr120378-3.c
b/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr120378-3.c
index d6b2ddd3f8d..c5b2b1c08ae 100644
--- a/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr120378-3.c
+++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr120378-3.c
@@ -1,5 +1,5 @@
/* { dg-do compile } */
-/* { dg-options "-march=rv64gcv -mabi=lp64d -O3" } */
+/* { dg-options "-march=rv64gcv -mabi=lp64d -O3 -fno-vect-cost-model" } */
#include <stdint.h>
diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr120378-4.c
b/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr120378-4.c
index 4657e52f883..fbf652ff866 100644
--- a/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr120378-4.c
+++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr120378-4.c
@@ -1,5 +1,5 @@
/* { dg-do compile } */
-/* { dg-options "-march=rv64gcv -mabi=lp64d -O3" } */
+/* { dg-options "-march=rv64gcv -mabi=lp64d -O3 -fno-vect-cost-model" } */
#include <stdint.h>
diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/vx_vf/vx-5-i64.c
b/gcc/testsuite/gcc.target/riscv/rvv/autovec/vx_vf/vx-5-i64.c
index f308b99a494..5c3db33b602 100644
--- a/gcc/testsuite/gcc.target/riscv/rvv/autovec/vx_vf/vx-5-i64.c
+++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/vx_vf/vx-5-i64.c
@@ -1,5 +1,5 @@
/* { dg-do compile } */
-/* { dg-options "-march=rv64gcv_zvl128b -mabi=lp64d --param=gpr2vr-cost=1
--param=vr2gpr-cost=1" } */
+/* { dg-options "-march=rv64gcv_zvl128b -mabi=lp64d --param=gpr2vr-cost=0
--param=vr2gpr-cost=0 " } */
#include "vx_binary.h"
diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/vx_vf/vx-5-u64.c
b/gcc/testsuite/gcc.target/riscv/rvv/autovec/vx_vf/vx-5-u64.c
index 2865e058fd4..ab6c4427f01 100644
--- a/gcc/testsuite/gcc.target/riscv/rvv/autovec/vx_vf/vx-5-u64.c
+++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/vx_vf/vx-5-u64.c
@@ -1,5 +1,5 @@
/* { dg-do compile } */
-/* { dg-options "-march=rv64gcv_zvl128b -mabi=lp64d --param=gpr2vr-cost=1
--param=vr2gpr-cost=1" } */
+/* { dg-options "-march=rv64gcv_zvl128b -mabi=lp64d --param=gpr2vr-cost=0
--param=vr2gpr-cost=0" } */
#include "vx_binary.h"
diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/vx_vf/vx-6-i64.c
b/gcc/testsuite/gcc.target/riscv/rvv/autovec/vx_vf/vx-6-i64.c
index 2dec50b1f76..609b4eb3227 100644
--- a/gcc/testsuite/gcc.target/riscv/rvv/autovec/vx_vf/vx-6-i64.c
+++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/vx_vf/vx-6-i64.c
@@ -1,5 +1,5 @@
/* { dg-do compile } */
-/* { dg-options "-march=rv64gcv_zvl128b -mabi=lp64d --param=gpr2vr-cost=2
--param=vr2gpr-cost=2" } */
+/* { dg-options "-march=rv64gcv_zvl128b -mabi=lp64d --param=gpr2vr-cost=1
--param=vr2gpr-cost=1" } */
#include "vx_binary.h"
@@ -23,21 +23,34 @@ DEF_VX_BINARY_CASE_3_WRAP(T, SAT_S_SUB_FUNC_WRAP(T),
sat_sub, VX_BINARY_FUNC_BOD
DEF_VX_BINARY_CASE_3_WRAP(T, AVG_FLOOR_FUNC_WRAP(T), avg_floor,
VX_BINARY_FUNC_BODY)
DEF_VX_BINARY_CASE_3_WRAP(T, AVG_CEIL_FUNC_WRAP(T), avg_ceil,
VX_BINARY_FUNC_BODY)
-/* { dg-final { scan-assembler-not {vadd.vx} } } */
-/* { dg-final { scan-assembler-not {vsub.vx} } } */
-/* { dg-final { scan-assembler-not {vrsub.vx} } } */
-/* { dg-final { scan-assembler-not {vand.vx} } } */
-/* { dg-final { scan-assembler-not {vor.vx} } } */
-/* { dg-final { scan-assembler-not {vxor.vx} } } */
-/* { dg-final { scan-assembler-not {vmul.vx} } } */
-/* { dg-final { scan-assembler-not {vdiv.vx} } } */
-/* { dg-final { scan-assembler-not {vrem.vx} } } */
-/* { dg-final { scan-assembler-not {vmax.vx} } } */
-/* { dg-final { scan-assembler-not {vmin.vx} } } */
-/* { dg-final { scan-assembler-not {vsadd.vx} } } */
-/* { dg-final { scan-assembler-not {vssub.vx} } } */
-/* { dg-final { scan-assembler {vaadd.vx} { target { any-opts {
+/* { dg-final { scan-assembler {vadd.vx} } } */
+/* { dg-final { scan-assembler {vsub.vx} } } */
+/* { dg-final { scan-assembler {vrsub.vx} } } */
+/* { dg-final { scan-assembler {vand.vx} } } */
+/* { dg-final { scan-assembler {vor.vx} } } */
+/* { dg-final { scan-assembler {vxor.vx} } } */
+/* { dg-final { scan-assembler {vmul.vx} } } */
+/* { dg-final { scan-assembler {vdiv.vx} } } */
+/* { dg-final { scan-assembler {vrem.vx} } } */
+/* { dg-final { scan-assembler {vmax.vx} } } */
+/* { dg-final { scan-assembler {vmin.vx} } } */
+/* { dg-final { scan-assembler {vsadd.vx} { target { no-opts {
"-mrvv-vector-bits=zvl -mrvv-max-lmul=m1"
"-mrvv-vector-bits=zvl -mrvv-max-lmul=m2"
"-mrvv-vector-bits=zvl -mrvv-max-lmul=m4"
+ "-mrvv-vector-bits=zvl -mrvv-max-lmul=m8"
+ "-mrvv-vector-bits=zvl -mrvv-max-lmul=dynamic"
+ } } } } } */
+/* { dg-final { scan-assembler {vssub.vx} { target { no-opts {
+ "-mrvv-vector-bits=zvl -mrvv-max-lmul=m1"
+ "-mrvv-vector-bits=zvl -mrvv-max-lmul=m2"
+ "-mrvv-vector-bits=zvl -mrvv-max-lmul=m4"
+ "-mrvv-vector-bits=zvl -mrvv-max-lmul=m8"
+ "-mrvv-vector-bits=zvl -mrvv-max-lmul=dynamic"
+ } } } } } */
+/* { dg-final { scan-assembler {vaadd.vx} { target { no-opts {
+ "-mrvv-max-lmul=m1"
+ "-mrvv-max-lmul=m2"
+ "-mrvv-max-lmul=m8"
+ "-mrvv-max-lmul=dynamic"
} } } } } */
diff --git a/gcc/testsuite/gcc.target/riscv/rvv/vsetvl/pr112988-1.c
b/gcc/testsuite/gcc.target/riscv/rvv/vsetvl/pr112988-1.c
index 1facfd55d79..e151324f5b0 100644
--- a/gcc/testsuite/gcc.target/riscv/rvv/vsetvl/pr112988-1.c
+++ b/gcc/testsuite/gcc.target/riscv/rvv/vsetvl/pr112988-1.c
@@ -1,5 +1,5 @@
/* { dg-do compile } */
-/* { dg-options "-mrvv-vector-bits=scalable -march=rv64gcv -mabi=lp64d -O3" }
*/
+/* { dg-options "-mrvv-vector-bits=scalable -march=rv64gcv -mabi=lp64d -O3
-fno-vect-cost-model" } */
int a = 0;
int p, q, r, x = 230;
--
2.43.0
