From: Kyrylo Tkachov <[email protected]>
The documentation of the reduc_widen_ssum and reduc_widen_usum patterns says
only that operand 1 is added to operand 2, which reads as though each element
of operand 1 has to be accumulated into the element of operand 0 at the same
position. Nothing says the patterns are only ever used for a reassociable
reduction, so a port cannot tell whether it is allowed to regroup the input
elements, for example by using a pairwise widening add.
vect_recog_widen_sum_pattern only forms WIDEN_SUM_EXPR
through vect_reassociating_reduction_p, which requires the statement to be a
reduction, refuses a statement nested in the inner loop of an outer-loop
vectorization because the order of the computation matters there, and refuses
a type that needs a fold-left reduction. The comment on the pattern already
describes the idiom as producing N/2 results by summing up pairs of
intermediate results. The accumulator is only ever consumed by a horizontal
sum in the epilogue, so the distribution of input elements over accumulator
elements is not observable.
Spell that out in md.texi, along with the constraint that comes with it: an
implementation that adds elements together before they reach the element width
of operand 0 has to widen them first so that no intermediate sum can overflow.
Note that the widen_[us]sum names are left free for a lane preserving pattern.
Give WIDEN_SUM_EXPR the same statement in tree.def. DOT_PROD_EXPR and SAD_EXPR
are formed through vect_reassociating_reduction_p as well and read the same way,
so add a sentence to each pointing at the WIDEN_SUM_EXPR rule.
Ok for trunk?
Thanks,
Kyrill
gcc/ChangeLog:
* doc/md.texi (reduc_widen_ssum@var{n}@var{m}3)
(reduc_widen_usum@var{n}@var{m}3): Document that the assignment of
input elements to accumulator elements is unconstrained, that
intermediate sums must not overflow, and that the widen_[us]sum
names are reserved. Fix a typo.
* tree.def (WIDEN_SUM_EXPR): Document that the assignment of elements
of the first argument to elements of the second is unconstrained.
(DOT_PROD_EXPR, SAD_EXPR): Note the same freedom.
Signed-off-by: Kyrylo Tkachov <[email protected]>
---
gcc/doc/md.texi | 23 +++++++++++++++++++++--
gcc/tree.def | 19 +++++++++++++++++--
2 files changed, 38 insertions(+), 4 deletions(-)
diff --git a/gcc/doc/md.texi b/gcc/doc/md.texi
index 4b3dc950aeb..f86fa8038e8 100644
--- a/gcc/doc/md.texi
+++ b/gcc/doc/md.texi
@@ -5097,10 +5097,29 @@ in operand 0, which is of the same mode as operand 3.
@itemx @samp{reduc_widen_usum@var{n}@var{m}3}
Operands 0 and 2 are of the same mode, which is wider than the mode of
operand 1. Add operand 1 to operand 2 and place the widened result in
-operand 0. (This is used express accumulation of elements into an accumulator
-of a wider mode.)
+operand 0. (This is used to express accumulation of elements into an
+accumulator of a wider mode.)
@var{m} is the mode of operand 1 and @var{n} is the mode of operand 0.
+These patterns are only used for a reduction whose summation the vectorizer
+has already established may be reassociated. Operand 0 is the accumulator
+of that reduction and is itself reduced to a scalar by a horizontal sum once
+the loop is done, so which element of the accumulator a given input element
+ends up in is not observable. An implementation is therefore free to choose
+which elements of operand 1 it accumulates into which element of operand 0,
+provided that every element of operand 1 is accumulated exactly once. In
+particular it may add adjacent elements of operand 1 to each other before
+accumulating them, which is what a pairwise widening add instruction does.
+
+The intermediate sums must not overflow. An implementation that combines
+elements of operand 1 with each other before they reach the element width of
+operand 0 therefore has to widen them first.
+
+The names @samp{widen_ssum@var{n}@var{m}3} and
+@samp{widen_usum@var{n}@var{m}3} are deliberately left free for a future
+lane preserving pattern, which would have to add element @var{i} of
+operand 1 to element @var{i} of operand 2 for every @var{i}.
+
@mdindex smulhs@var{m}3
@mdindex umulhs@var{m}3
@item @samp{smulhs@var{m}3}
diff --git a/gcc/tree.def b/gcc/tree.def
index ad8e676e942..2e767dbe045 100644
--- a/gcc/tree.def
+++ b/gcc/tree.def
@@ -1465,7 +1465,10 @@ DEFTREECODE (TRANSACTION_EXPR, "transaction_expr",
tcc_expression, 1)
arg3 = PLUS_EXPR (tmp, arg3);
or:
tmp = WIDEN_MULT_EXPR(arg1, arg2);
- arg3 = WIDEN_SUM_EXPR (tmp, arg3); */
+ arg3 = WIDEN_SUM_EXPR (tmp, arg3);
+ Like WIDEN_SUM_EXPR, this code is only created for a reduction whose
+ summation may be reassociated, so which products are added into which
+ element of arg3 is not observable. */
DEFTREECODE (DOT_PROD_EXPR, "dot_prod_expr", tcc_expression, 3)
/* Widening summation.
@@ -1474,7 +1477,16 @@ DEFTREECODE (DOT_PROD_EXPR, "dot_prod_expr",
tcc_expression, 3)
the size of t1. The type of the entire expression is also t2.
WIDEN_SUM_EXPR is equivalent to first widening (promoting)
the first argument from type t1 to type t2, and then summing it
- with the second argument. */
+ with the second argument.
+
+ This code is only created for a reduction whose summation the vectorizer
+ has established may be reassociated, and the result is reduced to a scalar
+ by a horizontal sum once the loop is done. Which element of the second
+ argument a given element of the first argument is added to is therefore
+ not observable, and an expander may choose that assignment freely, so long
+ as every element of the first argument is accumulated exactly once and no
+ intermediate sum overflows. See the reduc_widen_ssum and reduc_widen_usum
+ patterns in md.texi. */
DEFTREECODE (WIDEN_SUM_EXPR, "widen_sum_expr", tcc_binary, 2)
/* Widening sad (sum of absolute differences).
@@ -1490,6 +1502,9 @@ DEFTREECODE (WIDEN_SUM_EXPR, "widen_sum_expr",
tcc_binary, 2)
tmp = IFN_VEC_WIDEN_MINUS_EXPR (arg1, arg2)
tmp2 = ABS_EXPR (tmp)
arg3 = WIDEN_SUM_EXPR (tmp2, arg3)
+ Like WIDEN_SUM_EXPR, this code is only created for a reduction whose
+ summation may be reassociated, so which absolute differences are added
+ into which element of arg3 is not observable.
*/
DEFTREECODE (SAD_EXPR, "sad_expr", tcc_expression, 3)
--
2.50.1 (Apple Git-155)
