https://gcc.gnu.org/bugzilla/show_bug.cgi?id=125750
Tamar Christina <tnfchris at gcc dot gnu.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Version|16.0 |17.0
Summary|Excessive vector unrolling |multi-lane SLP fails
|leads to excessive spilling |forcing bad Single lane SLP
| |vectorization.
--- Comment #7 from Tamar Christina <tnfchris at gcc dot gnu.org> ---
So there are three cases here. These testcases can all be compiled with -Ofast
-march=armv8-a+sve2.
I don't think that clang avoided the unrolling, it's just that it just didn't
vectorize the loops at all.
GCC also vectorizes the same part using normal linear loads, but we vectorized
the other loops with SLP=1 lanes.
So that's a costing issue on our end. The costing falls apart I think because
it doesn't cost the throughput
limitations of all LANE=1 instances together. So it doesn't realize it's
choking the predicate bandwidth.
However there are a couple of other things to get from this that clang also
doesn't do.
All of these are gated on the basic problem though in that we can't build the
full SLP
tree because the operations differ between nodes.
the part of the code that throws off SLP in both compilers is this
int m0 = ((i < n) & (r[i] == 0)) ? -1 : 0;
int m1 = ((i + 1 < n) & (r[i + 1] == 0)) ? -1 : 0;
int m2 = ((i + 2 < n) & (r[i + 2] == 0)) ? -1 : 0;
int m3 = ((i + 3 < n) & (r[i + 3] == 0)) ? -1 : 0;
because m0 does `i < n` whereas the other operations have a +.
semantically though we can make the SLP tree if we does
int m0 = ((i + 0 < n) & (r[i + 0] == 0)) ? -1 : 0;
int m1 = ((i + 1 < n) & (r[i + 1] == 0)) ? -1 : 0;
int m2 = ((i + 2 < n) & (r[i + 2] == 0)) ? -1 : 0;
int m3 = ((i + 3 < n) & (r[i + 3] == 0)) ? -1 : 0;
which makes the SLP lanes match. Me and Richi talked about this two cauldrons
ago and going SLP only was partially to try to do this in a way that doesn't
require as much backtracking.
If we are able to do that and see all the lanes then these improvements become
possible and should give a much better result than clang:
--- unpredicated main SVE loop
typedef unsigned char uint8_t;
void __attribute__((noinline, noclone))
f (uint8_t *r, float *x, float *y, float *z, int n, float out[3])
{
const int sign = (int) 0x80000000u;
float ax0 = 0, ax1 = 0, ax2 = 0, ax3 = 0;
float ay0 = 0, ay1 = 0, ay2 = 0, ay3 = 0;
float az0 = 0, az1 = 0, az2 = 0, az3 = 0;
for (int i = 0; i < n; i += 4)
{
int m0 = ((i < n) & (r[i] == 0)) ? -1 : 0;
int m1 = ((i + 1 < n) & (r[i + 1] == 0)) ? -1 : 0;
int m2 = ((i + 2 < n) & (r[i + 2] == 0)) ? -1 : 0;
int m3 = ((i + 3 < n) & (r[i + 3] == 0)) ? -1 : 0;
float x0 = x[i], x1 = x[i + 1], x2 = x[i + 2], x3 = x[i + 3];
float y0 = y[i], y1 = y[i + 1], y2 = y[i + 2], y3 = y[i + 3];
float z0 = z[i], z1 = z[i + 1], z2 = z[i + 2], z3 = z[i + 3];
ax0 += (m0 & sign) ? x0 : 0;
ax1 += (m1 & sign) ? x1 : 0;
ax2 += (m2 & sign) ? x2 : 0;
ax3 += (m3 & sign) ? x3 : 0;
ay0 += (m0 & sign) ? y0 : 0;
ay1 += (m1 & sign) ? y1 : 0;
ay2 += (m2 & sign) ? y2 : 0;
ay3 += (m3 & sign) ? y3 : 0;
az0 += (m0 & sign) ? z0 : 0;
az1 += (m1 & sign) ? z1 : 0;
az2 += (m2 & sign) ? z2 : 0;
az3 += (m3 & sign) ? z3 : 0;
}
out[0] = (ax0 + ax2) + (ax1 + ax3);
out[1] = (ay0 + ay2) + (ay1 + ay3);
out[2] = (az0 + az2) + (az1 + az3);
}
https://godbolt.org/z/MaTofrP9o
This example has three concurrent live out values, both GCC and clang generate
the same amount of LD4Ws however the Clang version is faster because the main
loop was unpredicated.
This means they don't have the 4 concurrent whilelo for every loop iteration.
Clang generates an unpredicated main SVE loop, Adv. SIMD epilog followed by
scalar.
we can probably do the same. However don't think we need the permutes here so
we can probably generate much better code without needing the epilog by using
SVE2p1's predicate as a counter and linear group loads. Since we don't
actually have or want a permute here.
--- live reduction ordering
this example
typedef unsigned char uint8_t;
void __attribute__((noinline, noclone))
f (uint8_t *r, float *x, int n, float *out)
{
const int sign = (int) 0x80000000u;
float a0 = 0;
float a1 = 0;
float a2 = 0;
float a3 = 0;
for (int i = 0; i < n; i += 4)
{
int m0 = ((i < n) & (r[i] == 0)) ? -1 : 0;
int m1 = ((i + 1 < n) & (r[i + 1] == 0)) ? -1 : 0;
int m2 = ((i + 2 < n) & (r[i + 2] == 0)) ? -1 : 0;
int m3 = ((i + 3 < n) & (r[i + 3] == 0)) ? -1 : 0;
float x0 = x[i];
float x1 = x[i + 1];
float x2 = x[i + 2];
float x3 = x[i + 3];
a0 += (m0 & sign) ? x0 : 0;
a1 += (m1 & sign) ? x1 : 0;
a2 += (m2 & sign) ? x2 : 0;
a3 += (m3 & sign) ? x3 : 0;
}
*out = (a0 + a2) + (a1 + a3);
}
https://godbolt.org/z/daans1f39
shows that both clang and gcc miss that the reduction values are live but used
in a way in which their orders are not important.
and as such the compiler is not forced to codegen the lane orders. i.e. the
above should be one LD1W not 4 LD4W.
--- over permute
typedef unsigned char uint8_t;
void __attribute__((noinline, noclone))
f (uint8_t *r, float *x, int n, float out[4])
{
const int sign = (int) 0x80000000u;
float a0 = 0;
float a1 = 0;
float a2 = 0;
float a3 = 0;
for (int i = 0; i < n; i += 4)
{
int m0 = ((i < n) & (r[i] == 0)) ? -1 : 0;
int m1 = ((i + 1 < n) & (r[i + 1] == 0)) ? -1 : 0;
int m2 = ((i + 2 < n) & (r[i + 2] == 0)) ? -1 : 0;
int m3 = ((i + 3 < n) & (r[i + 3] == 0)) ? -1 : 0;
float x0 = x[i];
float x1 = x[i + 1];
float x2 = x[i + 2];
float x3 = x[i + 3];
a0 += (m0 & sign) ? x0 : 0;
a1 += (m1 & sign) ? x1 : 0;
a2 += (m2 & sign) ? x2 : 0;
a3 += (m3 & sign) ? x3 : 0;
}
out[0] = a0;
out[1] = a1;
out[2] = a2;
out[3] = a3;
}
https://godbolt.org/z/6PcEaYe4Y
This is a case where the lanes are required, but we only have one stream here.
But the 4 lane SLP reduction fails
: missed: not vectorized: relevant stmt not supported: patt_178 =
(<signed-boolean:8>) _19;
: note: unsupported SLP instance starting from: patt_195 = .COND_ADD
(patt_194, x3_58, a3_80, a3_80);
: missed: unsupported SLP instances
: note: re-trying with single-lane SLP
and so we fall back to single lane SLP which creates the overunrolling effect.
This is failing because it looks like it's trying to truncate the mask of a
32-bit element to 8-bits
to perform the & in
int m0 = ((i < n) & (r[i] == 0)) ? -1 : 0;
as a boolean, these have the same precision
_71 = _72 < n_46(D);
_68 = *_69;
_67 = _68 == 0;
_66 = _71 & 67;
as a vector they don't. we would have been OK had we picked for the r the
partial vector VNx4QI.
using this we can avoid the nasty predicate unpacks.
