| Issue |
240667
|
| Summary |
[X86] AVX-VNNI: dot-product reduction folded to a single vpdpbusd drops 3 of the 4 lane sums
|
| Labels |
new issue
|
| Assignees |
|
| Reporter |
20000419
|
A 16-element i8 x u8 dot product written as a halving reduction is compiled into one `vpdpbusd`, and only lane 0 of the result is returned.
```c
#include <stdio.h>
typedef signed char s8x16 __attribute__((vector_size(16)));
typedef unsigned char u8x16 __attribute__((vector_size(16)));
typedef int i32x16 __attribute__((vector_size(64)));
typedef int i32x8 __attribute__((vector_size(32)));
typedef int i32x4 __attribute__((vector_size(16)));
typedef int i32x2 __attribute__((vector_size(8)));
typedef int i32x1 __attribute__((vector_size(4)));
__attribute__((noinline)) int dp16(s8x16 a,u8x16 b){
i32x16 m=__builtin_convertvector(a,i32x16)*__builtin_convertvector(b,i32x16);
i32x8 s8=__builtin_shufflevector(m,m,0,1,2,3,4,5,6,7)+__builtin_shufflevector(m,m,8,9,10,11,12,13,14,15);
i32x4 s4=__builtin_shufflevector(s8,s8,0,1,2,3)+__builtin_shufflevector(s8,s8,4,5,6,7);
i32x2 s2=__builtin_shufflevector(s4,s4,0,1)+__builtin_shufflevector(s4,s4,2,3);
i32x1 s1=__builtin_shufflevector(s2,s2,0)+__builtin_shufflevector(s2,s2,1); return s1[0];
}
int main(void){ s8x16 a; u8x16 b; for(int i=0;i<16;i++){a[i]=i;b[i]=i+1;} a[0]=(signed char)200;
int r=dp16(a,b),e=0; for(int i=0;i<16;i++) e+=a[i]*b[i]; printf("got=%d want=%d %s\n",r,e,r==e?"OK":"BAD"); return r!=e;}
```
`clang -mavxvnni -O2 t.c && ./a.out` (also `-march=sapphirerapids`) prints `got=-36 want=1304 BAD`. `-O0` and GCC print 1304.
```asm
dp16:
vpxor %xmm2, %xmm2, %xmm2
{vex} vpdpbusd %xmm0, %xmm1, %xmm2
vmovd %xmm2, %eax
retq
```
Godbolt: https://godbolt.org/z/9hv77T775 (the Godbolt runners have no AVX-VNNI, so only the `-O2` assembly and the `-O0` run are shown)
`vpdpbusd` gives four i32 lanes, each the sum of four adjacent byte products. The full reduction needs those four lanes added together, but the code returns lane 0 alone, which is -56*1 + 1*2 + 2*3 + 3*4 = -36.
The bug is in `combineVPDPBUSDPattern` (`llvm/lib/Target/X86/X86ISelLowering.cpp`). `DAG.matchBinOpReduction` also matches a reduction that narrows the vector with `extract_subvector` at each step, as in this source. Afterwards, the number of extra stages to emit is computed from the type of the extract's operand, not from the width of the multiply that was matched:
```cpp
EVT VT = Extract->getOperand(0).getValueType();
...
unsigned Stages = Log2_32(VT.getVectorNumElements());
...
if (Stages > StageBias) { // add the remaining lanes of DP
```
In this case `VT` is the last, narrowed vector (one or two elements), so `Stages <= StageBias` and none of the adds after `vpdpbusd` are emitted, even though `Root` (the `mul`) is `v16i32`. If the same reduction is written without narrowing (each step a 16-element `__builtin_shufflevector` with `-1` padding), the combine emits the extra shuffle/add stages and the result is correct (1304).
Suggested fix (not tested): compute `Stages` from `Root.getValueType().getVectorNumElements()` (the width of the matched multiply) instead of `VT`.
Versions: wrong at `-O1` and above with clang 15.0.6, 16.0.4, 17.0.6, 19, 20, 21, 22, 23 and trunk (24.0.0, 7bdabc8be443); clang 18.1.0 emits the same single `vpdpbusd` sequence. `-O0` is correct; GCC is correct.
See also #233803 (a missed VPDPBUSD match, not wrong code).
Disclaimer: This test case was found with LLM-assisted targeted testing, and the report was written with LLM assistance.
Assisted-by: Grok
_______________________________________________
llvm-bugs mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-bugs