https://gcc.gnu.org/bugzilla/show_bug.cgi?id=126429
Bug ID: 126429
Summary: [i386] _mm512_{mask,maskz}_dpbf16_ps only honor the
low 8 bits of the __mmask16 (kmovb instead of kmovw)
Product: gcc
Version: 16.1.0
Status: UNCONFIRMED
Keywords: wrong-code
Severity: normal
Priority: P3
Component: target
Assignee: unassigned at gcc dot gnu.org
Reporter: michael.crusoe at gmail dot com
Target Milestone: ---
Target: x86_64--
The below report is written by https://github.com/fo40225 (from
https://github.com/simd-everywhere/simde/pull/1419#issuecomment-5078585078 )
and I affirm that their workaround fixes a real problem I've seen as the SIMDe
maintainer.
---
The AVX512-BF16 masked dot-product intrinsics _mm512_mask_dpbf16_ps and
_mm512_maskz_dpbf16_ps take a __mmask16 -- the zmm destination holds 16
float32 lanes -- but the writemask operand of the underlying insn
patterns is QImode, so the upper 8 mask bits are dropped. Lanes 8-15
always behave as if their mask bit were clear: the merge form copies src
through and the zeroing form zeroes the lane, whatever the mask says.
The visible symptom is a kmovb where kmovw is required.
*** Please enable AVX512DQ when reproducing. ***
kmovb is an AVX512DQ instruction, so the QImode truncation only becomes
architecturally observable once AVX512DQ is on. Under a bare
-mavx512bf16 the QImode mask move is lowered to kmovw, which transfers
all 16 bits and hides the defect. All ten -march= values that imply
AVX512BF16 in GCC 16.1 also imply AVX512DQ -- cooperlake,
sapphirerapids, emeraldrapids, graniterapids, graniterapids-d,
diamondrapids, novalake, znver4, znver5, znver6 -- and every one of them
produces the kmovb. The wrong code is therefore what real builds get;
-mavx512bf16 on its own is the only configuration that escapes it.
This is a silent wrong-code bug present in every GCC release with
AVX512BF16 support. Reproduced on the official docker images 10.5.0,
11.5.0, 12.5.0, 13.4.0, 14.4.0, 15.3.0 and 16.1.0; on 16.1.0 the kmovb
comes out for all ten -march= values above at each of -O0, -O1, -O2, -O3
and -Os. The three operands responsible are unchanged on trunk
(f6b00aefc25a). Clang emits kmovd and is correct (checked 18.1.3 and
20.1.2).
It is not latent in practice: GitHub Actions' ubuntu-24.04 runner fleet
now contains AVX512-BF16-capable machines (AMD EPYC 9V74 / Zen4, Intel
Emerald Rapids 8573C, Granite Rapids 6973P-C), so any -march=native
build that lands on one of them silently computes wrong results. It was
found through intermittent CI failures in SIMDe.
https://github.com/simd-everywhere/simde/issues/902
https://github.com/simd-everywhere/simde/issues/988
https://github.com/simd-everywhere/simde/issues/1095
https://github.com/simd-everywhere/simde/issues/1201
https://github.com/simd-everywhere/simde/pull/1276
=== Reproducer (compile only) ===
$ cat asm.c
#include <immintrin.h>
__m512 f_mask(__m512 src, __mmask16 k, __m512bh a, __m512bh b) {
return _mm512_mask_dpbf16_ps(src, k, a, b);
}
__m512 f_maskz(__mmask16 k, __m512 src, __m512bh a, __m512bh b) {
return _mm512_maskz_dpbf16_ps(k, src, a, b);
}
$ gcc-16 -O2 -mavx512bf16 -mavx512dq -S -o - asm.c
f_mask:
kmovb %edi, %k1 <-- should be kmovw
vdpbf16ps %zmm2, %zmm1, %zmm0{%k1}
ret
f_maskz:
kmovb %edi, %k1 <-- should be kmovw
vdpbf16ps %zmm2, %zmm1, %zmm0{%k1}{z}
ret
-march=sapphirerapids / znver4 / cooperlake / ... give the same kmovb.
clang emits kmovd for all of them.
=== Reproducer (runtime, wrong-code) ===
$ cat runtime.c
#include <immintrin.h>
#include <stdio.h>
#include <string.h>
volatile unsigned short k_mask = 0xFFFF; /* opaque so the mask is not folded
*/
volatile unsigned short k_maskz = 0xFF00;
int main(void) {
float srcf[16], af[16], bf[16], r[16];
for (int i = 0; i < 16; i++) { srcf[i] = 1000.0f + i; af[i] = 2.0f; bf[i] =
3.0f; }
/* each f32 lane of a/b holds bf16 pair {lo=0, hi=2.0/3.0} -> dot adds 6.0 */
__m512 src = _mm512_loadu_ps(srcf);
__m512bh a, b;
memcpy(&a, af, 64); memcpy(&b, bf, 64);
int bad = 0;
_mm512_storeu_ps(r, _mm512_mask_dpbf16_ps(src, (__mmask16)k_mask, a, b));
for (int i = 8; i < 16; i++) /* k=0xFFFF: expect src+6 everywhere */
if (r[i] != srcf[i] + 6.0f) { bad++; printf("mask lane %2d: got %g want
%g\n", i, r[i], srcf[i]+6.0f); }
_mm512_storeu_ps(r, _mm512_maskz_dpbf16_ps((__mmask16)k_maskz, src, a, b));
for (int i = 8; i < 16; i++) /* k=0xFF00: expect src+6 in lanes 8-15 */
if (r[i] != srcf[i] + 6.0f) { bad++; printf("maskz lane %2d: got %g want
%g\n", i, r[i], srcf[i]+6.0f); }
printf(bad ? "FAIL: %d wrong lanes\n" : "OK\n", bad);
return bad != 0;
}
$ gcc-16 -O2 -march=native runtime.c && ./a.out # on avx512_bf16 hardware
mask lane 8: got 1008 want 1014 (src passthrough: mask bit read as 0)
...
maskz lane 8: got 0 want 1014 (zeroed: mask bit read as 0)
...
FAIL: 16 wrong lanes
All 16 lanes are wrong under Intel SDE 10.8.0 -gnr, and identically so
for -march=graniterapids, and on an AMD EPYC 9474F (Zen4) running -march=znver4
on real silicon. The same binary built with a bare -mavx512bf16 (the kmovw
form)
prints OK, so the defect is in the generated code, not in any CPU or emulator.
=== Root cause ===
(Line numbers from trunk, commit f6b00aefc25a.)
All three masked dpbf16 patterns carry the writemask in the half-width
mask mode <avx512fmaskhalfmode>:
1. gcc/config/i386/sse.md:32651, define_insn
"avx512f_dpbf16ps_<mode>_mask":
(vec_merge:VF1_AVX512VL
(unspec:VF1_AVX512VL [...] UNSPEC_VDPBF16PS)
(match_dup 1)
(match_operand:<avx512fmaskhalfmode> 4 "register_operand" "Yk"))
2. gcc/config/i386/sse.md:32628, define_expand
"avx512f_dpbf16ps_<mode>_maskz": operand 4 is
(match_operand:<avx512fmaskhalfmode> 4 "register_operand").
3. gcc/config/i386/sse.md:32641, define_insn
"avx512f_dpbf16ps_<mode><maskz_half_name>": the maskz variant is
generated by the dedicated define_subst "maskz_half"
(gcc/config/i386/subst.md:473-484), whose mask operand is likewise
(match_operand:<avx512fmaskhalfmode> 3 "register_operand" "Yk").
For V16SF <avx512fmaskhalfmode> is QImode, so only 8 mask bits survive.
The writemask governs the 16 float32 *output* lanes, so it must be
<avx512fmaskmode> (HImode for V16SF); "<avx512fmaskhalfmode>" appears to
have been chosen because the BF16 *inputs* have 32 elements, but
per-input-element masking does not exist for this instruction.
The builtin prototypes in i386-builtin.def already use UHI correctly
(i386-builtin.def:2874-2875, V16SF_FTYPE_V16SF_V32BF_V32BF_UHI); only
the insn operand mode is wrong. Comparing the two mode attributes:
avx512fmaskmode (sse.md:1087): V16SF -> HI, V8SF -> QI, V4SF -> QI
avx512fmaskhalfmode (sse.md:1113): V16SF -> QI, V8SF -> QI, V4SF -> QI
For V4SF/V8SF the two attributes coincide (QImode covers the 4/8 output
lanes), so the 128/256-bit variants are accidentally correct; only the
V16SF (512-bit) case misbehaves.
Apart from its definition, those three operands are the only uses of
<avx512fmaskhalfmode> anywhere in the i386 backend's machine
descriptions, and dpbf16 is the only user of the "maskz_half" subst.
Suggested fix: switch the writemask operands to <avx512fmaskmode> in all
three places -- the _mask insn, the _maskz expander, and the "maskz_half"
subst (which is dpbf16-only, so it can be corrected in place or the
pattern moved to the standard "maskz" subst machinery) -- plus a
wrong-code testcase like the runtime reproducer above. Once that is
done, both the "maskz_half" subst and the "avx512fmaskhalfmode" mode
attribute become entirely unused and can be removed.
