| Issue |
244132
|
| Summary |
[AMDGPU] Add `__bf16` vector variants of the bf16 `_1k` MFMA builtins
|
| Labels |
new issue
|
| Assignees |
|
| Reporter |
yingluAMD
|
The bf16 `_1k` MFMA builtins and intrinsics take their A/B operands as `short4` / `<4 x i16>`. To use them with real `__bf16` data, callers have to bitcast to `short4`. The newer gfx950 bf16 builtins (`mfma_f32_16x16x32_bf16`, `mfma_f32_32x32x16_bf16`, `smfmac_f32_*_bf16`) take `__bf16` vectors directly.
When the operand is produced by an `fptrunc` in registers, this mandatory bitcast also makes gfx950 codegen worse. The packed `v_cvt_pk_bf16_f32` gets split into one conversion per element, and `v_perm_b32` repacks the halves.
Request: add variants of these builtins and intrinsics that take `__bf16 x4` / `<4 x bfloat>` operands and select to the same instructions.
## Reproducer
```llvm
; RUN: llc -O3 -mtriple=amdgpu9.50-amd-amdhsa < %s
define <4 x float> @f(<4 x float> %a, <4 x float> %b, <4 x float> %c) {
%x = fptrunc <4 x float> %a to <4 x bfloat>
%y = fptrunc <4 x float> %b to <4 x bfloat>
%xi = bitcast <4 x bfloat> %x to <4 x i16>
%yi = bitcast <4 x bfloat> %y to <4 x i16>
%r = call <4 x float> @llvm.amdgcn.mfma.f32.16x16x16bf16.1k(<4 x i16> %xi, <4 x i16> %yi, <4 x float> %c, i32 0, i32 0, i32 0)
ret <4 x float> %r
}
```
Output:
```asm
v_cvt_pk_bf16_f32 v0, v0, s0
v_cvt_pk_bf16_f32 v12, v1, s0
v_cvt_pk_bf16_f32 v1, v2, s0
v_cvt_pk_bf16_f32 v2, v3, s0
v_cvt_pk_bf16_f32 v4, v4, s0
v_cvt_pk_bf16_f32 v5, v5, s0
s_mov_b32 s0, 0x5040100
v_perm_b32 v1, v2, v1, s0
v_perm_b32 v0, v12, v0, s0
v_cvt_pk_bf16_f32 v2, v6, s0
v_cvt_pk_bf16_f32 v3, v7, s0
v_perm_b32 v3, v3, v2, s0
v_perm_b32 v2, v5, v4, s0
s_nop 1
v_mfma_f32_16x16x16_bf16 v[0:3], v[0:1], v[2:3], v[8:11]
```
Expected: four packed conversions and no `v_perm_b32`. For example, `v_cvt_pk_bf16_f32 v0, v0, v1`.
The same happens from HIP:
```cpp
typedef float v4f __attribute__((ext_vector_type(4)));
typedef __bf16 v4bf __attribute__((ext_vector_type(4)));
typedef short v4i16 __attribute__((ext_vector_type(4)));
__attribute__((global)) void k(const v4f *a, const v4f *b, v4f *c) {
int i = __builtin_amdgcn_workitem_id_x();
v4bf x = __builtin_convertvector(a[i], v4bf);
v4bf y = __builtin_convertvector(b[i], v4bf);
// The builtin only accepts short4, so the bf16 values must be bitcast.
c[i] = __builtin_amdgcn_mfma_f32_16x16x16bf16_1k(
__builtin_bit_cast(v4i16, x), __builtin_bit_cast(v4i16, y), c[i], 0, 0, 0);
}
```
```
clang -x hip --offload-arch=gfx950 --cuda-device-only -nogpulib -nogpuinc -O3 -S repro.hip -o -
```
This gives 8 `v_cvt_pk_bf16_f32` and 4 `v_perm_b32`. Without the bitcast it would be 4 and 0.
## Details
Affected builtins (`clang/include/clang/Basic/BuiltinsAMDGPU.td`, all gated on `gfx90a-insts`):
- `__builtin_amdgcn_mfma_f32_32x32x4bf16_1k`
- `__builtin_amdgcn_mfma_f32_16x16x4bf16_1k`
- `__builtin_amdgcn_mfma_f32_4x4x4bf16_1k`
- `__builtin_amdgcn_mfma_f32_32x32x8bf16_1k`
- `__builtin_amdgcn_mfma_f32_16x16x16bf16_1k`
The cause of the codegen difference is the `ISD::BITCAST` combine in `AMDGPUISelLowering.cpp`. It pushes the same-element-count bitcast into the `BUILD_VECTOR` produced by expanding `FP_ROUND v4bf16`. The elements become `i16`, so the `v2bf16` `V_CVT_PK_BF16_F32` pattern no longer matches.
That combine could be fixed on its own. Even then, the `short4` signature keeps the bitcast in every caller, and it loses type checking: fp16 or integer data passed by mistake is not diagnosed.
Targets:
| target | `_1k` builtins | hardware bf16 cvt | effect of `__bf16` operands |
|---|---|---|---|
| gfx908 | no | no | — |
| gfx90a, gfx942, gfx9-4-generic | yes | no | type checking only; codegen is identical (software rounding) |
| gfx950 | yes | yes | type checking, and 8 cvt + 4 perm → 4 cvt + 0 perm per MFMA |
| gfx12+ | no | — | — |
Today, the workaround is to convert pairs to `<2 x bfloat>`, bitcast each pair to `i32`, and build the operand from `i32` words.
## Possible approach
Add new builtins and intrinsics with `<4 x bfloat>` operands, and keep the existing ones unchanged. The names could follow the gfx940+ assembly mnemonics, as other gfx940+ MFMA intrinsics do:
| existing (kept) | new, `__bf16 x4` operands |
|---|---|
| `mfma_f32_16x16x16bf16_1k` | `mfma_f32_16x16x16_bf16` |
| `mfma_f32_4x4x4bf16_1k` | `mfma_f32_4x4x4_16b_bf16` |
| `mfma_f32_32x32x8bf16_1k` | `mfma_f32_32x32x8_bf16` |
| `mfma_f32_16x16x4bf16_1k` | `mfma_f32_16x16x4_4b_bf16` |
| `mfma_f32_32x32x4bf16_1k` | `mfma_f32_32x32x4_2b_bf16` |
The `_2b`/`_4b`/`_16b` suffixes keep the names distinct from the gfx908 `mfma_f32_32x32x4bf16`.
With new names, existing source, IR, and bitcode keep working without AutoUpgrade, and the new builtins can be detected with `__has_builtin`.
Overloading the existing intrinsics instead would change their mangled names and require AutoUpgrade. It would also leave `__has_builtin` unable to tell whether `__bf16` operands are accepted.
## Version
- IR: `llc` built from llvm/llvm-project main at `05b37c72e4`.
- HIP: AMD clang 24.0.0git (ROCm/llvm-project `d6f6cb69`, ROCm 10.1.0).
This issue was drafted with AI assistance (Claude). I reviewed the text and reran the reproducers above.
_______________________________________________
llvm-bugs mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-bugs