Issue 244132
Summary [AMDGPU] Add `__bf16` vector variants of the bf16 `_1k` MFMA builtins
Labels new issue
Assignees
Reporter yingluAMD
    The bf16 `_1k` MFMA builtins and intrinsics take their A/B operands as `short4` / `<4 x i16>`. To use them with real `__bf16` data, callers have to bitcast to `short4`. The newer gfx950 bf16 builtins (`mfma_f32_16x16x32_bf16`, `mfma_f32_32x32x16_bf16`, `smfmac_f32_*_bf16`) take `__bf16` vectors directly.

When the operand is produced by an `fptrunc` in registers, this mandatory bitcast also makes gfx950 codegen worse. The packed `v_cvt_pk_bf16_f32` gets split into one conversion per element, and `v_perm_b32` repacks the halves.

Request: add variants of these builtins and intrinsics that take `__bf16 x4` / `<4 x bfloat>` operands and select to the same instructions.

## Reproducer

```llvm
; RUN: llc -O3 -mtriple=amdgpu9.50-amd-amdhsa < %s

define <4 x float> @f(<4 x float> %a, <4 x float> %b, <4 x float> %c) {
  %x  = fptrunc <4 x float> %a to <4 x bfloat>
  %y  = fptrunc <4 x float> %b to <4 x bfloat>
  %xi = bitcast <4 x bfloat> %x to <4 x i16>
  %yi = bitcast <4 x bfloat> %y to <4 x i16>
  %r  = call <4 x float> @llvm.amdgcn.mfma.f32.16x16x16bf16.1k(<4 x i16> %xi, <4 x i16> %yi, <4 x float> %c, i32 0, i32 0, i32 0)
  ret <4 x float> %r
}
```

Output:

```asm
	v_cvt_pk_bf16_f32 v0, v0, s0
	v_cvt_pk_bf16_f32 v12, v1, s0
	v_cvt_pk_bf16_f32 v1, v2, s0
	v_cvt_pk_bf16_f32 v2, v3, s0
	v_cvt_pk_bf16_f32 v4, v4, s0
	v_cvt_pk_bf16_f32 v5, v5, s0
	s_mov_b32 s0, 0x5040100
	v_perm_b32 v1, v2, v1, s0
	v_perm_b32 v0, v12, v0, s0
	v_cvt_pk_bf16_f32 v2, v6, s0
	v_cvt_pk_bf16_f32 v3, v7, s0
	v_perm_b32 v3, v3, v2, s0
	v_perm_b32 v2, v5, v4, s0
	s_nop 1
	v_mfma_f32_16x16x16_bf16 v[0:3], v[0:1], v[2:3], v[8:11]
```

Expected: four packed conversions and no `v_perm_b32`. For example, `v_cvt_pk_bf16_f32 v0, v0, v1`.

The same happens from HIP:

```cpp
typedef float  v4f   __attribute__((ext_vector_type(4)));
typedef __bf16 v4bf  __attribute__((ext_vector_type(4)));
typedef short  v4i16 __attribute__((ext_vector_type(4)));

__attribute__((global)) void k(const v4f *a, const v4f *b, v4f *c) {
  int i = __builtin_amdgcn_workitem_id_x();
  v4bf x = __builtin_convertvector(a[i], v4bf);
  v4bf y = __builtin_convertvector(b[i], v4bf);
  // The builtin only accepts short4, so the bf16 values must be bitcast.
  c[i] = __builtin_amdgcn_mfma_f32_16x16x16bf16_1k(
      __builtin_bit_cast(v4i16, x), __builtin_bit_cast(v4i16, y), c[i], 0, 0, 0);
}
```

```
clang -x hip --offload-arch=gfx950 --cuda-device-only -nogpulib -nogpuinc -O3 -S repro.hip -o -
```

This gives 8 `v_cvt_pk_bf16_f32` and 4 `v_perm_b32`. Without the bitcast it would be 4 and 0.

## Details

Affected builtins (`clang/include/clang/Basic/BuiltinsAMDGPU.td`, all gated on `gfx90a-insts`):

- `__builtin_amdgcn_mfma_f32_32x32x4bf16_1k`
- `__builtin_amdgcn_mfma_f32_16x16x4bf16_1k`
- `__builtin_amdgcn_mfma_f32_4x4x4bf16_1k`
- `__builtin_amdgcn_mfma_f32_32x32x8bf16_1k`
- `__builtin_amdgcn_mfma_f32_16x16x16bf16_1k`

The cause of the codegen difference is the `ISD::BITCAST` combine in `AMDGPUISelLowering.cpp`. It pushes the same-element-count bitcast into the `BUILD_VECTOR` produced by expanding `FP_ROUND v4bf16`. The elements become `i16`, so the `v2bf16` `V_CVT_PK_BF16_F32` pattern no longer matches.

That combine could be fixed on its own. Even then, the `short4` signature keeps the bitcast in every caller, and it loses type checking: fp16 or integer data passed by mistake is not diagnosed.

Targets:

| target | `_1k` builtins | hardware bf16 cvt | effect of `__bf16` operands |
|---|---|---|---|
| gfx908 | no | no | — |
| gfx90a, gfx942, gfx9-4-generic | yes | no | type checking only; codegen is identical (software rounding) |
| gfx950 | yes | yes | type checking, and 8 cvt + 4 perm → 4 cvt + 0 perm per MFMA |
| gfx12+ | no | — | — |

Today, the workaround is to convert pairs to `<2 x bfloat>`, bitcast each pair to `i32`, and build the operand from `i32` words.

## Possible approach

Add new builtins and intrinsics with `<4 x bfloat>` operands, and keep the existing ones unchanged. The names could follow the gfx940+ assembly mnemonics, as other gfx940+ MFMA intrinsics do:

| existing (kept) | new, `__bf16 x4` operands |
|---|---|
| `mfma_f32_16x16x16bf16_1k` | `mfma_f32_16x16x16_bf16` |
| `mfma_f32_4x4x4bf16_1k` | `mfma_f32_4x4x4_16b_bf16` |
| `mfma_f32_32x32x8bf16_1k` | `mfma_f32_32x32x8_bf16` |
| `mfma_f32_16x16x4bf16_1k` | `mfma_f32_16x16x4_4b_bf16` |
| `mfma_f32_32x32x4bf16_1k` | `mfma_f32_32x32x4_2b_bf16` |

The `_2b`/`_4b`/`_16b` suffixes keep the names distinct from the gfx908 `mfma_f32_32x32x4bf16`.

With new names, existing source, IR, and bitcode keep working without AutoUpgrade, and the new builtins can be detected with `__has_builtin`.

Overloading the existing intrinsics instead would change their mangled names and require AutoUpgrade. It would also leave `__has_builtin` unable to tell whether `__bf16` operands are accepted.

## Version

- IR: `llc` built from llvm/llvm-project main at `05b37c72e4`.
- HIP: AMD clang 24.0.0git (ROCm/llvm-project `d6f6cb69`, ROCm 10.1.0).

This issue was drafted with AI assistance (Claude). I reviewed the text and reran the reproducers above.

_______________________________________________
llvm-bugs mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-bugs

Reply via email to