Hello Stefan,
I tested MUL_MAT on A750 and got just these failures:
MUL_MAT(type_a=f16,type_b=f32,m=64,n=45,k=128,bs=[8,1],nr=[4,1],per=[0,1,2,3],k_v=0,o=1)
MUL_MAT(type_a=f16,type_b=f32,m=128,n=45,k=64,bs=[8,1],nr=[4,1],per=[0,1,2,3],k_v=0,o=1)
MUL_MAT(type_a=f32,type_b=f32,m=64,n=77,k=77,bs=[12,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1)
MUL_MAT(type_a=q4_0,type_b=f32,m=576,n=512,k=576,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1)
So this could be something isolated to the A702.
Regarding the `SPIR-V WARNING`. A702 doesn't support KHR_8bit_storage and
storageBuffer8BitAccess (A750 does have support for this), and some of
the tests
might be trying to compile a shader requiring that.
In ggml-vulkan.cpp I don't see VK_KHR_8bit_storage being enabled or
checked for (I can see
16bit is) and I can see mul_mat_vec_base.glsl requires
GL_EXT_shader_8bit_storage, so the
issue here could be that the tests aren't adhering to the Vulkan spec
and end up with broken
result. The way to check this would be to use Vulkan Validation Layers.
Anyway, it would be good if you could file the issue here instead:
https://gitlab.freedesktop.org/mesa/mesa/-/work_items
Regards,
Karmjit
On 21/07/2026 03:10, Stefan Rinass wrote:
Hi all,
I'm seeing incorrect numerical results from Vulkan compute matrix
multiplication on a Turnip-driven Adreno 702 (Qualcomm QRB2210, Arduino
Uno Q board), reproducible with ggml's upstream test-backend-ops suite.
Writing this up because the pattern is very consistent and I've been able
to rule out several likely causes already.
--------------------------------------------------------------------------
Hardware / software
--------------------------------------------------------------------------
GPU: Turnip Adreno (TM) 702
vendorID: 0x5143 (Qualcomm)
Mesa: built from git main, driverVersion reported as 26.2.99
(also reproduces on the distro-shipped Mesa version)
maxComputeSharedMemorySize: 16384 bytes
Test tool: ggml's test-backend-ops (from ggml-org/llama.cpp), Vulkan
backend, built natively/cross-compiled aarch64
MUL_MAT (general matrix x matrix / matrix x vector multiply) produces
numerically wrong output when the batch dimension n is small (n=1 through
roughly 8, depending on k), across every plain floating point type tested
(f32xf32, f16xf32, bf16xf32 - no quantization involved). Larger n (>=9,
or sufficiently large n*k) produces correct results in the same test run,
with the same operands.
This is significant in practice because n=1 is the standard shape for
single-token autoregressive LLM decoding, i.e. this hits ggml/llama.cpp's
most common real-world workload, not an edge case.
--------------------------------------------------------------------------
Reproduction
--------------------------------------------------------------------------
Build ggml/llama.cpp's test-backend-ops with the Vulkan backend enabled
and run:
./test-backend-ops test -o MUL_MAT
Representative output (trimmed, full log available on request):
MUL_MAT(type_a=f32,type_b=f32,m=16,n=1,k=256,...): FAIL ERR=1.214182167
MUL_MAT(type_a=f32,type_b=f32,m=16,n=2,k=256,...): FAIL ERR=0.738095533
MUL_MAT(type_a=f32,type_b=f32,m=16,n=3,k=256,...): FAIL ERR=0.964137248
MUL_MAT(type_a=f32,type_b=f32,m=16,n=4,k=256,...): FAIL ERR=1.433633782
MUL_MAT(type_a=f32,type_b=f32,m=16,n=5,k=256,...): FAIL ERR=1.523953343
MUL_MAT(type_a=f32,type_b=f32,m=16,n=6,k=256,...): FAIL ERR=0.961054406
MUL_MAT(type_a=f32,type_b=f32,m=16,n=7,k=256,...): FAIL ERR=0.725742246
MUL_MAT(type_a=f32,type_b=f32,m=16,n=8,k=256,...): FAIL ERR=0.977231894
MUL_MAT(type_a=f32,type_b=f32,m=16,n=9,k=256,...): OK
Same pattern for f16 and bf16 operands. Tolerance in these tests is
0.0005; observed errors are ~0.6-1.5, i.e. not float rounding noise, but
substantially wrong values.
The threshold is not a flat "n<9" rule - it interacts with total work
size. For example:
MUL_MAT(type_a=f32,type_b=f32,m=16,n=1,k=1024,bs=[3,2],...): FAIL
MUL_MAT(type_a=f32,type_b=f32,m=16,n=8,k=1024,bs=[3,2],...): OK
MUL_MAT(type_a=f32,type_b=f32,m=16,n=16,k=1024,bs=[3,2],...): OK
but
MUL_MAT(type_a=f32,type_b=f32,m=16,n=8,k=256,...): FAIL
suggesting the selection of a particular compute tile/pipeline variant
(rather than n alone) determines whether the result is correct - larger
total workloads appear to route through a different, working code path.
n=1 specifically fails almost unconditionally across every m/k/batch/
permutation combination tested in the suite (dozens of distinct shapes),
e.g.:
MUL_MAT(type_a=f32,type_b=f32,m=1056,n=1,k=128,...): FAIL
MUL_MAT(type_a=f32,type_b=f32,m=128,n=1,k=1056,...): FAIL
MUL_MAT(type_a=f32,type_b=f32,m=1057,n=1,k=129,...): FAIL
(and many more shapes/batch/permutation variants, all n=1, all FAIL)
--------------------------------------------------------------------------
What I've already ruled out
--------------------------------------------------------------------------
- Not quantization-specific: reproduces identically on plain f32xf32 with
no quantized types involved at all.
- Not an issue with a single dispatch path: reproduces whether the
mmvq quantized-matmul path is force-enabled or force-disabled via
ggml's GGML_VK_DISABLE_MMVQ (tested on a related quantized-MUL_MAT
investigation before broadening to this type-independent repro).
- Not fixed by forcing the smallest matmul tile size (mul_mat_s) via a
local patch adding a Qualcomm-specific override alongside the existing
Honeykrisp special-case in ggml-vulkan.cpp's device setup - the
numerical corruption persisted with the smallest tile forced.
- Not fixed on latest Mesa main (driverVersion 26.2.99 locally built),
reproduces identically to the distro-shipped Mesa version.
- A from-scratch, standalone Vulkan compute program (no ggml/llama.cpp
code at all) implementing the same dequant/dot-product math scalar,
with parallel shared-memory reduction, at batch sizes up to n=256 and
k=4096, across f32/f16/q8_0/q4_0 (including a byte-faithful ggml
block_q4_0 layout with real fp16-stored scale) all produced CORRECT
results. This suggests the bug is specific to ggml's actual generated
shaders/dispatch parameters (tile selection, workgroup sizing, etc.)
rather than a blanket compute-correctness problem with basic
arithmetic, barriers, or shared memory on this device.
--------------------------------------------------------------------------
Separately noticed (may be unrelated, mentioning for completeness)
--------------------------------------------------------------------------
During the same test-backend-ops run, MUL_MAT invocations with bf16 and
quantized type_a operands also emit:
SPIR-V WARNING:
In file ../src/compiler/spirv/spirv_to_nir.c:5651
Unsupported SPIR-V capability:
SpvCapabilityStorageBuffer8BitAccess (4448)
52 bytes into the SPIR-V binary
This is logged as a non-fatal warning rather than a hard failure, so
execution continues, but it suggests glslc/the driver's SPIR-V->NIR
consumer doesn't fully support a capability the shader declares
(VK_KHR_8bit_storage). Not sure if this is a contributing cause of the
n=1..8 MUL_MAT corruption above or a separate issue - flagging it since
it appears in the same log and involves the same driver/shader pipeline.
The same test run additionally showed CONV_2D producing large errors for
inputs with height>1, and GET_ROWS/SET_ROWS failing specifically for
q4_0/q8_0 types, plus a crash (uncaught exception in ggml_vk_submit)
partway through the CONV_2D tests. Happy to provide full logs for these
if useful, but keeping this report focused on the clearest, most
type-independent finding (MUL_MAT small-n).
Full test-backend-ops log, the ggml-vulkan.cpp patch I tried, and the
standalone Vulkan reproducer programs are all available on request. I'm
happy to test patches/build with extra debug output, or run the
IR3_SHADER_OVERRIDE_PATH / RenderDoc capture workflow described in the
freedreno docs if a maintainer can point me toward which shader variant
is selected for small-n MUL_MAT dispatches - I wasn't able to
conclusively
identify that from the ggml-vulkan.cpp tile-selection code on my own.
Thanks,
Stef
