Hi all,
I'm seeing incorrect numerical results from Vulkan compute matrix
multiplication on a Turnip-driven Adreno 702 (Qualcomm QRB2210, Arduino
Uno Q board), reproducible with ggml's upstream test-backend-ops suite.
Writing this up because the pattern is very consistent and I've been
able
to rule out several likely causes already.
--------------------------------------------------------------------------
Hardware / software
--------------------------------------------------------------------------
GPU: Turnip Adreno (TM) 702
vendorID: 0x5143 (Qualcomm)
Mesa: built from git main, driverVersion reported as 26.2.99
(also reproduces on the distro-shipped Mesa version)
maxComputeSharedMemorySize: 16384 bytes
Test tool: ggml's test-backend-ops (from ggml-org/llama.cpp),
Vulkan
backend, built natively/cross-compiled aarch64
MUL_MAT (general matrix x matrix / matrix x vector multiply) produces
numerically wrong output when the batch dimension n is small (n=1
through
roughly 8, depending on k), across every plain floating point type
tested
(f32xf32, f16xf32, bf16xf32 - no quantization involved). Larger n (>=9,
or sufficiently large n*k) produces correct results in the same test
run,
with the same operands.
This is significant in practice because n=1 is the standard shape for
single-token autoregressive LLM decoding, i.e. this hits
ggml/llama.cpp's
most common real-world workload, not an edge case.
--------------------------------------------------------------------------
Reproduction
--------------------------------------------------------------------------
Build ggml/llama.cpp's test-backend-ops with the Vulkan backend enabled
and run:
./test-backend-ops test -o MUL_MAT
Representative output (trimmed, full log available on request):
MUL_MAT(type_a=f32,type_b=f32,m=16,n=1,k=256,...): FAIL
ERR=1.214182167
MUL_MAT(type_a=f32,type_b=f32,m=16,n=2,k=256,...): FAIL
ERR=0.738095533
MUL_MAT(type_a=f32,type_b=f32,m=16,n=3,k=256,...): FAIL
ERR=0.964137248
MUL_MAT(type_a=f32,type_b=f32,m=16,n=4,k=256,...): FAIL
ERR=1.433633782
MUL_MAT(type_a=f32,type_b=f32,m=16,n=5,k=256,...): FAIL
ERR=1.523953343
MUL_MAT(type_a=f32,type_b=f32,m=16,n=6,k=256,...): FAIL
ERR=0.961054406
MUL_MAT(type_a=f32,type_b=f32,m=16,n=7,k=256,...): FAIL
ERR=0.725742246
MUL_MAT(type_a=f32,type_b=f32,m=16,n=8,k=256,...): FAIL
ERR=0.977231894
MUL_MAT(type_a=f32,type_b=f32,m=16,n=9,k=256,...): OK
Same pattern for f16 and bf16 operands. Tolerance in these tests is
0.0005; observed errors are ~0.6-1.5, i.e. not float rounding noise, but
substantially wrong values.
The threshold is not a flat "n<9" rule - it interacts with total work
size. For example:
MUL_MAT(type_a=f32,type_b=f32,m=16,n=1,k=1024,bs=[3,2],...): FAIL
MUL_MAT(type_a=f32,type_b=f32,m=16,n=8,k=1024,bs=[3,2],...): OK
MUL_MAT(type_a=f32,type_b=f32,m=16,n=16,k=1024,bs=[3,2],...): OK
but
MUL_MAT(type_a=f32,type_b=f32,m=16,n=8,k=256,...): FAIL
suggesting the selection of a particular compute tile/pipeline variant
(rather than n alone) determines whether the result is correct - larger
total workloads appear to route through a different, working code path.
n=1 specifically fails almost unconditionally across every m/k/batch/
permutation combination tested in the suite (dozens of distinct shapes),
e.g.:
MUL_MAT(type_a=f32,type_b=f32,m=1056,n=1,k=128,...): FAIL
MUL_MAT(type_a=f32,type_b=f32,m=128,n=1,k=1056,...): FAIL
MUL_MAT(type_a=f32,type_b=f32,m=1057,n=1,k=129,...): FAIL
(and many more shapes/batch/permutation variants, all n=1, all FAIL)
--------------------------------------------------------------------------
What I've already ruled out
--------------------------------------------------------------------------
- Not quantization-specific: reproduces identically on plain f32xf32
with
no quantized types involved at all.
- Not an issue with a single dispatch path: reproduces whether the
mmvq quantized-matmul path is force-enabled or force-disabled via
ggml's GGML_VK_DISABLE_MMVQ (tested on a related quantized-MUL_MAT
investigation before broadening to this type-independent repro).
- Not fixed by forcing the smallest matmul tile size (mul_mat_s) via a
local patch adding a Qualcomm-specific override alongside the existing
Honeykrisp special-case in ggml-vulkan.cpp's device setup - the
numerical corruption persisted with the smallest tile forced.
- Not fixed on latest Mesa main (driverVersion 26.2.99 locally built),
reproduces identically to the distro-shipped Mesa version.
- A from-scratch, standalone Vulkan compute program (no ggml/llama.cpp
code at all) implementing the same dequant/dot-product math scalar,
with parallel shared-memory reduction, at batch sizes up to n=256 and
k=4096, across f32/f16/q8_0/q4_0 (including a byte-faithful ggml
block_q4_0 layout with real fp16-stored scale) all produced CORRECT
results. This suggests the bug is specific to ggml's actual generated
shaders/dispatch parameters (tile selection, workgroup sizing, etc.)
rather than a blanket compute-correctness problem with basic
arithmetic, barriers, or shared memory on this device.
--------------------------------------------------------------------------
Separately noticed (may be unrelated, mentioning for completeness)
--------------------------------------------------------------------------
During the same test-backend-ops run, MUL_MAT invocations with bf16 and
quantized type_a operands also emit:
SPIR-V WARNING:
In file ../src/compiler/spirv/spirv_to_nir.c:5651
Unsupported SPIR-V capability:
SpvCapabilityStorageBuffer8BitAccess (4448)
52 bytes into the SPIR-V binary
This is logged as a non-fatal warning rather than a hard failure, so
execution continues, but it suggests glslc/the driver's SPIR-V->NIR
consumer doesn't fully support a capability the shader declares
(VK_KHR_8bit_storage). Not sure if this is a contributing cause of the
n=1..8 MUL_MAT corruption above or a separate issue - flagging it since
it appears in the same log and involves the same driver/shader pipeline.
The same test run additionally showed CONV_2D producing large errors for
inputs with height>1, and GET_ROWS/SET_ROWS failing specifically for
q4_0/q8_0 types, plus a crash (uncaught exception in ggml_vk_submit)
partway through the CONV_2D tests. Happy to provide full logs for these
if useful, but keeping this report focused on the clearest, most
type-independent finding (MUL_MAT small-n).
Full test-backend-ops log, the ggml-vulkan.cpp patch I tried, and the
standalone Vulkan reproducer programs are all available on request. I'm
happy to test patches/build with extra debug output, or run the
IR3_SHADER_OVERRIDE_PATH / RenderDoc capture workflow described in the
freedreno docs if a maintainer can point me toward which shader variant
is selected for small-n MUL_MAT dispatches - I wasn't able to
conclusively
identify that from the ggml-vulkan.cpp tile-selection code on my own.
Thanks,
Stef
