Public bug reported:
Package: rocm-hipamd 7.1.0-0ubuntu2 (binary libamdhip64-7)
Release: Ubuntu 26.04.1 LTS (resolute), kernel 7.0.0-31-generic
Hardware: 2x AMD Radeon AI PRO R9700 (gfx1201)
Also installed: rocr-runtime 7.1.0+dfsg-0ubuntu9 (libhsa-runtime64-1) — not at
fault, see attribution.
STATUS UPSTREAM
Not present in any AMD-released runtime tested: ROCm 7.2.4 (earliest known
good; noble .debs from
repo.radeon.com), 7.13.0 and 7.14.1 (TheRock wheels). Same test binary, same
hardware, runtime swapped
via LD_LIBRARY_PATH: 0.00 KB/call and no SharedSignalPool warning on all three.
The defect is specific
to the 7.1 branch that resolute packages.
SUMMARY
The HIP runtime in resolute
(/usr/lib/x86_64-linux-gnu/libamdhip64.so.7.1.52801) never releases the
completion signal it allocates for each hipEventRecord + hipStreamWaitEvent
pair (and, to a lesser
degree, for hipMemcpyPeerAsync). Every pair leaks ~0.5–1.2 KB of host heap
permanently. At process
exit the ROCR runtime reports it:
Warning: Resource leak detected by SharedSignalPool, 142909 Signals
leaked.
Kernel launches and same-device copies do not leak. Any application that
synchronises streams or
devices with events in a loop grows without bound. Real-world impact:
llama.cpp's llama-server with
--split-mode tensor on two GPUs issues hundreds of such pairs per decode step,
leaked ~1.2 MB per
decode call (~500 KB per generated token), reached 55 GB RSS and was killed by
the kernel OOM killer
after about two hours of normal use.
STEPS TO REPRODUCE (2 GPUs needed; apileak.hip attached, ~60 lines)
hipcc --offload-arch=gfx1201 -O2 -o apileak apileak.hip
HIP_VISIBLE_DEVICES=0,1 ./apileak
The program runs each primitive 20,000 times, twice, and prints the process's
RssAnon delta per call.
ACTUAL RESULT (Ubuntu libamdhip64 7.1.0-0ubuntu2; full output attached as
apileak-7.1.log)
kernel launch (again) 0.00 KB/iter
same-device memcpyAsync D2D (again) 0.00 KB/iter
memcpyPeerAsync only (again) 0.26 KB/iter
eventRecord + cross-device streamWaitEvent 1.18 KB/iter (second pass 0.51)
eventRecord + same-stream wait 0.26–0.46 KB/iter
Warning: Resource leak detected by SharedSignalPool, 142909 Signals leaked.
EXPECTED RESULT (same binary against AMD's own libamdhip64 via LD_LIBRARY_PATH:
ROCm 7.2.4 from
repo.radeon.com noble .debs, 7.13.0 and 7.14.1 from AMD's TheRock wheels)
memcpyPeerAsync only (again) 0.00 KB/iter
eventRecord + cross-device streamWaitEvent 0.00 KB/iter (after first-pass
pool warm-up)
eventRecord + same-stream wait 0.00 KB/iter
no SharedSignalPool warning
A 4 x 100,000-iteration loop of copy+event+wait on 7.2.4 and 7.14.1 grows only
during the first pass,
then reads 0.0 KB/iter.
REQUEST
Please update rocm-hipamd in resolute to a release without this defect.
Verified clean on this hardware with the same test: ROCm 7.2.4, 7.13.0
and 7.14.1.
** Affects: rocm-hipamd (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2167138
Title:
libamdhip64 7.1.0 leaks an HSA signal on every
hipEventRecord/hipStreamWaitEvent — unbounded host RSS growth in
multi-GPU HIP applications
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rocm-hipamd/+bug/2167138/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
