| Issue |
203890
|
| Summary |
[flang][OpenMP] `firstprivate` of a fixed-size array lowers copy-in via `_FortranAAssign`, spilling to scratch / failing to link on GPU offload
|
| Labels |
flang
|
| Assignees |
|
| Reporter |
sbryngelson
|
## Summary
On OpenMP GPU offload, a `firstprivate` clause carrying a small fixed-size array on a register-heavy `target teams distribute parallel do` kernel lowers the per-work-item copy-in through the Fortran runtime descriptor-assignment helper `_FortranAAssign`, emitted *into the target region*, instead of a plain value copy. The consequences on AMDGPU (gfx90a):
- On the released ROCm 7.2.0 flang (LLVM 22.0.0git) the device object fails to link: `ld.lld: error: undefined symbol: _FortranAAssign` (the runtime helper isn't on the device).
- On newer AMD flang drops (LLVM 23.0.0git) it links, but the helper is inlined into the kernel as a large scratch-spilling blob: ~35 KB/work-item scratch, AGPRs pinned at the hardware max, occupancy down to one wave/SIMD, and the kernel runs 30-50x slower.
Carrying the *same data* as scalars, or as a plain `private` array initialized from those scalars, does neither — it lowers to a clean register-resident copy.
## Caveat on scope
I could only verify on AMD's ROCm flang fork (ROCm 7.2.0 and AMD AFAR 23.1.0 / 23.2.0); I don't have an upstream-built flang with working AMDGPU offload to test against. The copy-in lowering that emits `_FortranAAssign` is shared flang OpenMP frontend code, so I expect this affects upstream flang offload as well, but I have not confirmed on a trunk build. Filing here because the lowering is not AMD-fork-specific; AMDGPU-side tracking and full traces are in [ROCm/llvm-project#2909](https://github.com/ROCm/llvm-project/issues/2909).
## Reproducer
One self-contained Fortran file, built five ways (one `-DVARIANT_*` each), plus build/run scripts and raw `LIBOMPTARGET_KERNEL_TRACE` logs:
https://github.com/sbryngelson/compiler-bugs/tree/main/amd/flang-firstprivate-array-occupancy
The kernel arithmetic is byte-identical across all five variants — a register-heavy blob (~90 private real scalars + a few small private arrays through a long dependent chain so nothing folds away). Only the clause carrying the two small integers differs.
## It's `firstprivate`-of-an-array, isolated
A 2x2 over {clause} x {how the array is indexed} separates "dynamic-index forces a spill (expected)" from "the firstprivate-array clause itself":
| | read `a(i)` (dynamic) | read `merge(a(1),a(2),..)` (constant) |
|----------------------------------------|------------------------|----------------------------------------|
| `firstprivate(a)` | **spills, 12% occ** | **spills, 12% occ** |
| `private`, seeded from firstprivate scalars | **0 scratch, 50% occ** | scalars: 0 scratch, 50% occ |
- A const-indexed `firstprivate` array still spills, so it's not the dynamic index.
- A dynamically-indexed `private` array is fine, so it's not the array storage or the indexing.
- The `private`-seeded-from-firstprivate-scalars variant expresses the *exact* semantics of `firstprivate(a)` by hand and lowers to clean code — so the back end handles the semantics fine; only the `firstprivate(<array>)` spelling takes the `_FortranAAssign` path.
## Measured (AMD AFAR 23.2.0, gfx90a, `LIBOMPTARGET_KERNEL_TRACE=1`)
```
variant ns/elem scratch AGPR SGPR-spill VGPR-spill occ
baseline, no clause 0.135 0 B 0 0 0 50%
firstprivate(a) [integer(2)] 6.330 35424 B 256 1155 451 12%
firstprivate(a1, a2) [scalars] 0.196 0 B 0 0 0 50%
private(p)+firstprivate(scalars) 0.203 0 B 0 0 0 50%
```
`a` is `integer, dimension(2)` = 8 bytes.
## Question for flang/OpenMP folks
Should the `firstprivate` copy-in for a fixed-size (non-allocatable, non-pointer) array be a value copy rather than a `_FortranAAssign` runtime call? The runtime-assign path is what makes it undefined on the device under one toolchain and a scratch-spilling inlined blob under another. Happy to provide FIR/HLFIR/LLVM IR dumps or test a patch.
![]()
_______________________________________________
llvm-bugs mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-bugs
