Issue 242295
Summary [BOLT] A function the compiler split into a hot part and a `.cold` part loses the `.cold` part from its subprogram, compile unit and location lists in the debug info
Labels BOLT
Assignees
Reporter not4juu
    ## Description

BOLT treats the `.cold` part of a function (the rarely or never executed blocks that Clang and GCC move into a separate `<name>.cold` symbol) as a fragment of the hot part and moves both. In the input, one subprogram describes both parts and its ranges list both; in the rewritten DWARF only the hot part is kept:

- the subprogram becomes `DW_AT_low_pc`/`DW_AT_high_pc` of the hot part alone;
- the compile unit's ranges (and `.debug_aranges`) no longer cover the `.cold` part;
- location-list entries inside the `.cold` part are dropped.

The code in the `.cold` part still runs, but a debugger no longer knows which function, scope or variables it belongs to. gdb reports no line information there, and `llvm-symbolizer` returns line 0. This happens on AArch64 and on x86-64.

**Environment:**

- **BOLT:** `llvm-bolt` built from `72f18f704c03c8bdac29ba54d387e7d08fd1bad4`
- **Toolchain:** `clang`, `lld`, `llvm-profdata`, `llvm-nm`, `llvm-objdump`, `llvm-dwarfdump` and `llvm-symbolizer` built from the same commit; `gdb` 16.3

## Reproducer

```cpp
// lib.cpp
void log_negative(int);

int counter;

__attribute__((cold, noinline)) void report(int x) { log_negative(x); }

__attribute__((noinline)) int work(int x) {
  if (x < 0) {
    report(x);
    counter = 0;
    return -1;
  }
  counter += x;
  return counter * 3;
}
```

```
# lib.proftext: work() ran 100000 times, never with x < 0
:ir
_Z4worki
# CFG hash; on a mismatch Clang warns and prints the expected value
146835647075900052
2
100000
0
```

>From this profile Clang keeps only the entry block of `work` in place. Everything else goes into `work(int) (.cold)`, including the branch that never ran:

```bash
llvm-profdata merge lib.proftext -o lib.profdata
clang++ --target=aarch64-linux-gnu -O2 -g -fPIC -shared -fprofile-use=lib.profdata -fsplit-machine-functions lib.cpp -o libt.so -fuse-ld=lld -Wl,--emit-relocs
llvm-bolt libt.so -o libt.bolt.so --update-debug-sections
# BOLT-WARNING: split function detected on input : _Z4worki.cold. The support is limited in relocation mode
```

### Result

Addresses may differ with another toolchain or linker; the symptoms are the same. BOLT moves `report` from `0x10764` to `0x4000cc`, `work` from `0x10768` to `0x4000d0` and `work.cold` from `0x1077c` to `0x4000e4`; in both files `work.cold` starts right where `work` ends:

```bash
$ llvm-nm -C -S libt.so | grep -E 'work|report'
0000000000010768 0000000000000014 T work(int)
000000000001077c 0000000000000038 t work(int) (.cold)
0000000000010764 0000000000000004 T report(int)

$ llvm-nm -C -S libt.bolt.so | grep -E 'work|report'
00000000004000d0 0000000000000014 T work(int)
00000000004000e4 0000000000000038 t work(int) (.cold)
00000000004000cc 0000000000000004 T report(int)
```

At the `bl report` in `work.cold` (`0x10798` in the input, `0x400100` after BOLT):

```bash
$ llvm-objdump -d -j .text --no-show-raw-insn libt.so | grep 'bl.*report'
   10798:      	bl	0x10830 <_Z6reporti@plt>

$ llvm-objdump -d -j .text --no-show-raw-insn libt.bolt.so | grep 'bl.*report'
  400100:      	bl	0x10830 <_Z6reporti@plt>


$ llvm-symbolizer --obj=libt.so 0x10798
work(int) (.cold)
lib.cpp:10:5

$ llvm-symbolizer --obj=libt.bolt.so 0x400100
work(int) (.cold)
lib.cpp:0:0


$ gdb -batch -ex 'info line *0x10798' -ex 'info scope *0x10798' libt.so
Line 10 of "lib.cpp" starts at address 0x10798 <_Z4worki+48> and ends at 0x107a8 <_Z4worki+64>.
Symbol x is multi-location: ...

$ gdb -batch -ex 'info line *0x400100' -ex 'info scope *0x400100' libt.bolt.so
No line number information available for address 0x400100 <_Z4worki.cold+28>
Scope for *0x400100 contains no locals or arguments.
```

In the DWARF itself (compile unit, subprogram `work` and the location list of `x`):

```bash
$ llvm-dwarfdump --name=work --show-children --show-parents libt.so libt.bolt.so | grep -E 'file format|DW_TAG_(compile_unit|subprogram|formal_parameter)|^ *\[|DW_AT_(low|high)_pc'

libt.so:	file format elf64-littleaarch64
0x0000000c: DW_TAG_compile_unit
 DW_AT_low_pc	(0x0000000000000000)
                 [0x0000000000010764, 0x0000000000010768)
                 [0x0000000000010768, 0x000000000001077c)
                 [0x000000000001077c, 0x00000000000107b4))
0x00000063:   DW_TAG_subprogram
 [0x0000000000010768, 0x000000000001077c)
 [0x000000000001077c, 0x00000000000107b4))
0x0000006f: DW_TAG_formal_parameter
                     [0x0000000000010768, 0x0000000000010774): DW_OP_reg0 W0
                     [0x000000000001077c, 0x0000000000010788): DW_OP_reg0 W0
                     [0x0000000000010788, 0x0000000000010790): DW_OP_entry_value(DW_OP_reg0 W0), DW_OP_stack_value
 [0x0000000000010790, 0x000000000001079c): DW_OP_reg0 W0
 [0x000000000001079c, 0x00000000000107b4): DW_OP_entry_value(DW_OP_reg0 W0), DW_OP_stack_value)

libt.bolt.so:	file format elf64-littleaarch64
0x0000000c: DW_TAG_compile_unit
 DW_AT_low_pc	(0x0000000000000000)
                 [0x00000000004000cc, 0x00000000004000d0)
                 [0x00000000004000d0, 0x00000000004000e4))
0x00000063:   DW_TAG_subprogram
 DW_AT_low_pc	(0x00000000004000d0)
 DW_AT_high_pc	(0x00000000004000e4)
0x00000073: DW_TAG_formal_parameter
                     [0x00000000004000d0, 0x00000000004000dc): DW_OP_reg0 W0)
```

| | `libt.so` | `libt.bolt.so` | expected |
|---|---|---|---|
| subprogram `work` | `[0x10768, 0x1077c)`, `[0x1077c, 0x107b4)` | `[0x4000d0, 0x4000e4)` | covers `work` and `work.cold`: `[0x4000d0, 0x4000e4)`, `[0x4000e4, 0x40011c)` |
| CU ranges | `report`, `work`, `work.cold` | `report`, `work` | `report`, `work`, `work.cold` |
| location list of `x` | 5 entries, 4 in `work.cold` | 1 entry, none in `work.cold` | keeps the entries in `work.cold`: 5 entries, 4 in `work.cold` `[0x4000e4, 0x40011c)` |
| `llvm-symbolizer` at `bl report` | `lib.cpp:10:5` | `lib.cpp:0:0` | `lib.cpp:10:5` |
| gdb `info line` at `bl report` | line 10 | no line information | line 10 |
| gdb `info scope` at `bl report` | `x` | no locals or arguments | `x` |


`.debug_line` still has the row for line 10 at `0x400100`. gdb and `llvm-symbolizer` do not reach it because no unit covers the address. `llvm-dwarfdump --verify` reports no errors.

## Cause

The `.cold` part is a separate `BinaryFunction` with `IsFragment` set, and the hot part lists it in `Fragments`.

- `BinaryFunction::getOutputAddressRanges()` returns nothing for a fragment, and adds the ranges of `Fragments` only when the parent is **not** simple. When BOLT rewrites the parent, the fragment's range is in neither result. The subprogram (`DWARFRewriter`, `DW_TAG_subprogram`) and the compile unit (`BinaryContext::translateModuleAddressRanges()`) both build their ranges from it.
- `translateInputToOutputRanges()` and `translateInputToOutputLocationList()` in `DWARFRewriter.cpp` translate every range or entry through one function, the one holding the first range. `translateInputToOutputRange()` returns nothing for an address outside that function, so every range or entry that falls in the `.cold` part is dropped. This applies to location lists and to `DW_TAG_lexical_block` / `DW_TAG_inlined_subroutine` ranges that span both parts.

## Possible fix

Add the fragments' output ranges for a simple parent too, and translate each range or entry through the fragment that holds it:

```diff
--- a/bolt/lib/Core/BinaryFunction.cpp
+++ b/bolt/lib/Core/BinaryFunction.cpp
@@ DebugAddressRangesVector BinaryFunction::getOutputAddressRanges() const {
-  if (isSimple())
- return OutputRanges;
-
   for (BinaryFunction *Frag : Fragments) {
- assert(!Frag->isSimple() &&
-           "fragment of non-simple function should also be non-simple");
 OutputRanges.emplace_back(Frag->getOutputAddress(),
 Frag->getOutputAddress() + Frag->getOutputSize());
+    if (Frag->isSplit())
+      for (const FunctionFragment &FF : Frag->getLayout().getSplitFragments())
+ OutputRanges.emplace_back(FF.getAddress(),
+ FF.getAddress() + FF.getImageSize());
   }
--- a/bolt/lib/Rewrite/DWARFRewriter.cpp
+++ b/bolt/lib/Rewrite/DWARFRewriter.cpp
@@ DWARFRewriter::getOrCreateDebugInfoThreadPool(unsigned ThreadsCount) {
+/// The function holding input \p Address: \p BF itself, or a fragment of the
+/// same function, such as a compiler-generated .cold part.
+static const BinaryFunction &getFragmentFor(const BinaryFunction &BF,
+ uint64_t Address) {
+  if (BF.containsAddress(Address))
+    return BF;
+  if (const BinaryFunction *Other =
+ BF.getBinaryContext().getBinaryFunctionContainingAddress(Address))
+    if (Other->isChildOf(BF) || BF.isChildOf(*Other))
+      return *Other;
+ return BF;
+}
+
@@ translateInputToOutputRanges(const BinaryFunction &BF,
   for (const DWARFAddressRange &Range : InputRanges)
- llvm::append_range(OutputRanges, BF.translateInputToOutputRange(
- {Range.LowPC, Range.HighPC}));
+ llvm::append_range(OutputRanges,
+                       getFragmentFor(BF, Range.LowPC)
+                           .translateInputToOutputRange(
+ {Range.LowPC, Range.HighPC}));
@@ translateInputToOutputLocationList(const BinaryFunction &BF,
 DebugAddressRangesVector OutRanges =
- BF.translateInputToOutputRange({Entry.LowPC, Entry.HighPC});
+ getFragmentFor(BF, Entry.LowPC)
+ .translateInputToOutputRange({Entry.LowPC, Entry.HighPC});
```

With this change `libt.bolt.so` matches the expected column of the table above. Input built by GCC with a `.cold` part is fixed the same way. `check-bolt`, `check-large-bolt` and `check-large-bolt-arm` give the same results with and without the change.
_______________________________________________
llvm-bugs mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-bugs

Reply via email to