| Issue |
208182
|
| Summary |
[llvm-cov] `llvm-cov report` intermittently crashes with `-num-threads > 1`
|
| Labels |
|
| Assignees |
|
| Reporter |
LucaCiucci
|
`llvm-cov-20 report` intermittently crashes when generating a summary from multiple instrumented executables and an indexed profile. The crash only appears when more than one worker thread is used. `-num-threads=1` seems to be a reliable workaround.
## Environment
I have currently tested this behavior on Ubuntu 24.04 with LLVM 20.1.8 and 22.1.8.
## Reproducer
The attached `reproducer` directory contains:
- `Overall.profdata`
- four instrumented test executables with debug information
- `crash.sh`, which repeatedly runs `llvm-cov-20 report`
Run:
```sh
cd reproducer
./crash.sh
```
The minimized command is:
```sh
llvm-cov-22 report \
ut_Date_Time \
-object ut_Unit_Conversion \
-object ut_Isolette \
-object ut_Room_Air \
-instr-profile="" \
-dump \
-num-threads=8 \
> /dev/null
```
The crash is nondeterministic. On my machine it usually happens within a few iterations with `-num-threads=$(nproc)`.
## Expected behavior
`llvm-cov report` should either complete successfully or report a normal diagnostic. It should not crash or corrupt heap state.
## Actual behavior
The failure is nondeterministic and the exact symptom varies. I have observed heap corruption diagnostics, segmentation faults, aborts, and in one run the LLVM crash-report message was printed but the process did not exit at all.
A few representative crashes are shown below.
<details>
<summary>Example 1</summary>
```plain
+ llvm-cov-22 report ut_Date_Time -object ut_Unit_Conversion -object ut_Isolette -object ut_Room_Air -instr-profile="" -dump -num-threads=4
double free or corruption (!prev)
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace and instructions to reproduce the bug.
#0 0x00007588094dd17d llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/home/luca/micromamba/envs/clang-22/bin/../lib/libLLVM.so.22.1+0xcdd17d)
#1 0x00007588094da120 SignalHandler(int, siginfo_t*, void*) Signals.cpp:0:0
#2 0x0000758808045330 (/lib/x86_64-linux-gnu/libc.so.6+0x45330)
#3 0x000075880809eb2c __pthread_kill_implementation ./nptl/pthread_kill.c:44:76
#4 0x000075880809eb2c __pthread_kill_internal ./nptl/pthread_kill.c:78:10
#5 0x000075880809eb2c pthread_kill ./nptl/pthread_kill.c:89:10
#6 0x000075880804527e raise ./signal/../sysdeps/posix/raise.c:27:6
#7 0x00007588080288ff abort ./stdlib/abort.c:81:7
#8 0x00007588080297b6 _IO_acquire_lock_fct ./libio/libioP.h:979:6
#9 0x00007588080297b6 _IO_peekc_locked ./libio/peekc.c:37:3
#10 0x00007588080a8ff5 ./malloc/malloc.c:5775:3
#11 0x00007588080ab12c _int_free_merge_chunk ./malloc/malloc.c:4679:5
#12 0x00007588080ab529 _int_free ./malloc/malloc.c:4648:5
#13 0x00007588080addce cfree ./malloc/malloc.c:3401:3
#14 0x00007588094adae7 llvm::raw_ostream::SetBuffered() (/home/luca/micromamba/envs/clang-22/bin/../lib/libLLVM.so.22.1+0xcadae7)
#15 0x00007588094ae537 llvm::raw_ostream::write(char const*, unsigned long) (/home/luca/micromamba/envs/clang-22/bin/../lib/libLLVM.so.22.1+0xcae537)
#16 0x000058ab3f596306 llvm::CoverageReport::prepareSingleFileReport(llvm::StringRef, llvm::coverage::CoverageMapping const*, llvm::CoverageViewOptions const&, unsigned int, llvm::FileCoverageSummary*, llvm::CoverageFilter const*) (/home/luca/micromamba/envs/clang-22/bin/llvm-cov-22+0x48306)
#17 0x000058ab3f594f1d void llvm::detail::UniqueFunctionBase<void>::CallImpl<std::_Bind<void (* (llvm::StringRef, llvm::coverage::CoverageMapping const*, llvm::CoverageViewOptions, unsigned int, llvm::FileCoverageSummary*, llvm::CoverageFilter const*))(llvm::StringRef, llvm::coverage::CoverageMapping const*, llvm::CoverageViewOptions const&, unsigned int, llvm::FileCoverageSummary*, llvm::CoverageFilter const*)>>(void*) (/home/luca/micromamba/envs/clang-22/bin/llvm-cov-22+0x46f1d)
#18 0x000058ab3f5652e7 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<llvm::unique_function<void ()>>>, void>>::_M_invoke(std::_Any_data const&) (/home/luca/micromamba/envs/clang-22/bin/llvm-cov-22+0x172e7)
#19 0x000058ab3f567249 std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) (/home/luca/micromamba/envs/clang-22/bin/llvm-cov-22+0x19249)
#20 0x00007588080a1ed3 __pthread_once_slow ./nptl/pthread_once.c:118:7
#21 0x000058ab3f56719c std::__future_base::_Deferred_state<std::thread::_Invoker<std::tuple<llvm::unique_function<void ()>>>, void>::_M_complete_async() (/home/luca/micromamba/envs/clang-22/bin/llvm-cov-22+0x1919c)
#22 0x000058ab3f5664ec void llvm::detail::UniqueFunctionBase<void>::CallImpl<std::shared_future<void> llvm::ThreadPoolInterface::asyncImpl<void>(llvm::unique_function<void ()>, llvm::ThreadPoolTaskGroup*)::'lambda'()>(void*) (/home/luca/micromamba/envs/clang-22/bin/llvm-cov-22+0x184ec)
#23 0x000075880944d639 llvm::StdThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) (/home/luca/micromamba/envs/clang-22/bin/../lib/libLLVM.so.22.1+0xc4d639)
#24 0x000075880944e2e7 void* llvm::thread::ThreadProxy<std::tuple<llvm::StdThreadPool::grow(int)::'lambda'()>>(void*) ThreadPool.cpp:0:0
#25 0x000075880809caa4 start_thread ./nptl/pthread_create.c:447:8
#26 0x0000758808129c6c clone3 ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:80:0
./crash.sh: line 20: 214489 Aborted llvm-cov-22 report "${overall_objects[@]}" -instr-profile="" -dump "-num-threads=4" > /dev/null
```
</details>
<details>
<summary>Example 2</summary>
```plain
double free or corruption (out)
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace and instructions to reproduce the bug.
malloc(): unaligned tcache chunk detected
./crash.sh: line 20: 217872 Aborted llvm-cov-22 report "${overall_objects[@]}" -instr-profile="" -dump "-num-threads=4" > /dev/null
```
</details>
<details>
<summary>Example 3</summary>
```plain
+ llvm-cov-22 report ut_Date_Time -object ut_Unit_Conversion -object ut_Isolette -object ut_Room_Air -instr-profile="" -dump -num-threads=4
LLVM ERROR: IO failure on output stream: Bad address
```
</details>
<details>
<summary>Example 4</summary>
```plain
corrupted size vs. prev_size
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0 libLLVM.so.20.1 0x000078adcac6343f llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 63
1 libLLVM.so.20.1 0x000078adcac61149 llvm::sys::RunSignalHandlers() + 89
2 libLLVM.so.20.1 0x000078adcac63b50
3 libc.so.6 0x000078adc9602330
4 libc.so.6 0x000078adc965bb2c pthread_kill + 284
5 libc.so.6 0x000078adc960227e gsignal + 30
6 libc.so.6 0x000078adc95e58ff abort + 223
7 libc.so.6 0x000078adc95e67b6
8 libc.so.6 0x000078adc9665ff5
9 libc.so.6 0x000078adc9666b96
10 libc.so.6 0x000078adc96680e3
11 libc.so.6 0x000078adc9668529
12 libc.so.6 0x000078adc966adce __libc_free + 126
13 libLLVM.so.20.1 0x000078adcac4256b llvm::raw_ostream::SetBuffered() + 91
14 libLLVM.so.20.1 0x000078adcac432b7 llvm::raw_ostream::write(char const*, unsigned long) + 167
15 llvm-cov-20 0x000058550904c6cf
16 llvm-cov-20 0x000058550904f50a
17 llvm-cov-20 0x000058550903a10f
18 llvm-cov-20 0x000058550903a060
19 libc.so.6 0x000078adc965eed3
20 llvm-cov-20 0x000058550903a54e
21 llvm-cov-20 0x000058550903a5fe
22 libLLVM.so.20.1 0x000078adcac044df llvm::StdThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) + 655
23 libLLVM.so.20.1 0x000078adcac05aa8
24 libc.so.6 0x000078adc9659aa4
25 libc.so.6 0x000078adc96e6a64 __clone + 68
./test_coverage.sh: line 164: 10466 Aborted (core dumped) llvm-cov-20 report "${overall_objects[@]}" -instr-profile="" --show-mcdc-summary "${common_opt[@]}" -dump > "$output_txt"
```
</details>
## Workaround
Passing `-num-threads=1` avoids the crash in my testing.
[llvm-cov-reproducer.zip](https://github.com/user-attachments/files/29794220/llvm-cov-reproducer.zip)
![]()
_______________________________________________
llvm-bugs mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-bugs
