LuciferYang opened a new issue, #67871:
URL: https://github.com/apache/doris/issues/67871
### Search before asking
- [X] I had searched in the issues and found no similar issues.
### Version
master. Observed on BE UT builds of at least five unrelated pull requests on
2026-09-10 and 2026-09-11: builds 1042616, 1042618 (pull/67779), 1042619
(pull/67469), 1042676, 1042774 (pull/67774), 1042975.
### What's Wrong?
`doris_be_test` dies with a core dump under ASAN. The report is `SEGV on
unknown address 0x000000000000 (pc 0x000000000000)`, and frame #1 is absl's
seed reader, so the process calls through a null pointer while absl seeds a
PRNG:
```
[ RUN ]
RemoteDorisV2ReaderTest.RuntimeCancellationInterruptsBlockedFlightDoGet
AddressSanitizer:DEADLYSIGNAL
==133211==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000
(pc 0x000000000000 bp 0x7b5cd37b4e90 sp 0x7b5cd37b4658 T1846)
==133211==Hint: pc points to the zero page.
../test/format_v2/table/remote_doris_reader_test.cpp:626: Failure
Value of: entered
Actual: false
Expected: true
#0 0x000000000000 (<unknown module>)
#1 0x559acac06beb in
absl::lts_20250512::random_internal::ReadSeedMaterialFromOSEntropy(absl::lts_20250512::Span<unsigned
int>)
```
The dying thread is one the test spawns, not the main thread:
```
Thread T1846 created by T1843 here:
#0 pthread_create
#1 std::thread::_M_start_thread(...)
#2 doris::format::remote_doris::(anonymous
namespace)::FlightRemoteDorisStream::open()
#3 doris::format::remote_doris::(anonymous
namespace)::create_flight_stream(...)
#4 doris::format::remote_doris::RemoteDorisFileReader::_open_stream()
#5 doris::format::remote_doris::RemoteDorisFileReader::open(...)
#6
RemoteDorisV2ReaderTest_RuntimeCancellationInterruptsBlockedFlightDoGet_Test::TestBody()::$_0::operator()()
```
The gtest failure printed just above the SEGV is the test's own `entered`
flag: the background thread dies before it can report that it reached the
blocking DoGet.
Two things make this expensive for pull requests that have nothing to do
with this code.
The whole binary aborts, so no test registered after
`RemoteDorisV2ReaderTest` runs at all. A build that hits this says nothing
about the rest of the suite, including tests the pull request itself added.
TeamCity fails the build on its `CORE DUMPED` condition and attaches no test
name: `ABORTED, PLZ CHECK (new); build stopped: CORE DUMPED (new); exit code
143 (Step: Command Line)`. The REST API lists zero failed test occurrences for
such a build. So the author sees a red BE UT with no test name on it, and
separating this from a real failure in their own change costs a log download
per build.
How often, counting BE UT builds by day over the last 400 builds on all
branches, any core dump:
| day | builds | core dumped |
|---|---|---|
| 2026-09-05 | 7 | 0 |
| 2026-09-06 | 21 | 1 |
| 2026-09-07 | 50 | 2 |
| 2026-09-08 | 42 | 2 |
| 2026-09-09 | 59 | 3 |
| 2026-09-10 | 76 | 26 |
| 2026-09-11 | 63 | 51 |
Six of the eight core dumps I sampled in that window carry this signature.
The other two are separate crashes, so BE UT has more than one crasher right
now and this issue is only about the absl one: build 1041858 dies on `Check
failed: _seg_compact_mem_tracker != nullptr` in
`SegcompactionWorker::~SegcompactionWorker()` during `DeltaWriterCancelTest`,
and build 1040773 from 09-09 dies on a null dereference in
`IcebergTableReader::_read_equality_delete_file`. Neither of the two pre-09-10
core dumps I sampled shows the absl signature.
The crash is not deterministic. TeamCity has 40 recorded SUCCESS results for
this test, including builds at 15:06 and 15:10 on 09-11, while builds at 15:09
and 15:11 core dumped on it. Every build gets a fresh single-use agent
(`20260911150847agent_172.16.0.172_1` and the like), and the addresses differ
across builds, so this looks machine-dependent or racy rather than tied to one
commit or one bad host.
### What You Expected?
BE UT either runs to completion or fails on a test assertion instead of
taking the process down. Separately, a build that aborts should name the test
that was running, so authors are not left guessing whether the red build is
theirs.
### How to Reproduce?
Run the ASAN UT build of current master on x86 Linux and let it reach
`RemoteDorisV2ReaderTest.RuntimeCancellationInterruptsBlockedFlightDoGet`. It
may take a few runs, since the same test also passes on some agents. I have not
reproduced it outside CI and I have not bisected it.
Guest access to the builds needs no login:
- status: `http://43.132.222.7:8111/guestAuth/app/rest/builds/id:1042619`
- full log:
`http://43.132.222.7:8111/guestAuth/downloadBuildLog.html?buildId=1042619`
### Anything Else?
A hypothesis worth checking first, from someone who cannot build the ASAN UT
target. `be/src/glibc-compatibility/musl/getrandom.c` defines `getrandom` in
the binary itself, and the CMake comment there says the file is deliberately
kept in an OBJECT library rather than the static archive because sanitizers can
emit `getrandom` libcalls that arrive too late for another archive scan. When
the binary defines the symbol itself, ASAN's own interceptor resolves the real
one through `dlsym(RTLD_NEXT, "getrandom")`, which can come back null, and
calling that is exactly a jump to address 0 from inside absl's seed reader.
Checking what `getrandom` and `getentropy` resolve to in `doris_be_test`, and
whether ASAN installs an interceptor over the interposed definition, should
settle it quickly.
The timing also points at that area, weakly: `lance_symbol_versions.c` sits
in the same `glibc-compatibility-explicit` target and was rewritten on
2026-09-10 by #67701, which is when the rate jumps. That is a correlation and
nothing more. `be/CMakeLists.txt:859` links both the object library and the
archive globally, so the link line looks right from the outside.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
