morningman opened a new pull request, #67451:
URL: https://github.com/apache/doris/pull/67451
### What problem does this PR solve?
Issue Number: close #67448, close #67445
Related PR: #66615
Problem Summary:
Two BE build failures on arm64, both found while building on Apple Silicon.
**1. `kuromoji_build_dict` fails to link on macOS arm64 (#67448)**
`sh build.sh --be` fails while linking `bin/kuromoji_build_dict`:
```
ld: fixup error (kind=arm64_b26) at
'__ZN8tcmallocL14memalign_pagesEmmbb'+0x1DC from
libtcmalloc.a[2](libtcmalloc_la-tcmalloc.o),
B/BL out of range (displacement=-135695688, max is +/-128MB), from
0x10816FF98 to 0x100007250 ('___clang_call_terminate')
__TEXT addr=0x100000000, size=0x008310000
google_malloc addr=0x10816eac0, size=0x0000014fc
malloc_hook addr=0x10816ffbc, size=0x0000001dc
```
The offline dictionary converter only calls the kuromoji builder/parser, but
those return `Status`, and `Status` reaches `config.cpp` (-> `ExecEnv`),
`status.cpp` (-> thrift/protobuf/`BackendOptions`) and `stack_util.cpp`, so the
tool's link closure is effectively the whole BE and its `__TEXT` exceeds
arm64's +/-128 MB direct-branch reach. Apple's linker lays tcmalloc's custom
`google_malloc` / `malloc_hook` sections out after `__text` and cannot insert
branch islands there, so the branch from tcmalloc back to
`___clang_call_terminate` at the start of `__TEXT` cannot be relaxed. The
failed link leaves the four dictionary files ungenerated, the install-time
guard fires, and no `output/` is produced. The triage comment on #67448 reports
the same failure on the `macos-15` runner of the `BE UT (macOS)` workflow.
Trimming the tool's link line (the issue's first suggestion) is not possible
without refactoring `Status`, so this applies the #66615 treatment per target:
on macOS arm64 the tool links against the system allocator, and
`gperftools_stubs.cpp` supplies the few gperftools symbols that are still
referenced unconditionally. `doris_be` keeps tcmalloc, Linux link lines are
unchanged, `kuromoji_dict` stays in `ALL` and the install-time file check stays.
**2. `SNII_CRC32C_X86` is undefined on non-x86 targets (#67445)**
`be/src/storage/index/snii/encoding/crc32c.cpp` tests `SNII_CRC32C_X86` with
`#if` but only defines it on x86_64. The BE compiles with `-Wundef -Werror`, so
on aarch64 the four `#if` sites fail:
```
crc32c.cpp:105:5: error: 'SNII_CRC32C_X86' is not defined, evaluates to 0
[-Werror,-Wundef]
```
The file is `BE_TEST`-only, so `run-be-ut.sh` cannot build on Apple Silicon
or Linux aarch64, while CI never sees it (the only aarch64 workflow builds with
`MAKE_TEST=OFF`). Define the flag as `0` on the other branch: the x86 branch is
unchanged, and the non-x86 build keeps only the portable slice-by-8 reference
path, which the `hw_*` seams already fall back to.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
macOS 26.5 arm64, Homebrew clang 20.1.8, on master `72071af801d`:
- Relinking `kuromoji_build_dict` in the same build directory with the
pristine `be/CMakeLists.txt` reproduces the `arm64_b26` fixup error above
(`__TEXT` size `0x08310000`, tcmalloc five times on the link line). With this
change `ninja -v kuromoji_build_dict` links: no `tcmalloc` on the link line,
`gperftools_stubs.cpp.o` present, `__TEXT` `0x082fc000` with no `google_malloc`
/ `malloc_hook` sections.
- `ninja kuromoji_dict` regenerates `system.bin` / `matrix.bin` /
`chardef.bin` / `unkdict.bin` (325871 surfaces, 392126 lexicon rows, matrix
1316x1316), identical in size to the previous output.
- A full `DISABLE_BUILD_UI=ON bash build.sh --be` ends with
`Successfully build Doris` and `output/be/dict/kuromoji/` contains all four
non-empty files.
- `crc32c.cpp`: `clang++ -std=c++20 -DBE_TEST -Wundef -Werror
-fsyntax-only` on arm64 reproduces the four errors before this change and
passes after it; `clang-format` is clean. A standalone arm64 program compiled
with `-DBE_TEST` compares `crc32c_slice8_extend` / `crc32c_hw_serial_extend` /
`crc32c_hw3_extend` against the bundled Google crc32c over 15 sizes x 6
alignments plus split-extend cases: 168 comparisons identical,
`crc32c_has_hw()` reports false, `"123456789"` -> `0xE3069283`.
- Linux is covered by the regular CI builds; the CMake change is guarded
by `OS_MACOSX AND ARCH_ARM`, so Linux link lines are byte-identical.
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_015masEhncRptQgqg3JPaRyC
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
