This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 383be27d963 [fix](build) Fix two arm64 BE build failures:
kuromoji_build_dict link and SNII_CRC32C_X86 -Wundef (#67451)
383be27d963 is described below
commit 383be27d963c6bac3595e3528510348a65da0551
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Thu Sep 3 10:58:49 2026 +0800
[fix](build) Fix two arm64 BE build failures: kuromoji_build_dict link and
SNII_CRC32C_X86 -Wundef (#67451)
### What problem does this PR solve?
Issue Number: close #67448, close #67445
Related PR: #64667 (introduced `kuromoji_build_dict`), #66052
(introduced the `SNII_CRC32C_X86` test seam), #66615 (the macOS arm64
allocator workaround this PR reuses)
Problem Summary:
Two BE build failures on arm64, both found while building on Apple
Silicon.
**1. `kuromoji_build_dict` fails to link on macOS arm64 (#67448,
introduced by #64667)**
`sh build.sh --be` fails while linking `bin/kuromoji_build_dict`:
```
ld: fixup error (kind=arm64_b26) at
'__ZN8tcmallocL14memalign_pagesEmmbb'+0x1DC from
libtcmalloc.a[2](libtcmalloc_la-tcmalloc.o),
B/BL out of range (displacement=-135695688, max is +/-128MB), from
0x10816FF98 to 0x100007250 ('___clang_call_terminate')
__TEXT addr=0x100000000, size=0x008310000
google_malloc addr=0x10816eac0, size=0x0000014fc
malloc_hook addr=0x10816ffbc, size=0x0000001dc
```
The offline dictionary converter only calls the kuromoji builder/parser,
but those return `Status`, and `Status` reaches `config.cpp` (->
`ExecEnv`), `status.cpp` (-> thrift/protobuf/`BackendOptions`) and
`stack_util.cpp`, so the tool's link closure is effectively the whole BE
and its `__TEXT` exceeds arm64's +/-128 MB direct-branch reach. Apple's
linker lays tcmalloc's custom `google_malloc` / `malloc_hook` sections
out after `__text` and cannot insert branch islands there, so the branch
from tcmalloc back to `___clang_call_terminate` at the start of `__TEXT`
cannot be relaxed. The failed link leaves the four dictionary files
ungenerated, the install-time guard fires, and no `output/` is produced.
The triage comment on #67448 reports the same failure on the `macos-15`
runner of the `BE UT (macOS)` workflow.
Trimming the tool's link line (the issue's first suggestion) is not
possible without refactoring `Status`, so this applies the #66615
treatment per target: on macOS arm64 the tool links against the system
allocator, and `gperftools_stubs.cpp` supplies the few gperftools
symbols that are still referenced unconditionally. `doris_be` keeps
tcmalloc, Linux link lines are unchanged, `kuromoji_dict` stays in `ALL`
and the install-time file check stays.
**2. `SNII_CRC32C_X86` is undefined on non-x86 targets (#67445,
introduced by #66052)**
`be/src/storage/index/snii/encoding/crc32c.cpp` tests `SNII_CRC32C_X86`
with `#if` but only defines it on x86_64. The BE compiles with `-Wundef
-Werror`, so on aarch64 the four `#if` sites fail:
```
crc32c.cpp:105:5: error: 'SNII_CRC32C_X86' is not defined, evaluates to 0
[-Werror,-Wundef]
```
The file is `BE_TEST`-only, so `run-be-ut.sh` cannot build on Apple
Silicon or Linux aarch64, while CI never sees it (the only aarch64
workflow builds with `MAKE_TEST=OFF`). Define the flag as `0` on the
other branch: the x86 branch is unchanged, and the non-x86 build keeps
only the portable slice-by-8 reference path, which the `hw_*` seams
already fall back to.
---
be/CMakeLists.txt | 25 ++++++++++++++++++++++---
be/src/storage/index/snii/encoding/crc32c.cpp | 5 +++++
be/test/testutil/gperftools_stubs.cpp | 15 ++++++++-------
3 files changed, 35 insertions(+), 10 deletions(-)
diff --git a/be/CMakeLists.txt b/be/CMakeLists.txt
index 55f8166b200..90af07e966b 100644
--- a/be/CMakeLists.txt
+++ b/be/CMakeLists.txt
@@ -867,7 +867,8 @@ if ("${CMAKE_BUILD_TYPE}" STREQUAL "DEBUG" OR
"${CMAKE_BUILD_TYPE}" STREQUAL "RE
# need a custom allocator; fall back to system malloc here. ASAN
# builds never linked ${MALLOCLIB} anyway, which is why they never
# hit this. gperftools_stubs.cpp supplies the few symbols still
- # referenced unconditionally.
+ # referenced unconditionally. kuromoji_build_dict is a normal-build
+ # target this branch cannot cover; it opts out per target below.
else()
set(DORIS_LINK_LIBS ${DORIS_LINK_LIBS} ${MALLOCLIB})
endif()
@@ -1100,7 +1101,25 @@ endif()
if (NOT MAKE_TEST)
# Offline generator: compiles the UTF-8 mecab-ipadic source into binary
files.
- add_executable(kuromoji_build_dict EXCLUDE_FROM_ALL
${SRC_DIR}/tools/kuromoji_build_dict.cpp)
+ set(KUROMOJI_BUILD_DICT_SRCS ${SRC_DIR}/tools/kuromoji_build_dict.cpp)
+ set(KUROMOJI_BUILD_DICT_LINK_LIBS ${DORIS_LINK_LIBS})
+ if (OS_MACOSX AND ARCH_ARM AND MALLOCLIB STREQUAL "tcmalloc")
+ # The tool only calls the dictionary builder/parser, but they return
+ # Status, and Status reaches config.cpp (-> ExecEnv), status.cpp
+ # (-> thrift/protobuf/BackendOptions) and stack_util.cpp, so the link
+ # closure is effectively the whole BE and __TEXT exceeds arm64's
+ # +/-128MB direct-branch reach. Apple's linker lays tcmalloc's custom
+ # google_malloc/malloc_hook sections out after __text and cannot
+ # insert branch islands there ("fixup error (kind=arm64_b26) ... B/BL
+ # out of range") -- the failure doris_be_test and benchmark_test hit
+ # (see the MALLOCLIB branch above). An offline converter does not need
+ # a custom allocator: link it against the system malloc and let
+ # gperftools_stubs.cpp supply the few gperftools symbols that are still
+ # referenced unconditionally. doris_be keeps tcmalloc.
+ list(REMOVE_ITEM KUROMOJI_BUILD_DICT_LINK_LIBS ${MALLOCLIB})
+ list(APPEND KUROMOJI_BUILD_DICT_SRCS
${BASE_DIR}/test/testutil/gperftools_stubs.cpp)
+ endif()
+ add_executable(kuromoji_build_dict EXCLUDE_FROM_ALL
${KUROMOJI_BUILD_DICT_SRCS})
target_include_directories(kuromoji_build_dict PRIVATE
${PROJECT_SOURCE_DIR}/..)
pch_reuse(kuromoji_build_dict)
set_target_properties(kuromoji_build_dict PROPERTIES ENABLE_EXPORTS 1)
@@ -1109,7 +1128,7 @@ if (NOT MAKE_TEST)
-Wno-implicit-int-conversion
-Wno-shorten-64-to-32)
endif()
- target_link_libraries(kuromoji_build_dict ${DORIS_LINK_LIBS})
+ target_link_libraries(kuromoji_build_dict ${KUROMOJI_BUILD_DICT_LINK_LIBS})
set(KUROMOJI_IPADIC_SRC
"${THIRDPARTY_DIR}/share/mecab-ipadic-2.7.0-20250920"
CACHE PATH "UTF-8 mecab-ipadic source directory used to generate the
kuromoji dictionary")
diff --git a/be/src/storage/index/snii/encoding/crc32c.cpp
b/be/src/storage/index/snii/encoding/crc32c.cpp
index 39d7c6f58fe..180f2ea315a 100644
--- a/be/src/storage/index/snii/encoding/crc32c.cpp
+++ b/be/src/storage/index/snii/encoding/crc32c.cpp
@@ -34,10 +34,15 @@
#include <cstddef>
#include <cstring>
+// Value-style feature flag: the BE compiles with -Wundef -Werror, so it must
be
+// defined on every target. Non-x86 builds get 0 and keep only the portable
+// slice-by-8 reference path (the hw_* seams then fall back to it).
#if defined(__x86_64__) || defined(_M_X64)
#define SNII_CRC32C_X86 1
#include <cpuid.h> // __get_cpuid, bit_SSE4_2
#include <nmmintrin.h> // _mm_crc32_u8/u32/u64 (SSE4.2)
+#else
+#define SNII_CRC32C_X86 0
#endif
namespace doris::snii {
diff --git a/be/test/testutil/gperftools_stubs.cpp
b/be/test/testutil/gperftools_stubs.cpp
index 49e5016ee8c..3938492268c 100644
--- a/be/test/testutil/gperftools_stubs.cpp
+++ b/be/test/testutil/gperftools_stubs.cpp
@@ -15,13 +15,14 @@
// specific language governing permissions and limitations
// under the License.
-// doris_be_test on macOS/arm64 links against the system allocator instead of
-// tcmalloc: the Debug test binary's .text exceeds arm64's +/-128MB direct
-// branch reach and Apple's linker emits no branch islands for the prebuilt
-// gperftools archive. These no-op definitions satisfy the few gperftools
-// symbols still referenced unconditionally (HeapAction's HTTP handler, which
-// no unit test invokes, and brpc's periodic MallocExtension release hint).
-// Everywhere else the real libtcmalloc.a provides them and this TU is empty.
+// doris_be_test, benchmark_test and the kuromoji_build_dict tool on
macOS/arm64
+// link against the system allocator instead of tcmalloc: their .text exceeds
+// arm64's +/-128MB direct branch reach and Apple's linker emits no branch
+// islands for the prebuilt gperftools archive. These no-op definitions satisfy
+// the few gperftools symbols still referenced unconditionally (HeapAction's
+// HTTP handler, which none of them invokes, and brpc's periodic
MallocExtension
+// release hint). Everywhere else the real libtcmalloc.a provides them and this
+// TU is empty.
#if defined(__APPLE__) && defined(__aarch64__)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
